Oct 30 20:38:25.010777 ip-10-0-133-192 systemd[1]: Starting Kubernetes Kubelet... Oct 30 20:38:25.409651 ip-10-0-133-192 kubenswrapper[2578]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 20:38:25.409651 ip-10-0-133-192 kubenswrapper[2578]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 30 20:38:25.409651 ip-10-0-133-192 kubenswrapper[2578]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 20:38:25.409651 ip-10-0-133-192 kubenswrapper[2578]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 20:38:25.414953 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.414866 2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 20:38:25.414953 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.414950 2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 20:38:25.414953 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.414954 2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 20:38:25.414953 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.414958 2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 20:38:25.415116 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.414961 2578 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 20:38:25.415116 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.414965 2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 20:38:25.415116 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.414967 2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 20:38:25.415116 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.414970 2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 20:38:25.415116 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.414973 2578 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 20:38:25.415116 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.414976 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 20:38:25.415116 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.414978 2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 20:38:25.415116 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.414981 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 20:38:25.415116 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.414984 2578 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 20:38:25.415116 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.414987 2578 feature_gate.go:386] unrecognized feature gate: Example Oct 30 20:38:25.415116 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.414989 2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 20:38:25.415116 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.414992 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 20:38:25.415116 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.414995 2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 20:38:25.415116 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.414998 2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 20:38:25.415116 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.415001 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 20:38:25.415116 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.415003 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 20:38:25.415116 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.415006 2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 20:38:25.415116 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.415009 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 20:38:25.415116 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.415011 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 20:38:25.415582 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.415014 2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 20:38:25.415582 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.415017 2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 20:38:25.415582 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.415020 2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 20:38:25.415582 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.415023 2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 20:38:25.415582 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.415025 2578 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 20:38:25.415582 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.415028 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 20:38:25.415582 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.415030 2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 20:38:25.415582 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.415033 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 20:38:25.415582 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.415036 2578 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 20:38:25.415582 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.415045 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 20:38:25.415582 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.415048 2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 20:38:25.415582 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.415050 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 20:38:25.415582 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.415053 2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 20:38:25.415582 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.415056 2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 20:38:25.415582 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.415059 2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 20:38:25.415582 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.415062 2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 20:38:25.415582 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.415064 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 20:38:25.415582 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.415067 2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 20:38:25.415582 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.415070 2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 20:38:25.415582 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.415073 2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 20:38:25.416114 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.415075 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 20:38:25.416114 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.415078 2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 20:38:25.416114 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.415080 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 20:38:25.416114 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.415083 2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 20:38:25.416114 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.415086 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 20:38:25.416114 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.415088 2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 20:38:25.416114 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.415091 2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 20:38:25.416114 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.415094 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 20:38:25.416114 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.415097 2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 20:38:25.416114 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.415100 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 20:38:25.416114 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.415102 2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 20:38:25.416114 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.415105 2578 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 20:38:25.416114 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.415107 2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 20:38:25.416114 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.415110 2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 20:38:25.416114 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.415112 2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 20:38:25.416114 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.415115 2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 20:38:25.416114 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.415117 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 20:38:25.416114 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.415120 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 20:38:25.416114 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.415123 2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 20:38:25.416576 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.415125 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 20:38:25.416576 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.415128 2578 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 20:38:25.416576 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.415131 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 20:38:25.416576 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.415133 2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 20:38:25.416576 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.415135 2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 20:38:25.416576 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.415138 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 20:38:25.416576 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.415141 2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 20:38:25.416576 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.415144 2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 20:38:25.416576 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.415147 2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 20:38:25.416576 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.415152 2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 20:38:25.416576 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.415156 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 20:38:25.416576 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.415160 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 20:38:25.416576 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.415163 2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 20:38:25.416576 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.415166 2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 20:38:25.416576 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.415169 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 20:38:25.416576 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.415172 2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 20:38:25.416576 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.415174 2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 20:38:25.416576 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.415177 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 20:38:25.416576 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.415180 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 20:38:25.417064 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.415182 2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 20:38:25.417064 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.415185 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 20:38:25.417064 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.415195 2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 20:38:25.417064 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.415199 2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 20:38:25.417064 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.415202 2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 20:38:25.417064 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.415205 2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 20:38:25.417064 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.415207 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 20:38:25.417064 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.415210 2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 20:38:25.417064 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.415213 2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 20:38:25.417064 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.415215 2578 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 20:38:25.417064 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.415218 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 20:38:25.417064 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.415220 2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 20:38:25.417064 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.415223 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 20:38:25.417064 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.415226 2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 20:38:25.417064 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.415228 2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 20:38:25.417064 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.415231 2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 20:38:25.417064 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.415233 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 20:38:25.417064 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.415236 2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 20:38:25.417064 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.415238 2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 20:38:25.417064 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.415241 2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 20:38:25.417550 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.415245 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 20:38:25.417550 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.415249 2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 20:38:25.417550 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.415252 2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 20:38:25.417550 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.415254 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 20:38:25.417550 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.415257 2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 20:38:25.418027 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.418013 2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 20:38:25.418027 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.418026 2578 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 20:38:25.418081 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.418031 2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 20:38:25.418081 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.418034 2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 20:38:25.418081 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.418037 2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 20:38:25.418081 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.418040 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 20:38:25.418081 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.418043 2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 20:38:25.418081 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.418045 2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 20:38:25.418081 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.418048 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 20:38:25.418081 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.418051 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 20:38:25.418081 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.418059 2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 20:38:25.418081 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.418062 2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 20:38:25.418081 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.418064 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 20:38:25.418081 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.418067 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 20:38:25.418081 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.418070 2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 20:38:25.418081 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.418072 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 20:38:25.418081 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.418075 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 20:38:25.418081 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.418077 2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 20:38:25.418081 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.418080 2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 20:38:25.418081 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.418083 2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 20:38:25.418081 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.418085 2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 20:38:25.418547 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.418088 2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 20:38:25.418547 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.418091 2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 20:38:25.418547 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.418095 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 20:38:25.418547 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.418097 2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 20:38:25.418547 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.418102 2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 20:38:25.418547 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.418106 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 20:38:25.418547 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.418109 2578 feature_gate.go:386] unrecognized feature gate: Example Oct 30 20:38:25.418547 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.418112 2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 20:38:25.418547 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.418115 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 20:38:25.418547 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.418117 2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 20:38:25.418547 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.418120 2578 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 20:38:25.418547 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.418122 2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 20:38:25.418547 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.418125 2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 20:38:25.418547 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.418127 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 20:38:25.418547 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.418130 2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 20:38:25.418547 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.418133 2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 20:38:25.418547 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.418135 2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 20:38:25.418547 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.418138 2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 20:38:25.418547 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.418141 2578 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 20:38:25.418547 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.418143 2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 20:38:25.419012 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.418146 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 20:38:25.419012 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.418148 2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 20:38:25.419012 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.418156 2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 20:38:25.419012 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.418159 2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 20:38:25.419012 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.418162 2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 20:38:25.419012 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.418164 2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 20:38:25.419012 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.418167 2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 20:38:25.419012 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.418170 2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 20:38:25.419012 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.418172 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 20:38:25.419012 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.418175 2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 20:38:25.419012 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.418177 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 20:38:25.419012 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.418180 2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 20:38:25.419012 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.418182 2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 20:38:25.419012 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.418185 2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 20:38:25.419012 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.418189 2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 20:38:25.419012 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.418191 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 20:38:25.419012 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.418194 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 20:38:25.419012 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.418196 2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 20:38:25.419012 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.418199 2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 20:38:25.419466 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.418202 2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 20:38:25.419466 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.418204 2578 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 20:38:25.419466 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.418207 2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 20:38:25.419466 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.418210 2578 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 20:38:25.419466 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.418212 2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 20:38:25.419466 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.418215 2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 20:38:25.419466 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.418217 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 20:38:25.419466 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.418220 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 20:38:25.419466 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.418222 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 20:38:25.419466 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.418225 2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 20:38:25.419466 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.418227 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 20:38:25.419466 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.418230 2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 20:38:25.419466 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.418232 2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 20:38:25.419466 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.418235 2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 20:38:25.419466 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.418237 2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 20:38:25.419466 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.418240 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 20:38:25.419466 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.418248 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 20:38:25.419466 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.418250 2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 20:38:25.419466 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.418253 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 20:38:25.419466 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.418255 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 20:38:25.419958 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.418258 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 20:38:25.419958 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.418260 2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 20:38:25.419958 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.418263 2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 20:38:25.419958 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.418265 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 20:38:25.419958 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.418268 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 20:38:25.419958 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.418270 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 20:38:25.419958 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.418275 2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 20:38:25.419958 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.418278 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 20:38:25.419958 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.418281 2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 20:38:25.419958 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.418284 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 20:38:25.419958 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.418287 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 20:38:25.419958 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.418289 2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 20:38:25.419958 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.418293 2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 20:38:25.419958 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.418295 2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 20:38:25.419958 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.418298 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 20:38:25.419958 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.418300 2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 20:38:25.419958 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.418303 2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 20:38:25.419958 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.418305 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 20:38:25.419958 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.418308 2578 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 20:38:25.420414 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.418310 2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 20:38:25.420414 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.418313 2578 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 20:38:25.420414 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.418315 2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 20:38:25.420414 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.418318 2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 20:38:25.420414 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.418320 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 20:38:25.420414 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.418323 2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 20:38:25.420414 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.418326 2578 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 20:38:25.420414 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.418897 2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 20:38:25.420414 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.418904 2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 20:38:25.420414 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.418911 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 20:38:25.420414 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.418914 2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 20:38:25.420414 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.418917 2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 20:38:25.420414 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.418920 2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 20:38:25.420414 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.418922 2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 20:38:25.420414 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.418925 2578 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 20:38:25.420414 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.418928 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 20:38:25.420414 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.418930 2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 20:38:25.420414 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.418933 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 20:38:25.420414 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.418936 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 20:38:25.420414 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.418939 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 20:38:25.420905 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.418942 2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 20:38:25.420905 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.418944 2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 20:38:25.420905 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.418947 2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 20:38:25.420905 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.418949 2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 20:38:25.420905 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.418952 2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 20:38:25.420905 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.418955 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 20:38:25.420905 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.418957 2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 20:38:25.420905 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.418960 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 20:38:25.420905 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.418963 2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 20:38:25.420905 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.418965 2578 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 20:38:25.420905 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.418968 2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 20:38:25.420905 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.418970 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 20:38:25.420905 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.418973 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 20:38:25.420905 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.418977 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 20:38:25.420905 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.418979 2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 20:38:25.420905 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.418982 2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 20:38:25.420905 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.418984 2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 20:38:25.420905 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.418986 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 20:38:25.420905 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.418989 2578 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 20:38:25.420905 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.418992 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 20:38:25.421372 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.418994 2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 20:38:25.421372 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.418997 2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 20:38:25.421372 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.419004 2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 20:38:25.421372 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.419008 2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 20:38:25.421372 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.419011 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 20:38:25.421372 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.419014 2578 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 20:38:25.421372 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.419017 2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 20:38:25.421372 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.419019 2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 20:38:25.421372 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.419022 2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 20:38:25.421372 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.419025 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 20:38:25.421372 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.419028 2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 20:38:25.421372 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.419030 2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 20:38:25.421372 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.419033 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 20:38:25.421372 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.419035 2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 20:38:25.421372 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.419038 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 20:38:25.421372 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.419041 2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 20:38:25.421372 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.419043 2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 20:38:25.421372 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.419046 2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 20:38:25.421372 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.419048 2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 20:38:25.421840 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.419051 2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 20:38:25.421840 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.419053 2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 20:38:25.421840 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.419056 2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 20:38:25.421840 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.419058 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 20:38:25.421840 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.419061 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 20:38:25.421840 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.419063 2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 20:38:25.421840 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.419066 2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 20:38:25.421840 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.419069 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 20:38:25.421840 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.419071 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 20:38:25.421840 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.419074 2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 20:38:25.421840 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.419077 2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 20:38:25.421840 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.419079 2578 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 20:38:25.421840 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.419082 2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 20:38:25.421840 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.419085 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 20:38:25.421840 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.419087 2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 20:38:25.421840 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.419089 2578 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 20:38:25.421840 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.419093 2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 20:38:25.421840 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.419095 2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 20:38:25.421840 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.419098 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 20:38:25.421840 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.419100 2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 20:38:25.422336 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.419103 2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 20:38:25.422336 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.419105 2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 20:38:25.422336 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.419108 2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 20:38:25.422336 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.419110 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 20:38:25.422336 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.419113 2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 20:38:25.422336 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.419115 2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 20:38:25.422336 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.419119 2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 20:38:25.422336 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.419122 2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 20:38:25.422336 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.419126 2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 20:38:25.422336 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.419129 2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 20:38:25.422336 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.419132 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 20:38:25.422336 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.419134 2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 20:38:25.422336 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.419137 2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 20:38:25.422336 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.419140 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 20:38:25.422336 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.419143 2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 20:38:25.422336 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.419145 2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 20:38:25.422336 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.419147 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 20:38:25.422336 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.419150 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 20:38:25.422336 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.419153 2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 20:38:25.422796 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.419155 2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 20:38:25.422796 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.419158 2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 20:38:25.422796 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.419160 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 20:38:25.422796 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.419162 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 20:38:25.422796 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.419165 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 20:38:25.422796 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.419167 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 20:38:25.422796 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.419170 2578 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 20:38:25.422796 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.419173 2578 feature_gate.go:386] unrecognized feature gate: Example Oct 30 20:38:25.422796 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.419175 2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 20:38:25.422796 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.419178 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 20:38:25.422796 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.419181 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 20:38:25.422796 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.419184 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 20:38:25.422796 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.419187 2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 20:38:25.422796 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.419189 2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 20:38:25.422796 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.419193 2578 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 20:38:25.422796 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:25.419276 2578 flags.go:64] FLAG: --address="0.0.0.0" Oct 30 20:38:25.422796 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:25.419283 2578 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 30 20:38:25.422796 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:25.419289 2578 flags.go:64] FLAG: --anonymous-auth="true" Oct 30 20:38:25.422796 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:25.419295 2578 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 30 20:38:25.422796 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:25.419309 2578 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 30 20:38:25.422796 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:25.419312 2578 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 30 20:38:25.423354 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:25.419316 2578 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 30 20:38:25.423354 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:25.419321 2578 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 30 20:38:25.423354 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:25.419324 2578 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 30 20:38:25.423354 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:25.419327 2578 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 30 20:38:25.423354 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:25.419330 2578 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 30 20:38:25.423354 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:25.419333 2578 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 30 20:38:25.423354 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:25.419337 2578 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 30 20:38:25.423354 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:25.419340 2578 flags.go:64] FLAG: --cgroup-root="" Oct 30 20:38:25.423354 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:25.419343 2578 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 30 20:38:25.423354 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:25.419346 2578 flags.go:64] FLAG: --client-ca-file="" Oct 30 20:38:25.423354 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:25.419349 2578 flags.go:64] FLAG: --cloud-provider="external" Oct 30 20:38:25.423354 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:25.419352 2578 flags.go:64] FLAG: --cluster-dns="[]" Oct 30 20:38:25.423354 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:25.419356 2578 flags.go:64] FLAG: --cluster-domain="" Oct 30 20:38:25.423354 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:25.419359 2578 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 30 20:38:25.423354 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:25.419362 2578 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 30 20:38:25.423354 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:25.419365 2578 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 30 20:38:25.423354 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:25.419369 2578 flags.go:64] FLAG: --container-log-max-files="5" Oct 30 20:38:25.423354 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:25.419373 2578 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 30 20:38:25.423354 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:25.419376 2578 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 30 20:38:25.423354 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:25.419379 2578 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 30 20:38:25.423354 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:25.419382 2578 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 30 20:38:25.423354 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:25.419385 2578 flags.go:64] FLAG: --contention-profiling="false" Oct 30 20:38:25.423354 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:25.419389 2578 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 30 20:38:25.423354 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:25.419392 2578 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 30 20:38:25.423964 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:25.419398 2578 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 30 20:38:25.423964 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:25.419401 2578 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 30 20:38:25.423964 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:25.419405 2578 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 30 20:38:25.423964 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:25.419408 2578 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 30 20:38:25.423964 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:25.419411 2578 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 30 20:38:25.423964 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:25.419414 2578 flags.go:64] FLAG: --enable-load-reader="false" Oct 30 20:38:25.423964 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:25.419417 2578 flags.go:64] FLAG: --enable-server="true" Oct 30 20:38:25.423964 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:25.419420 2578 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 30 20:38:25.423964 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:25.419425 2578 flags.go:64] FLAG: --event-burst="100" Oct 30 20:38:25.423964 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:25.419428 2578 flags.go:64] FLAG: --event-qps="50" Oct 30 20:38:25.423964 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:25.419431 2578 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 30 20:38:25.423964 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:25.419434 2578 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 30 20:38:25.423964 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:25.419437 2578 flags.go:64] FLAG: --eviction-hard="" Oct 30 20:38:25.423964 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:25.419445 2578 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 30 20:38:25.423964 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:25.419448 2578 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 30 20:38:25.423964 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:25.419453 2578 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 30 20:38:25.423964 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:25.419456 2578 flags.go:64] FLAG: --eviction-soft="" Oct 30 20:38:25.423964 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:25.419459 2578 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 30 20:38:25.423964 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:25.419462 2578 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 30 20:38:25.423964 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:25.419465 2578 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 30 20:38:25.423964 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:25.419468 2578 flags.go:64] FLAG: --experimental-mounter-path="" Oct 30 20:38:25.423964 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:25.419471 2578 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 30 20:38:25.423964 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:25.419474 2578 flags.go:64] FLAG: --fail-swap-on="true" Oct 30 20:38:25.423964 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:25.419477 2578 flags.go:64] FLAG: --feature-gates="" Oct 30 20:38:25.423964 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:25.419482 2578 flags.go:64] FLAG: --file-check-frequency="20s" Oct 30 20:38:25.424579 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:25.419485 2578 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 30 20:38:25.424579 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:25.419488 2578 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 30 20:38:25.424579 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:25.419491 2578 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 30 20:38:25.424579 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:25.419494 2578 flags.go:64] FLAG: --healthz-port="10248" Oct 30 20:38:25.424579 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:25.419497 2578 flags.go:64] FLAG: --help="false" Oct 30 20:38:25.424579 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:25.419500 2578 flags.go:64] FLAG: --hostname-override="ip-10-0-133-192.us-east-2.compute.internal" Oct 30 20:38:25.424579 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:25.419517 2578 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 30 20:38:25.424579 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:25.419524 2578 flags.go:64] FLAG: --http-check-frequency="20s" Oct 30 20:38:25.424579 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:25.419527 2578 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 30 20:38:25.424579 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:25.419531 2578 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 30 20:38:25.424579 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:25.419534 2578 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 30 20:38:25.424579 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:25.419537 2578 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 30 20:38:25.424579 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:25.419540 2578 flags.go:64] FLAG: --image-service-endpoint="" Oct 30 20:38:25.424579 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:25.419543 2578 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 30 20:38:25.424579 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:25.419546 2578 flags.go:64] FLAG: --kube-api-burst="100" Oct 30 20:38:25.424579 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:25.419549 2578 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 30 20:38:25.424579 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:25.419552 2578 flags.go:64] FLAG: --kube-api-qps="50" Oct 30 20:38:25.424579 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:25.419555 2578 flags.go:64] FLAG: --kube-reserved="" Oct 30 20:38:25.424579 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:25.419558 2578 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 30 20:38:25.424579 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:25.419560 2578 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 30 20:38:25.424579 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:25.419563 2578 flags.go:64] FLAG: --kubelet-cgroups="" Oct 30 20:38:25.424579 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:25.419566 2578 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 30 20:38:25.424579 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:25.419570 2578 flags.go:64] FLAG: --lock-file="" Oct 30 20:38:25.425120 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:25.419573 2578 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 30 20:38:25.425120 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:25.419576 2578 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 30 20:38:25.425120 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:25.419579 2578 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 30 20:38:25.425120 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:25.419585 2578 flags.go:64] FLAG: --log-json-split-stream="false" Oct 30 20:38:25.425120 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:25.419588 2578 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 30 20:38:25.425120 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:25.419591 2578 flags.go:64] FLAG: --log-text-split-stream="false" Oct 30 20:38:25.425120 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:25.419593 2578 flags.go:64] FLAG: --logging-format="text" Oct 30 20:38:25.425120 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:25.419596 2578 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 30 20:38:25.425120 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:25.419600 2578 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 30 20:38:25.425120 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:25.419603 2578 flags.go:64] FLAG: --manifest-url="" Oct 30 20:38:25.425120 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:25.419606 2578 flags.go:64] FLAG: --manifest-url-header="" Oct 30 20:38:25.425120 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:25.419610 2578 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 30 20:38:25.425120 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:25.419613 2578 flags.go:64] FLAG: --max-open-files="1000000" Oct 30 20:38:25.425120 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:25.419617 2578 flags.go:64] FLAG: --max-pods="110" Oct 30 20:38:25.425120 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:25.419620 2578 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 30 20:38:25.425120 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:25.419623 2578 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 30 20:38:25.425120 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:25.419632 2578 flags.go:64] FLAG: --memory-manager-policy="None" Oct 30 20:38:25.425120 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:25.419635 2578 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 30 20:38:25.425120 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:25.419638 2578 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 30 20:38:25.425120 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:25.419641 2578 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 30 20:38:25.425120 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:25.419644 2578 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 30 20:38:25.425120 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:25.419651 2578 flags.go:64] FLAG: --node-status-max-images="50" Oct 30 20:38:25.425120 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:25.419654 2578 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 30 20:38:25.425120 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:25.419657 2578 flags.go:64] FLAG: --oom-score-adj="-999" Oct 30 20:38:25.425120 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:25.419660 2578 flags.go:64] FLAG: --pod-cidr="" Oct 30 20:38:25.425746 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:25.419663 2578 flags.go:64] FLAG: --pod-manifest-path="" Oct 30 20:38:25.425746 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:25.419666 2578 flags.go:64] FLAG: --pod-max-pids="-1" Oct 30 20:38:25.425746 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:25.419669 2578 flags.go:64] FLAG: --pods-per-core="0" Oct 30 20:38:25.425746 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:25.419672 2578 flags.go:64] FLAG: --port="10250" Oct 30 20:38:25.425746 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:25.419675 2578 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 30 20:38:25.425746 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:25.419678 2578 flags.go:64] FLAG: --provider-id="aws:///us-east-2a/i-01b1d486861e78080" Oct 30 20:38:25.425746 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:25.419683 2578 flags.go:64] FLAG: --qos-reserved="" Oct 30 20:38:25.425746 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:25.419686 2578 flags.go:64] FLAG: --read-only-port="10255" Oct 30 20:38:25.425746 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:25.419689 2578 flags.go:64] FLAG: --register-node="true" Oct 30 20:38:25.425746 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:25.419692 2578 flags.go:64] FLAG: --register-with-taints="" Oct 30 20:38:25.425746 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:25.419696 2578 flags.go:64] FLAG: --registry-burst="10" Oct 30 20:38:25.425746 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:25.419699 2578 flags.go:64] FLAG: --registry-qps="5" Oct 30 20:38:25.425746 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:25.419701 2578 flags.go:64] FLAG: --reserved-cpus="" Oct 30 20:38:25.425746 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:25.419704 2578 flags.go:64] FLAG: --reserved-memory="" Oct 30 20:38:25.425746 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:25.419708 2578 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 30 20:38:25.425746 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:25.419711 2578 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 30 20:38:25.425746 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:25.419714 2578 flags.go:64] FLAG: --rotate-certificates="false" Oct 30 20:38:25.425746 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:25.419717 2578 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 30 20:38:25.425746 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:25.419720 2578 flags.go:64] FLAG: --runonce="false" Oct 30 20:38:25.425746 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:25.419723 2578 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 30 20:38:25.425746 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:25.419726 2578 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 30 20:38:25.425746 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:25.419729 2578 flags.go:64] FLAG: --seccomp-default="false" Oct 30 20:38:25.425746 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:25.419732 2578 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 30 20:38:25.425746 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:25.419736 2578 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 30 20:38:25.425746 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:25.419745 2578 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 30 20:38:25.425746 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:25.419748 2578 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 30 20:38:25.426352 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:25.419751 2578 flags.go:64] FLAG: --storage-driver-password="root" Oct 30 20:38:25.426352 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:25.419755 2578 flags.go:64] FLAG: --storage-driver-secure="false" Oct 30 20:38:25.426352 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:25.419757 2578 flags.go:64] FLAG: --storage-driver-table="stats" Oct 30 20:38:25.426352 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:25.419760 2578 flags.go:64] FLAG: --storage-driver-user="root" Oct 30 20:38:25.426352 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:25.419763 2578 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 30 20:38:25.426352 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:25.419766 2578 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 30 20:38:25.426352 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:25.419769 2578 flags.go:64] FLAG: --system-cgroups="" Oct 30 20:38:25.426352 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:25.419772 2578 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 30 20:38:25.426352 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:25.419778 2578 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 30 20:38:25.426352 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:25.419780 2578 flags.go:64] FLAG: --tls-cert-file="" Oct 30 20:38:25.426352 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:25.419783 2578 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 30 20:38:25.426352 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:25.419787 2578 flags.go:64] FLAG: --tls-min-version="" Oct 30 20:38:25.426352 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:25.419791 2578 flags.go:64] FLAG: --tls-private-key-file="" Oct 30 20:38:25.426352 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:25.419794 2578 flags.go:64] FLAG: --topology-manager-policy="none" Oct 30 20:38:25.426352 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:25.419796 2578 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 30 20:38:25.426352 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:25.419799 2578 flags.go:64] FLAG: --topology-manager-scope="container" Oct 30 20:38:25.426352 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:25.419803 2578 flags.go:64] FLAG: --v="2" Oct 30 20:38:25.426352 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:25.419807 2578 flags.go:64] FLAG: --version="false" Oct 30 20:38:25.426352 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:25.419811 2578 flags.go:64] FLAG: --vmodule="" Oct 30 20:38:25.426352 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:25.419815 2578 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 30 20:38:25.426352 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:25.419818 2578 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 30 20:38:25.426352 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.419990 2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 20:38:25.426352 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.419995 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 20:38:25.426352 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.419998 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 20:38:25.426941 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.420001 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 20:38:25.426941 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.420004 2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 20:38:25.426941 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.420006 2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 20:38:25.426941 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.420009 2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 20:38:25.426941 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.420012 2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 20:38:25.426941 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.420015 2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 20:38:25.426941 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.420019 2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 20:38:25.426941 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.420023 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 20:38:25.426941 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.420026 2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 20:38:25.426941 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.420029 2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 20:38:25.426941 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.420032 2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 20:38:25.426941 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.420034 2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 20:38:25.426941 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.420037 2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 20:38:25.426941 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.420040 2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 20:38:25.426941 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.420042 2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 20:38:25.426941 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.420045 2578 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 20:38:25.426941 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.420048 2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 20:38:25.426941 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.420050 2578 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 20:38:25.426941 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.420053 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 20:38:25.427395 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.420055 2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 20:38:25.427395 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.420059 2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 20:38:25.427395 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.420061 2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 20:38:25.427395 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.420064 2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 20:38:25.427395 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.420066 2578 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 20:38:25.427395 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.420069 2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 20:38:25.427395 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.420071 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 20:38:25.427395 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.420074 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 20:38:25.427395 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.420076 2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 20:38:25.427395 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.420079 2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 20:38:25.427395 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.420081 2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 20:38:25.427395 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.420084 2578 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 20:38:25.427395 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.420086 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 20:38:25.427395 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.420089 2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 20:38:25.427395 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.420091 2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 20:38:25.427395 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.420094 2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 20:38:25.427395 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.420096 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 20:38:25.427395 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.420099 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 20:38:25.427395 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.420102 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 20:38:25.427395 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.420104 2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 20:38:25.427898 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.420107 2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 20:38:25.427898 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.420116 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 20:38:25.427898 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.420119 2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 20:38:25.427898 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.420122 2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 20:38:25.427898 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.420124 2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 20:38:25.427898 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.420128 2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 20:38:25.427898 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.420131 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 20:38:25.427898 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.420134 2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 20:38:25.427898 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.420137 2578 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 20:38:25.427898 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.420140 2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 20:38:25.427898 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.420143 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 20:38:25.427898 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.420145 2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 20:38:25.427898 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.420148 2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 20:38:25.427898 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.420151 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 20:38:25.427898 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.420153 2578 feature_gate.go:386] unrecognized feature gate: Example Oct 30 20:38:25.427898 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.420156 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 20:38:25.427898 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.420159 2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 20:38:25.427898 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.420161 2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 20:38:25.427898 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.420163 2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 20:38:25.428345 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.420166 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 20:38:25.428345 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.420168 2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 20:38:25.428345 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.420171 2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 20:38:25.428345 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.420173 2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 20:38:25.428345 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.420176 2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 20:38:25.428345 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.420178 2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 20:38:25.428345 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.420181 2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 20:38:25.428345 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.420183 2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 20:38:25.428345 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.420186 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 20:38:25.428345 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.420188 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 20:38:25.428345 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.420191 2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 20:38:25.428345 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.420193 2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 20:38:25.428345 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.420196 2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 20:38:25.428345 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.420198 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 20:38:25.428345 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.420201 2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 20:38:25.428345 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.420204 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 20:38:25.428345 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.420206 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 20:38:25.428345 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.420209 2578 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 20:38:25.428345 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.420212 2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 20:38:25.428345 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.420214 2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 20:38:25.428861 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.420217 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 20:38:25.428861 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.420219 2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 20:38:25.428861 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.420222 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 20:38:25.428861 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.420225 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 20:38:25.428861 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.420227 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 20:38:25.428861 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.420230 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 20:38:25.428861 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.420233 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 20:38:25.428861 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.420235 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 20:38:25.428861 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.420238 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 20:38:25.428861 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.420240 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 20:38:25.428861 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.420243 2578 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 20:38:25.428861 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.420245 2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 20:38:25.428861 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.420248 2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 20:38:25.428861 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.420251 2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 20:38:25.428861 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.420253 2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 20:38:25.428861 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.420256 2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 20:38:25.428861 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.420258 2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 20:38:25.428861 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.420261 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 20:38:25.428861 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.420263 2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 20:38:25.429312 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.420266 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 20:38:25.429312 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.420268 2578 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 20:38:25.429312 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.420271 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 20:38:25.429312 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.420273 2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 20:38:25.429312 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.420276 2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 20:38:25.429312 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.420278 2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 20:38:25.429312 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:25.420303 2578 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 20:38:25.429312 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.420349 2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 20:38:25.429312 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.420353 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 20:38:25.429312 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.420356 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 20:38:25.429312 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.420359 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 20:38:25.429312 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.420362 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 20:38:25.429312 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.420365 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 20:38:25.429312 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.420367 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 20:38:25.429312 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.420369 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 20:38:25.429312 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.420372 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 20:38:25.429707 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.420375 2578 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 20:38:25.429707 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.420377 2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 20:38:25.429707 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.420380 2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 20:38:25.429707 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.420382 2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 20:38:25.429707 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.420385 2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 20:38:25.429707 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.420389 2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 20:38:25.429707 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.420392 2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 20:38:25.429707 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.420394 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 20:38:25.429707 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.420397 2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 20:38:25.429707 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.420400 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 20:38:25.429707 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.420402 2578 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 20:38:25.429707 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.420405 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 20:38:25.429707 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.420407 2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 20:38:25.429707 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.420410 2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 20:38:25.429707 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.420412 2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 20:38:25.429707 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.420415 2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 20:38:25.429707 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.420418 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 20:38:25.429707 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.420420 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 20:38:25.429707 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.420423 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 20:38:25.429707 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.420426 2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 20:38:25.430173 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.420429 2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 20:38:25.430173 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.420431 2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 20:38:25.430173 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.420434 2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 20:38:25.430173 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.420437 2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 20:38:25.430173 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.420440 2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 20:38:25.430173 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.420444 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 20:38:25.430173 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.420447 2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 20:38:25.430173 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.420450 2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 20:38:25.430173 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.420452 2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 20:38:25.430173 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.420455 2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 20:38:25.430173 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.420457 2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 20:38:25.430173 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.420460 2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 20:38:25.430173 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.420462 2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 20:38:25.430173 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.420465 2578 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 20:38:25.430173 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.420470 2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 20:38:25.430173 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.420472 2578 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 20:38:25.430173 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.420475 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 20:38:25.430173 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.420478 2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 20:38:25.430173 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.420481 2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 20:38:25.430645 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.420483 2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 20:38:25.430645 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.420486 2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 20:38:25.430645 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.420489 2578 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 20:38:25.430645 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.420491 2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 20:38:25.430645 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.420494 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 20:38:25.430645 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.420496 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 20:38:25.430645 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.420499 2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 20:38:25.430645 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.420501 2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 20:38:25.430645 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.420517 2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 20:38:25.430645 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.420521 2578 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 20:38:25.430645 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.420525 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 20:38:25.430645 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.420527 2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 20:38:25.430645 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.420530 2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 20:38:25.430645 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.420533 2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 20:38:25.430645 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.420535 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 20:38:25.430645 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.420538 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 20:38:25.430645 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.420540 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 20:38:25.430645 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.420543 2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 20:38:25.430645 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.420546 2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 20:38:25.430645 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.420549 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 20:38:25.431145 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.420552 2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 20:38:25.431145 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.420555 2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 20:38:25.431145 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.420557 2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 20:38:25.431145 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.420561 2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 20:38:25.431145 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.420565 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 20:38:25.431145 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.420568 2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 20:38:25.431145 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.420571 2578 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 20:38:25.431145 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.420575 2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 20:38:25.431145 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.420578 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 20:38:25.431145 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.420582 2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 20:38:25.431145 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.420586 2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 20:38:25.431145 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.420589 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 20:38:25.431145 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.420591 2578 feature_gate.go:386] unrecognized feature gate: Example Oct 30 20:38:25.431145 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.420594 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 20:38:25.431145 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.420597 2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 20:38:25.431145 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.420599 2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 20:38:25.431145 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.420602 2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 20:38:25.431145 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.420604 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 20:38:25.431145 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.420607 2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 20:38:25.431708 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.420610 2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 20:38:25.431708 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.420612 2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 20:38:25.431708 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.420615 2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 20:38:25.431708 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.420617 2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 20:38:25.431708 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.420620 2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 20:38:25.431708 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.420623 2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 20:38:25.431708 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.420625 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 20:38:25.431708 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.420628 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 20:38:25.431708 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.420630 2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 20:38:25.431708 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.420633 2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 20:38:25.431708 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.420635 2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 20:38:25.431708 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.420638 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 20:38:25.431708 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.420641 2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 20:38:25.431708 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.420643 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 20:38:25.431708 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.420646 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 20:38:25.431708 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.420649 2578 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 20:38:25.431708 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.420651 2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 20:38:25.431708 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.420654 2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 20:38:25.431708 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.420656 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 20:38:25.431708 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:25.427086 2578 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 30 20:38:25.432338 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:25.427101 2578 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 30 20:38:25.432338 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:25.427115 2578 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 30 20:38:25.432338 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:25.427120 2578 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 30 20:38:25.432338 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.427180 2578 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 20:38:25.432338 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.427184 2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 20:38:25.432338 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.427188 2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 20:38:25.432338 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.427191 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 20:38:25.432338 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.427193 2578 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 20:38:25.432338 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.427196 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 20:38:25.432338 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.427199 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 20:38:25.432338 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.427201 2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 20:38:25.432338 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.427204 2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 20:38:25.432338 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.427207 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 20:38:25.432338 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.427209 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 20:38:25.432338 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.427212 2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 20:38:25.432338 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.427215 2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 20:38:25.432338 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.427217 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 20:38:25.432338 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.427220 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 20:38:25.432338 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.427223 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 20:38:25.433075 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.427225 2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 20:38:25.433075 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.427227 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 20:38:25.433075 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.427230 2578 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 20:38:25.433075 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.427233 2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 20:38:25.433075 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.427236 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 20:38:25.433075 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.427239 2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 20:38:25.433075 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.427241 2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 20:38:25.433075 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.427244 2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 20:38:25.433075 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.427246 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 20:38:25.433075 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.427249 2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 20:38:25.433075 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.427251 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 20:38:25.433075 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.427254 2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 20:38:25.433075 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.427257 2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 20:38:25.433075 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.427268 2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 20:38:25.433075 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.427271 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 20:38:25.433075 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.427274 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 20:38:25.433075 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.427276 2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 20:38:25.433075 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.427279 2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 20:38:25.433075 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.427282 2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 20:38:25.433075 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.427285 2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 20:38:25.433615 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.427287 2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 20:38:25.433615 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.427290 2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 20:38:25.433615 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.427292 2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 20:38:25.433615 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.427295 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 20:38:25.433615 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.427298 2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 20:38:25.433615 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.427301 2578 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 20:38:25.433615 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.427303 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 20:38:25.433615 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.427306 2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 20:38:25.433615 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.427308 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 20:38:25.433615 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.427311 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 20:38:25.433615 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.427313 2578 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 20:38:25.433615 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.427316 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 20:38:25.433615 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.427318 2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 20:38:25.433615 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.427321 2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 20:38:25.433615 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.427324 2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 20:38:25.433615 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.427326 2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 20:38:25.433615 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.427329 2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 20:38:25.433615 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.427332 2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 20:38:25.433615 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.427334 2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 20:38:25.433615 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.427337 2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 20:38:25.434294 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.427339 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 20:38:25.434294 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.427342 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 20:38:25.434294 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.427345 2578 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 20:38:25.434294 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.427348 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 20:38:25.434294 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.427351 2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 20:38:25.434294 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.427354 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 20:38:25.434294 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.427362 2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 20:38:25.434294 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.427365 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 20:38:25.434294 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.427368 2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 20:38:25.434294 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.427370 2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 20:38:25.434294 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.427373 2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 20:38:25.434294 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.427375 2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 20:38:25.434294 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.427378 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 20:38:25.434294 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.427381 2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 20:38:25.434294 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.427384 2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 20:38:25.434294 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.427386 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 20:38:25.434294 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.427389 2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 20:38:25.434294 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.427392 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 20:38:25.434294 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.427394 2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 20:38:25.434768 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.427397 2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 20:38:25.434768 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.427399 2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 20:38:25.434768 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.427402 2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 20:38:25.434768 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.427405 2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 20:38:25.434768 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.427408 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 20:38:25.434768 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.427412 2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 20:38:25.434768 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.427416 2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 20:38:25.434768 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.427419 2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 20:38:25.434768 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.427422 2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 20:38:25.434768 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.427426 2578 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 20:38:25.434768 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.427429 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 20:38:25.434768 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.427432 2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 20:38:25.434768 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.427435 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 20:38:25.434768 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.427437 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 20:38:25.434768 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.427440 2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 20:38:25.434768 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.427443 2578 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 20:38:25.434768 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.427446 2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 20:38:25.434768 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.427448 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 20:38:25.434768 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.427451 2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 20:38:25.435208 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.427454 2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 20:38:25.435208 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.427462 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 20:38:25.435208 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.427466 2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 20:38:25.435208 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.427468 2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 20:38:25.435208 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.427471 2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 20:38:25.435208 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.427473 2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 20:38:25.435208 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.427476 2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 20:38:25.435208 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.427479 2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 20:38:25.435208 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.427483 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 20:38:25.435208 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.427486 2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 20:38:25.435208 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.427488 2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 20:38:25.435208 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.427491 2578 feature_gate.go:386] unrecognized feature gate: Example Oct 30 20:38:25.435208 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:25.427537 2578 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 20:38:25.435208 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.427722 2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 20:38:25.435208 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.427727 2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 20:38:25.435208 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.427729 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 20:38:25.435594 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.427732 2578 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 20:38:25.435594 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.427736 2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 20:38:25.435594 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.427738 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 20:38:25.435594 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.427741 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 20:38:25.435594 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.427743 2578 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 20:38:25.435594 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.427747 2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 20:38:25.435594 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.427749 2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 20:38:25.435594 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.427752 2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 20:38:25.435594 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.427755 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 20:38:25.435594 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.427758 2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 20:38:25.435594 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.427760 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 20:38:25.435594 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.427763 2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 20:38:25.435594 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.427766 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 20:38:25.435594 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.427768 2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 20:38:25.435594 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.427771 2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 20:38:25.435594 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.427776 2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 20:38:25.435594 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.427780 2578 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 20:38:25.435594 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.427783 2578 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 20:38:25.435594 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.427792 2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 20:38:25.435594 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.427795 2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 20:38:25.436070 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.427797 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 20:38:25.436070 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.427800 2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 20:38:25.436070 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.427803 2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 20:38:25.436070 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.427807 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 20:38:25.436070 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.427811 2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 20:38:25.436070 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.427813 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 20:38:25.436070 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.427816 2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 20:38:25.436070 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.427818 2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 20:38:25.436070 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.427821 2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 20:38:25.436070 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.427824 2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 20:38:25.436070 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.427826 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 20:38:25.436070 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.427828 2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 20:38:25.436070 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.427831 2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 20:38:25.436070 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.427833 2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 20:38:25.436070 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.427836 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 20:38:25.436070 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.427838 2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 20:38:25.436070 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.427841 2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 20:38:25.436070 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.427844 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 20:38:25.436070 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.427846 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 20:38:25.436523 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.427849 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 20:38:25.436523 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.427851 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 20:38:25.436523 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.427854 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 20:38:25.436523 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.427856 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 20:38:25.436523 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.427859 2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 20:38:25.436523 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.427862 2578 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 20:38:25.436523 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.427865 2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 20:38:25.436523 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.427867 2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 20:38:25.436523 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.427870 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 20:38:25.436523 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.427873 2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 20:38:25.436523 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.427875 2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 20:38:25.436523 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.427878 2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 20:38:25.436523 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.427886 2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 20:38:25.436523 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.427897 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 20:38:25.436523 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.427900 2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 20:38:25.436523 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.427903 2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 20:38:25.436523 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.427906 2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 20:38:25.436523 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.427908 2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 20:38:25.436523 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.427911 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 20:38:25.437037 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.427913 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 20:38:25.437037 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.427916 2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 20:38:25.437037 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.427919 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 20:38:25.437037 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.427921 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 20:38:25.437037 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.427924 2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 20:38:25.437037 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.427927 2578 feature_gate.go:386] unrecognized feature gate: Example Oct 30 20:38:25.437037 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.427929 2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 20:38:25.437037 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.427932 2578 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 20:38:25.437037 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.427934 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 20:38:25.437037 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.427937 2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 20:38:25.437037 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.427939 2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 20:38:25.437037 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.427942 2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 20:38:25.437037 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.427944 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 20:38:25.437037 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.427947 2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 20:38:25.437037 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.427949 2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 20:38:25.437037 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.427958 2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 20:38:25.437037 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.427961 2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 20:38:25.437037 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.427963 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 20:38:25.437037 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.427966 2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 20:38:25.437482 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.427968 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 20:38:25.437482 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.427971 2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 20:38:25.437482 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.427974 2578 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 20:38:25.437482 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.427976 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 20:38:25.437482 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.427979 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 20:38:25.437482 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.427981 2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 20:38:25.437482 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.427984 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 20:38:25.437482 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.427992 2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 20:38:25.437482 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.427994 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 20:38:25.437482 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.427997 2578 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 20:38:25.437482 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.427999 2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 20:38:25.437482 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.428002 2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 20:38:25.437482 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.428004 2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 20:38:25.437482 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.428007 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 20:38:25.437482 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.428009 2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 20:38:25.437482 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.428012 2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 20:38:25.437482 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.428014 2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 20:38:25.437482 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.428017 2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 20:38:25.437482 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.428019 2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 20:38:25.437482 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.428022 2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 20:38:25.437966 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.428024 2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 20:38:25.437966 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.428027 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 20:38:25.437966 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.428029 2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 20:38:25.437966 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.428032 2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 20:38:25.437966 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.428035 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 20:38:25.437966 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.428037 2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 20:38:25.437966 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:25.428060 2578 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 20:38:25.437966 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.428098 2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 20:38:25.437966 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.428102 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 20:38:25.437966 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.428105 2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 20:38:25.437966 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.428108 2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 20:38:25.437966 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.428111 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 20:38:25.437966 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.428114 2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 20:38:25.437966 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.428116 2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 20:38:25.437966 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.428119 2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 20:38:25.437966 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.428122 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 20:38:25.438379 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.428124 2578 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 20:38:25.438379 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.428127 2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 20:38:25.438379 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.428129 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 20:38:25.438379 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.428132 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 20:38:25.438379 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.428141 2578 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 20:38:25.438379 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.428144 2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 20:38:25.438379 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.428146 2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 20:38:25.438379 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.428149 2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 20:38:25.438379 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.428152 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 20:38:25.438379 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.428154 2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 20:38:25.438379 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.428157 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 20:38:25.438379 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.428159 2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 20:38:25.438379 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.428161 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 20:38:25.438379 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.428164 2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 20:38:25.438379 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.428167 2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 20:38:25.438379 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.428170 2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 20:38:25.438379 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.428174 2578 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 20:38:25.438379 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.428177 2578 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 20:38:25.438379 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.428180 2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 20:38:25.438379 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.428183 2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 20:38:25.439042 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.428185 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 20:38:25.439042 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.428188 2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 20:38:25.439042 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.428191 2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 20:38:25.439042 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.428195 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 20:38:25.439042 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.428198 2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 20:38:25.439042 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.428201 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 20:38:25.439042 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.428205 2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 20:38:25.439042 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.428207 2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 20:38:25.439042 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.428210 2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 20:38:25.439042 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.428213 2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 20:38:25.439042 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.428215 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 20:38:25.439042 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.428218 2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 20:38:25.439042 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.428220 2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 20:38:25.439042 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.428223 2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 20:38:25.439042 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.428226 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 20:38:25.439042 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.428228 2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 20:38:25.439042 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.428231 2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 20:38:25.439042 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.428239 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 20:38:25.439042 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.428241 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 20:38:25.439484 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.428244 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 20:38:25.439484 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.428246 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 20:38:25.439484 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.428249 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 20:38:25.439484 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.428251 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 20:38:25.439484 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.428254 2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 20:38:25.439484 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.428256 2578 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 20:38:25.439484 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.428259 2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 20:38:25.439484 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.428262 2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 20:38:25.439484 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.428264 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 20:38:25.439484 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.428266 2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 20:38:25.439484 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.428269 2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 20:38:25.439484 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.428272 2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 20:38:25.439484 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.428275 2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 20:38:25.439484 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.428277 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 20:38:25.439484 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.428280 2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 20:38:25.439484 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.428282 2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 20:38:25.439484 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.428285 2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 20:38:25.439484 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.428287 2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 20:38:25.439484 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.428290 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 20:38:25.439994 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.428292 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 20:38:25.439994 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.428295 2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 20:38:25.439994 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.428297 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 20:38:25.439994 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.428300 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 20:38:25.439994 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.428302 2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 20:38:25.439994 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.428304 2578 feature_gate.go:386] unrecognized feature gate: Example Oct 30 20:38:25.439994 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.428307 2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 20:38:25.439994 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.428310 2578 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 20:38:25.439994 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.428312 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 20:38:25.439994 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.428314 2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 20:38:25.439994 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.428317 2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 20:38:25.439994 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.428319 2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 20:38:25.439994 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.428327 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 20:38:25.439994 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.428330 2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 20:38:25.439994 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.428333 2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 20:38:25.439994 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.428335 2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 20:38:25.439994 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.428339 2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 20:38:25.439994 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.428342 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 20:38:25.439994 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.428345 2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 20:38:25.440553 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.428347 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 20:38:25.440553 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.428350 2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 20:38:25.440553 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.428352 2578 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 20:38:25.440553 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.428355 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 20:38:25.440553 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.428360 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 20:38:25.440553 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.428363 2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 20:38:25.440553 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.428365 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 20:38:25.440553 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.428368 2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 20:38:25.440553 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.428370 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 20:38:25.440553 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.428373 2578 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 20:38:25.440553 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.428375 2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 20:38:25.440553 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.428379 2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 20:38:25.440553 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.428381 2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 20:38:25.440553 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.428384 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 20:38:25.440553 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.428387 2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 20:38:25.440553 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.428390 2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 20:38:25.440553 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.428392 2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 20:38:25.440553 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.428395 2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 20:38:25.440553 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.428398 2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 20:38:25.440553 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:25.428400 2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 20:38:25.441018 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:25.428462 2578 server.go:957] "Client rotation is on, will bootstrap in background" Oct 30 20:38:25.441018 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:25.430430 2578 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Oct 30 20:38:25.441018 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:25.431263 2578 server.go:1014] "Starting client certificate rotation" Oct 30 20:38:25.441018 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:25.431362 2578 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 20:38:25.441018 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:25.432665 2578 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 20:38:25.441018 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:25.434256 2578 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 20:38:25.441018 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:25.436868 2578 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 20:38:25.448892 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:25.448875 2578 log.go:25] "Validated CRI v1 runtime API" Oct 30 20:38:25.452913 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:25.452889 2578 log.go:25] "Validated CRI v1 image API" Oct 30 20:38:25.453811 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:25.453785 2578 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 30 20:38:25.456440 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:25.456414 2578 fs.go:135] Filesystem UUIDs: map[039b0299-bc26-4617-bc85-4278d8a25752:/dev/nvme0n1p4 42cbd088-c4f9-4973-972e-0963ada6e718:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2] Oct 30 20:38:25.456521 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:25.456438 2578 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Oct 30 20:38:25.463102 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:25.462909 2578 manager.go:217] Machine: {Timestamp:2026-10-30 20:38:25.461623896 +0000 UTC m=+0.332452468 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100081 MemoryCapacity:33164333056 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec283823ff5dcaf4490b53bdc3778ed4 SystemUUID:ec283823-ff5d-caf4-490b-53bdc3778ed4 BootID:5cd9162a-cf97-4c4e-b2c3-f0cc131338ae Filesystems:[{Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582164480 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:ea:cf:72:b1 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:ea:cf:72:b1 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:66:eb:9d:3a:12:e6 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164333056 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 30 20:38:25.463102 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:25.463099 2578 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 30 20:38:25.463250 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:25.463229 2578 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 30 20:38:25.464809 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:25.464790 2578 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 30 20:38:25.464943 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:25.464810 2578 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-133-192.us-east-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 30 20:38:25.464990 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:25.464949 2578 topology_manager.go:143] "Creating topology manager with none policy" Oct 30 20:38:25.464990 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:25.464957 2578 container_manager_linux.go:311] "Creating device plugin manager" Oct 30 20:38:25.465040 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:25.464994 2578 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 20:38:25.465040 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:25.465011 2578 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 20:38:25.465774 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:25.465750 2578 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 30 20:38:25.465774 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:25.465759 2578 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 30 20:38:25.469668 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:25.469650 2578 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 30 20:38:25.469788 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:25.469778 2578 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 30 20:38:25.469822 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:25.469807 2578 kubelet.go:493] "Attempting to sync node with API server" Oct 30 20:38:25.469822 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:25.469818 2578 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 30 20:38:25.469872 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:25.469827 2578 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 30 20:38:25.469872 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:25.469835 2578 kubelet.go:405] "Adding apiserver pod source" Oct 30 20:38:25.469872 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:25.469842 2578 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 30 20:38:25.470241 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:25.470230 2578 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 20:38:25.472341 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:25.472318 2578 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 30 20:38:25.476815 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:25.476802 2578 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 20:38:25.477538 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:25.477502 2578 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 30 20:38:25.477586 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:25.477556 2578 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 30 20:38:25.477714 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:25.477702 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 30 20:38:25.477743 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:25.477720 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 30 20:38:25.477743 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:25.477727 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 30 20:38:25.477743 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:25.477733 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 30 20:38:25.477743 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:25.477738 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 30 20:38:25.477743 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:25.477743 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 30 20:38:25.478173 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:25.478165 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 30 20:38:25.478201 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:25.478174 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 30 20:38:25.478201 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:25.478181 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 30 20:38:25.478201 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:25.478187 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 30 20:38:25.478201 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:25.478195 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 30 20:38:25.478320 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:25.478204 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 30 20:38:25.478688 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:25.478678 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 30 20:38:25.478717 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:25.478688 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 30 20:38:25.482439 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:25.482426 2578 server.go:1263] "Started kubelet" Oct 30 20:38:25.482630 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:25.482551 2578 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 30 20:38:25.482856 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:25.482550 2578 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 30 20:38:25.482916 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:25.482882 2578 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 30 20:38:25.483180 ip-10-0-133-192 systemd[1]: Started Kubernetes Kubelet. Oct 30 20:38:25.484106 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:25.483954 2578 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 30 20:38:25.484822 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:25.484809 2578 server.go:317] "Adding debug handlers to kubelet server" Oct 30 20:38:25.486053 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:25.486034 2578 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-133-192.us-east-2.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Oct 30 20:38:25.489756 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:25.489095 2578 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-jlz6q" Oct 30 20:38:25.489756 ip-10-0-133-192 kubenswrapper[2578]: E1030 20:38:25.489451 2578 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" Oct 30 20:38:25.489756 ip-10-0-133-192 kubenswrapper[2578]: E1030 20:38:25.489562 2578 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-133-192.us-east-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node" Oct 30 20:38:25.489994 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:25.489979 2578 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 30 20:38:25.490069 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:25.490052 2578 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 30 20:38:25.490405 ip-10-0-133-192 kubenswrapper[2578]: E1030 20:38:25.490386 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-192.us-east-2.compute.internal\" not found" Oct 30 20:38:25.490560 ip-10-0-133-192 kubenswrapper[2578]: E1030 20:38:25.489681 2578 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-192.us-east-2.compute.internal.18e36929f10e12f3 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-192.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-133-192.us-east-2.compute.internal,},FirstTimestamp:2026-10-30 20:38:25.482396403 +0000 UTC m=+0.353224953,LastTimestamp:2026-10-30 20:38:25.482396403 +0000 UTC m=+0.353224953,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-192.us-east-2.compute.internal,}" Oct 30 20:38:25.490697 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:25.490672 2578 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 30 20:38:25.490783 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:25.490675 2578 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 30 20:38:25.490783 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:25.490738 2578 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 30 20:38:25.490880 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:25.490841 2578 reconstruct.go:97] "Volume reconstruction finished" Oct 30 20:38:25.490880 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:25.490855 2578 reconciler.go:29] "Reconciler: start to sync state" Oct 30 20:38:25.492580 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:25.492562 2578 factory.go:153] Registering CRI-O factory Oct 30 20:38:25.492580 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:25.492581 2578 factory.go:223] Registration of the crio container factory successfully Oct 30 20:38:25.492691 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:25.492635 2578 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 30 20:38:25.492691 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:25.492645 2578 factory.go:55] Registering systemd factory Oct 30 20:38:25.492691 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:25.492652 2578 factory.go:223] Registration of the systemd container factory successfully Oct 30 20:38:25.492691 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:25.492666 2578 factory.go:103] Registering Raw factory Oct 30 20:38:25.492691 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:25.492679 2578 manager.go:1199] Started watching for new ooms in manager Oct 30 20:38:25.493668 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:25.493568 2578 manager.go:319] Starting recovery of all containers Oct 30 20:38:25.493668 ip-10-0-133-192 kubenswrapper[2578]: E1030 20:38:25.493572 2578 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Oct 30 20:38:25.498408 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:25.498368 2578 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-jlz6q" Oct 30 20:38:25.503737 ip-10-0-133-192 kubenswrapper[2578]: E1030 20:38:25.502633 2578 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-133-192.us-east-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Oct 30 20:38:25.504695 ip-10-0-133-192 kubenswrapper[2578]: E1030 20:38:25.504671 2578 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" Oct 30 20:38:25.506058 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:25.506040 2578 manager.go:324] Recovery completed Oct 30 20:38:25.510201 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:25.510189 2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 20:38:25.512651 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:25.512635 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-192.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 30 20:38:25.512711 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:25.512662 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-192.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 30 20:38:25.512711 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:25.512677 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-192.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 30 20:38:25.513201 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:25.513186 2578 cpu_manager.go:226] "Starting" policy="none" Oct 30 20:38:25.513201 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:25.513196 2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 20:38:25.513320 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:25.513199 2578 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 30 20:38:25.513320 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:25.513231 2578 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 30 20:38:25.513859 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:25.513845 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-192.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 30 20:38:25.513925 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:25.513872 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-192.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 30 20:38:25.513925 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:25.513886 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-192.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 30 20:38:25.515266 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:25.515250 2578 policy_none.go:50] "Start" Oct 30 20:38:25.515266 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:25.515266 2578 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 30 20:38:25.515407 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:25.515275 2578 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 30 20:38:25.517362 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:25.517348 2578 policy_none.go:44] "Start" Oct 30 20:38:25.562101 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:25.562070 2578 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 30 20:38:25.566025 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:25.566009 2578 manager.go:341] "Starting Device Plugin manager" Oct 30 20:38:25.566114 ip-10-0-133-192 kubenswrapper[2578]: E1030 20:38:25.566045 2578 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 30 20:38:25.566114 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:25.566058 2578 server.go:87] "Starting device plugin registration server" Oct 30 20:38:25.566345 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:25.566331 2578 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 30 20:38:25.566406 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:25.566381 2578 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 30 20:38:25.568034 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:25.566481 2578 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 30 20:38:25.568034 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:25.566582 2578 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 30 20:38:25.568034 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:25.566595 2578 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 30 20:38:25.568034 ip-10-0-133-192 kubenswrapper[2578]: E1030 20:38:25.567027 2578 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 30 20:38:25.568034 ip-10-0-133-192 kubenswrapper[2578]: E1030 20:38:25.567073 2578 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-133-192.us-east-2.compute.internal\" not found" Oct 30 20:38:25.666810 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:25.666761 2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 20:38:25.668708 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:25.667786 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-192.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 30 20:38:25.668708 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:25.667817 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-192.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 30 20:38:25.668708 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:25.667829 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-192.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 30 20:38:25.668708 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:25.667855 2578 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-133-192.us-east-2.compute.internal" Oct 30 20:38:25.678013 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:25.678000 2578 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-133-192.us-east-2.compute.internal" Oct 30 20:38:25.678087 ip-10-0-133-192 kubenswrapper[2578]: E1030 20:38:25.678020 2578 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-133-192.us-east-2.compute.internal\": node \"ip-10-0-133-192.us-east-2.compute.internal\" not found" Oct 30 20:38:25.697296 ip-10-0-133-192 kubenswrapper[2578]: E1030 20:38:25.697281 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-192.us-east-2.compute.internal\" not found" Oct 30 20:38:25.727807 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:25.727782 2578 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 30 20:38:25.728936 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:25.728918 2578 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 30 20:38:25.728936 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:25.728935 2578 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 30 20:38:25.729067 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:25.728952 2578 kubelet.go:2521] "Starting kubelet main sync loop" Oct 30 20:38:25.729622 ip-10-0-133-192 kubenswrapper[2578]: E1030 20:38:25.728978 2578 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Oct 30 20:38:25.735729 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:25.735714 2578 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 20:38:25.798003 ip-10-0-133-192 kubenswrapper[2578]: E1030 20:38:25.797977 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-192.us-east-2.compute.internal\" not found" Oct 30 20:38:25.830149 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:25.830125 2578 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-133-192.us-east-2.compute.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-192.us-east-2.compute.internal"] Oct 30 20:38:25.830236 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:25.830184 2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 20:38:25.831051 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:25.831035 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-192.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 30 20:38:25.831123 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:25.831066 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-192.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 30 20:38:25.831123 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:25.831077 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-192.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 30 20:38:25.833247 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:25.833235 2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 20:38:25.833393 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:25.833378 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-192.us-east-2.compute.internal" Oct 30 20:38:25.833434 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:25.833405 2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 20:38:25.833943 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:25.833927 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-192.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 30 20:38:25.834003 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:25.833958 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-192.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 30 20:38:25.834003 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:25.833968 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-192.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 30 20:38:25.834069 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:25.833931 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-192.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 30 20:38:25.834069 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:25.834032 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-192.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 30 20:38:25.834069 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:25.834050 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-192.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 30 20:38:25.836144 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:25.836128 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-192.us-east-2.compute.internal" Oct 30 20:38:25.836228 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:25.836154 2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 20:38:25.836814 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:25.836794 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-192.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 30 20:38:25.836882 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:25.836826 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-192.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 30 20:38:25.836882 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:25.836843 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-192.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 30 20:38:25.861104 ip-10-0-133-192 kubenswrapper[2578]: E1030 20:38:25.861086 2578 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-192.us-east-2.compute.internal\" not found" node="ip-10-0-133-192.us-east-2.compute.internal" Oct 30 20:38:25.867398 ip-10-0-133-192 kubenswrapper[2578]: E1030 20:38:25.867384 2578 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-192.us-east-2.compute.internal\" not found" node="ip-10-0-133-192.us-east-2.compute.internal" Oct 30 20:38:25.892866 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:25.892848 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a211a81db7289ccbd1f6896c27ff51c5-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-192.us-east-2.compute.internal\" (UID: \"a211a81db7289ccbd1f6896c27ff51c5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-192.us-east-2.compute.internal" Oct 30 20:38:25.892927 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:25.892871 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a211a81db7289ccbd1f6896c27ff51c5-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-192.us-east-2.compute.internal\" (UID: \"a211a81db7289ccbd1f6896c27ff51c5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-192.us-east-2.compute.internal" Oct 30 20:38:25.892927 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:25.892889 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/10e325ce488448e0259da3eb69cd60ba-config\") pod \"kube-apiserver-proxy-ip-10-0-133-192.us-east-2.compute.internal\" (UID: \"10e325ce488448e0259da3eb69cd60ba\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-192.us-east-2.compute.internal" Oct 30 20:38:25.898958 ip-10-0-133-192 kubenswrapper[2578]: E1030 20:38:25.898937 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-192.us-east-2.compute.internal\" not found" Oct 30 20:38:25.993635 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:25.993592 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/10e325ce488448e0259da3eb69cd60ba-config\") pod \"kube-apiserver-proxy-ip-10-0-133-192.us-east-2.compute.internal\" (UID: \"10e325ce488448e0259da3eb69cd60ba\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-192.us-east-2.compute.internal" Oct 30 20:38:25.993635 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:25.993615 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a211a81db7289ccbd1f6896c27ff51c5-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-192.us-east-2.compute.internal\" (UID: \"a211a81db7289ccbd1f6896c27ff51c5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-192.us-east-2.compute.internal" Oct 30 20:38:25.993749 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:25.993638 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a211a81db7289ccbd1f6896c27ff51c5-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-192.us-east-2.compute.internal\" (UID: \"a211a81db7289ccbd1f6896c27ff51c5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-192.us-east-2.compute.internal" Oct 30 20:38:25.993749 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:25.993663 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a211a81db7289ccbd1f6896c27ff51c5-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-192.us-east-2.compute.internal\" (UID: \"a211a81db7289ccbd1f6896c27ff51c5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-192.us-east-2.compute.internal" Oct 30 20:38:25.993749 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:25.993684 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/10e325ce488448e0259da3eb69cd60ba-config\") pod \"kube-apiserver-proxy-ip-10-0-133-192.us-east-2.compute.internal\" (UID: \"10e325ce488448e0259da3eb69cd60ba\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-192.us-east-2.compute.internal" Oct 30 20:38:25.993749 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:25.993691 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a211a81db7289ccbd1f6896c27ff51c5-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-192.us-east-2.compute.internal\" (UID: \"a211a81db7289ccbd1f6896c27ff51c5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-192.us-east-2.compute.internal" Oct 30 20:38:25.999720 ip-10-0-133-192 kubenswrapper[2578]: E1030 20:38:25.999695 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-192.us-east-2.compute.internal\" not found" Oct 30 20:38:26.100181 ip-10-0-133-192 kubenswrapper[2578]: E1030 20:38:26.100158 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-192.us-east-2.compute.internal\" not found" Oct 30 20:38:26.164395 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:26.164358 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-192.us-east-2.compute.internal" Oct 30 20:38:26.170162 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:26.170045 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-192.us-east-2.compute.internal" Oct 30 20:38:26.201100 ip-10-0-133-192 kubenswrapper[2578]: E1030 20:38:26.201079 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-192.us-east-2.compute.internal\" not found" Oct 30 20:38:26.301624 ip-10-0-133-192 kubenswrapper[2578]: E1030 20:38:26.301556 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-192.us-east-2.compute.internal\" not found" Oct 30 20:38:26.402030 ip-10-0-133-192 kubenswrapper[2578]: E1030 20:38:26.402010 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-192.us-east-2.compute.internal\" not found" Oct 30 20:38:26.431184 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:26.431164 2578 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Oct 30 20:38:26.431595 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:26.431296 2578 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Oct 30 20:38:26.502289 ip-10-0-133-192 kubenswrapper[2578]: E1030 20:38:26.502266 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-192.us-east-2.compute.internal\" not found" Oct 30 20:38:26.505727 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:26.504149 2578 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 20:38:26.507597 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:26.507549 2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 20:33:25 +0000 UTC" deadline="2028-03-29 17:38:10.476435866 +0000 UTC" Oct 30 20:38:26.507645 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:26.507596 2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12380h59m43.96884211s" Oct 30 20:38:26.602829 ip-10-0-133-192 kubenswrapper[2578]: E1030 20:38:26.602767 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-192.us-east-2.compute.internal\" not found" Oct 30 20:38:26.703687 ip-10-0-133-192 kubenswrapper[2578]: E1030 20:38:26.703650 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-192.us-east-2.compute.internal\" not found" Oct 30 20:38:26.803810 ip-10-0-133-192 kubenswrapper[2578]: E1030 20:38:26.803785 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-192.us-east-2.compute.internal\" not found" Oct 30 20:38:26.818066 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:26.818037 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10e325ce488448e0259da3eb69cd60ba.slice/crio-fb0f2c227bb739438aaf12d4ffc31d80eab2c7f4b5776fd30ffc16448932514e WatchSource:0}: Error finding container fb0f2c227bb739438aaf12d4ffc31d80eab2c7f4b5776fd30ffc16448932514e: Status 404 returned error can't find the container with id fb0f2c227bb739438aaf12d4ffc31d80eab2c7f4b5776fd30ffc16448932514e Oct 30 20:38:26.819975 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:26.819949 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda211a81db7289ccbd1f6896c27ff51c5.slice/crio-160c0397f4741ec3a44de2d9a323a0a3713a7ab5c7ba1ac1f92e56c2b4711663 WatchSource:0}: Error finding container 160c0397f4741ec3a44de2d9a323a0a3713a7ab5c7ba1ac1f92e56c2b4711663: Status 404 returned error can't find the container with id 160c0397f4741ec3a44de2d9a323a0a3713a7ab5c7ba1ac1f92e56c2b4711663 Oct 30 20:38:26.824411 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:26.824396 2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 20:38:26.903943 ip-10-0-133-192 kubenswrapper[2578]: E1030 20:38:26.903881 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-192.us-east-2.compute.internal\" not found" Oct 30 20:38:26.999340 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:26.999316 2578 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 20:38:27.063853 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:27.063831 2578 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 20:38:27.092075 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:27.092050 2578 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-192.us-east-2.compute.internal" Oct 30 20:38:27.104576 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:27.104557 2578 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 20:38:27.105602 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:27.105582 2578 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-192.us-east-2.compute.internal" Oct 30 20:38:27.120043 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:27.120023 2578 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 20:38:27.378825 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:27.378749 2578 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 20:38:27.470784 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:27.470755 2578 apiserver.go:52] "Watching apiserver" Oct 30 20:38:27.484136 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:27.484096 2578 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 30 20:38:27.484520 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:27.484492 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-additional-cni-plugins-nxmjg","openshift-multus/multus-c4fvc","kube-system/global-pull-secret-syncer-857r2","openshift-dns/node-resolver-lc599","openshift-network-operator/iptables-alerter-kn2t9","kube-system/kube-apiserver-proxy-ip-10-0-133-192.us-east-2.compute.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-192.us-east-2.compute.internal","openshift-multus/network-metrics-daemon-pdcfk","kube-system/konnectivity-agent-jfxnj","openshift-network-diagnostics/network-check-target-jp2cr","openshift-ovn-kubernetes/ovnkube-node-7w5wz","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t7h2z","openshift-cluster-node-tuning-operator/tuned-2qgck","openshift-image-registry/node-ca-hsv4j"] Oct 30 20:38:27.487180 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:27.487155 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jp2cr" Oct 30 20:38:27.487290 ip-10-0-133-192 kubenswrapper[2578]: E1030 20:38:27.487248 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jp2cr" podUID="0dde7eae-1962-4559-89e9-0c67b1165914" Oct 30 20:38:27.489633 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:27.489611 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-2qgck" Oct 30 20:38:27.494321 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:27.491311 2578 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Oct 30 20:38:27.494321 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:27.493677 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-857r2" Oct 30 20:38:27.494321 ip-10-0-133-192 kubenswrapper[2578]: E1030 20:38:27.493741 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-857r2" podUID="bf21cf19-4475-41c3-9153-44d8fb329cdb" Oct 30 20:38:27.494898 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:27.494645 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-lc599" Oct 30 20:38:27.496949 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:27.496825 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-g4v6z\"" Oct 30 20:38:27.497046 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:27.496954 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 30 20:38:27.497046 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:27.497000 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 30 20:38:27.497170 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:27.497067 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-kn2t9" Oct 30 20:38:27.499893 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:27.499665 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-hsv4j" Oct 30 20:38:27.500619 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:27.500477 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6b88d0a3-97c0-4adf-97a7-2e05d5cef781-var-lib-kubelet\") pod \"tuned-2qgck\" (UID: \"6b88d0a3-97c0-4adf-97a7-2e05d5cef781\") " pod="openshift-cluster-node-tuning-operator/tuned-2qgck" Oct 30 20:38:27.500619 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:27.500509 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6b88d0a3-97c0-4adf-97a7-2e05d5cef781-host\") pod \"tuned-2qgck\" (UID: \"6b88d0a3-97c0-4adf-97a7-2e05d5cef781\") " pod="openshift-cluster-node-tuning-operator/tuned-2qgck" Oct 30 20:38:27.500619 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:27.500534 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/6b88d0a3-97c0-4adf-97a7-2e05d5cef781-etc-tuned\") pod \"tuned-2qgck\" (UID: \"6b88d0a3-97c0-4adf-97a7-2e05d5cef781\") " pod="openshift-cluster-node-tuning-operator/tuned-2qgck" Oct 30 20:38:27.500915 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:27.500613 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6b88d0a3-97c0-4adf-97a7-2e05d5cef781-tmp\") pod \"tuned-2qgck\" (UID: \"6b88d0a3-97c0-4adf-97a7-2e05d5cef781\") " pod="openshift-cluster-node-tuning-operator/tuned-2qgck" Oct 30 20:38:27.500915 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:27.500660 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/6b88d0a3-97c0-4adf-97a7-2e05d5cef781-etc-sysconfig\") pod \"tuned-2qgck\" (UID: \"6b88d0a3-97c0-4adf-97a7-2e05d5cef781\") " pod="openshift-cluster-node-tuning-operator/tuned-2qgck" Oct 30 20:38:27.500915 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:27.500683 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/6b88d0a3-97c0-4adf-97a7-2e05d5cef781-etc-sysctl-d\") pod \"tuned-2qgck\" (UID: \"6b88d0a3-97c0-4adf-97a7-2e05d5cef781\") " pod="openshift-cluster-node-tuning-operator/tuned-2qgck" Oct 30 20:38:27.500915 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:27.500702 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/bf21cf19-4475-41c3-9153-44d8fb329cdb-kubelet-config\") pod \"global-pull-secret-syncer-857r2\" (UID: \"bf21cf19-4475-41c3-9153-44d8fb329cdb\") " pod="kube-system/global-pull-secret-syncer-857r2" Oct 30 20:38:27.500915 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:27.500761 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bf21cf19-4475-41c3-9153-44d8fb329cdb-original-pull-secret\") pod \"global-pull-secret-syncer-857r2\" (UID: \"bf21cf19-4475-41c3-9153-44d8fb329cdb\") " pod="kube-system/global-pull-secret-syncer-857r2" Oct 30 20:38:27.500915 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:27.500816 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6hvqv\" (UniqueName: \"kubernetes.io/projected/0dde7eae-1962-4559-89e9-0c67b1165914-kube-api-access-6hvqv\") pod \"network-check-target-jp2cr\" (UID: \"0dde7eae-1962-4559-89e9-0c67b1165914\") " pod="openshift-network-diagnostics/network-check-target-jp2cr" Oct 30 20:38:27.500915 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:27.500877 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/6b88d0a3-97c0-4adf-97a7-2e05d5cef781-etc-modprobe-d\") pod \"tuned-2qgck\" (UID: \"6b88d0a3-97c0-4adf-97a7-2e05d5cef781\") " pod="openshift-cluster-node-tuning-operator/tuned-2qgck" Oct 30 20:38:27.500915 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:27.500907 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mslf8\" (UniqueName: \"kubernetes.io/projected/6b88d0a3-97c0-4adf-97a7-2e05d5cef781-kube-api-access-mslf8\") pod \"tuned-2qgck\" (UID: \"6b88d0a3-97c0-4adf-97a7-2e05d5cef781\") " pod="openshift-cluster-node-tuning-operator/tuned-2qgck" Oct 30 20:38:27.501328 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:27.500975 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c9257033-53bf-40fc-a1e9-67fd5901678b-hosts-file\") pod \"node-resolver-lc599\" (UID: \"c9257033-53bf-40fc-a1e9-67fd5901678b\") " pod="openshift-dns/node-resolver-lc599" Oct 30 20:38:27.501328 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:27.501043 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c9257033-53bf-40fc-a1e9-67fd5901678b-tmp-dir\") pod \"node-resolver-lc599\" (UID: \"c9257033-53bf-40fc-a1e9-67fd5901678b\") " pod="openshift-dns/node-resolver-lc599" Oct 30 20:38:27.501328 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:27.501069 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9jszj\" (UniqueName: \"kubernetes.io/projected/c9257033-53bf-40fc-a1e9-67fd5901678b-kube-api-access-9jszj\") pod \"node-resolver-lc599\" (UID: \"c9257033-53bf-40fc-a1e9-67fd5901678b\") " pod="openshift-dns/node-resolver-lc599" Oct 30 20:38:27.501328 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:27.501131 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/6b88d0a3-97c0-4adf-97a7-2e05d5cef781-etc-sysctl-conf\") pod \"tuned-2qgck\" (UID: \"6b88d0a3-97c0-4adf-97a7-2e05d5cef781\") " pod="openshift-cluster-node-tuning-operator/tuned-2qgck" Oct 30 20:38:27.501328 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:27.501175 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/6b88d0a3-97c0-4adf-97a7-2e05d5cef781-etc-systemd\") pod \"tuned-2qgck\" (UID: \"6b88d0a3-97c0-4adf-97a7-2e05d5cef781\") " pod="openshift-cluster-node-tuning-operator/tuned-2qgck" Oct 30 20:38:27.501328 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:27.501205 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/6b88d0a3-97c0-4adf-97a7-2e05d5cef781-run\") pod \"tuned-2qgck\" (UID: \"6b88d0a3-97c0-4adf-97a7-2e05d5cef781\") " pod="openshift-cluster-node-tuning-operator/tuned-2qgck" Oct 30 20:38:27.501328 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:27.501247 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6b88d0a3-97c0-4adf-97a7-2e05d5cef781-sys\") pod \"tuned-2qgck\" (UID: \"6b88d0a3-97c0-4adf-97a7-2e05d5cef781\") " pod="openshift-cluster-node-tuning-operator/tuned-2qgck" Oct 30 20:38:27.501328 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:27.501272 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/bf21cf19-4475-41c3-9153-44d8fb329cdb-dbus\") pod \"global-pull-secret-syncer-857r2\" (UID: \"bf21cf19-4475-41c3-9153-44d8fb329cdb\") " pod="kube-system/global-pull-secret-syncer-857r2" Oct 30 20:38:27.501328 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:27.501298 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6b88d0a3-97c0-4adf-97a7-2e05d5cef781-etc-kubernetes\") pod \"tuned-2qgck\" (UID: \"6b88d0a3-97c0-4adf-97a7-2e05d5cef781\") " pod="openshift-cluster-node-tuning-operator/tuned-2qgck" Oct 30 20:38:27.501328 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:27.501324 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6b88d0a3-97c0-4adf-97a7-2e05d5cef781-lib-modules\") pod \"tuned-2qgck\" (UID: \"6b88d0a3-97c0-4adf-97a7-2e05d5cef781\") " pod="openshift-cluster-node-tuning-operator/tuned-2qgck" Oct 30 20:38:27.502322 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:27.502033 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-jfxnj" Oct 30 20:38:27.505689 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:27.505396 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pdcfk" Oct 30 20:38:27.505689 ip-10-0-133-192 kubenswrapper[2578]: E1030 20:38:27.505476 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pdcfk" podUID="0915cfe8-4606-4f92-877b-ebe7f52f4187" Oct 30 20:38:27.509404 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:27.508990 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-c4fvc" Oct 30 20:38:27.511767 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:27.511744 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-nxmjg" Oct 30 20:38:27.518229 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:27.518208 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-7w5wz" Oct 30 20:38:27.521175 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:27.521146 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t7h2z" Oct 30 20:38:27.529464 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:27.529386 2578 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 30 20:38:27.529464 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:27.529445 2578 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-h6ctc" Oct 30 20:38:27.536915 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:27.536893 2578 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-h6ctc" Oct 30 20:38:27.591811 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:27.591791 2578 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 30 20:38:27.601612 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:27.601584 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cv9nx\" (UniqueName: \"kubernetes.io/projected/f5f694f2-6f18-409b-b777-2dd7a0eb12b2-kube-api-access-cv9nx\") pod \"aws-ebs-csi-driver-node-t7h2z\" (UID: \"f5f694f2-6f18-409b-b777-2dd7a0eb12b2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t7h2z" Oct 30 20:38:27.601727 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:27.601626 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e277a3d9-3fe9-4454-9420-28afea76f745-host-run-k8s-cni-cncf-io\") pod \"multus-c4fvc\" (UID: \"e277a3d9-3fe9-4454-9420-28afea76f745\") " pod="openshift-multus/multus-c4fvc" Oct 30 20:38:27.601727 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:27.601655 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e277a3d9-3fe9-4454-9420-28afea76f745-host-run-netns\") pod \"multus-c4fvc\" (UID: \"e277a3d9-3fe9-4454-9420-28afea76f745\") " pod="openshift-multus/multus-c4fvc" Oct 30 20:38:27.601848 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:27.601716 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/ea0e353b-8cb1-4759-9f47-03f788f256d3-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-nxmjg\" (UID: \"ea0e353b-8cb1-4759-9f47-03f788f256d3\") " pod="openshift-multus/multus-additional-cni-plugins-nxmjg" Oct 30 20:38:27.601848 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:27.601755 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/ea0e353b-8cb1-4759-9f47-03f788f256d3-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-nxmjg\" (UID: \"ea0e353b-8cb1-4759-9f47-03f788f256d3\") " pod="openshift-multus/multus-additional-cni-plugins-nxmjg" Oct 30 20:38:27.601848 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:27.601809 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/bf21cf19-4475-41c3-9153-44d8fb329cdb-kubelet-config\") pod \"global-pull-secret-syncer-857r2\" (UID: \"bf21cf19-4475-41c3-9153-44d8fb329cdb\") " pod="kube-system/global-pull-secret-syncer-857r2" Oct 30 20:38:27.601848 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:27.601839 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f5f694f2-6f18-409b-b777-2dd7a0eb12b2-registration-dir\") pod \"aws-ebs-csi-driver-node-t7h2z\" (UID: \"f5f694f2-6f18-409b-b777-2dd7a0eb12b2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t7h2z" Oct 30 20:38:27.602063 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:27.601866 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f5f694f2-6f18-409b-b777-2dd7a0eb12b2-device-dir\") pod \"aws-ebs-csi-driver-node-t7h2z\" (UID: \"f5f694f2-6f18-409b-b777-2dd7a0eb12b2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t7h2z" Oct 30 20:38:27.602063 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:27.601927 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mslf8\" (UniqueName: \"kubernetes.io/projected/6b88d0a3-97c0-4adf-97a7-2e05d5cef781-kube-api-access-mslf8\") pod \"tuned-2qgck\" (UID: \"6b88d0a3-97c0-4adf-97a7-2e05d5cef781\") " pod="openshift-cluster-node-tuning-operator/tuned-2qgck" Oct 30 20:38:27.602063 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:27.601956 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c9257033-53bf-40fc-a1e9-67fd5901678b-hosts-file\") pod \"node-resolver-lc599\" (UID: \"c9257033-53bf-40fc-a1e9-67fd5901678b\") " pod="openshift-dns/node-resolver-lc599" Oct 30 20:38:27.602063 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:27.601983 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e277a3d9-3fe9-4454-9420-28afea76f745-host-var-lib-cni-bin\") pod \"multus-c4fvc\" (UID: \"e277a3d9-3fe9-4454-9420-28afea76f745\") " pod="openshift-multus/multus-c4fvc" Oct 30 20:38:27.602063 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:27.602009 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/7e2574c1-b59c-453c-9501-25ec13a1ccad-env-overrides\") pod \"ovnkube-node-7w5wz\" (UID: \"7e2574c1-b59c-453c-9501-25ec13a1ccad\") " pod="openshift-ovn-kubernetes/ovnkube-node-7w5wz" Oct 30 20:38:27.602063 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:27.602033 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/7e2574c1-b59c-453c-9501-25ec13a1ccad-ovnkube-script-lib\") pod \"ovnkube-node-7w5wz\" (UID: \"7e2574c1-b59c-453c-9501-25ec13a1ccad\") " pod="openshift-ovn-kubernetes/ovnkube-node-7w5wz" Oct 30 20:38:27.602063 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:27.602060 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/6b88d0a3-97c0-4adf-97a7-2e05d5cef781-etc-systemd\") pod \"tuned-2qgck\" (UID: \"6b88d0a3-97c0-4adf-97a7-2e05d5cef781\") " pod="openshift-cluster-node-tuning-operator/tuned-2qgck" Oct 30 20:38:27.602465 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:27.602142 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f5f694f2-6f18-409b-b777-2dd7a0eb12b2-kubelet-dir\") pod \"aws-ebs-csi-driver-node-t7h2z\" (UID: \"f5f694f2-6f18-409b-b777-2dd7a0eb12b2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t7h2z" Oct 30 20:38:27.602465 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:27.602170 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6b88d0a3-97c0-4adf-97a7-2e05d5cef781-etc-kubernetes\") pod \"tuned-2qgck\" (UID: \"6b88d0a3-97c0-4adf-97a7-2e05d5cef781\") " pod="openshift-cluster-node-tuning-operator/tuned-2qgck" Oct 30 20:38:27.602465 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:27.602194 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6b88d0a3-97c0-4adf-97a7-2e05d5cef781-var-lib-kubelet\") pod \"tuned-2qgck\" (UID: \"6b88d0a3-97c0-4adf-97a7-2e05d5cef781\") " pod="openshift-cluster-node-tuning-operator/tuned-2qgck" Oct 30 20:38:27.602465 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:27.602222 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/acf9d9fb-1ae4-40ca-a83e-420a9fe40a01-host-slash\") pod \"iptables-alerter-kn2t9\" (UID: \"acf9d9fb-1ae4-40ca-a83e-420a9fe40a01\") " pod="openshift-network-operator/iptables-alerter-kn2t9" Oct 30 20:38:27.602465 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:27.602255 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f5f694f2-6f18-409b-b777-2dd7a0eb12b2-sys-fs\") pod \"aws-ebs-csi-driver-node-t7h2z\" (UID: \"f5f694f2-6f18-409b-b777-2dd7a0eb12b2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t7h2z" Oct 30 20:38:27.602465 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:27.602282 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/6b88d0a3-97c0-4adf-97a7-2e05d5cef781-etc-sysconfig\") pod \"tuned-2qgck\" (UID: \"6b88d0a3-97c0-4adf-97a7-2e05d5cef781\") " pod="openshift-cluster-node-tuning-operator/tuned-2qgck" Oct 30 20:38:27.602465 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:27.602307 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/6b88d0a3-97c0-4adf-97a7-2e05d5cef781-etc-sysctl-d\") pod \"tuned-2qgck\" (UID: \"6b88d0a3-97c0-4adf-97a7-2e05d5cef781\") " pod="openshift-cluster-node-tuning-operator/tuned-2qgck" Oct 30 20:38:27.602465 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:27.602333 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4ntjf\" (UniqueName: \"kubernetes.io/projected/35d51f79-80a9-4f9d-970e-6a35a34976a2-kube-api-access-4ntjf\") pod \"node-ca-hsv4j\" (UID: \"35d51f79-80a9-4f9d-970e-6a35a34976a2\") " pod="openshift-image-registry/node-ca-hsv4j" Oct 30 20:38:27.602465 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:27.602359 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/7e2574c1-b59c-453c-9501-25ec13a1ccad-ovnkube-config\") pod \"ovnkube-node-7w5wz\" (UID: \"7e2574c1-b59c-453c-9501-25ec13a1ccad\") " pod="openshift-ovn-kubernetes/ovnkube-node-7w5wz" Oct 30 20:38:27.602465 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:27.602394 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/e277a3d9-3fe9-4454-9420-28afea76f745-multus-daemon-config\") pod \"multus-c4fvc\" (UID: \"e277a3d9-3fe9-4454-9420-28afea76f745\") " pod="openshift-multus/multus-c4fvc" Oct 30 20:38:27.602465 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:27.602419 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ea0e353b-8cb1-4759-9f47-03f788f256d3-cni-binary-copy\") pod \"multus-additional-cni-plugins-nxmjg\" (UID: \"ea0e353b-8cb1-4759-9f47-03f788f256d3\") " pod="openshift-multus/multus-additional-cni-plugins-nxmjg" Oct 30 20:38:27.602465 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:27.602451 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/7e2574c1-b59c-453c-9501-25ec13a1ccad-host-kubelet\") pod \"ovnkube-node-7w5wz\" (UID: \"7e2574c1-b59c-453c-9501-25ec13a1ccad\") " pod="openshift-ovn-kubernetes/ovnkube-node-7w5wz" Oct 30 20:38:27.603086 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:27.602564 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/7e2574c1-b59c-453c-9501-25ec13a1ccad-systemd-units\") pod \"ovnkube-node-7w5wz\" (UID: \"7e2574c1-b59c-453c-9501-25ec13a1ccad\") " pod="openshift-ovn-kubernetes/ovnkube-node-7w5wz" Oct 30 20:38:27.603086 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:27.602642 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7e2574c1-b59c-453c-9501-25ec13a1ccad-host-run-ovn-kubernetes\") pod \"ovnkube-node-7w5wz\" (UID: \"7e2574c1-b59c-453c-9501-25ec13a1ccad\") " pod="openshift-ovn-kubernetes/ovnkube-node-7w5wz" Oct 30 20:38:27.603086 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:27.602683 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/6b88d0a3-97c0-4adf-97a7-2e05d5cef781-etc-sysctl-conf\") pod \"tuned-2qgck\" (UID: \"6b88d0a3-97c0-4adf-97a7-2e05d5cef781\") " pod="openshift-cluster-node-tuning-operator/tuned-2qgck" Oct 30 20:38:27.603086 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:27.602713 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/6b88d0a3-97c0-4adf-97a7-2e05d5cef781-run\") pod \"tuned-2qgck\" (UID: \"6b88d0a3-97c0-4adf-97a7-2e05d5cef781\") " pod="openshift-cluster-node-tuning-operator/tuned-2qgck" Oct 30 20:38:27.603086 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:27.602740 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e277a3d9-3fe9-4454-9420-28afea76f745-multus-cni-dir\") pod \"multus-c4fvc\" (UID: \"e277a3d9-3fe9-4454-9420-28afea76f745\") " pod="openshift-multus/multus-c4fvc" Oct 30 20:38:27.603086 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:27.602767 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e277a3d9-3fe9-4454-9420-28afea76f745-etc-kubernetes\") pod \"multus-c4fvc\" (UID: \"e277a3d9-3fe9-4454-9420-28afea76f745\") " pod="openshift-multus/multus-c4fvc" Oct 30 20:38:27.603086 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:27.602793 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/7e2574c1-b59c-453c-9501-25ec13a1ccad-ovn-node-metrics-cert\") pod \"ovnkube-node-7w5wz\" (UID: \"7e2574c1-b59c-453c-9501-25ec13a1ccad\") " pod="openshift-ovn-kubernetes/ovnkube-node-7w5wz" Oct 30 20:38:27.603086 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:27.602819 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f5f694f2-6f18-409b-b777-2dd7a0eb12b2-etc-selinux\") pod \"aws-ebs-csi-driver-node-t7h2z\" (UID: \"f5f694f2-6f18-409b-b777-2dd7a0eb12b2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t7h2z" Oct 30 20:38:27.603086 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:27.602847 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6b88d0a3-97c0-4adf-97a7-2e05d5cef781-lib-modules\") pod \"tuned-2qgck\" (UID: \"6b88d0a3-97c0-4adf-97a7-2e05d5cef781\") " pod="openshift-cluster-node-tuning-operator/tuned-2qgck" Oct 30 20:38:27.603086 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:27.602874 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/6b88d0a3-97c0-4adf-97a7-2e05d5cef781-etc-tuned\") pod \"tuned-2qgck\" (UID: \"6b88d0a3-97c0-4adf-97a7-2e05d5cef781\") " pod="openshift-cluster-node-tuning-operator/tuned-2qgck" Oct 30 20:38:27.603086 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:27.602901 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/35d51f79-80a9-4f9d-970e-6a35a34976a2-serviceca\") pod \"node-ca-hsv4j\" (UID: \"35d51f79-80a9-4f9d-970e-6a35a34976a2\") " pod="openshift-image-registry/node-ca-hsv4j" Oct 30 20:38:27.603086 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:27.602941 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e277a3d9-3fe9-4454-9420-28afea76f745-hostroot\") pod \"multus-c4fvc\" (UID: \"e277a3d9-3fe9-4454-9420-28afea76f745\") " pod="openshift-multus/multus-c4fvc" Oct 30 20:38:27.603086 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:27.602968 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7e2574c1-b59c-453c-9501-25ec13a1ccad-etc-openvswitch\") pod \"ovnkube-node-7w5wz\" (UID: \"7e2574c1-b59c-453c-9501-25ec13a1ccad\") " pod="openshift-ovn-kubernetes/ovnkube-node-7w5wz" Oct 30 20:38:27.603086 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:27.602995 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7e2574c1-b59c-453c-9501-25ec13a1ccad-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-7w5wz\" (UID: \"7e2574c1-b59c-453c-9501-25ec13a1ccad\") " pod="openshift-ovn-kubernetes/ovnkube-node-7w5wz" Oct 30 20:38:27.603086 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:27.603026 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bf21cf19-4475-41c3-9153-44d8fb329cdb-original-pull-secret\") pod \"global-pull-secret-syncer-857r2\" (UID: \"bf21cf19-4475-41c3-9153-44d8fb329cdb\") " pod="kube-system/global-pull-secret-syncer-857r2" Oct 30 20:38:27.603086 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:27.603054 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bg7g6\" (UniqueName: \"kubernetes.io/projected/e277a3d9-3fe9-4454-9420-28afea76f745-kube-api-access-bg7g6\") pod \"multus-c4fvc\" (UID: \"e277a3d9-3fe9-4454-9420-28afea76f745\") " pod="openshift-multus/multus-c4fvc" Oct 30 20:38:27.603086 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:27.603081 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ea0e353b-8cb1-4759-9f47-03f788f256d3-cnibin\") pod \"multus-additional-cni-plugins-nxmjg\" (UID: \"ea0e353b-8cb1-4759-9f47-03f788f256d3\") " pod="openshift-multus/multus-additional-cni-plugins-nxmjg" Oct 30 20:38:27.603946 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:27.603107 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ea0e353b-8cb1-4759-9f47-03f788f256d3-tuning-conf-dir\") pod \"multus-additional-cni-plugins-nxmjg\" (UID: \"ea0e353b-8cb1-4759-9f47-03f788f256d3\") " pod="openshift-multus/multus-additional-cni-plugins-nxmjg" Oct 30 20:38:27.603946 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:27.603134 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rjj44\" (UniqueName: \"kubernetes.io/projected/ea0e353b-8cb1-4759-9f47-03f788f256d3-kube-api-access-rjj44\") pod \"multus-additional-cni-plugins-nxmjg\" (UID: \"ea0e353b-8cb1-4759-9f47-03f788f256d3\") " pod="openshift-multus/multus-additional-cni-plugins-nxmjg" Oct 30 20:38:27.603946 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:27.603160 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7e2574c1-b59c-453c-9501-25ec13a1ccad-host-slash\") pod \"ovnkube-node-7w5wz\" (UID: \"7e2574c1-b59c-453c-9501-25ec13a1ccad\") " pod="openshift-ovn-kubernetes/ovnkube-node-7w5wz" Oct 30 20:38:27.603946 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:27.603188 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7e2574c1-b59c-453c-9501-25ec13a1ccad-run-openvswitch\") pod \"ovnkube-node-7w5wz\" (UID: \"7e2574c1-b59c-453c-9501-25ec13a1ccad\") " pod="openshift-ovn-kubernetes/ovnkube-node-7w5wz" Oct 30 20:38:27.603946 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:27.603218 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6hvqv\" (UniqueName: \"kubernetes.io/projected/0dde7eae-1962-4559-89e9-0c67b1165914-kube-api-access-6hvqv\") pod \"network-check-target-jp2cr\" (UID: \"0dde7eae-1962-4559-89e9-0c67b1165914\") " pod="openshift-network-diagnostics/network-check-target-jp2cr" Oct 30 20:38:27.603946 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:27.603415 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/6b88d0a3-97c0-4adf-97a7-2e05d5cef781-etc-modprobe-d\") pod \"tuned-2qgck\" (UID: \"6b88d0a3-97c0-4adf-97a7-2e05d5cef781\") " pod="openshift-cluster-node-tuning-operator/tuned-2qgck" Oct 30 20:38:27.603946 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:27.603445 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c9257033-53bf-40fc-a1e9-67fd5901678b-tmp-dir\") pod \"node-resolver-lc599\" (UID: \"c9257033-53bf-40fc-a1e9-67fd5901678b\") " pod="openshift-dns/node-resolver-lc599" Oct 30 20:38:27.603946 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:27.603473 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/35d51f79-80a9-4f9d-970e-6a35a34976a2-host\") pod \"node-ca-hsv4j\" (UID: \"35d51f79-80a9-4f9d-970e-6a35a34976a2\") " pod="openshift-image-registry/node-ca-hsv4j" Oct 30 20:38:27.603946 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:27.603513 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e277a3d9-3fe9-4454-9420-28afea76f745-multus-socket-dir-parent\") pod \"multus-c4fvc\" (UID: \"e277a3d9-3fe9-4454-9420-28afea76f745\") " pod="openshift-multus/multus-c4fvc" Oct 30 20:38:27.603946 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:27.603552 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e277a3d9-3fe9-4454-9420-28afea76f745-host-var-lib-cni-multus\") pod \"multus-c4fvc\" (UID: \"e277a3d9-3fe9-4454-9420-28afea76f745\") " pod="openshift-multus/multus-c4fvc" Oct 30 20:38:27.603946 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:27.603579 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e277a3d9-3fe9-4454-9420-28afea76f745-host-var-lib-kubelet\") pod \"multus-c4fvc\" (UID: \"e277a3d9-3fe9-4454-9420-28afea76f745\") " pod="openshift-multus/multus-c4fvc" Oct 30 20:38:27.603946 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:27.603624 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e277a3d9-3fe9-4454-9420-28afea76f745-host-run-multus-certs\") pod \"multus-c4fvc\" (UID: \"e277a3d9-3fe9-4454-9420-28afea76f745\") " pod="openshift-multus/multus-c4fvc" Oct 30 20:38:27.603946 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:27.603653 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6b88d0a3-97c0-4adf-97a7-2e05d5cef781-sys\") pod \"tuned-2qgck\" (UID: \"6b88d0a3-97c0-4adf-97a7-2e05d5cef781\") " pod="openshift-cluster-node-tuning-operator/tuned-2qgck" Oct 30 20:38:27.603946 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:27.603692 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/bf21cf19-4475-41c3-9153-44d8fb329cdb-dbus\") pod \"global-pull-secret-syncer-857r2\" (UID: \"bf21cf19-4475-41c3-9153-44d8fb329cdb\") " pod="kube-system/global-pull-secret-syncer-857r2" Oct 30 20:38:27.603946 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:27.603719 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-szhgd\" (UniqueName: \"kubernetes.io/projected/acf9d9fb-1ae4-40ca-a83e-420a9fe40a01-kube-api-access-szhgd\") pod \"iptables-alerter-kn2t9\" (UID: \"acf9d9fb-1ae4-40ca-a83e-420a9fe40a01\") " pod="openshift-network-operator/iptables-alerter-kn2t9" Oct 30 20:38:27.603946 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:27.603746 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e277a3d9-3fe9-4454-9420-28afea76f745-cnibin\") pod \"multus-c4fvc\" (UID: \"e277a3d9-3fe9-4454-9420-28afea76f745\") " pod="openshift-multus/multus-c4fvc" Oct 30 20:38:27.603946 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:27.603777 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6b88d0a3-97c0-4adf-97a7-2e05d5cef781-host\") pod \"tuned-2qgck\" (UID: \"6b88d0a3-97c0-4adf-97a7-2e05d5cef781\") " pod="openshift-cluster-node-tuning-operator/tuned-2qgck" Oct 30 20:38:27.604801 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:27.603802 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6b88d0a3-97c0-4adf-97a7-2e05d5cef781-tmp\") pod \"tuned-2qgck\" (UID: \"6b88d0a3-97c0-4adf-97a7-2e05d5cef781\") " pod="openshift-cluster-node-tuning-operator/tuned-2qgck" Oct 30 20:38:27.604801 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:27.603830 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/da50258d-eef3-4976-8e22-be0e936b9993-agent-certs\") pod \"konnectivity-agent-jfxnj\" (UID: \"da50258d-eef3-4976-8e22-be0e936b9993\") " pod="kube-system/konnectivity-agent-jfxnj" Oct 30 20:38:27.604801 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:27.603859 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0915cfe8-4606-4f92-877b-ebe7f52f4187-metrics-certs\") pod \"network-metrics-daemon-pdcfk\" (UID: \"0915cfe8-4606-4f92-877b-ebe7f52f4187\") " pod="openshift-multus/network-metrics-daemon-pdcfk" Oct 30 20:38:27.604801 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:27.603888 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7e2574c1-b59c-453c-9501-25ec13a1ccad-host-cni-bin\") pod \"ovnkube-node-7w5wz\" (UID: \"7e2574c1-b59c-453c-9501-25ec13a1ccad\") " pod="openshift-ovn-kubernetes/ovnkube-node-7w5wz" Oct 30 20:38:27.604801 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:27.603914 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/acf9d9fb-1ae4-40ca-a83e-420a9fe40a01-iptables-alerter-script\") pod \"iptables-alerter-kn2t9\" (UID: \"acf9d9fb-1ae4-40ca-a83e-420a9fe40a01\") " pod="openshift-network-operator/iptables-alerter-kn2t9" Oct 30 20:38:27.604801 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:27.603942 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e277a3d9-3fe9-4454-9420-28afea76f745-system-cni-dir\") pod \"multus-c4fvc\" (UID: \"e277a3d9-3fe9-4454-9420-28afea76f745\") " pod="openshift-multus/multus-c4fvc" Oct 30 20:38:27.604801 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:27.603969 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ea0e353b-8cb1-4759-9f47-03f788f256d3-system-cni-dir\") pod \"multus-additional-cni-plugins-nxmjg\" (UID: \"ea0e353b-8cb1-4759-9f47-03f788f256d3\") " pod="openshift-multus/multus-additional-cni-plugins-nxmjg" Oct 30 20:38:27.604801 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:27.603996 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7e2574c1-b59c-453c-9501-25ec13a1ccad-host-run-netns\") pod \"ovnkube-node-7w5wz\" (UID: \"7e2574c1-b59c-453c-9501-25ec13a1ccad\") " pod="openshift-ovn-kubernetes/ovnkube-node-7w5wz" Oct 30 20:38:27.604801 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:27.604023 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/7e2574c1-b59c-453c-9501-25ec13a1ccad-run-systemd\") pod \"ovnkube-node-7w5wz\" (UID: \"7e2574c1-b59c-453c-9501-25ec13a1ccad\") " pod="openshift-ovn-kubernetes/ovnkube-node-7w5wz" Oct 30 20:38:27.604801 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:27.604051 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7e2574c1-b59c-453c-9501-25ec13a1ccad-var-lib-openvswitch\") pod \"ovnkube-node-7w5wz\" (UID: \"7e2574c1-b59c-453c-9501-25ec13a1ccad\") " pod="openshift-ovn-kubernetes/ovnkube-node-7w5wz" Oct 30 20:38:27.604801 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:27.604103 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/7e2574c1-b59c-453c-9501-25ec13a1ccad-run-ovn\") pod \"ovnkube-node-7w5wz\" (UID: \"7e2574c1-b59c-453c-9501-25ec13a1ccad\") " pod="openshift-ovn-kubernetes/ovnkube-node-7w5wz" Oct 30 20:38:27.604801 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:27.604158 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f5f694f2-6f18-409b-b777-2dd7a0eb12b2-socket-dir\") pod \"aws-ebs-csi-driver-node-t7h2z\" (UID: \"f5f694f2-6f18-409b-b777-2dd7a0eb12b2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t7h2z" Oct 30 20:38:27.604801 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:27.604189 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9jszj\" (UniqueName: \"kubernetes.io/projected/c9257033-53bf-40fc-a1e9-67fd5901678b-kube-api-access-9jszj\") pod \"node-resolver-lc599\" (UID: \"c9257033-53bf-40fc-a1e9-67fd5901678b\") " pod="openshift-dns/node-resolver-lc599" Oct 30 20:38:27.604801 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:27.604220 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/da50258d-eef3-4976-8e22-be0e936b9993-konnectivity-ca\") pod \"konnectivity-agent-jfxnj\" (UID: \"da50258d-eef3-4976-8e22-be0e936b9993\") " pod="kube-system/konnectivity-agent-jfxnj" Oct 30 20:38:27.604801 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:27.604250 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mjbz2\" (UniqueName: \"kubernetes.io/projected/0915cfe8-4606-4f92-877b-ebe7f52f4187-kube-api-access-mjbz2\") pod \"network-metrics-daemon-pdcfk\" (UID: \"0915cfe8-4606-4f92-877b-ebe7f52f4187\") " pod="openshift-multus/network-metrics-daemon-pdcfk" Oct 30 20:38:27.604801 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:27.604277 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e277a3d9-3fe9-4454-9420-28afea76f745-cni-binary-copy\") pod \"multus-c4fvc\" (UID: \"e277a3d9-3fe9-4454-9420-28afea76f745\") " pod="openshift-multus/multus-c4fvc" Oct 30 20:38:27.605609 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:27.604304 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e277a3d9-3fe9-4454-9420-28afea76f745-multus-conf-dir\") pod \"multus-c4fvc\" (UID: \"e277a3d9-3fe9-4454-9420-28afea76f745\") " pod="openshift-multus/multus-c4fvc" Oct 30 20:38:27.605609 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:27.604330 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/7e2574c1-b59c-453c-9501-25ec13a1ccad-log-socket\") pod \"ovnkube-node-7w5wz\" (UID: \"7e2574c1-b59c-453c-9501-25ec13a1ccad\") " pod="openshift-ovn-kubernetes/ovnkube-node-7w5wz" Oct 30 20:38:27.605609 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:27.604356 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/7e2574c1-b59c-453c-9501-25ec13a1ccad-node-log\") pod \"ovnkube-node-7w5wz\" (UID: \"7e2574c1-b59c-453c-9501-25ec13a1ccad\") " pod="openshift-ovn-kubernetes/ovnkube-node-7w5wz" Oct 30 20:38:27.605609 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:27.604384 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/7e2574c1-b59c-453c-9501-25ec13a1ccad-host-cni-netd\") pod \"ovnkube-node-7w5wz\" (UID: \"7e2574c1-b59c-453c-9501-25ec13a1ccad\") " pod="openshift-ovn-kubernetes/ovnkube-node-7w5wz" Oct 30 20:38:27.605609 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:27.604423 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jk5kq\" (UniqueName: \"kubernetes.io/projected/7e2574c1-b59c-453c-9501-25ec13a1ccad-kube-api-access-jk5kq\") pod \"ovnkube-node-7w5wz\" (UID: \"7e2574c1-b59c-453c-9501-25ec13a1ccad\") " pod="openshift-ovn-kubernetes/ovnkube-node-7w5wz" Oct 30 20:38:27.605609 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:27.604637 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/6b88d0a3-97c0-4adf-97a7-2e05d5cef781-etc-systemd\") pod \"tuned-2qgck\" (UID: \"6b88d0a3-97c0-4adf-97a7-2e05d5cef781\") " pod="openshift-cluster-node-tuning-operator/tuned-2qgck" Oct 30 20:38:27.605609 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:27.604710 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/bf21cf19-4475-41c3-9153-44d8fb329cdb-kubelet-config\") pod \"global-pull-secret-syncer-857r2\" (UID: \"bf21cf19-4475-41c3-9153-44d8fb329cdb\") " pod="kube-system/global-pull-secret-syncer-857r2" Oct 30 20:38:27.605609 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:27.604720 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/6b88d0a3-97c0-4adf-97a7-2e05d5cef781-etc-modprobe-d\") pod \"tuned-2qgck\" (UID: \"6b88d0a3-97c0-4adf-97a7-2e05d5cef781\") " pod="openshift-cluster-node-tuning-operator/tuned-2qgck" Oct 30 20:38:27.605609 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:27.604819 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6b88d0a3-97c0-4adf-97a7-2e05d5cef781-etc-kubernetes\") pod \"tuned-2qgck\" (UID: \"6b88d0a3-97c0-4adf-97a7-2e05d5cef781\") " pod="openshift-cluster-node-tuning-operator/tuned-2qgck" Oct 30 20:38:27.605609 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:27.604831 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6b88d0a3-97c0-4adf-97a7-2e05d5cef781-sys\") pod \"tuned-2qgck\" (UID: \"6b88d0a3-97c0-4adf-97a7-2e05d5cef781\") " pod="openshift-cluster-node-tuning-operator/tuned-2qgck" Oct 30 20:38:27.605609 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:27.604865 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6b88d0a3-97c0-4adf-97a7-2e05d5cef781-var-lib-kubelet\") pod \"tuned-2qgck\" (UID: \"6b88d0a3-97c0-4adf-97a7-2e05d5cef781\") " pod="openshift-cluster-node-tuning-operator/tuned-2qgck" Oct 30 20:38:27.605609 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:27.604940 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/bf21cf19-4475-41c3-9153-44d8fb329cdb-dbus\") pod \"global-pull-secret-syncer-857r2\" (UID: \"bf21cf19-4475-41c3-9153-44d8fb329cdb\") " pod="kube-system/global-pull-secret-syncer-857r2" Oct 30 20:38:27.605609 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:27.604958 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/6b88d0a3-97c0-4adf-97a7-2e05d5cef781-etc-sysconfig\") pod \"tuned-2qgck\" (UID: \"6b88d0a3-97c0-4adf-97a7-2e05d5cef781\") " pod="openshift-cluster-node-tuning-operator/tuned-2qgck" Oct 30 20:38:27.605609 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:27.605041 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/6b88d0a3-97c0-4adf-97a7-2e05d5cef781-etc-sysctl-d\") pod \"tuned-2qgck\" (UID: \"6b88d0a3-97c0-4adf-97a7-2e05d5cef781\") " pod="openshift-cluster-node-tuning-operator/tuned-2qgck" Oct 30 20:38:27.605609 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:27.605087 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c9257033-53bf-40fc-a1e9-67fd5901678b-hosts-file\") pod \"node-resolver-lc599\" (UID: \"c9257033-53bf-40fc-a1e9-67fd5901678b\") " pod="openshift-dns/node-resolver-lc599" Oct 30 20:38:27.605609 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:27.605168 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/6b88d0a3-97c0-4adf-97a7-2e05d5cef781-etc-sysctl-conf\") pod \"tuned-2qgck\" (UID: \"6b88d0a3-97c0-4adf-97a7-2e05d5cef781\") " pod="openshift-cluster-node-tuning-operator/tuned-2qgck" Oct 30 20:38:27.605609 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:27.605239 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/6b88d0a3-97c0-4adf-97a7-2e05d5cef781-run\") pod \"tuned-2qgck\" (UID: \"6b88d0a3-97c0-4adf-97a7-2e05d5cef781\") " pod="openshift-cluster-node-tuning-operator/tuned-2qgck" Oct 30 20:38:27.605609 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:27.605256 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6b88d0a3-97c0-4adf-97a7-2e05d5cef781-host\") pod \"tuned-2qgck\" (UID: \"6b88d0a3-97c0-4adf-97a7-2e05d5cef781\") " pod="openshift-cluster-node-tuning-operator/tuned-2qgck" Oct 30 20:38:27.606328 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:27.605361 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6b88d0a3-97c0-4adf-97a7-2e05d5cef781-lib-modules\") pod \"tuned-2qgck\" (UID: \"6b88d0a3-97c0-4adf-97a7-2e05d5cef781\") " pod="openshift-cluster-node-tuning-operator/tuned-2qgck" Oct 30 20:38:27.606328 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:27.605381 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c9257033-53bf-40fc-a1e9-67fd5901678b-tmp-dir\") pod \"node-resolver-lc599\" (UID: \"c9257033-53bf-40fc-a1e9-67fd5901678b\") " pod="openshift-dns/node-resolver-lc599" Oct 30 20:38:27.606328 ip-10-0-133-192 kubenswrapper[2578]: E1030 20:38:27.605642 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 20:38:27.606328 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:27.605672 2578 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 30 20:38:27.606328 ip-10-0-133-192 kubenswrapper[2578]: E1030 20:38:27.605723 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bf21cf19-4475-41c3-9153-44d8fb329cdb-original-pull-secret podName:bf21cf19-4475-41c3-9153-44d8fb329cdb nodeName:}" failed. No retries permitted until 2026-10-30 20:38:28.105702187 +0000 UTC m=+2.976530745 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/bf21cf19-4475-41c3-9153-44d8fb329cdb-original-pull-secret") pod "global-pull-secret-syncer-857r2" (UID: "bf21cf19-4475-41c3-9153-44d8fb329cdb") : object "kube-system"/"original-pull-secret" not registered Oct 30 20:38:27.606328 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:27.605753 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 30 20:38:27.606328 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:27.605940 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 30 20:38:27.606328 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:27.605961 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 30 20:38:27.606727 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:27.606357 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 30 20:38:27.606727 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:27.606402 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 30 20:38:27.606727 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:27.606533 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-4gdbd\"" Oct 30 20:38:27.608541 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:27.608207 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 30 20:38:27.608541 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:27.608264 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 30 20:38:27.608541 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:27.608321 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-w89q8\"" Oct 30 20:38:27.608541 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:27.608369 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-7chw8\"" Oct 30 20:38:27.608541 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:27.608441 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-b7wk5\"" Oct 30 20:38:27.608541 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:27.608472 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 30 20:38:27.608541 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:27.608509 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 30 20:38:27.608955 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:27.608786 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 30 20:38:27.608955 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:27.608835 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 30 20:38:27.608955 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:27.608924 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 30 20:38:27.608955 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:27.608951 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 30 20:38:27.609149 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:27.609040 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-8b74p\"" Oct 30 20:38:27.609149 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:27.609087 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 30 20:38:27.609149 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:27.609120 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 30 20:38:27.609307 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:27.609257 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 30 20:38:27.609359 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:27.609343 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 30 20:38:27.609415 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:27.609384 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 30 20:38:27.609453 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:27.609419 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 30 20:38:27.609491 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:27.609483 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 30 20:38:27.609536 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:27.609516 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 30 20:38:27.610223 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:27.610198 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6b88d0a3-97c0-4adf-97a7-2e05d5cef781-tmp\") pod \"tuned-2qgck\" (UID: \"6b88d0a3-97c0-4adf-97a7-2e05d5cef781\") " pod="openshift-cluster-node-tuning-operator/tuned-2qgck" Oct 30 20:38:27.610328 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:27.610309 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/6b88d0a3-97c0-4adf-97a7-2e05d5cef781-etc-tuned\") pod \"tuned-2qgck\" (UID: \"6b88d0a3-97c0-4adf-97a7-2e05d5cef781\") " pod="openshift-cluster-node-tuning-operator/tuned-2qgck" Oct 30 20:38:27.610649 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:27.610617 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 30 20:38:27.610649 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:27.610628 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 30 20:38:27.611546 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:27.611446 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-kszdw\"" Oct 30 20:38:27.611546 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:27.611485 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-hh9qp\"" Oct 30 20:38:27.611546 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:27.611518 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 30 20:38:27.611546 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:27.611523 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-4khcx\"" Oct 30 20:38:27.611774 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:27.611660 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 30 20:38:27.620261 ip-10-0-133-192 kubenswrapper[2578]: E1030 20:38:27.620238 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 20:38:27.620385 ip-10-0-133-192 kubenswrapper[2578]: E1030 20:38:27.620267 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 20:38:27.620385 ip-10-0-133-192 kubenswrapper[2578]: E1030 20:38:27.620278 2578 projected.go:196] Error preparing data for projected volume kube-api-access-6hvqv for pod openshift-network-diagnostics/network-check-target-jp2cr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 20:38:27.620385 ip-10-0-133-192 kubenswrapper[2578]: E1030 20:38:27.620333 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0dde7eae-1962-4559-89e9-0c67b1165914-kube-api-access-6hvqv podName:0dde7eae-1962-4559-89e9-0c67b1165914 nodeName:}" failed. No retries permitted until 2026-10-30 20:38:28.120317623 +0000 UTC m=+2.991146166 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-6hvqv" (UniqueName: "kubernetes.io/projected/0dde7eae-1962-4559-89e9-0c67b1165914-kube-api-access-6hvqv") pod "network-check-target-jp2cr" (UID: "0dde7eae-1962-4559-89e9-0c67b1165914") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 20:38:27.622780 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:27.622760 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9jszj\" (UniqueName: \"kubernetes.io/projected/c9257033-53bf-40fc-a1e9-67fd5901678b-kube-api-access-9jszj\") pod \"node-resolver-lc599\" (UID: \"c9257033-53bf-40fc-a1e9-67fd5901678b\") " pod="openshift-dns/node-resolver-lc599" Oct 30 20:38:27.623014 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:27.622996 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mslf8\" (UniqueName: \"kubernetes.io/projected/6b88d0a3-97c0-4adf-97a7-2e05d5cef781-kube-api-access-mslf8\") pod \"tuned-2qgck\" (UID: \"6b88d0a3-97c0-4adf-97a7-2e05d5cef781\") " pod="openshift-cluster-node-tuning-operator/tuned-2qgck" Oct 30 20:38:27.705730 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:27.705658 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bg7g6\" (UniqueName: \"kubernetes.io/projected/e277a3d9-3fe9-4454-9420-28afea76f745-kube-api-access-bg7g6\") pod \"multus-c4fvc\" (UID: \"e277a3d9-3fe9-4454-9420-28afea76f745\") " pod="openshift-multus/multus-c4fvc" Oct 30 20:38:27.705730 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:27.705701 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ea0e353b-8cb1-4759-9f47-03f788f256d3-cnibin\") pod \"multus-additional-cni-plugins-nxmjg\" (UID: \"ea0e353b-8cb1-4759-9f47-03f788f256d3\") " pod="openshift-multus/multus-additional-cni-plugins-nxmjg" Oct 30 20:38:27.705939 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:27.705729 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ea0e353b-8cb1-4759-9f47-03f788f256d3-tuning-conf-dir\") pod \"multus-additional-cni-plugins-nxmjg\" (UID: \"ea0e353b-8cb1-4759-9f47-03f788f256d3\") " pod="openshift-multus/multus-additional-cni-plugins-nxmjg" Oct 30 20:38:27.705939 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:27.705758 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rjj44\" (UniqueName: \"kubernetes.io/projected/ea0e353b-8cb1-4759-9f47-03f788f256d3-kube-api-access-rjj44\") pod \"multus-additional-cni-plugins-nxmjg\" (UID: \"ea0e353b-8cb1-4759-9f47-03f788f256d3\") " pod="openshift-multus/multus-additional-cni-plugins-nxmjg" Oct 30 20:38:27.705939 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:27.705786 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7e2574c1-b59c-453c-9501-25ec13a1ccad-host-slash\") pod \"ovnkube-node-7w5wz\" (UID: \"7e2574c1-b59c-453c-9501-25ec13a1ccad\") " pod="openshift-ovn-kubernetes/ovnkube-node-7w5wz" Oct 30 20:38:27.705939 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:27.705786 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ea0e353b-8cb1-4759-9f47-03f788f256d3-cnibin\") pod \"multus-additional-cni-plugins-nxmjg\" (UID: \"ea0e353b-8cb1-4759-9f47-03f788f256d3\") " pod="openshift-multus/multus-additional-cni-plugins-nxmjg" Oct 30 20:38:27.705939 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:27.705810 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7e2574c1-b59c-453c-9501-25ec13a1ccad-run-openvswitch\") pod \"ovnkube-node-7w5wz\" (UID: \"7e2574c1-b59c-453c-9501-25ec13a1ccad\") " pod="openshift-ovn-kubernetes/ovnkube-node-7w5wz" Oct 30 20:38:27.705939 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:27.705845 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/35d51f79-80a9-4f9d-970e-6a35a34976a2-host\") pod \"node-ca-hsv4j\" (UID: \"35d51f79-80a9-4f9d-970e-6a35a34976a2\") " pod="openshift-image-registry/node-ca-hsv4j" Oct 30 20:38:27.705939 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:27.705865 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7e2574c1-b59c-453c-9501-25ec13a1ccad-host-slash\") pod \"ovnkube-node-7w5wz\" (UID: \"7e2574c1-b59c-453c-9501-25ec13a1ccad\") " pod="openshift-ovn-kubernetes/ovnkube-node-7w5wz" Oct 30 20:38:27.705939 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:27.705868 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e277a3d9-3fe9-4454-9420-28afea76f745-multus-socket-dir-parent\") pod \"multus-c4fvc\" (UID: \"e277a3d9-3fe9-4454-9420-28afea76f745\") " pod="openshift-multus/multus-c4fvc" Oct 30 20:38:27.705939 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:27.705905 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7e2574c1-b59c-453c-9501-25ec13a1ccad-run-openvswitch\") pod \"ovnkube-node-7w5wz\" (UID: \"7e2574c1-b59c-453c-9501-25ec13a1ccad\") " pod="openshift-ovn-kubernetes/ovnkube-node-7w5wz" Oct 30 20:38:27.705939 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:27.705920 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e277a3d9-3fe9-4454-9420-28afea76f745-multus-socket-dir-parent\") pod \"multus-c4fvc\" (UID: \"e277a3d9-3fe9-4454-9420-28afea76f745\") " pod="openshift-multus/multus-c4fvc" Oct 30 20:38:27.705939 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:27.705914 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e277a3d9-3fe9-4454-9420-28afea76f745-host-var-lib-cni-multus\") pod \"multus-c4fvc\" (UID: \"e277a3d9-3fe9-4454-9420-28afea76f745\") " pod="openshift-multus/multus-c4fvc" Oct 30 20:38:27.706431 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:27.705949 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e277a3d9-3fe9-4454-9420-28afea76f745-host-var-lib-cni-multus\") pod \"multus-c4fvc\" (UID: \"e277a3d9-3fe9-4454-9420-28afea76f745\") " pod="openshift-multus/multus-c4fvc" Oct 30 20:38:27.706431 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:27.705966 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/35d51f79-80a9-4f9d-970e-6a35a34976a2-host\") pod \"node-ca-hsv4j\" (UID: \"35d51f79-80a9-4f9d-970e-6a35a34976a2\") " pod="openshift-image-registry/node-ca-hsv4j" Oct 30 20:38:27.706431 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:27.705966 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e277a3d9-3fe9-4454-9420-28afea76f745-host-var-lib-kubelet\") pod \"multus-c4fvc\" (UID: \"e277a3d9-3fe9-4454-9420-28afea76f745\") " pod="openshift-multus/multus-c4fvc" Oct 30 20:38:27.706431 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:27.706003 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e277a3d9-3fe9-4454-9420-28afea76f745-host-var-lib-kubelet\") pod \"multus-c4fvc\" (UID: \"e277a3d9-3fe9-4454-9420-28afea76f745\") " pod="openshift-multus/multus-c4fvc" Oct 30 20:38:27.706431 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:27.706005 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e277a3d9-3fe9-4454-9420-28afea76f745-host-run-multus-certs\") pod \"multus-c4fvc\" (UID: \"e277a3d9-3fe9-4454-9420-28afea76f745\") " pod="openshift-multus/multus-c4fvc" Oct 30 20:38:27.706431 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:27.706023 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ea0e353b-8cb1-4759-9f47-03f788f256d3-tuning-conf-dir\") pod \"multus-additional-cni-plugins-nxmjg\" (UID: \"ea0e353b-8cb1-4759-9f47-03f788f256d3\") " pod="openshift-multus/multus-additional-cni-plugins-nxmjg" Oct 30 20:38:27.706431 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:27.706036 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e277a3d9-3fe9-4454-9420-28afea76f745-host-run-multus-certs\") pod \"multus-c4fvc\" (UID: \"e277a3d9-3fe9-4454-9420-28afea76f745\") " pod="openshift-multus/multus-c4fvc" Oct 30 20:38:27.706431 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:27.706041 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-szhgd\" (UniqueName: \"kubernetes.io/projected/acf9d9fb-1ae4-40ca-a83e-420a9fe40a01-kube-api-access-szhgd\") pod \"iptables-alerter-kn2t9\" (UID: \"acf9d9fb-1ae4-40ca-a83e-420a9fe40a01\") " pod="openshift-network-operator/iptables-alerter-kn2t9" Oct 30 20:38:27.706431 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:27.706154 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e277a3d9-3fe9-4454-9420-28afea76f745-cnibin\") pod \"multus-c4fvc\" (UID: \"e277a3d9-3fe9-4454-9420-28afea76f745\") " pod="openshift-multus/multus-c4fvc" Oct 30 20:38:27.706431 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:27.706196 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/da50258d-eef3-4976-8e22-be0e936b9993-agent-certs\") pod \"konnectivity-agent-jfxnj\" (UID: \"da50258d-eef3-4976-8e22-be0e936b9993\") " pod="kube-system/konnectivity-agent-jfxnj" Oct 30 20:38:27.706431 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:27.706221 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0915cfe8-4606-4f92-877b-ebe7f52f4187-metrics-certs\") pod \"network-metrics-daemon-pdcfk\" (UID: \"0915cfe8-4606-4f92-877b-ebe7f52f4187\") " pod="openshift-multus/network-metrics-daemon-pdcfk" Oct 30 20:38:27.706431 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:27.706246 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7e2574c1-b59c-453c-9501-25ec13a1ccad-host-cni-bin\") pod \"ovnkube-node-7w5wz\" (UID: \"7e2574c1-b59c-453c-9501-25ec13a1ccad\") " pod="openshift-ovn-kubernetes/ovnkube-node-7w5wz" Oct 30 20:38:27.706431 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:27.706271 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/acf9d9fb-1ae4-40ca-a83e-420a9fe40a01-iptables-alerter-script\") pod \"iptables-alerter-kn2t9\" (UID: \"acf9d9fb-1ae4-40ca-a83e-420a9fe40a01\") " pod="openshift-network-operator/iptables-alerter-kn2t9" Oct 30 20:38:27.706431 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:27.706279 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e277a3d9-3fe9-4454-9420-28afea76f745-cnibin\") pod \"multus-c4fvc\" (UID: \"e277a3d9-3fe9-4454-9420-28afea76f745\") " pod="openshift-multus/multus-c4fvc" Oct 30 20:38:27.706431 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:27.706296 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e277a3d9-3fe9-4454-9420-28afea76f745-system-cni-dir\") pod \"multus-c4fvc\" (UID: \"e277a3d9-3fe9-4454-9420-28afea76f745\") " pod="openshift-multus/multus-c4fvc" Oct 30 20:38:27.706431 ip-10-0-133-192 kubenswrapper[2578]: E1030 20:38:27.706331 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 20:38:27.706431 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:27.706346 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7e2574c1-b59c-453c-9501-25ec13a1ccad-host-cni-bin\") pod \"ovnkube-node-7w5wz\" (UID: \"7e2574c1-b59c-453c-9501-25ec13a1ccad\") " pod="openshift-ovn-kubernetes/ovnkube-node-7w5wz" Oct 30 20:38:27.706431 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:27.706352 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e277a3d9-3fe9-4454-9420-28afea76f745-system-cni-dir\") pod \"multus-c4fvc\" (UID: \"e277a3d9-3fe9-4454-9420-28afea76f745\") " pod="openshift-multus/multus-c4fvc" Oct 30 20:38:27.707289 ip-10-0-133-192 kubenswrapper[2578]: E1030 20:38:27.706390 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0915cfe8-4606-4f92-877b-ebe7f52f4187-metrics-certs podName:0915cfe8-4606-4f92-877b-ebe7f52f4187 nodeName:}" failed. No retries permitted until 2026-10-30 20:38:28.206372345 +0000 UTC m=+3.077200903 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0915cfe8-4606-4f92-877b-ebe7f52f4187-metrics-certs") pod "network-metrics-daemon-pdcfk" (UID: "0915cfe8-4606-4f92-877b-ebe7f52f4187") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 20:38:27.707289 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:27.706414 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ea0e353b-8cb1-4759-9f47-03f788f256d3-system-cni-dir\") pod \"multus-additional-cni-plugins-nxmjg\" (UID: \"ea0e353b-8cb1-4759-9f47-03f788f256d3\") " pod="openshift-multus/multus-additional-cni-plugins-nxmjg" Oct 30 20:38:27.707289 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:27.706444 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7e2574c1-b59c-453c-9501-25ec13a1ccad-host-run-netns\") pod \"ovnkube-node-7w5wz\" (UID: \"7e2574c1-b59c-453c-9501-25ec13a1ccad\") " pod="openshift-ovn-kubernetes/ovnkube-node-7w5wz" Oct 30 20:38:27.707289 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:27.706469 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/7e2574c1-b59c-453c-9501-25ec13a1ccad-run-systemd\") pod \"ovnkube-node-7w5wz\" (UID: \"7e2574c1-b59c-453c-9501-25ec13a1ccad\") " pod="openshift-ovn-kubernetes/ovnkube-node-7w5wz" Oct 30 20:38:27.707289 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:27.706494 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7e2574c1-b59c-453c-9501-25ec13a1ccad-var-lib-openvswitch\") pod \"ovnkube-node-7w5wz\" (UID: \"7e2574c1-b59c-453c-9501-25ec13a1ccad\") " pod="openshift-ovn-kubernetes/ovnkube-node-7w5wz" Oct 30 20:38:27.707289 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:27.706518 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/7e2574c1-b59c-453c-9501-25ec13a1ccad-run-ovn\") pod \"ovnkube-node-7w5wz\" (UID: \"7e2574c1-b59c-453c-9501-25ec13a1ccad\") " pod="openshift-ovn-kubernetes/ovnkube-node-7w5wz" Oct 30 20:38:27.707289 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:27.706541 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f5f694f2-6f18-409b-b777-2dd7a0eb12b2-socket-dir\") pod \"aws-ebs-csi-driver-node-t7h2z\" (UID: \"f5f694f2-6f18-409b-b777-2dd7a0eb12b2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t7h2z" Oct 30 20:38:27.707289 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:27.706581 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/da50258d-eef3-4976-8e22-be0e936b9993-konnectivity-ca\") pod \"konnectivity-agent-jfxnj\" (UID: \"da50258d-eef3-4976-8e22-be0e936b9993\") " pod="kube-system/konnectivity-agent-jfxnj" Oct 30 20:38:27.707289 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:27.706602 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mjbz2\" (UniqueName: \"kubernetes.io/projected/0915cfe8-4606-4f92-877b-ebe7f52f4187-kube-api-access-mjbz2\") pod \"network-metrics-daemon-pdcfk\" (UID: \"0915cfe8-4606-4f92-877b-ebe7f52f4187\") " pod="openshift-multus/network-metrics-daemon-pdcfk" Oct 30 20:38:27.707289 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:27.706619 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e277a3d9-3fe9-4454-9420-28afea76f745-cni-binary-copy\") pod \"multus-c4fvc\" (UID: \"e277a3d9-3fe9-4454-9420-28afea76f745\") " pod="openshift-multus/multus-c4fvc" Oct 30 20:38:27.707289 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:27.706636 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e277a3d9-3fe9-4454-9420-28afea76f745-multus-conf-dir\") pod \"multus-c4fvc\" (UID: \"e277a3d9-3fe9-4454-9420-28afea76f745\") " pod="openshift-multus/multus-c4fvc" Oct 30 20:38:27.707289 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:27.706652 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/7e2574c1-b59c-453c-9501-25ec13a1ccad-log-socket\") pod \"ovnkube-node-7w5wz\" (UID: \"7e2574c1-b59c-453c-9501-25ec13a1ccad\") " pod="openshift-ovn-kubernetes/ovnkube-node-7w5wz" Oct 30 20:38:27.707289 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:27.706666 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/7e2574c1-b59c-453c-9501-25ec13a1ccad-node-log\") pod \"ovnkube-node-7w5wz\" (UID: \"7e2574c1-b59c-453c-9501-25ec13a1ccad\") " pod="openshift-ovn-kubernetes/ovnkube-node-7w5wz" Oct 30 20:38:27.707289 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:27.706681 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/7e2574c1-b59c-453c-9501-25ec13a1ccad-host-cni-netd\") pod \"ovnkube-node-7w5wz\" (UID: \"7e2574c1-b59c-453c-9501-25ec13a1ccad\") " pod="openshift-ovn-kubernetes/ovnkube-node-7w5wz" Oct 30 20:38:27.707289 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:27.706696 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jk5kq\" (UniqueName: \"kubernetes.io/projected/7e2574c1-b59c-453c-9501-25ec13a1ccad-kube-api-access-jk5kq\") pod \"ovnkube-node-7w5wz\" (UID: \"7e2574c1-b59c-453c-9501-25ec13a1ccad\") " pod="openshift-ovn-kubernetes/ovnkube-node-7w5wz" Oct 30 20:38:27.707289 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:27.706713 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cv9nx\" (UniqueName: \"kubernetes.io/projected/f5f694f2-6f18-409b-b777-2dd7a0eb12b2-kube-api-access-cv9nx\") pod \"aws-ebs-csi-driver-node-t7h2z\" (UID: \"f5f694f2-6f18-409b-b777-2dd7a0eb12b2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t7h2z" Oct 30 20:38:27.707887 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:27.706730 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e277a3d9-3fe9-4454-9420-28afea76f745-host-run-k8s-cni-cncf-io\") pod \"multus-c4fvc\" (UID: \"e277a3d9-3fe9-4454-9420-28afea76f745\") " pod="openshift-multus/multus-c4fvc" Oct 30 20:38:27.707887 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:27.706746 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e277a3d9-3fe9-4454-9420-28afea76f745-host-run-netns\") pod \"multus-c4fvc\" (UID: \"e277a3d9-3fe9-4454-9420-28afea76f745\") " pod="openshift-multus/multus-c4fvc" Oct 30 20:38:27.707887 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:27.706790 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/7e2574c1-b59c-453c-9501-25ec13a1ccad-host-cni-netd\") pod \"ovnkube-node-7w5wz\" (UID: \"7e2574c1-b59c-453c-9501-25ec13a1ccad\") " pod="openshift-ovn-kubernetes/ovnkube-node-7w5wz" Oct 30 20:38:27.707887 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:27.706795 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/7e2574c1-b59c-453c-9501-25ec13a1ccad-run-systemd\") pod \"ovnkube-node-7w5wz\" (UID: \"7e2574c1-b59c-453c-9501-25ec13a1ccad\") " pod="openshift-ovn-kubernetes/ovnkube-node-7w5wz" Oct 30 20:38:27.707887 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:27.706838 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/acf9d9fb-1ae4-40ca-a83e-420a9fe40a01-iptables-alerter-script\") pod \"iptables-alerter-kn2t9\" (UID: \"acf9d9fb-1ae4-40ca-a83e-420a9fe40a01\") " pod="openshift-network-operator/iptables-alerter-kn2t9" Oct 30 20:38:27.707887 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:27.706845 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e277a3d9-3fe9-4454-9420-28afea76f745-host-run-netns\") pod \"multus-c4fvc\" (UID: \"e277a3d9-3fe9-4454-9420-28afea76f745\") " pod="openshift-multus/multus-c4fvc" Oct 30 20:38:27.707887 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:27.706877 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ea0e353b-8cb1-4759-9f47-03f788f256d3-system-cni-dir\") pod \"multus-additional-cni-plugins-nxmjg\" (UID: \"ea0e353b-8cb1-4759-9f47-03f788f256d3\") " pod="openshift-multus/multus-additional-cni-plugins-nxmjg" Oct 30 20:38:27.707887 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:27.706910 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7e2574c1-b59c-453c-9501-25ec13a1ccad-var-lib-openvswitch\") pod \"ovnkube-node-7w5wz\" (UID: \"7e2574c1-b59c-453c-9501-25ec13a1ccad\") " pod="openshift-ovn-kubernetes/ovnkube-node-7w5wz" Oct 30 20:38:27.707887 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:27.706960 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/7e2574c1-b59c-453c-9501-25ec13a1ccad-run-ovn\") pod \"ovnkube-node-7w5wz\" (UID: \"7e2574c1-b59c-453c-9501-25ec13a1ccad\") " pod="openshift-ovn-kubernetes/ovnkube-node-7w5wz" Oct 30 20:38:27.707887 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:27.706979 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/7e2574c1-b59c-453c-9501-25ec13a1ccad-node-log\") pod \"ovnkube-node-7w5wz\" (UID: \"7e2574c1-b59c-453c-9501-25ec13a1ccad\") " pod="openshift-ovn-kubernetes/ovnkube-node-7w5wz" Oct 30 20:38:27.707887 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:27.706795 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/7e2574c1-b59c-453c-9501-25ec13a1ccad-log-socket\") pod \"ovnkube-node-7w5wz\" (UID: \"7e2574c1-b59c-453c-9501-25ec13a1ccad\") " pod="openshift-ovn-kubernetes/ovnkube-node-7w5wz" Oct 30 20:38:27.707887 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:27.707037 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e277a3d9-3fe9-4454-9420-28afea76f745-host-run-k8s-cni-cncf-io\") pod \"multus-c4fvc\" (UID: \"e277a3d9-3fe9-4454-9420-28afea76f745\") " pod="openshift-multus/multus-c4fvc" Oct 30 20:38:27.707887 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:27.707052 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f5f694f2-6f18-409b-b777-2dd7a0eb12b2-socket-dir\") pod \"aws-ebs-csi-driver-node-t7h2z\" (UID: \"f5f694f2-6f18-409b-b777-2dd7a0eb12b2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t7h2z" Oct 30 20:38:27.707887 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:27.707073 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7e2574c1-b59c-453c-9501-25ec13a1ccad-host-run-netns\") pod \"ovnkube-node-7w5wz\" (UID: \"7e2574c1-b59c-453c-9501-25ec13a1ccad\") " pod="openshift-ovn-kubernetes/ovnkube-node-7w5wz" Oct 30 20:38:27.707887 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:27.707107 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/ea0e353b-8cb1-4759-9f47-03f788f256d3-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-nxmjg\" (UID: \"ea0e353b-8cb1-4759-9f47-03f788f256d3\") " pod="openshift-multus/multus-additional-cni-plugins-nxmjg" Oct 30 20:38:27.707887 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:27.707109 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e277a3d9-3fe9-4454-9420-28afea76f745-multus-conf-dir\") pod \"multus-c4fvc\" (UID: \"e277a3d9-3fe9-4454-9420-28afea76f745\") " pod="openshift-multus/multus-c4fvc" Oct 30 20:38:27.707887 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:27.707225 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/ea0e353b-8cb1-4759-9f47-03f788f256d3-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-nxmjg\" (UID: \"ea0e353b-8cb1-4759-9f47-03f788f256d3\") " pod="openshift-multus/multus-additional-cni-plugins-nxmjg" Oct 30 20:38:27.708722 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:27.707256 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f5f694f2-6f18-409b-b777-2dd7a0eb12b2-registration-dir\") pod \"aws-ebs-csi-driver-node-t7h2z\" (UID: \"f5f694f2-6f18-409b-b777-2dd7a0eb12b2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t7h2z" Oct 30 20:38:27.708722 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:27.707282 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f5f694f2-6f18-409b-b777-2dd7a0eb12b2-device-dir\") pod \"aws-ebs-csi-driver-node-t7h2z\" (UID: \"f5f694f2-6f18-409b-b777-2dd7a0eb12b2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t7h2z" Oct 30 20:38:27.708722 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:27.707311 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e277a3d9-3fe9-4454-9420-28afea76f745-host-var-lib-cni-bin\") pod \"multus-c4fvc\" (UID: \"e277a3d9-3fe9-4454-9420-28afea76f745\") " pod="openshift-multus/multus-c4fvc" Oct 30 20:38:27.708722 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:27.707334 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/7e2574c1-b59c-453c-9501-25ec13a1ccad-env-overrides\") pod \"ovnkube-node-7w5wz\" (UID: \"7e2574c1-b59c-453c-9501-25ec13a1ccad\") " pod="openshift-ovn-kubernetes/ovnkube-node-7w5wz" Oct 30 20:38:27.708722 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:27.707357 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/7e2574c1-b59c-453c-9501-25ec13a1ccad-ovnkube-script-lib\") pod \"ovnkube-node-7w5wz\" (UID: \"7e2574c1-b59c-453c-9501-25ec13a1ccad\") " pod="openshift-ovn-kubernetes/ovnkube-node-7w5wz" Oct 30 20:38:27.708722 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:27.707401 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f5f694f2-6f18-409b-b777-2dd7a0eb12b2-kubelet-dir\") pod \"aws-ebs-csi-driver-node-t7h2z\" (UID: \"f5f694f2-6f18-409b-b777-2dd7a0eb12b2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t7h2z" Oct 30 20:38:27.708722 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:27.707407 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e277a3d9-3fe9-4454-9420-28afea76f745-cni-binary-copy\") pod \"multus-c4fvc\" (UID: \"e277a3d9-3fe9-4454-9420-28afea76f745\") " pod="openshift-multus/multus-c4fvc" Oct 30 20:38:27.708722 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:27.707427 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/acf9d9fb-1ae4-40ca-a83e-420a9fe40a01-host-slash\") pod \"iptables-alerter-kn2t9\" (UID: \"acf9d9fb-1ae4-40ca-a83e-420a9fe40a01\") " pod="openshift-network-operator/iptables-alerter-kn2t9" Oct 30 20:38:27.708722 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:27.707456 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/da50258d-eef3-4976-8e22-be0e936b9993-konnectivity-ca\") pod \"konnectivity-agent-jfxnj\" (UID: \"da50258d-eef3-4976-8e22-be0e936b9993\") " pod="kube-system/konnectivity-agent-jfxnj" Oct 30 20:38:27.708722 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:27.707468 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f5f694f2-6f18-409b-b777-2dd7a0eb12b2-sys-fs\") pod \"aws-ebs-csi-driver-node-t7h2z\" (UID: \"f5f694f2-6f18-409b-b777-2dd7a0eb12b2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t7h2z" Oct 30 20:38:27.708722 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:27.707471 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e277a3d9-3fe9-4454-9420-28afea76f745-host-var-lib-cni-bin\") pod \"multus-c4fvc\" (UID: \"e277a3d9-3fe9-4454-9420-28afea76f745\") " pod="openshift-multus/multus-c4fvc" Oct 30 20:38:27.708722 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:27.707512 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4ntjf\" (UniqueName: \"kubernetes.io/projected/35d51f79-80a9-4f9d-970e-6a35a34976a2-kube-api-access-4ntjf\") pod \"node-ca-hsv4j\" (UID: \"35d51f79-80a9-4f9d-970e-6a35a34976a2\") " pod="openshift-image-registry/node-ca-hsv4j" Oct 30 20:38:27.708722 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:27.707569 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/7e2574c1-b59c-453c-9501-25ec13a1ccad-ovnkube-config\") pod \"ovnkube-node-7w5wz\" (UID: \"7e2574c1-b59c-453c-9501-25ec13a1ccad\") " pod="openshift-ovn-kubernetes/ovnkube-node-7w5wz" Oct 30 20:38:27.708722 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:27.707612 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/e277a3d9-3fe9-4454-9420-28afea76f745-multus-daemon-config\") pod \"multus-c4fvc\" (UID: \"e277a3d9-3fe9-4454-9420-28afea76f745\") " pod="openshift-multus/multus-c4fvc" Oct 30 20:38:27.708722 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:27.707638 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ea0e353b-8cb1-4759-9f47-03f788f256d3-cni-binary-copy\") pod \"multus-additional-cni-plugins-nxmjg\" (UID: \"ea0e353b-8cb1-4759-9f47-03f788f256d3\") " pod="openshift-multus/multus-additional-cni-plugins-nxmjg" Oct 30 20:38:27.708722 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:27.707661 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/7e2574c1-b59c-453c-9501-25ec13a1ccad-host-kubelet\") pod \"ovnkube-node-7w5wz\" (UID: \"7e2574c1-b59c-453c-9501-25ec13a1ccad\") " pod="openshift-ovn-kubernetes/ovnkube-node-7w5wz" Oct 30 20:38:27.708722 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:27.707686 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/7e2574c1-b59c-453c-9501-25ec13a1ccad-systemd-units\") pod \"ovnkube-node-7w5wz\" (UID: \"7e2574c1-b59c-453c-9501-25ec13a1ccad\") " pod="openshift-ovn-kubernetes/ovnkube-node-7w5wz" Oct 30 20:38:27.709531 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:27.707711 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7e2574c1-b59c-453c-9501-25ec13a1ccad-host-run-ovn-kubernetes\") pod \"ovnkube-node-7w5wz\" (UID: \"7e2574c1-b59c-453c-9501-25ec13a1ccad\") " pod="openshift-ovn-kubernetes/ovnkube-node-7w5wz" Oct 30 20:38:27.709531 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:27.707735 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/ea0e353b-8cb1-4759-9f47-03f788f256d3-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-nxmjg\" (UID: \"ea0e353b-8cb1-4759-9f47-03f788f256d3\") " pod="openshift-multus/multus-additional-cni-plugins-nxmjg" Oct 30 20:38:27.709531 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:27.707795 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f5f694f2-6f18-409b-b777-2dd7a0eb12b2-registration-dir\") pod \"aws-ebs-csi-driver-node-t7h2z\" (UID: \"f5f694f2-6f18-409b-b777-2dd7a0eb12b2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t7h2z" Oct 30 20:38:27.709531 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:27.707740 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e277a3d9-3fe9-4454-9420-28afea76f745-multus-cni-dir\") pod \"multus-c4fvc\" (UID: \"e277a3d9-3fe9-4454-9420-28afea76f745\") " pod="openshift-multus/multus-c4fvc" Oct 30 20:38:27.709531 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:27.707810 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/ea0e353b-8cb1-4759-9f47-03f788f256d3-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-nxmjg\" (UID: \"ea0e353b-8cb1-4759-9f47-03f788f256d3\") " pod="openshift-multus/multus-additional-cni-plugins-nxmjg" Oct 30 20:38:27.709531 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:27.707860 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/acf9d9fb-1ae4-40ca-a83e-420a9fe40a01-host-slash\") pod \"iptables-alerter-kn2t9\" (UID: \"acf9d9fb-1ae4-40ca-a83e-420a9fe40a01\") " pod="openshift-network-operator/iptables-alerter-kn2t9" Oct 30 20:38:27.709531 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:27.707880 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/7e2574c1-b59c-453c-9501-25ec13a1ccad-host-kubelet\") pod \"ovnkube-node-7w5wz\" (UID: \"7e2574c1-b59c-453c-9501-25ec13a1ccad\") " pod="openshift-ovn-kubernetes/ovnkube-node-7w5wz" Oct 30 20:38:27.709531 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:27.707933 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/7e2574c1-b59c-453c-9501-25ec13a1ccad-systemd-units\") pod \"ovnkube-node-7w5wz\" (UID: \"7e2574c1-b59c-453c-9501-25ec13a1ccad\") " pod="openshift-ovn-kubernetes/ovnkube-node-7w5wz" Oct 30 20:38:27.709531 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:27.707975 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7e2574c1-b59c-453c-9501-25ec13a1ccad-host-run-ovn-kubernetes\") pod \"ovnkube-node-7w5wz\" (UID: \"7e2574c1-b59c-453c-9501-25ec13a1ccad\") " pod="openshift-ovn-kubernetes/ovnkube-node-7w5wz" Oct 30 20:38:27.709531 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:27.707799 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e277a3d9-3fe9-4454-9420-28afea76f745-multus-cni-dir\") pod \"multus-c4fvc\" (UID: \"e277a3d9-3fe9-4454-9420-28afea76f745\") " pod="openshift-multus/multus-c4fvc" Oct 30 20:38:27.709531 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:27.708030 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f5f694f2-6f18-409b-b777-2dd7a0eb12b2-device-dir\") pod \"aws-ebs-csi-driver-node-t7h2z\" (UID: \"f5f694f2-6f18-409b-b777-2dd7a0eb12b2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t7h2z" Oct 30 20:38:27.709531 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:27.708070 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f5f694f2-6f18-409b-b777-2dd7a0eb12b2-sys-fs\") pod \"aws-ebs-csi-driver-node-t7h2z\" (UID: \"f5f694f2-6f18-409b-b777-2dd7a0eb12b2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t7h2z" Oct 30 20:38:27.709531 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:27.708084 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f5f694f2-6f18-409b-b777-2dd7a0eb12b2-kubelet-dir\") pod \"aws-ebs-csi-driver-node-t7h2z\" (UID: \"f5f694f2-6f18-409b-b777-2dd7a0eb12b2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t7h2z" Oct 30 20:38:27.709531 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:27.708109 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e277a3d9-3fe9-4454-9420-28afea76f745-etc-kubernetes\") pod \"multus-c4fvc\" (UID: \"e277a3d9-3fe9-4454-9420-28afea76f745\") " pod="openshift-multus/multus-c4fvc" Oct 30 20:38:27.709531 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:27.708130 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/7e2574c1-b59c-453c-9501-25ec13a1ccad-ovn-node-metrics-cert\") pod \"ovnkube-node-7w5wz\" (UID: \"7e2574c1-b59c-453c-9501-25ec13a1ccad\") " pod="openshift-ovn-kubernetes/ovnkube-node-7w5wz" Oct 30 20:38:27.709531 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:27.708146 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f5f694f2-6f18-409b-b777-2dd7a0eb12b2-etc-selinux\") pod \"aws-ebs-csi-driver-node-t7h2z\" (UID: \"f5f694f2-6f18-409b-b777-2dd7a0eb12b2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t7h2z" Oct 30 20:38:27.709531 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:27.708163 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/35d51f79-80a9-4f9d-970e-6a35a34976a2-serviceca\") pod \"node-ca-hsv4j\" (UID: \"35d51f79-80a9-4f9d-970e-6a35a34976a2\") " pod="openshift-image-registry/node-ca-hsv4j" Oct 30 20:38:27.710374 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:27.708178 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e277a3d9-3fe9-4454-9420-28afea76f745-hostroot\") pod \"multus-c4fvc\" (UID: \"e277a3d9-3fe9-4454-9420-28afea76f745\") " pod="openshift-multus/multus-c4fvc" Oct 30 20:38:27.710374 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:27.708199 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7e2574c1-b59c-453c-9501-25ec13a1ccad-etc-openvswitch\") pod \"ovnkube-node-7w5wz\" (UID: \"7e2574c1-b59c-453c-9501-25ec13a1ccad\") " pod="openshift-ovn-kubernetes/ovnkube-node-7w5wz" Oct 30 20:38:27.710374 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:27.708224 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7e2574c1-b59c-453c-9501-25ec13a1ccad-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-7w5wz\" (UID: \"7e2574c1-b59c-453c-9501-25ec13a1ccad\") " pod="openshift-ovn-kubernetes/ovnkube-node-7w5wz" Oct 30 20:38:27.710374 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:27.708264 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e277a3d9-3fe9-4454-9420-28afea76f745-hostroot\") pod \"multus-c4fvc\" (UID: \"e277a3d9-3fe9-4454-9420-28afea76f745\") " pod="openshift-multus/multus-c4fvc" Oct 30 20:38:27.710374 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:27.708295 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f5f694f2-6f18-409b-b777-2dd7a0eb12b2-etc-selinux\") pod \"aws-ebs-csi-driver-node-t7h2z\" (UID: \"f5f694f2-6f18-409b-b777-2dd7a0eb12b2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t7h2z" Oct 30 20:38:27.710374 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:27.708337 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7e2574c1-b59c-453c-9501-25ec13a1ccad-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-7w5wz\" (UID: \"7e2574c1-b59c-453c-9501-25ec13a1ccad\") " pod="openshift-ovn-kubernetes/ovnkube-node-7w5wz" Oct 30 20:38:27.710374 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:27.708343 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e277a3d9-3fe9-4454-9420-28afea76f745-etc-kubernetes\") pod \"multus-c4fvc\" (UID: \"e277a3d9-3fe9-4454-9420-28afea76f745\") " pod="openshift-multus/multus-c4fvc" Oct 30 20:38:27.710374 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:27.708361 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7e2574c1-b59c-453c-9501-25ec13a1ccad-etc-openvswitch\") pod \"ovnkube-node-7w5wz\" (UID: \"7e2574c1-b59c-453c-9501-25ec13a1ccad\") " pod="openshift-ovn-kubernetes/ovnkube-node-7w5wz" Oct 30 20:38:27.710374 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:27.708497 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/7e2574c1-b59c-453c-9501-25ec13a1ccad-env-overrides\") pod \"ovnkube-node-7w5wz\" (UID: \"7e2574c1-b59c-453c-9501-25ec13a1ccad\") " pod="openshift-ovn-kubernetes/ovnkube-node-7w5wz" Oct 30 20:38:27.710374 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:27.708671 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/7e2574c1-b59c-453c-9501-25ec13a1ccad-ovnkube-script-lib\") pod \"ovnkube-node-7w5wz\" (UID: \"7e2574c1-b59c-453c-9501-25ec13a1ccad\") " pod="openshift-ovn-kubernetes/ovnkube-node-7w5wz" Oct 30 20:38:27.710374 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:27.708896 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/e277a3d9-3fe9-4454-9420-28afea76f745-multus-daemon-config\") pod \"multus-c4fvc\" (UID: \"e277a3d9-3fe9-4454-9420-28afea76f745\") " pod="openshift-multus/multus-c4fvc" Oct 30 20:38:27.710374 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:27.709302 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/35d51f79-80a9-4f9d-970e-6a35a34976a2-serviceca\") pod \"node-ca-hsv4j\" (UID: \"35d51f79-80a9-4f9d-970e-6a35a34976a2\") " pod="openshift-image-registry/node-ca-hsv4j" Oct 30 20:38:27.710374 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:27.709367 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ea0e353b-8cb1-4759-9f47-03f788f256d3-cni-binary-copy\") pod \"multus-additional-cni-plugins-nxmjg\" (UID: \"ea0e353b-8cb1-4759-9f47-03f788f256d3\") " pod="openshift-multus/multus-additional-cni-plugins-nxmjg" Oct 30 20:38:27.710374 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:27.709396 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/7e2574c1-b59c-453c-9501-25ec13a1ccad-ovnkube-config\") pod \"ovnkube-node-7w5wz\" (UID: \"7e2574c1-b59c-453c-9501-25ec13a1ccad\") " pod="openshift-ovn-kubernetes/ovnkube-node-7w5wz" Oct 30 20:38:27.710374 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:27.710365 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/da50258d-eef3-4976-8e22-be0e936b9993-agent-certs\") pod \"konnectivity-agent-jfxnj\" (UID: \"da50258d-eef3-4976-8e22-be0e936b9993\") " pod="kube-system/konnectivity-agent-jfxnj" Oct 30 20:38:27.711157 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:27.710812 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/7e2574c1-b59c-453c-9501-25ec13a1ccad-ovn-node-metrics-cert\") pod \"ovnkube-node-7w5wz\" (UID: \"7e2574c1-b59c-453c-9501-25ec13a1ccad\") " pod="openshift-ovn-kubernetes/ovnkube-node-7w5wz" Oct 30 20:38:27.721093 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:27.721046 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bg7g6\" (UniqueName: \"kubernetes.io/projected/e277a3d9-3fe9-4454-9420-28afea76f745-kube-api-access-bg7g6\") pod \"multus-c4fvc\" (UID: \"e277a3d9-3fe9-4454-9420-28afea76f745\") " pod="openshift-multus/multus-c4fvc" Oct 30 20:38:27.721674 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:27.721615 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-szhgd\" (UniqueName: \"kubernetes.io/projected/acf9d9fb-1ae4-40ca-a83e-420a9fe40a01-kube-api-access-szhgd\") pod \"iptables-alerter-kn2t9\" (UID: \"acf9d9fb-1ae4-40ca-a83e-420a9fe40a01\") " pod="openshift-network-operator/iptables-alerter-kn2t9" Oct 30 20:38:27.721920 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:27.721875 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rjj44\" (UniqueName: \"kubernetes.io/projected/ea0e353b-8cb1-4759-9f47-03f788f256d3-kube-api-access-rjj44\") pod \"multus-additional-cni-plugins-nxmjg\" (UID: \"ea0e353b-8cb1-4759-9f47-03f788f256d3\") " pod="openshift-multus/multus-additional-cni-plugins-nxmjg" Oct 30 20:38:27.722500 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:27.722478 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jk5kq\" (UniqueName: \"kubernetes.io/projected/7e2574c1-b59c-453c-9501-25ec13a1ccad-kube-api-access-jk5kq\") pod \"ovnkube-node-7w5wz\" (UID: \"7e2574c1-b59c-453c-9501-25ec13a1ccad\") " pod="openshift-ovn-kubernetes/ovnkube-node-7w5wz" Oct 30 20:38:27.724423 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:27.724399 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cv9nx\" (UniqueName: \"kubernetes.io/projected/f5f694f2-6f18-409b-b777-2dd7a0eb12b2-kube-api-access-cv9nx\") pod \"aws-ebs-csi-driver-node-t7h2z\" (UID: \"f5f694f2-6f18-409b-b777-2dd7a0eb12b2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t7h2z" Oct 30 20:38:27.724629 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:27.724609 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mjbz2\" (UniqueName: \"kubernetes.io/projected/0915cfe8-4606-4f92-877b-ebe7f52f4187-kube-api-access-mjbz2\") pod \"network-metrics-daemon-pdcfk\" (UID: \"0915cfe8-4606-4f92-877b-ebe7f52f4187\") " pod="openshift-multus/network-metrics-daemon-pdcfk" Oct 30 20:38:27.726405 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:27.726377 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4ntjf\" (UniqueName: \"kubernetes.io/projected/35d51f79-80a9-4f9d-970e-6a35a34976a2-kube-api-access-4ntjf\") pod \"node-ca-hsv4j\" (UID: \"35d51f79-80a9-4f9d-970e-6a35a34976a2\") " pod="openshift-image-registry/node-ca-hsv4j" Oct 30 20:38:27.734900 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:27.734860 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-192.us-east-2.compute.internal" event={"ID":"a211a81db7289ccbd1f6896c27ff51c5","Type":"ContainerStarted","Data":"160c0397f4741ec3a44de2d9a323a0a3713a7ab5c7ba1ac1f92e56c2b4711663"} Oct 30 20:38:27.736865 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:27.736838 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-192.us-east-2.compute.internal" event={"ID":"10e325ce488448e0259da3eb69cd60ba","Type":"ContainerStarted","Data":"fb0f2c227bb739438aaf12d4ffc31d80eab2c7f4b5776fd30ffc16448932514e"} Oct 30 20:38:27.811243 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:27.811212 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-2qgck" Oct 30 20:38:27.818409 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:27.818240 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-lc599" Oct 30 20:38:27.826820 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:27.826788 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b88d0a3_97c0_4adf_97a7_2e05d5cef781.slice/crio-e5f9d53e85a04cd314b74a77306f8d45a156af9ec9cf91a805f738978c98d2d4 WatchSource:0}: Error finding container e5f9d53e85a04cd314b74a77306f8d45a156af9ec9cf91a805f738978c98d2d4: Status 404 returned error can't find the container with id e5f9d53e85a04cd314b74a77306f8d45a156af9ec9cf91a805f738978c98d2d4 Oct 30 20:38:27.830491 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:27.830469 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9257033_53bf_40fc_a1e9_67fd5901678b.slice/crio-8dcc88a7fd48996068bc3d76be510d02ef2d218a81f7bc76d6722c39bd915e66 WatchSource:0}: Error finding container 8dcc88a7fd48996068bc3d76be510d02ef2d218a81f7bc76d6722c39bd915e66: Status 404 returned error can't find the container with id 8dcc88a7fd48996068bc3d76be510d02ef2d218a81f7bc76d6722c39bd915e66 Oct 30 20:38:27.831491 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:27.831471 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-kn2t9" Oct 30 20:38:27.841666 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:27.841644 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-hsv4j" Oct 30 20:38:27.843267 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:27.843241 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podacf9d9fb_1ae4_40ca_a83e_420a9fe40a01.slice/crio-3f323fafdad03f14abde257617593b193fdb5f4224c3a96a9af7ef7171ce52e2 WatchSource:0}: Error finding container 3f323fafdad03f14abde257617593b193fdb5f4224c3a96a9af7ef7171ce52e2: Status 404 returned error can't find the container with id 3f323fafdad03f14abde257617593b193fdb5f4224c3a96a9af7ef7171ce52e2 Oct 30 20:38:27.849358 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:27.849336 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-jfxnj" Oct 30 20:38:27.852509 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:27.852413 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35d51f79_80a9_4f9d_970e_6a35a34976a2.slice/crio-f51c59026b3d0d64b59aa49ecb7828bc8a105f9afe7bc7cd031ba55b03628fcd WatchSource:0}: Error finding container f51c59026b3d0d64b59aa49ecb7828bc8a105f9afe7bc7cd031ba55b03628fcd: Status 404 returned error can't find the container with id f51c59026b3d0d64b59aa49ecb7828bc8a105f9afe7bc7cd031ba55b03628fcd Oct 30 20:38:27.857557 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:27.857477 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-c4fvc" Oct 30 20:38:27.860382 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:27.860358 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda50258d_eef3_4976_8e22_be0e936b9993.slice/crio-e050f81a233aef1a82b2b11133bdd960ffe8aa369417dd2ec80ee56b7f3943d3 WatchSource:0}: Error finding container e050f81a233aef1a82b2b11133bdd960ffe8aa369417dd2ec80ee56b7f3943d3: Status 404 returned error can't find the container with id e050f81a233aef1a82b2b11133bdd960ffe8aa369417dd2ec80ee56b7f3943d3 Oct 30 20:38:27.868272 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:27.868239 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode277a3d9_3fe9_4454_9420_28afea76f745.slice/crio-010c147ffee86dab26403a897c7b0869149d45cdcb3e80147d42ff89dbd53f0c WatchSource:0}: Error finding container 010c147ffee86dab26403a897c7b0869149d45cdcb3e80147d42ff89dbd53f0c: Status 404 returned error can't find the container with id 010c147ffee86dab26403a897c7b0869149d45cdcb3e80147d42ff89dbd53f0c Oct 30 20:38:27.886241 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:27.886217 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-nxmjg" Oct 30 20:38:27.895317 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:27.895285 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea0e353b_8cb1_4759_9f47_03f788f256d3.slice/crio-1989faae8b908dfcf68e0f0d36993b7d6dca11487e1a4da908baed6982c99bd0 WatchSource:0}: Error finding container 1989faae8b908dfcf68e0f0d36993b7d6dca11487e1a4da908baed6982c99bd0: Status 404 returned error can't find the container with id 1989faae8b908dfcf68e0f0d36993b7d6dca11487e1a4da908baed6982c99bd0 Oct 30 20:38:27.898962 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:27.898944 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t7h2z" Oct 30 20:38:27.908601 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:27.908557 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-7w5wz" Oct 30 20:38:27.910737 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:27.910709 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5f694f2_6f18_409b_b777_2dd7a0eb12b2.slice/crio-45629083268acdae1d03c5389cac435087f01f6bf584b04d7ea7163c7cb3286a WatchSource:0}: Error finding container 45629083268acdae1d03c5389cac435087f01f6bf584b04d7ea7163c7cb3286a: Status 404 returned error can't find the container with id 45629083268acdae1d03c5389cac435087f01f6bf584b04d7ea7163c7cb3286a Oct 30 20:38:27.920300 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:27.920265 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e2574c1_b59c_453c_9501_25ec13a1ccad.slice/crio-5dceb587cfe62bf5917086004d513955231e7c975d9ab6c60bcd2e597d055e2c WatchSource:0}: Error finding container 5dceb587cfe62bf5917086004d513955231e7c975d9ab6c60bcd2e597d055e2c: Status 404 returned error can't find the container with id 5dceb587cfe62bf5917086004d513955231e7c975d9ab6c60bcd2e597d055e2c Oct 30 20:38:28.111137 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:28.111045 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bf21cf19-4475-41c3-9153-44d8fb329cdb-original-pull-secret\") pod \"global-pull-secret-syncer-857r2\" (UID: \"bf21cf19-4475-41c3-9153-44d8fb329cdb\") " pod="kube-system/global-pull-secret-syncer-857r2" Oct 30 20:38:28.111306 ip-10-0-133-192 kubenswrapper[2578]: E1030 20:38:28.111218 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 20:38:28.111306 ip-10-0-133-192 kubenswrapper[2578]: E1030 20:38:28.111284 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bf21cf19-4475-41c3-9153-44d8fb329cdb-original-pull-secret podName:bf21cf19-4475-41c3-9153-44d8fb329cdb nodeName:}" failed. No retries permitted until 2026-10-30 20:38:29.11126536 +0000 UTC m=+3.982093918 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/bf21cf19-4475-41c3-9153-44d8fb329cdb-original-pull-secret") pod "global-pull-secret-syncer-857r2" (UID: "bf21cf19-4475-41c3-9153-44d8fb329cdb") : object "kube-system"/"original-pull-secret" not registered Oct 30 20:38:28.211779 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:28.211745 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6hvqv\" (UniqueName: \"kubernetes.io/projected/0dde7eae-1962-4559-89e9-0c67b1165914-kube-api-access-6hvqv\") pod \"network-check-target-jp2cr\" (UID: \"0dde7eae-1962-4559-89e9-0c67b1165914\") " pod="openshift-network-diagnostics/network-check-target-jp2cr" Oct 30 20:38:28.211963 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:28.211791 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0915cfe8-4606-4f92-877b-ebe7f52f4187-metrics-certs\") pod \"network-metrics-daemon-pdcfk\" (UID: \"0915cfe8-4606-4f92-877b-ebe7f52f4187\") " pod="openshift-multus/network-metrics-daemon-pdcfk" Oct 30 20:38:28.211963 ip-10-0-133-192 kubenswrapper[2578]: E1030 20:38:28.211934 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 20:38:28.211963 ip-10-0-133-192 kubenswrapper[2578]: E1030 20:38:28.211956 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 20:38:28.212137 ip-10-0-133-192 kubenswrapper[2578]: E1030 20:38:28.212028 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0915cfe8-4606-4f92-877b-ebe7f52f4187-metrics-certs podName:0915cfe8-4606-4f92-877b-ebe7f52f4187 nodeName:}" failed. No retries permitted until 2026-10-30 20:38:29.212008904 +0000 UTC m=+4.082837458 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0915cfe8-4606-4f92-877b-ebe7f52f4187-metrics-certs") pod "network-metrics-daemon-pdcfk" (UID: "0915cfe8-4606-4f92-877b-ebe7f52f4187") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 20:38:28.212137 ip-10-0-133-192 kubenswrapper[2578]: E1030 20:38:28.211958 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 20:38:28.212137 ip-10-0-133-192 kubenswrapper[2578]: E1030 20:38:28.212074 2578 projected.go:196] Error preparing data for projected volume kube-api-access-6hvqv for pod openshift-network-diagnostics/network-check-target-jp2cr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 20:38:28.212137 ip-10-0-133-192 kubenswrapper[2578]: E1030 20:38:28.212133 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0dde7eae-1962-4559-89e9-0c67b1165914-kube-api-access-6hvqv podName:0dde7eae-1962-4559-89e9-0c67b1165914 nodeName:}" failed. No retries permitted until 2026-10-30 20:38:29.212115668 +0000 UTC m=+4.082944212 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-6hvqv" (UniqueName: "kubernetes.io/projected/0dde7eae-1962-4559-89e9-0c67b1165914-kube-api-access-6hvqv") pod "network-check-target-jp2cr" (UID: "0dde7eae-1962-4559-89e9-0c67b1165914") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 20:38:28.541440 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:28.541371 2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 20:33:27 +0000 UTC" deadline="2028-08-15 00:38:07.169249366 +0000 UTC" Oct 30 20:38:28.541440 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:28.541432 2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15699h59m38.627821571s" Oct 30 20:38:28.729960 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:28.729867 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-857r2" Oct 30 20:38:28.729960 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:28.729891 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pdcfk" Oct 30 20:38:28.730184 ip-10-0-133-192 kubenswrapper[2578]: E1030 20:38:28.730012 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-857r2" podUID="bf21cf19-4475-41c3-9153-44d8fb329cdb" Oct 30 20:38:28.730184 ip-10-0-133-192 kubenswrapper[2578]: E1030 20:38:28.730127 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pdcfk" podUID="0915cfe8-4606-4f92-877b-ebe7f52f4187" Oct 30 20:38:28.730289 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:28.730201 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jp2cr" Oct 30 20:38:28.730289 ip-10-0-133-192 kubenswrapper[2578]: E1030 20:38:28.730274 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jp2cr" podUID="0dde7eae-1962-4559-89e9-0c67b1165914" Oct 30 20:38:28.749478 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:28.749093 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7w5wz" event={"ID":"7e2574c1-b59c-453c-9501-25ec13a1ccad","Type":"ContainerStarted","Data":"5dceb587cfe62bf5917086004d513955231e7c975d9ab6c60bcd2e597d055e2c"} Oct 30 20:38:28.751560 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:28.751465 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-jfxnj" event={"ID":"da50258d-eef3-4976-8e22-be0e936b9993","Type":"ContainerStarted","Data":"e050f81a233aef1a82b2b11133bdd960ffe8aa369417dd2ec80ee56b7f3943d3"} Oct 30 20:38:28.756936 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:28.756855 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-kn2t9" event={"ID":"acf9d9fb-1ae4-40ca-a83e-420a9fe40a01","Type":"ContainerStarted","Data":"3f323fafdad03f14abde257617593b193fdb5f4224c3a96a9af7ef7171ce52e2"} Oct 30 20:38:28.765216 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:28.765158 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-lc599" event={"ID":"c9257033-53bf-40fc-a1e9-67fd5901678b","Type":"ContainerStarted","Data":"8dcc88a7fd48996068bc3d76be510d02ef2d218a81f7bc76d6722c39bd915e66"} Oct 30 20:38:28.768000 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:28.767907 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t7h2z" event={"ID":"f5f694f2-6f18-409b-b777-2dd7a0eb12b2","Type":"ContainerStarted","Data":"45629083268acdae1d03c5389cac435087f01f6bf584b04d7ea7163c7cb3286a"} Oct 30 20:38:28.772341 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:28.772262 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nxmjg" event={"ID":"ea0e353b-8cb1-4759-9f47-03f788f256d3","Type":"ContainerStarted","Data":"1989faae8b908dfcf68e0f0d36993b7d6dca11487e1a4da908baed6982c99bd0"} Oct 30 20:38:28.776560 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:28.776534 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-c4fvc" event={"ID":"e277a3d9-3fe9-4454-9420-28afea76f745","Type":"ContainerStarted","Data":"010c147ffee86dab26403a897c7b0869149d45cdcb3e80147d42ff89dbd53f0c"} Oct 30 20:38:28.780108 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:28.780080 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-hsv4j" event={"ID":"35d51f79-80a9-4f9d-970e-6a35a34976a2","Type":"ContainerStarted","Data":"f51c59026b3d0d64b59aa49ecb7828bc8a105f9afe7bc7cd031ba55b03628fcd"} Oct 30 20:38:28.785301 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:28.785279 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-2qgck" event={"ID":"6b88d0a3-97c0-4adf-97a7-2e05d5cef781","Type":"ContainerStarted","Data":"e5f9d53e85a04cd314b74a77306f8d45a156af9ec9cf91a805f738978c98d2d4"} Oct 30 20:38:29.120396 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:29.119872 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bf21cf19-4475-41c3-9153-44d8fb329cdb-original-pull-secret\") pod \"global-pull-secret-syncer-857r2\" (UID: \"bf21cf19-4475-41c3-9153-44d8fb329cdb\") " pod="kube-system/global-pull-secret-syncer-857r2" Oct 30 20:38:29.120396 ip-10-0-133-192 kubenswrapper[2578]: E1030 20:38:29.119997 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 20:38:29.120396 ip-10-0-133-192 kubenswrapper[2578]: E1030 20:38:29.120063 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bf21cf19-4475-41c3-9153-44d8fb329cdb-original-pull-secret podName:bf21cf19-4475-41c3-9153-44d8fb329cdb nodeName:}" failed. No retries permitted until 2026-10-30 20:38:31.120045039 +0000 UTC m=+5.990873594 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/bf21cf19-4475-41c3-9153-44d8fb329cdb-original-pull-secret") pod "global-pull-secret-syncer-857r2" (UID: "bf21cf19-4475-41c3-9153-44d8fb329cdb") : object "kube-system"/"original-pull-secret" not registered Oct 30 20:38:29.221219 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:29.220987 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6hvqv\" (UniqueName: \"kubernetes.io/projected/0dde7eae-1962-4559-89e9-0c67b1165914-kube-api-access-6hvqv\") pod \"network-check-target-jp2cr\" (UID: \"0dde7eae-1962-4559-89e9-0c67b1165914\") " pod="openshift-network-diagnostics/network-check-target-jp2cr" Oct 30 20:38:29.221337 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:29.221251 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0915cfe8-4606-4f92-877b-ebe7f52f4187-metrics-certs\") pod \"network-metrics-daemon-pdcfk\" (UID: \"0915cfe8-4606-4f92-877b-ebe7f52f4187\") " pod="openshift-multus/network-metrics-daemon-pdcfk" Oct 30 20:38:29.221415 ip-10-0-133-192 kubenswrapper[2578]: E1030 20:38:29.221387 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 20:38:29.221473 ip-10-0-133-192 kubenswrapper[2578]: E1030 20:38:29.221447 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0915cfe8-4606-4f92-877b-ebe7f52f4187-metrics-certs podName:0915cfe8-4606-4f92-877b-ebe7f52f4187 nodeName:}" failed. No retries permitted until 2026-10-30 20:38:31.221426165 +0000 UTC m=+6.092254731 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0915cfe8-4606-4f92-877b-ebe7f52f4187-metrics-certs") pod "network-metrics-daemon-pdcfk" (UID: "0915cfe8-4606-4f92-877b-ebe7f52f4187") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 20:38:29.221891 ip-10-0-133-192 kubenswrapper[2578]: E1030 20:38:29.221870 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 20:38:29.221976 ip-10-0-133-192 kubenswrapper[2578]: E1030 20:38:29.221896 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 20:38:29.221976 ip-10-0-133-192 kubenswrapper[2578]: E1030 20:38:29.221910 2578 projected.go:196] Error preparing data for projected volume kube-api-access-6hvqv for pod openshift-network-diagnostics/network-check-target-jp2cr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 20:38:29.221976 ip-10-0-133-192 kubenswrapper[2578]: E1030 20:38:29.221969 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0dde7eae-1962-4559-89e9-0c67b1165914-kube-api-access-6hvqv podName:0dde7eae-1962-4559-89e9-0c67b1165914 nodeName:}" failed. No retries permitted until 2026-10-30 20:38:31.221952086 +0000 UTC m=+6.092780640 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-6hvqv" (UniqueName: "kubernetes.io/projected/0dde7eae-1962-4559-89e9-0c67b1165914-kube-api-access-6hvqv") pod "network-check-target-jp2cr" (UID: "0dde7eae-1962-4559-89e9-0c67b1165914") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 20:38:29.543474 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:29.541680 2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 20:33:27 +0000 UTC" deadline="2028-07-01 17:36:20.590610714 +0000 UTC" Oct 30 20:38:29.543474 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:29.541720 2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14636h57m51.048895563s" Oct 30 20:38:29.796105 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:29.795996 2578 generic.go:356] "Generic (PLEG): container finished" podID="a211a81db7289ccbd1f6896c27ff51c5" containerID="16291fa955d1e42357e24beb74fbb648db543686399f9a79bdcf58dcd749e9bb" exitCode=0 Oct 30 20:38:29.796293 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:29.796098 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-192.us-east-2.compute.internal" event={"ID":"a211a81db7289ccbd1f6896c27ff51c5","Type":"ContainerDied","Data":"16291fa955d1e42357e24beb74fbb648db543686399f9a79bdcf58dcd749e9bb"} Oct 30 20:38:29.820609 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:29.818910 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-192.us-east-2.compute.internal" event={"ID":"10e325ce488448e0259da3eb69cd60ba","Type":"ContainerStarted","Data":"fa27a51d81fd9981d0e02bd8f149f5d9570bdf79dea62f8ce876c8bbf05eb483"} Oct 30 20:38:30.730135 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:30.730104 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-857r2" Oct 30 20:38:30.730640 ip-10-0-133-192 kubenswrapper[2578]: E1030 20:38:30.730251 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-857r2" podUID="bf21cf19-4475-41c3-9153-44d8fb329cdb" Oct 30 20:38:30.730640 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:30.730574 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pdcfk" Oct 30 20:38:30.730777 ip-10-0-133-192 kubenswrapper[2578]: E1030 20:38:30.730687 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pdcfk" podUID="0915cfe8-4606-4f92-877b-ebe7f52f4187" Oct 30 20:38:30.730777 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:30.730724 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jp2cr" Oct 30 20:38:30.730890 ip-10-0-133-192 kubenswrapper[2578]: E1030 20:38:30.730796 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jp2cr" podUID="0dde7eae-1962-4559-89e9-0c67b1165914" Oct 30 20:38:30.832396 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:30.832269 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-192.us-east-2.compute.internal" event={"ID":"a211a81db7289ccbd1f6896c27ff51c5","Type":"ContainerStarted","Data":"54bf3937147dd87f3afb0aac3fe6c6a72f66f808c1e4ea40958793c00ebdacbb"} Oct 30 20:38:30.855529 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:30.854758 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-192.us-east-2.compute.internal" podStartSLOduration=3.854737346 podStartE2EDuration="3.854737346s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 20:38:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:38:29.844835673 +0000 UTC m=+4.715664237" watchObservedRunningTime="2026-10-30 20:38:30.854737346 +0000 UTC m=+5.725565912" Oct 30 20:38:31.137059 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:31.137020 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bf21cf19-4475-41c3-9153-44d8fb329cdb-original-pull-secret\") pod \"global-pull-secret-syncer-857r2\" (UID: \"bf21cf19-4475-41c3-9153-44d8fb329cdb\") " pod="kube-system/global-pull-secret-syncer-857r2" Oct 30 20:38:31.137355 ip-10-0-133-192 kubenswrapper[2578]: E1030 20:38:31.137334 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 20:38:31.137428 ip-10-0-133-192 kubenswrapper[2578]: E1030 20:38:31.137406 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bf21cf19-4475-41c3-9153-44d8fb329cdb-original-pull-secret podName:bf21cf19-4475-41c3-9153-44d8fb329cdb nodeName:}" failed. No retries permitted until 2026-10-30 20:38:35.137386584 +0000 UTC m=+10.008215138 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/bf21cf19-4475-41c3-9153-44d8fb329cdb-original-pull-secret") pod "global-pull-secret-syncer-857r2" (UID: "bf21cf19-4475-41c3-9153-44d8fb329cdb") : object "kube-system"/"original-pull-secret" not registered Oct 30 20:38:31.239108 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:31.238006 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6hvqv\" (UniqueName: \"kubernetes.io/projected/0dde7eae-1962-4559-89e9-0c67b1165914-kube-api-access-6hvqv\") pod \"network-check-target-jp2cr\" (UID: \"0dde7eae-1962-4559-89e9-0c67b1165914\") " pod="openshift-network-diagnostics/network-check-target-jp2cr" Oct 30 20:38:31.239108 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:31.238074 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0915cfe8-4606-4f92-877b-ebe7f52f4187-metrics-certs\") pod \"network-metrics-daemon-pdcfk\" (UID: \"0915cfe8-4606-4f92-877b-ebe7f52f4187\") " pod="openshift-multus/network-metrics-daemon-pdcfk" Oct 30 20:38:31.239108 ip-10-0-133-192 kubenswrapper[2578]: E1030 20:38:31.238260 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 20:38:31.239108 ip-10-0-133-192 kubenswrapper[2578]: E1030 20:38:31.238324 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0915cfe8-4606-4f92-877b-ebe7f52f4187-metrics-certs podName:0915cfe8-4606-4f92-877b-ebe7f52f4187 nodeName:}" failed. No retries permitted until 2026-10-30 20:38:35.238305582 +0000 UTC m=+10.109134140 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0915cfe8-4606-4f92-877b-ebe7f52f4187-metrics-certs") pod "network-metrics-daemon-pdcfk" (UID: "0915cfe8-4606-4f92-877b-ebe7f52f4187") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 20:38:31.239108 ip-10-0-133-192 kubenswrapper[2578]: E1030 20:38:31.238790 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 20:38:31.239108 ip-10-0-133-192 kubenswrapper[2578]: E1030 20:38:31.238868 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 20:38:31.239108 ip-10-0-133-192 kubenswrapper[2578]: E1030 20:38:31.238910 2578 projected.go:196] Error preparing data for projected volume kube-api-access-6hvqv for pod openshift-network-diagnostics/network-check-target-jp2cr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 20:38:31.239108 ip-10-0-133-192 kubenswrapper[2578]: E1030 20:38:31.238993 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0dde7eae-1962-4559-89e9-0c67b1165914-kube-api-access-6hvqv podName:0dde7eae-1962-4559-89e9-0c67b1165914 nodeName:}" failed. No retries permitted until 2026-10-30 20:38:35.238957309 +0000 UTC m=+10.109785850 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-6hvqv" (UniqueName: "kubernetes.io/projected/0dde7eae-1962-4559-89e9-0c67b1165914-kube-api-access-6hvqv") pod "network-check-target-jp2cr" (UID: "0dde7eae-1962-4559-89e9-0c67b1165914") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 20:38:31.837966 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:31.837931 2578 generic.go:356] "Generic (PLEG): container finished" podID="ea0e353b-8cb1-4759-9f47-03f788f256d3" containerID="c7fcd96fde6e28a72528d2556b5f3039d7e5ce747b8b21b2cf8ca64769055262" exitCode=0 Oct 30 20:38:31.838917 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:31.838893 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nxmjg" event={"ID":"ea0e353b-8cb1-4759-9f47-03f788f256d3","Type":"ContainerDied","Data":"c7fcd96fde6e28a72528d2556b5f3039d7e5ce747b8b21b2cf8ca64769055262"} Oct 30 20:38:31.871970 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:31.871915 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-192.us-east-2.compute.internal" podStartSLOduration=4.871898226 podStartE2EDuration="4.871898226s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 20:38:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:38:30.855457739 +0000 UTC m=+5.726286299" watchObservedRunningTime="2026-10-30 20:38:31.871898226 +0000 UTC m=+6.742726792" Oct 30 20:38:32.731731 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:32.731683 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jp2cr" Oct 30 20:38:32.731927 ip-10-0-133-192 kubenswrapper[2578]: E1030 20:38:32.731789 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jp2cr" podUID="0dde7eae-1962-4559-89e9-0c67b1165914" Oct 30 20:38:32.731927 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:32.731816 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-857r2" Oct 30 20:38:32.732009 ip-10-0-133-192 kubenswrapper[2578]: E1030 20:38:32.731941 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-857r2" podUID="bf21cf19-4475-41c3-9153-44d8fb329cdb" Oct 30 20:38:32.732065 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:32.732023 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pdcfk" Oct 30 20:38:32.732160 ip-10-0-133-192 kubenswrapper[2578]: E1030 20:38:32.732112 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pdcfk" podUID="0915cfe8-4606-4f92-877b-ebe7f52f4187" Oct 30 20:38:32.855725 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:32.854990 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-jfxnj" event={"ID":"da50258d-eef3-4976-8e22-be0e936b9993","Type":"ContainerStarted","Data":"00627e6501cc319ef6d267542d9be85920f58bc628a6ce4fed6755c260b95fdd"} Oct 30 20:38:33.865908 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:33.865873 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-hsv4j" event={"ID":"35d51f79-80a9-4f9d-970e-6a35a34976a2","Type":"ContainerStarted","Data":"f9ce1ea80f56a1b94d50c6c8a492eddd2139e3f97f771632bf320472d0d29bae"} Oct 30 20:38:33.883445 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:33.883404 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-jfxnj" podStartSLOduration=3.589277333 podStartE2EDuration="7.883390524s" totalImagesPullingTime="4.294113191s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:38:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:38:32.876108291 +0000 UTC m=+7.746936849" watchObservedRunningTime="2026-10-30 20:38:33.883390524 +0000 UTC m=+8.754219076" Oct 30 20:38:34.729551 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:34.729501 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-857r2" Oct 30 20:38:34.729717 ip-10-0-133-192 kubenswrapper[2578]: E1030 20:38:34.729645 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-857r2" podUID="bf21cf19-4475-41c3-9153-44d8fb329cdb" Oct 30 20:38:34.730056 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:34.730033 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pdcfk" Oct 30 20:38:34.730319 ip-10-0-133-192 kubenswrapper[2578]: E1030 20:38:34.730146 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pdcfk" podUID="0915cfe8-4606-4f92-877b-ebe7f52f4187" Oct 30 20:38:34.730319 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:34.730210 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jp2cr" Oct 30 20:38:34.730319 ip-10-0-133-192 kubenswrapper[2578]: E1030 20:38:34.730277 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jp2cr" podUID="0dde7eae-1962-4559-89e9-0c67b1165914" Oct 30 20:38:34.873930 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:34.873809 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t7h2z" event={"ID":"f5f694f2-6f18-409b-b777-2dd7a0eb12b2","Type":"ContainerStarted","Data":"4b0dcd5f63b08681e6ab54b308cd95a1dbd02e6c1641e33ce6a1678a841978a6"} Oct 30 20:38:35.172438 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:35.171953 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bf21cf19-4475-41c3-9153-44d8fb329cdb-original-pull-secret\") pod \"global-pull-secret-syncer-857r2\" (UID: \"bf21cf19-4475-41c3-9153-44d8fb329cdb\") " pod="kube-system/global-pull-secret-syncer-857r2" Oct 30 20:38:35.172438 ip-10-0-133-192 kubenswrapper[2578]: E1030 20:38:35.172079 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 20:38:35.172438 ip-10-0-133-192 kubenswrapper[2578]: E1030 20:38:35.172125 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bf21cf19-4475-41c3-9153-44d8fb329cdb-original-pull-secret podName:bf21cf19-4475-41c3-9153-44d8fb329cdb nodeName:}" failed. No retries permitted until 2026-10-30 20:38:43.172113818 +0000 UTC m=+18.042942367 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/bf21cf19-4475-41c3-9153-44d8fb329cdb-original-pull-secret") pod "global-pull-secret-syncer-857r2" (UID: "bf21cf19-4475-41c3-9153-44d8fb329cdb") : object "kube-system"/"original-pull-secret" not registered Oct 30 20:38:35.273746 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:35.273258 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6hvqv\" (UniqueName: \"kubernetes.io/projected/0dde7eae-1962-4559-89e9-0c67b1165914-kube-api-access-6hvqv\") pod \"network-check-target-jp2cr\" (UID: \"0dde7eae-1962-4559-89e9-0c67b1165914\") " pod="openshift-network-diagnostics/network-check-target-jp2cr" Oct 30 20:38:35.273746 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:35.273315 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0915cfe8-4606-4f92-877b-ebe7f52f4187-metrics-certs\") pod \"network-metrics-daemon-pdcfk\" (UID: \"0915cfe8-4606-4f92-877b-ebe7f52f4187\") " pod="openshift-multus/network-metrics-daemon-pdcfk" Oct 30 20:38:35.273746 ip-10-0-133-192 kubenswrapper[2578]: E1030 20:38:35.273445 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 20:38:35.273746 ip-10-0-133-192 kubenswrapper[2578]: E1030 20:38:35.273648 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 20:38:35.273746 ip-10-0-133-192 kubenswrapper[2578]: E1030 20:38:35.273666 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 20:38:35.273746 ip-10-0-133-192 kubenswrapper[2578]: E1030 20:38:35.273679 2578 projected.go:196] Error preparing data for projected volume kube-api-access-6hvqv for pod openshift-network-diagnostics/network-check-target-jp2cr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 20:38:35.274758 ip-10-0-133-192 kubenswrapper[2578]: E1030 20:38:35.274658 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0915cfe8-4606-4f92-877b-ebe7f52f4187-metrics-certs podName:0915cfe8-4606-4f92-877b-ebe7f52f4187 nodeName:}" failed. No retries permitted until 2026-10-30 20:38:43.273492633 +0000 UTC m=+18.144321172 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0915cfe8-4606-4f92-877b-ebe7f52f4187-metrics-certs") pod "network-metrics-daemon-pdcfk" (UID: "0915cfe8-4606-4f92-877b-ebe7f52f4187") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 20:38:35.274758 ip-10-0-133-192 kubenswrapper[2578]: E1030 20:38:35.274700 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0dde7eae-1962-4559-89e9-0c67b1165914-kube-api-access-6hvqv podName:0dde7eae-1962-4559-89e9-0c67b1165914 nodeName:}" failed. No retries permitted until 2026-10-30 20:38:43.274685504 +0000 UTC m=+18.145514041 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-6hvqv" (UniqueName: "kubernetes.io/projected/0dde7eae-1962-4559-89e9-0c67b1165914-kube-api-access-6hvqv") pod "network-check-target-jp2cr" (UID: "0dde7eae-1962-4559-89e9-0c67b1165914") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 20:38:35.495702 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:35.495635 2578 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 30 20:38:35.571728 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:35.571602 2578 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T20:38:35.495665438Z","UUID":"625ebb7c-f5e6-468a-abee-722b88b142e5","Handler":null,"Name":"","Endpoint":""} Oct 30 20:38:35.573482 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:35.573454 2578 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 30 20:38:35.573604 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:35.573501 2578 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 30 20:38:35.879442 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:35.879360 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t7h2z" event={"ID":"f5f694f2-6f18-409b-b777-2dd7a0eb12b2","Type":"ContainerStarted","Data":"bd33a2848d18acf0579c7b7430884dd823da9885bc4535294be9d133ab823e1c"} Oct 30 20:38:36.729498 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:36.729311 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-857r2" Oct 30 20:38:36.729634 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:36.729347 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pdcfk" Oct 30 20:38:36.729634 ip-10-0-133-192 kubenswrapper[2578]: E1030 20:38:36.729607 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-857r2" podUID="bf21cf19-4475-41c3-9153-44d8fb329cdb" Oct 30 20:38:36.729736 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:36.729420 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jp2cr" Oct 30 20:38:36.729736 ip-10-0-133-192 kubenswrapper[2578]: E1030 20:38:36.729671 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pdcfk" podUID="0915cfe8-4606-4f92-877b-ebe7f52f4187" Oct 30 20:38:36.729824 ip-10-0-133-192 kubenswrapper[2578]: E1030 20:38:36.729774 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jp2cr" podUID="0dde7eae-1962-4559-89e9-0c67b1165914" Oct 30 20:38:36.883164 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:36.883130 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-lc599" event={"ID":"c9257033-53bf-40fc-a1e9-67fd5901678b","Type":"ContainerStarted","Data":"4c58804304d30b221212ef2cb69fe713506b60bb99bcbd3c8b7ee7f421dacd60"} Oct 30 20:38:36.887124 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:36.887098 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t7h2z" event={"ID":"f5f694f2-6f18-409b-b777-2dd7a0eb12b2","Type":"ContainerStarted","Data":"1e8130d4f21e29e5f63115dc81527bae0241420d4c74851caab7bfb21f18433b"} Oct 30 20:38:36.889714 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:36.889691 2578 generic.go:356] "Generic (PLEG): container finished" podID="ea0e353b-8cb1-4759-9f47-03f788f256d3" containerID="4c9ade534cca4a562c797d122466d8a245afc6209b7c9d92bca07553602c797b" exitCode=0 Oct 30 20:38:36.889815 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:36.889770 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nxmjg" event={"ID":"ea0e353b-8cb1-4759-9f47-03f788f256d3","Type":"ContainerDied","Data":"4c9ade534cca4a562c797d122466d8a245afc6209b7c9d92bca07553602c797b"} Oct 30 20:38:36.891242 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:36.891222 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-c4fvc" event={"ID":"e277a3d9-3fe9-4454-9420-28afea76f745","Type":"ContainerStarted","Data":"54710c223300ef7cb40d61a98aa40b707824e45f9b459005a5a5d98b5eb042b5"} Oct 30 20:38:36.902453 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:36.901969 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-hsv4j" podStartSLOduration=5.636180014 podStartE2EDuration="10.901954548s" totalImagesPullingTime="5.265774534s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 20:38:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:38:33.884353401 +0000 UTC m=+8.755181951" watchObservedRunningTime="2026-10-30 20:38:36.901954548 +0000 UTC m=+11.772783107" Oct 30 20:38:36.902453 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:36.902301 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-lc599" podStartSLOduration=2.72129407 podStartE2EDuration="10.902288784s" totalImagesPullingTime="8.180994714s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 20:38:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:38:36.901555855 +0000 UTC m=+11.772384416" watchObservedRunningTime="2026-10-30 20:38:36.902288784 +0000 UTC m=+11.773117344" Oct 30 20:38:36.926483 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:36.926436 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-c4fvc" podStartSLOduration=2.906810433 podStartE2EDuration="10.926421503s" totalImagesPullingTime="8.01961107s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 20:38:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:38:36.925889517 +0000 UTC m=+11.796718079" watchObservedRunningTime="2026-10-30 20:38:36.926421503 +0000 UTC m=+11.797250063" Oct 30 20:38:36.975018 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:36.974940 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t7h2z" podStartSLOduration=2.629947461 podStartE2EDuration="10.974923096s" totalImagesPullingTime="8.344975635s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 20:38:26 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:38:36.947896298 +0000 UTC m=+11.818724877" watchObservedRunningTime="2026-10-30 20:38:36.974923096 +0000 UTC m=+11.845751658" Oct 30 20:38:37.066355 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:37.066328 2578 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-jfxnj" Oct 30 20:38:37.067063 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:37.067039 2578 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-jfxnj" Oct 30 20:38:37.895605 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:37.895569 2578 generic.go:356] "Generic (PLEG): container finished" podID="ea0e353b-8cb1-4759-9f47-03f788f256d3" containerID="f0cbfc41c587c7da1088a5340745e9d8d0aa4d404ae1b997b301149751763a6b" exitCode=0 Oct 30 20:38:37.896024 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:37.895698 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nxmjg" event={"ID":"ea0e353b-8cb1-4759-9f47-03f788f256d3","Type":"ContainerDied","Data":"f0cbfc41c587c7da1088a5340745e9d8d0aa4d404ae1b997b301149751763a6b"} Oct 30 20:38:37.897568 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:37.897414 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-kn2t9" event={"ID":"acf9d9fb-1ae4-40ca-a83e-420a9fe40a01","Type":"ContainerStarted","Data":"305ae4ab0309a245101664b4c8a5f91e542d35467046b0284302bdf79dce7f21"} Oct 30 20:38:37.945095 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:37.945049 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-kn2t9" podStartSLOduration=3.392597645 podStartE2EDuration="11.945036625s" totalImagesPullingTime="8.55243898s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 20:38:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:38:37.944631516 +0000 UTC m=+12.815460076" watchObservedRunningTime="2026-10-30 20:38:37.945036625 +0000 UTC m=+12.815865183" Oct 30 20:38:38.729377 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:38.729342 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jp2cr" Oct 30 20:38:38.729377 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:38.729374 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-857r2" Oct 30 20:38:38.729582 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:38.729574 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pdcfk" Oct 30 20:38:38.729616 ip-10-0-133-192 kubenswrapper[2578]: E1030 20:38:38.729581 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jp2cr" podUID="0dde7eae-1962-4559-89e9-0c67b1165914" Oct 30 20:38:38.729678 ip-10-0-133-192 kubenswrapper[2578]: E1030 20:38:38.729659 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-857r2" podUID="bf21cf19-4475-41c3-9153-44d8fb329cdb" Oct 30 20:38:38.729756 ip-10-0-133-192 kubenswrapper[2578]: E1030 20:38:38.729739 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pdcfk" podUID="0915cfe8-4606-4f92-877b-ebe7f52f4187" Oct 30 20:38:38.904937 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:38.904913 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-2qgck" event={"ID":"6b88d0a3-97c0-4adf-97a7-2e05d5cef781","Type":"ContainerStarted","Data":"e27cfe28e31f8c6061dbfb2c45bb0dc7d7a27ea31480a55ef0e2a80da33ad6fa"} Oct 30 20:38:39.908918 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:39.908876 2578 generic.go:356] "Generic (PLEG): container finished" podID="ea0e353b-8cb1-4759-9f47-03f788f256d3" containerID="b68ac9cc1f72f49d199521fc0937d68ad3790a9dd89d9b1f724816fa2de873c7" exitCode=0 Oct 30 20:38:39.909484 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:39.908954 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nxmjg" event={"ID":"ea0e353b-8cb1-4759-9f47-03f788f256d3","Type":"ContainerDied","Data":"b68ac9cc1f72f49d199521fc0937d68ad3790a9dd89d9b1f724816fa2de873c7"} Oct 30 20:38:39.938561 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:39.938525 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-2qgck" podStartSLOduration=3.760737721 podStartE2EDuration="13.938497606s" totalImagesPullingTime="10.177759885s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 20:38:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:38:38.92975941 +0000 UTC m=+13.800587971" watchObservedRunningTime="2026-10-30 20:38:39.938497606 +0000 UTC m=+14.809326166" Oct 30 20:38:40.729357 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:40.729322 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-857r2" Oct 30 20:38:40.729539 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:40.729363 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jp2cr" Oct 30 20:38:40.729539 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:40.729329 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pdcfk" Oct 30 20:38:40.729539 ip-10-0-133-192 kubenswrapper[2578]: E1030 20:38:40.729452 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-857r2" podUID="bf21cf19-4475-41c3-9153-44d8fb329cdb" Oct 30 20:38:40.729705 ip-10-0-133-192 kubenswrapper[2578]: E1030 20:38:40.729549 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jp2cr" podUID="0dde7eae-1962-4559-89e9-0c67b1165914" Oct 30 20:38:40.729705 ip-10-0-133-192 kubenswrapper[2578]: E1030 20:38:40.729616 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pdcfk" podUID="0915cfe8-4606-4f92-877b-ebe7f52f4187" Oct 30 20:38:41.915991 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:41.915718 2578 generic.go:356] "Generic (PLEG): container finished" podID="ea0e353b-8cb1-4759-9f47-03f788f256d3" containerID="356f44059a64aae0d8e96477590362d61a9f5e8ff5231740ca428714df820265" exitCode=0 Oct 30 20:38:41.915991 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:41.915811 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nxmjg" event={"ID":"ea0e353b-8cb1-4759-9f47-03f788f256d3","Type":"ContainerDied","Data":"356f44059a64aae0d8e96477590362d61a9f5e8ff5231740ca428714df820265"} Oct 30 20:38:42.729117 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:42.729090 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-857r2" Oct 30 20:38:42.729203 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:42.729122 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pdcfk" Oct 30 20:38:42.729203 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:42.729188 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jp2cr" Oct 30 20:38:42.729269 ip-10-0-133-192 kubenswrapper[2578]: E1030 20:38:42.729201 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-857r2" podUID="bf21cf19-4475-41c3-9153-44d8fb329cdb" Oct 30 20:38:42.729269 ip-10-0-133-192 kubenswrapper[2578]: E1030 20:38:42.729252 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jp2cr" podUID="0dde7eae-1962-4559-89e9-0c67b1165914" Oct 30 20:38:42.729335 ip-10-0-133-192 kubenswrapper[2578]: E1030 20:38:42.729315 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pdcfk" podUID="0915cfe8-4606-4f92-877b-ebe7f52f4187" Oct 30 20:38:42.921667 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:42.921448 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7w5wz_7e2574c1-b59c-453c-9501-25ec13a1ccad/ovn-acl-logging/0.log" Oct 30 20:38:42.922071 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:42.922045 2578 generic.go:356] "Generic (PLEG): container finished" podID="7e2574c1-b59c-453c-9501-25ec13a1ccad" containerID="18eb59f6bb86ae195c78e6f8c8bad822a23d47fd7a6e91c9379af2f71aea686c" exitCode=1 Oct 30 20:38:42.922170 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:42.922076 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7w5wz" event={"ID":"7e2574c1-b59c-453c-9501-25ec13a1ccad","Type":"ContainerStarted","Data":"a2a06ed9586e9088e65d17f8fadc6f9409ae71a37b947eed277be88d61931555"} Oct 30 20:38:42.922170 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:42.922139 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7w5wz" event={"ID":"7e2574c1-b59c-453c-9501-25ec13a1ccad","Type":"ContainerStarted","Data":"e9c689dca3b5a44db234d7b05e7f1d990aff2d29caa0549231ee6d2a15d6db8c"} Oct 30 20:38:42.922277 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:42.922184 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7w5wz" event={"ID":"7e2574c1-b59c-453c-9501-25ec13a1ccad","Type":"ContainerStarted","Data":"4d654f62e7775bab9e8d329afc42dec153618b35e9762efd89aee2a26d3bf60a"} Oct 30 20:38:42.922277 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:42.922224 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7w5wz" event={"ID":"7e2574c1-b59c-453c-9501-25ec13a1ccad","Type":"ContainerStarted","Data":"3eaa75b252f1b41c8c37c9dbd4f4c639ecaa131715dda80047ce760d8b5c514b"} Oct 30 20:38:42.922277 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:42.922271 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7w5wz" event={"ID":"7e2574c1-b59c-453c-9501-25ec13a1ccad","Type":"ContainerDied","Data":"18eb59f6bb86ae195c78e6f8c8bad822a23d47fd7a6e91c9379af2f71aea686c"} Oct 30 20:38:42.922428 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:42.922306 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7w5wz" event={"ID":"7e2574c1-b59c-453c-9501-25ec13a1ccad","Type":"ContainerStarted","Data":"a4c68bd0459faaf8dbde675daa5034791e62c6c4b8e0911f7b7ee002a684c660"} Oct 30 20:38:42.925286 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:42.925266 2578 generic.go:356] "Generic (PLEG): container finished" podID="ea0e353b-8cb1-4759-9f47-03f788f256d3" containerID="9d9de1dac3c841a5c65f0990eb6038317a82d39b037aeb1700ab7401e474df06" exitCode=0 Oct 30 20:38:42.925359 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:42.925306 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nxmjg" event={"ID":"ea0e353b-8cb1-4759-9f47-03f788f256d3","Type":"ContainerDied","Data":"9d9de1dac3c841a5c65f0990eb6038317a82d39b037aeb1700ab7401e474df06"} Oct 30 20:38:43.233225 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:43.233194 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bf21cf19-4475-41c3-9153-44d8fb329cdb-original-pull-secret\") pod \"global-pull-secret-syncer-857r2\" (UID: \"bf21cf19-4475-41c3-9153-44d8fb329cdb\") " pod="kube-system/global-pull-secret-syncer-857r2" Oct 30 20:38:43.233334 ip-10-0-133-192 kubenswrapper[2578]: E1030 20:38:43.233315 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 20:38:43.233387 ip-10-0-133-192 kubenswrapper[2578]: E1030 20:38:43.233373 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bf21cf19-4475-41c3-9153-44d8fb329cdb-original-pull-secret podName:bf21cf19-4475-41c3-9153-44d8fb329cdb nodeName:}" failed. No retries permitted until 2026-10-30 20:38:59.233358602 +0000 UTC m=+34.104187156 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/bf21cf19-4475-41c3-9153-44d8fb329cdb-original-pull-secret") pod "global-pull-secret-syncer-857r2" (UID: "bf21cf19-4475-41c3-9153-44d8fb329cdb") : object "kube-system"/"original-pull-secret" not registered Oct 30 20:38:43.333620 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:43.333571 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6hvqv\" (UniqueName: \"kubernetes.io/projected/0dde7eae-1962-4559-89e9-0c67b1165914-kube-api-access-6hvqv\") pod \"network-check-target-jp2cr\" (UID: \"0dde7eae-1962-4559-89e9-0c67b1165914\") " pod="openshift-network-diagnostics/network-check-target-jp2cr" Oct 30 20:38:43.333620 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:43.333599 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0915cfe8-4606-4f92-877b-ebe7f52f4187-metrics-certs\") pod \"network-metrics-daemon-pdcfk\" (UID: \"0915cfe8-4606-4f92-877b-ebe7f52f4187\") " pod="openshift-multus/network-metrics-daemon-pdcfk" Oct 30 20:38:43.333763 ip-10-0-133-192 kubenswrapper[2578]: E1030 20:38:43.333670 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 20:38:43.333763 ip-10-0-133-192 kubenswrapper[2578]: E1030 20:38:43.333698 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 20:38:43.333763 ip-10-0-133-192 kubenswrapper[2578]: E1030 20:38:43.333710 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0915cfe8-4606-4f92-877b-ebe7f52f4187-metrics-certs podName:0915cfe8-4606-4f92-877b-ebe7f52f4187 nodeName:}" failed. No retries permitted until 2026-10-30 20:38:59.333697693 +0000 UTC m=+34.204526230 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0915cfe8-4606-4f92-877b-ebe7f52f4187-metrics-certs") pod "network-metrics-daemon-pdcfk" (UID: "0915cfe8-4606-4f92-877b-ebe7f52f4187") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 20:38:43.333763 ip-10-0-133-192 kubenswrapper[2578]: E1030 20:38:43.333713 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 20:38:43.333763 ip-10-0-133-192 kubenswrapper[2578]: E1030 20:38:43.333722 2578 projected.go:196] Error preparing data for projected volume kube-api-access-6hvqv for pod openshift-network-diagnostics/network-check-target-jp2cr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 20:38:43.333763 ip-10-0-133-192 kubenswrapper[2578]: E1030 20:38:43.333755 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0dde7eae-1962-4559-89e9-0c67b1165914-kube-api-access-6hvqv podName:0dde7eae-1962-4559-89e9-0c67b1165914 nodeName:}" failed. No retries permitted until 2026-10-30 20:38:59.333745164 +0000 UTC m=+34.204573701 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-6hvqv" (UniqueName: "kubernetes.io/projected/0dde7eae-1962-4559-89e9-0c67b1165914-kube-api-access-6hvqv") pod "network-check-target-jp2cr" (UID: "0dde7eae-1962-4559-89e9-0c67b1165914") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 20:38:43.931990 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:43.931956 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nxmjg" event={"ID":"ea0e353b-8cb1-4759-9f47-03f788f256d3","Type":"ContainerStarted","Data":"6df5689f833cca9127889c1bdbee0191976985023cab95ec48107e1eb70da407"} Oct 30 20:38:43.963705 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:43.963666 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-nxmjg" podStartSLOduration=7.260447222 podStartE2EDuration="17.963654658s" totalImagesPullingTime="10.703207436s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 20:38:26 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:38:43.961178504 +0000 UTC m=+18.832007064" watchObservedRunningTime="2026-10-30 20:38:43.963654658 +0000 UTC m=+18.834483216" Oct 30 20:38:44.729663 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:44.729631 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jp2cr" Oct 30 20:38:44.729809 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:44.729668 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-857r2" Oct 30 20:38:44.729809 ip-10-0-133-192 kubenswrapper[2578]: E1030 20:38:44.729721 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jp2cr" podUID="0dde7eae-1962-4559-89e9-0c67b1165914" Oct 30 20:38:44.729809 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:44.729775 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pdcfk" Oct 30 20:38:44.729901 ip-10-0-133-192 kubenswrapper[2578]: E1030 20:38:44.729852 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pdcfk" podUID="0915cfe8-4606-4f92-877b-ebe7f52f4187" Oct 30 20:38:44.729932 ip-10-0-133-192 kubenswrapper[2578]: E1030 20:38:44.729904 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-857r2" podUID="bf21cf19-4475-41c3-9153-44d8fb329cdb" Oct 30 20:38:45.937928 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:45.937901 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7w5wz_7e2574c1-b59c-453c-9501-25ec13a1ccad/ovn-acl-logging/0.log" Oct 30 20:38:45.938412 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:45.938200 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7w5wz" event={"ID":"7e2574c1-b59c-453c-9501-25ec13a1ccad","Type":"ContainerStarted","Data":"797e2afea2b230fa03d26e73979d0841c6bcbf9d338106fa29c03b40bceee2e0"} Oct 30 20:38:46.729558 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:46.729528 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jp2cr" Oct 30 20:38:46.729558 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:46.729542 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-857r2" Oct 30 20:38:46.729558 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:46.729559 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pdcfk" Oct 30 20:38:46.729783 ip-10-0-133-192 kubenswrapper[2578]: E1030 20:38:46.729625 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jp2cr" podUID="0dde7eae-1962-4559-89e9-0c67b1165914" Oct 30 20:38:46.729783 ip-10-0-133-192 kubenswrapper[2578]: E1030 20:38:46.729669 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pdcfk" podUID="0915cfe8-4606-4f92-877b-ebe7f52f4187" Oct 30 20:38:46.729783 ip-10-0-133-192 kubenswrapper[2578]: E1030 20:38:46.729759 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-857r2" podUID="bf21cf19-4475-41c3-9153-44d8fb329cdb" Oct 30 20:38:46.815905 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:46.815875 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:38:47.045786 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:47.045763 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:38:47.944946 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:47.944915 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7w5wz_7e2574c1-b59c-453c-9501-25ec13a1ccad/ovn-acl-logging/0.log" Oct 30 20:38:47.945295 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:47.945270 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7w5wz" event={"ID":"7e2574c1-b59c-453c-9501-25ec13a1ccad","Type":"ContainerStarted","Data":"1b20ff084790103465814e0b2655d0e71ffaf44ca7f071baf74ece3da8bd3b19"} Oct 30 20:38:47.945554 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:47.945526 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-7w5wz" Oct 30 20:38:47.945659 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:47.945606 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-7w5wz" Oct 30 20:38:47.945716 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:47.945691 2578 scope.go:122] "RemoveContainer" containerID="18eb59f6bb86ae195c78e6f8c8bad822a23d47fd7a6e91c9379af2f71aea686c" Oct 30 20:38:47.960792 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:47.960769 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-7w5wz" Oct 30 20:38:47.961254 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:47.961242 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-7w5wz" Oct 30 20:38:48.729358 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:48.729175 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jp2cr" Oct 30 20:38:48.729927 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:48.729217 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-857r2" Oct 30 20:38:48.729927 ip-10-0-133-192 kubenswrapper[2578]: E1030 20:38:48.729471 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jp2cr" podUID="0dde7eae-1962-4559-89e9-0c67b1165914" Oct 30 20:38:48.729927 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:48.729217 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pdcfk" Oct 30 20:38:48.729927 ip-10-0-133-192 kubenswrapper[2578]: E1030 20:38:48.729554 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-857r2" podUID="bf21cf19-4475-41c3-9153-44d8fb329cdb" Oct 30 20:38:48.729927 ip-10-0-133-192 kubenswrapper[2578]: E1030 20:38:48.729605 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pdcfk" podUID="0915cfe8-4606-4f92-877b-ebe7f52f4187" Oct 30 20:38:48.733359 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:48.733332 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-857r2"] Oct 30 20:38:48.737907 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:48.737882 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-jp2cr"] Oct 30 20:38:48.738418 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:48.738400 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-pdcfk"] Oct 30 20:38:48.951054 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:48.950992 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7w5wz_7e2574c1-b59c-453c-9501-25ec13a1ccad/ovn-acl-logging/0.log" Oct 30 20:38:48.951335 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:48.951313 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7w5wz" event={"ID":"7e2574c1-b59c-453c-9501-25ec13a1ccad","Type":"ContainerStarted","Data":"46b31481efdc7c56c5e75074f39b68a8fb56c85db0327c5b7e9a7c6b66ae77c4"} Oct 30 20:38:48.951411 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:48.951353 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pdcfk" Oct 30 20:38:48.951411 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:48.951404 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jp2cr" Oct 30 20:38:48.951528 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:48.951404 2578 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 20:38:48.951528 ip-10-0-133-192 kubenswrapper[2578]: E1030 20:38:48.951468 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jp2cr" podUID="0dde7eae-1962-4559-89e9-0c67b1165914" Oct 30 20:38:48.951635 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:48.951589 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-857r2" Oct 30 20:38:48.951705 ip-10-0-133-192 kubenswrapper[2578]: E1030 20:38:48.951686 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-857r2" podUID="bf21cf19-4475-41c3-9153-44d8fb329cdb" Oct 30 20:38:48.951787 ip-10-0-133-192 kubenswrapper[2578]: E1030 20:38:48.951760 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pdcfk" podUID="0915cfe8-4606-4f92-877b-ebe7f52f4187" Oct 30 20:38:48.988428 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:48.988389 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-7w5wz" podStartSLOduration=8.40407158 podStartE2EDuration="22.98837894s" totalImagesPullingTime="14.58430736s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 20:38:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:38:48.988379338 +0000 UTC m=+23.859207908" watchObservedRunningTime="2026-10-30 20:38:48.98837894 +0000 UTC m=+23.859207498" Oct 30 20:38:49.247631 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:49.247561 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-7w5wz" Oct 30 20:38:50.729491 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:50.729456 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pdcfk" Oct 30 20:38:50.729491 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:50.729477 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jp2cr" Oct 30 20:38:50.729982 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:50.729454 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-857r2" Oct 30 20:38:50.729982 ip-10-0-133-192 kubenswrapper[2578]: E1030 20:38:50.729616 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pdcfk" podUID="0915cfe8-4606-4f92-877b-ebe7f52f4187" Oct 30 20:38:50.729982 ip-10-0-133-192 kubenswrapper[2578]: E1030 20:38:50.729711 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jp2cr" podUID="0dde7eae-1962-4559-89e9-0c67b1165914" Oct 30 20:38:50.729982 ip-10-0-133-192 kubenswrapper[2578]: E1030 20:38:50.729765 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-857r2" podUID="bf21cf19-4475-41c3-9153-44d8fb329cdb" Oct 30 20:38:50.850242 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:50.850213 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:38:51.051555 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:51.051445 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:38:52.730033 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:52.729999 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-857r2" Oct 30 20:38:52.730033 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:52.730012 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pdcfk" Oct 30 20:38:52.730640 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:52.730093 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jp2cr" Oct 30 20:38:52.730640 ip-10-0-133-192 kubenswrapper[2578]: E1030 20:38:52.730109 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-857r2" podUID="bf21cf19-4475-41c3-9153-44d8fb329cdb" Oct 30 20:38:52.730640 ip-10-0-133-192 kubenswrapper[2578]: E1030 20:38:52.730162 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jp2cr" podUID="0dde7eae-1962-4559-89e9-0c67b1165914" Oct 30 20:38:52.730640 ip-10-0-133-192 kubenswrapper[2578]: E1030 20:38:52.730234 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pdcfk" podUID="0915cfe8-4606-4f92-877b-ebe7f52f4187" Oct 30 20:38:53.183215 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:53.183187 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:38:53.336210 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:53.336173 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:38:53.605168 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:53.605134 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:38:53.870262 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:53.870192 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:38:54.016321 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:54.016286 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:38:54.415653 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:54.415598 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-192.us-east-2.compute.internal" event="NodeReady" Oct 30 20:38:54.415748 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:54.415706 2578 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Oct 30 20:38:54.466630 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:54.466603 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-2sccz"] Oct 30 20:38:54.468992 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:54.468976 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-gmq97"] Oct 30 20:38:54.469128 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:54.469111 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-2sccz" Oct 30 20:38:54.471644 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:54.471626 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-gmq97" Oct 30 20:38:54.476691 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:54.476677 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-pl5jd\"" Oct 30 20:38:54.479716 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:54.479701 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-cr4sq\"" Oct 30 20:38:54.479888 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:54.479843 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 30 20:38:54.481540 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:54.481504 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-gmq97"] Oct 30 20:38:54.482731 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:54.482599 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 30 20:38:54.482731 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:54.482701 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 30 20:38:54.482872 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:54.482738 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-2sccz"] Oct 30 20:38:54.490217 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:54.490199 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-hwzc6"] Oct 30 20:38:54.492919 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:54.492903 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-hwzc6" Oct 30 20:38:54.505232 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:54.505211 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-hwzc6"] Oct 30 20:38:54.576923 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:54.576900 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 30 20:38:54.577017 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:54.576943 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 30 20:38:54.577017 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:54.577004 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 30 20:38:54.577082 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:54.577019 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-2twjb\"" Oct 30 20:38:54.579911 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:54.579900 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 30 20:38:54.580043 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:54.580026 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 30 20:38:54.580118 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:54.580108 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 30 20:38:54.622130 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:54.622108 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6c3ba176-c9a5-4b6c-ae06-5593f68cf510-cert\") pod \"ingress-canary-gmq97\" (UID: \"6c3ba176-c9a5-4b6c-ae06-5593f68cf510\") " pod="openshift-ingress-canary/ingress-canary-gmq97" Oct 30 20:38:54.622209 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:54.622141 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0aa2bb04-0552-45da-a54f-7e5064cf3113-metrics-tls\") pod \"dns-default-2sccz\" (UID: \"0aa2bb04-0552-45da-a54f-7e5064cf3113\") " pod="openshift-dns/dns-default-2sccz" Oct 30 20:38:54.622255 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:54.622219 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c1a9e89f-5d46-4414-b0e0-7adff658510e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hwzc6\" (UID: \"c1a9e89f-5d46-4414-b0e0-7adff658510e\") " pod="openshift-insights/insights-runtime-extractor-hwzc6" Oct 30 20:38:54.622255 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:54.622242 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9d557\" (UniqueName: \"kubernetes.io/projected/c1a9e89f-5d46-4414-b0e0-7adff658510e-kube-api-access-9d557\") pod \"insights-runtime-extractor-hwzc6\" (UID: \"c1a9e89f-5d46-4414-b0e0-7adff658510e\") " pod="openshift-insights/insights-runtime-extractor-hwzc6" Oct 30 20:38:54.622328 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:54.622303 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0aa2bb04-0552-45da-a54f-7e5064cf3113-tmp-dir\") pod \"dns-default-2sccz\" (UID: \"0aa2bb04-0552-45da-a54f-7e5064cf3113\") " pod="openshift-dns/dns-default-2sccz" Oct 30 20:38:54.622328 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:54.622325 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c1a9e89f-5d46-4414-b0e0-7adff658510e-crio-socket\") pod \"insights-runtime-extractor-hwzc6\" (UID: \"c1a9e89f-5d46-4414-b0e0-7adff658510e\") " pod="openshift-insights/insights-runtime-extractor-hwzc6" Oct 30 20:38:54.622401 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:54.622342 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qhljq\" (UniqueName: \"kubernetes.io/projected/6c3ba176-c9a5-4b6c-ae06-5593f68cf510-kube-api-access-qhljq\") pod \"ingress-canary-gmq97\" (UID: \"6c3ba176-c9a5-4b6c-ae06-5593f68cf510\") " pod="openshift-ingress-canary/ingress-canary-gmq97" Oct 30 20:38:54.622401 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:54.622356 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c1a9e89f-5d46-4414-b0e0-7adff658510e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hwzc6\" (UID: \"c1a9e89f-5d46-4414-b0e0-7adff658510e\") " pod="openshift-insights/insights-runtime-extractor-hwzc6" Oct 30 20:38:54.622401 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:54.622378 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/0aa2bb04-0552-45da-a54f-7e5064cf3113-config-volume\") pod \"dns-default-2sccz\" (UID: \"0aa2bb04-0552-45da-a54f-7e5064cf3113\") " pod="openshift-dns/dns-default-2sccz" Oct 30 20:38:54.622524 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:54.622474 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hbqzh\" (UniqueName: \"kubernetes.io/projected/0aa2bb04-0552-45da-a54f-7e5064cf3113-kube-api-access-hbqzh\") pod \"dns-default-2sccz\" (UID: \"0aa2bb04-0552-45da-a54f-7e5064cf3113\") " pod="openshift-dns/dns-default-2sccz" Oct 30 20:38:54.622561 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:54.622528 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/c1a9e89f-5d46-4414-b0e0-7adff658510e-data-volume\") pod \"insights-runtime-extractor-hwzc6\" (UID: \"c1a9e89f-5d46-4414-b0e0-7adff658510e\") " pod="openshift-insights/insights-runtime-extractor-hwzc6" Oct 30 20:38:54.723434 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:54.723390 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6c3ba176-c9a5-4b6c-ae06-5593f68cf510-cert\") pod \"ingress-canary-gmq97\" (UID: \"6c3ba176-c9a5-4b6c-ae06-5593f68cf510\") " pod="openshift-ingress-canary/ingress-canary-gmq97" Oct 30 20:38:54.723434 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:54.723418 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0aa2bb04-0552-45da-a54f-7e5064cf3113-metrics-tls\") pod \"dns-default-2sccz\" (UID: \"0aa2bb04-0552-45da-a54f-7e5064cf3113\") " pod="openshift-dns/dns-default-2sccz" Oct 30 20:38:54.723570 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:54.723453 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c1a9e89f-5d46-4414-b0e0-7adff658510e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hwzc6\" (UID: \"c1a9e89f-5d46-4414-b0e0-7adff658510e\") " pod="openshift-insights/insights-runtime-extractor-hwzc6" Oct 30 20:38:54.723570 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:54.723469 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9d557\" (UniqueName: \"kubernetes.io/projected/c1a9e89f-5d46-4414-b0e0-7adff658510e-kube-api-access-9d557\") pod \"insights-runtime-extractor-hwzc6\" (UID: \"c1a9e89f-5d46-4414-b0e0-7adff658510e\") " pod="openshift-insights/insights-runtime-extractor-hwzc6" Oct 30 20:38:54.723570 ip-10-0-133-192 kubenswrapper[2578]: E1030 20:38:54.723543 2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 20:38:54.723726 ip-10-0-133-192 kubenswrapper[2578]: E1030 20:38:54.723576 2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 20:38:54.723726 ip-10-0-133-192 kubenswrapper[2578]: E1030 20:38:54.723596 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0aa2bb04-0552-45da-a54f-7e5064cf3113-metrics-tls podName:0aa2bb04-0552-45da-a54f-7e5064cf3113 nodeName:}" failed. No retries permitted until 2026-10-30 20:38:55.22358 +0000 UTC m=+30.094408563 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0aa2bb04-0552-45da-a54f-7e5064cf3113-metrics-tls") pod "dns-default-2sccz" (UID: "0aa2bb04-0552-45da-a54f-7e5064cf3113") : secret "dns-default-metrics-tls" not found Oct 30 20:38:54.723726 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:54.723617 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0aa2bb04-0552-45da-a54f-7e5064cf3113-tmp-dir\") pod \"dns-default-2sccz\" (UID: \"0aa2bb04-0552-45da-a54f-7e5064cf3113\") " pod="openshift-dns/dns-default-2sccz" Oct 30 20:38:54.723726 ip-10-0-133-192 kubenswrapper[2578]: E1030 20:38:54.723638 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c1a9e89f-5d46-4414-b0e0-7adff658510e-insights-runtime-extractor-tls podName:c1a9e89f-5d46-4414-b0e0-7adff658510e nodeName:}" failed. No retries permitted until 2026-10-30 20:38:55.223613868 +0000 UTC m=+30.094442419 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c1a9e89f-5d46-4414-b0e0-7adff658510e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hwzc6" (UID: "c1a9e89f-5d46-4414-b0e0-7adff658510e") : secret "insights-runtime-extractor-tls" not found Oct 30 20:38:54.723726 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:54.723684 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c1a9e89f-5d46-4414-b0e0-7adff658510e-crio-socket\") pod \"insights-runtime-extractor-hwzc6\" (UID: \"c1a9e89f-5d46-4414-b0e0-7adff658510e\") " pod="openshift-insights/insights-runtime-extractor-hwzc6" Oct 30 20:38:54.723726 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:54.723716 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qhljq\" (UniqueName: \"kubernetes.io/projected/6c3ba176-c9a5-4b6c-ae06-5593f68cf510-kube-api-access-qhljq\") pod \"ingress-canary-gmq97\" (UID: \"6c3ba176-c9a5-4b6c-ae06-5593f68cf510\") " pod="openshift-ingress-canary/ingress-canary-gmq97" Oct 30 20:38:54.724021 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:54.723741 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c1a9e89f-5d46-4414-b0e0-7adff658510e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hwzc6\" (UID: \"c1a9e89f-5d46-4414-b0e0-7adff658510e\") " pod="openshift-insights/insights-runtime-extractor-hwzc6" Oct 30 20:38:54.724021 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:54.723769 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/0aa2bb04-0552-45da-a54f-7e5064cf3113-config-volume\") pod \"dns-default-2sccz\" (UID: \"0aa2bb04-0552-45da-a54f-7e5064cf3113\") " pod="openshift-dns/dns-default-2sccz" Oct 30 20:38:54.724021 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:54.723796 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hbqzh\" (UniqueName: \"kubernetes.io/projected/0aa2bb04-0552-45da-a54f-7e5064cf3113-kube-api-access-hbqzh\") pod \"dns-default-2sccz\" (UID: \"0aa2bb04-0552-45da-a54f-7e5064cf3113\") " pod="openshift-dns/dns-default-2sccz" Oct 30 20:38:54.724021 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:54.723822 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/c1a9e89f-5d46-4414-b0e0-7adff658510e-data-volume\") pod \"insights-runtime-extractor-hwzc6\" (UID: \"c1a9e89f-5d46-4414-b0e0-7adff658510e\") " pod="openshift-insights/insights-runtime-extractor-hwzc6" Oct 30 20:38:54.724021 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:54.723896 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0aa2bb04-0552-45da-a54f-7e5064cf3113-tmp-dir\") pod \"dns-default-2sccz\" (UID: \"0aa2bb04-0552-45da-a54f-7e5064cf3113\") " pod="openshift-dns/dns-default-2sccz" Oct 30 20:38:54.724021 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:54.723902 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c1a9e89f-5d46-4414-b0e0-7adff658510e-crio-socket\") pod \"insights-runtime-extractor-hwzc6\" (UID: \"c1a9e89f-5d46-4414-b0e0-7adff658510e\") " pod="openshift-insights/insights-runtime-extractor-hwzc6" Oct 30 20:38:54.724021 ip-10-0-133-192 kubenswrapper[2578]: E1030 20:38:54.723931 2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 30 20:38:54.724021 ip-10-0-133-192 kubenswrapper[2578]: E1030 20:38:54.724008 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c1a9e89f-5d46-4414-b0e0-7adff658510e-kube-rbac-proxy-cm podName:c1a9e89f-5d46-4414-b0e0-7adff658510e nodeName:}" failed. No retries permitted until 2026-10-30 20:38:55.223992671 +0000 UTC m=+30.094821256 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/c1a9e89f-5d46-4414-b0e0-7adff658510e-kube-rbac-proxy-cm") pod "insights-runtime-extractor-hwzc6" (UID: "c1a9e89f-5d46-4414-b0e0-7adff658510e") : configmap "kube-rbac-proxy" not found Oct 30 20:38:54.724325 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:54.724147 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/c1a9e89f-5d46-4414-b0e0-7adff658510e-data-volume\") pod \"insights-runtime-extractor-hwzc6\" (UID: \"c1a9e89f-5d46-4414-b0e0-7adff658510e\") " pod="openshift-insights/insights-runtime-extractor-hwzc6" Oct 30 20:38:54.724483 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:54.724466 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/0aa2bb04-0552-45da-a54f-7e5064cf3113-config-volume\") pod \"dns-default-2sccz\" (UID: \"0aa2bb04-0552-45da-a54f-7e5064cf3113\") " pod="openshift-dns/dns-default-2sccz" Oct 30 20:38:54.727574 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:54.727430 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6c3ba176-c9a5-4b6c-ae06-5593f68cf510-cert\") pod \"ingress-canary-gmq97\" (UID: \"6c3ba176-c9a5-4b6c-ae06-5593f68cf510\") " pod="openshift-ingress-canary/ingress-canary-gmq97" Oct 30 20:38:54.730169 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:54.730150 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-857r2" Oct 30 20:38:54.730261 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:54.730166 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jp2cr" Oct 30 20:38:54.730441 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:54.730386 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pdcfk" Oct 30 20:38:54.735053 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:54.735030 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 30 20:38:54.735053 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:54.735030 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 20:38:54.735228 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:54.735127 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-d5kr7\"" Oct 30 20:38:54.735228 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:54.735166 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 30 20:38:54.735228 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:54.735202 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-5p7gr\"" Oct 30 20:38:54.735402 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:54.735298 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-ndwvd\"" Oct 30 20:38:54.735402 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:54.735378 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 20:38:54.739561 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:54.739541 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9d557\" (UniqueName: \"kubernetes.io/projected/c1a9e89f-5d46-4414-b0e0-7adff658510e-kube-api-access-9d557\") pod \"insights-runtime-extractor-hwzc6\" (UID: \"c1a9e89f-5d46-4414-b0e0-7adff658510e\") " pod="openshift-insights/insights-runtime-extractor-hwzc6" Oct 30 20:38:54.739986 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:54.739962 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hbqzh\" (UniqueName: \"kubernetes.io/projected/0aa2bb04-0552-45da-a54f-7e5064cf3113-kube-api-access-hbqzh\") pod \"dns-default-2sccz\" (UID: \"0aa2bb04-0552-45da-a54f-7e5064cf3113\") " pod="openshift-dns/dns-default-2sccz" Oct 30 20:38:54.741309 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:54.741290 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qhljq\" (UniqueName: \"kubernetes.io/projected/6c3ba176-c9a5-4b6c-ae06-5593f68cf510-kube-api-access-qhljq\") pod \"ingress-canary-gmq97\" (UID: \"6c3ba176-c9a5-4b6c-ae06-5593f68cf510\") " pod="openshift-ingress-canary/ingress-canary-gmq97" Oct 30 20:38:54.787956 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:54.787935 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-gmq97" Oct 30 20:38:54.970450 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:54.970419 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:38:55.054596 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:55.054544 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-jfxnj" Oct 30 20:38:55.054703 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:55.054662 2578 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 20:38:55.055091 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:55.055078 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-jfxnj" Oct 30 20:38:55.067611 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:55.067589 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-gmq97"] Oct 30 20:38:55.070350 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:55.070324 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c3ba176_c9a5_4b6c_ae06_5593f68cf510.slice/crio-b26584b16a622d31abdf0ac29e363b7db4853bdaf37ec6c855ea80444314bbf5 WatchSource:0}: Error finding container b26584b16a622d31abdf0ac29e363b7db4853bdaf37ec6c855ea80444314bbf5: Status 404 returned error can't find the container with id b26584b16a622d31abdf0ac29e363b7db4853bdaf37ec6c855ea80444314bbf5 Oct 30 20:38:55.124776 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:55.124754 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:38:55.226680 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:55.226659 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0aa2bb04-0552-45da-a54f-7e5064cf3113-metrics-tls\") pod \"dns-default-2sccz\" (UID: \"0aa2bb04-0552-45da-a54f-7e5064cf3113\") " pod="openshift-dns/dns-default-2sccz" Oct 30 20:38:55.226775 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:55.226705 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c1a9e89f-5d46-4414-b0e0-7adff658510e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hwzc6\" (UID: \"c1a9e89f-5d46-4414-b0e0-7adff658510e\") " pod="openshift-insights/insights-runtime-extractor-hwzc6" Oct 30 20:38:55.226775 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:55.226762 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c1a9e89f-5d46-4414-b0e0-7adff658510e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hwzc6\" (UID: \"c1a9e89f-5d46-4414-b0e0-7adff658510e\") " pod="openshift-insights/insights-runtime-extractor-hwzc6" Oct 30 20:38:55.226844 ip-10-0-133-192 kubenswrapper[2578]: E1030 20:38:55.226797 2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 20:38:55.226844 ip-10-0-133-192 kubenswrapper[2578]: E1030 20:38:55.226810 2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 30 20:38:55.226901 ip-10-0-133-192 kubenswrapper[2578]: E1030 20:38:55.226854 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c1a9e89f-5d46-4414-b0e0-7adff658510e-kube-rbac-proxy-cm podName:c1a9e89f-5d46-4414-b0e0-7adff658510e nodeName:}" failed. No retries permitted until 2026-10-30 20:38:56.226842715 +0000 UTC m=+31.097671265 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/c1a9e89f-5d46-4414-b0e0-7adff658510e-kube-rbac-proxy-cm") pod "insights-runtime-extractor-hwzc6" (UID: "c1a9e89f-5d46-4414-b0e0-7adff658510e") : configmap "kube-rbac-proxy" not found Oct 30 20:38:55.226901 ip-10-0-133-192 kubenswrapper[2578]: E1030 20:38:55.226856 2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 20:38:55.226901 ip-10-0-133-192 kubenswrapper[2578]: E1030 20:38:55.226867 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0aa2bb04-0552-45da-a54f-7e5064cf3113-metrics-tls podName:0aa2bb04-0552-45da-a54f-7e5064cf3113 nodeName:}" failed. No retries permitted until 2026-10-30 20:38:56.226861474 +0000 UTC m=+31.097690024 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0aa2bb04-0552-45da-a54f-7e5064cf3113-metrics-tls") pod "dns-default-2sccz" (UID: "0aa2bb04-0552-45da-a54f-7e5064cf3113") : secret "dns-default-metrics-tls" not found Oct 30 20:38:55.227002 ip-10-0-133-192 kubenswrapper[2578]: E1030 20:38:55.226923 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c1a9e89f-5d46-4414-b0e0-7adff658510e-insights-runtime-extractor-tls podName:c1a9e89f-5d46-4414-b0e0-7adff658510e nodeName:}" failed. No retries permitted until 2026-10-30 20:38:56.226908897 +0000 UTC m=+31.097737449 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c1a9e89f-5d46-4414-b0e0-7adff658510e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hwzc6" (UID: "c1a9e89f-5d46-4414-b0e0-7adff658510e") : secret "insights-runtime-extractor-tls" not found Oct 30 20:38:55.332582 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:55.332525 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:38:55.518129 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:55.518080 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:38:55.671312 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:55.671234 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:38:55.967034 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:55.966959 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-gmq97" event={"ID":"6c3ba176-c9a5-4b6c-ae06-5593f68cf510","Type":"ContainerStarted","Data":"b26584b16a622d31abdf0ac29e363b7db4853bdaf37ec6c855ea80444314bbf5"} Oct 30 20:38:56.074688 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:56.074657 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-lc599_c9257033-53bf-40fc-a1e9-67fd5901678b/dns-node-resolver/0.log" Oct 30 20:38:56.157057 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:56.157022 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:38:56.236071 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:56.235988 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0aa2bb04-0552-45da-a54f-7e5064cf3113-metrics-tls\") pod \"dns-default-2sccz\" (UID: \"0aa2bb04-0552-45da-a54f-7e5064cf3113\") " pod="openshift-dns/dns-default-2sccz" Oct 30 20:38:56.236071 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:56.236046 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c1a9e89f-5d46-4414-b0e0-7adff658510e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hwzc6\" (UID: \"c1a9e89f-5d46-4414-b0e0-7adff658510e\") " pod="openshift-insights/insights-runtime-extractor-hwzc6" Oct 30 20:38:56.236271 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:56.236099 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c1a9e89f-5d46-4414-b0e0-7adff658510e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hwzc6\" (UID: \"c1a9e89f-5d46-4414-b0e0-7adff658510e\") " pod="openshift-insights/insights-runtime-extractor-hwzc6" Oct 30 20:38:56.236271 ip-10-0-133-192 kubenswrapper[2578]: E1030 20:38:56.236196 2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 20:38:56.236271 ip-10-0-133-192 kubenswrapper[2578]: E1030 20:38:56.236204 2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 30 20:38:56.236420 ip-10-0-133-192 kubenswrapper[2578]: E1030 20:38:56.236281 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c1a9e89f-5d46-4414-b0e0-7adff658510e-insights-runtime-extractor-tls podName:c1a9e89f-5d46-4414-b0e0-7adff658510e nodeName:}" failed. No retries permitted until 2026-10-30 20:38:58.236261071 +0000 UTC m=+33.107089622 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c1a9e89f-5d46-4414-b0e0-7adff658510e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hwzc6" (UID: "c1a9e89f-5d46-4414-b0e0-7adff658510e") : secret "insights-runtime-extractor-tls" not found Oct 30 20:38:56.236420 ip-10-0-133-192 kubenswrapper[2578]: E1030 20:38:56.236330 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c1a9e89f-5d46-4414-b0e0-7adff658510e-kube-rbac-proxy-cm podName:c1a9e89f-5d46-4414-b0e0-7adff658510e nodeName:}" failed. No retries permitted until 2026-10-30 20:38:58.236310155 +0000 UTC m=+33.107138711 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/c1a9e89f-5d46-4414-b0e0-7adff658510e-kube-rbac-proxy-cm") pod "insights-runtime-extractor-hwzc6" (UID: "c1a9e89f-5d46-4414-b0e0-7adff658510e") : configmap "kube-rbac-proxy" not found Oct 30 20:38:56.238997 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:56.238968 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0aa2bb04-0552-45da-a54f-7e5064cf3113-metrics-tls\") pod \"dns-default-2sccz\" (UID: \"0aa2bb04-0552-45da-a54f-7e5064cf3113\") " pod="openshift-dns/dns-default-2sccz" Oct 30 20:38:56.284225 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:56.284186 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-2sccz" Oct 30 20:38:56.304081 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:56.304055 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:38:56.614795 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:56.614734 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-2sccz"] Oct 30 20:38:56.617858 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:56.617832 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0aa2bb04_0552_45da_a54f_7e5064cf3113.slice/crio-ef4fe99de9fe62a9de95ae83fb4b6dccc0ea8e7edb7c3c4ec2475d6d4e3740e3 WatchSource:0}: Error finding container ef4fe99de9fe62a9de95ae83fb4b6dccc0ea8e7edb7c3c4ec2475d6d4e3740e3: Status 404 returned error can't find the container with id ef4fe99de9fe62a9de95ae83fb4b6dccc0ea8e7edb7c3c4ec2475d6d4e3740e3 Oct 30 20:38:56.888204 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:56.888122 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:38:56.970430 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:56.970393 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-2sccz" event={"ID":"0aa2bb04-0552-45da-a54f-7e5064cf3113","Type":"ContainerStarted","Data":"ef4fe99de9fe62a9de95ae83fb4b6dccc0ea8e7edb7c3c4ec2475d6d4e3740e3"} Oct 30 20:38:57.034982 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:57.034949 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:38:57.219152 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:57.219079 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:38:57.274232 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:57.274205 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-hsv4j_35d51f79-80a9-4f9d-970e-6a35a34976a2/node-ca/0.log" Oct 30 20:38:57.412797 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:57.412628 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:38:57.614333 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:57.614254 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:38:57.755122 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:57.755078 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:38:57.976288 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:57.976256 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-gmq97" event={"ID":"6c3ba176-c9a5-4b6c-ae06-5593f68cf510","Type":"ContainerStarted","Data":"e370ddddc171618430c0195314511534f6d00e044cebe9797d30e07bf5b17fdb"} Oct 30 20:38:57.996787 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:57.996734 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-gmq97" podStartSLOduration=1.229505209 podStartE2EDuration="3.996717056s" totalImagesPullingTime="2.767211847s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:38:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:38:57.996435018 +0000 UTC m=+32.867263605" watchObservedRunningTime="2026-10-30 20:38:57.996717056 +0000 UTC m=+32.867545616" Oct 30 20:38:58.255771 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:58.255691 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c1a9e89f-5d46-4414-b0e0-7adff658510e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hwzc6\" (UID: \"c1a9e89f-5d46-4414-b0e0-7adff658510e\") " pod="openshift-insights/insights-runtime-extractor-hwzc6" Oct 30 20:38:58.255771 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:58.255759 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c1a9e89f-5d46-4414-b0e0-7adff658510e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hwzc6\" (UID: \"c1a9e89f-5d46-4414-b0e0-7adff658510e\") " pod="openshift-insights/insights-runtime-extractor-hwzc6" Oct 30 20:38:58.256203 ip-10-0-133-192 kubenswrapper[2578]: E1030 20:38:58.255865 2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 20:38:58.256203 ip-10-0-133-192 kubenswrapper[2578]: E1030 20:38:58.255942 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c1a9e89f-5d46-4414-b0e0-7adff658510e-insights-runtime-extractor-tls podName:c1a9e89f-5d46-4414-b0e0-7adff658510e nodeName:}" failed. No retries permitted until 2026-10-30 20:39:02.255921625 +0000 UTC m=+37.126750166 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c1a9e89f-5d46-4414-b0e0-7adff658510e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hwzc6" (UID: "c1a9e89f-5d46-4414-b0e0-7adff658510e") : secret "insights-runtime-extractor-tls" not found Oct 30 20:38:58.256320 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:58.256303 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c1a9e89f-5d46-4414-b0e0-7adff658510e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hwzc6\" (UID: \"c1a9e89f-5d46-4414-b0e0-7adff658510e\") " pod="openshift-insights/insights-runtime-extractor-hwzc6" Oct 30 20:38:58.843027 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:58.842992 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:38:58.980441 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:58.980383 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-2sccz" event={"ID":"0aa2bb04-0552-45da-a54f-7e5064cf3113","Type":"ContainerStarted","Data":"d7e4a35edb588843d90b10186658dc646236705d06f2f6e8be262b53880213d8"} Oct 30 20:38:58.980441 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:58.980434 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-2sccz" event={"ID":"0aa2bb04-0552-45da-a54f-7e5064cf3113","Type":"ContainerStarted","Data":"f714be7ae96495cc268122d3e1e1a53105e917de6d44f36e0a01255f8a322601"} Oct 30 20:38:59.003141 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:59.003096 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/dns-default-2sccz" podStartSLOduration=2.995500431 podStartE2EDuration="5.00308365s" totalImagesPullingTime="2.007583219s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:38:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:38:59.002735436 +0000 UTC m=+33.873564002" watchObservedRunningTime="2026-10-30 20:38:59.00308365 +0000 UTC m=+33.873912209" Oct 30 20:38:59.009321 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:59.009295 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:38:59.218333 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:59.218308 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:38:59.264396 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:59.264347 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bf21cf19-4475-41c3-9153-44d8fb329cdb-original-pull-secret\") pod \"global-pull-secret-syncer-857r2\" (UID: \"bf21cf19-4475-41c3-9153-44d8fb329cdb\") " pod="kube-system/global-pull-secret-syncer-857r2" Oct 30 20:38:59.266469 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:59.266447 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bf21cf19-4475-41c3-9153-44d8fb329cdb-original-pull-secret\") pod \"global-pull-secret-syncer-857r2\" (UID: \"bf21cf19-4475-41c3-9153-44d8fb329cdb\") " pod="kube-system/global-pull-secret-syncer-857r2" Oct 30 20:38:59.364916 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:59.364892 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6hvqv\" (UniqueName: \"kubernetes.io/projected/0dde7eae-1962-4559-89e9-0c67b1165914-kube-api-access-6hvqv\") pod \"network-check-target-jp2cr\" (UID: \"0dde7eae-1962-4559-89e9-0c67b1165914\") " pod="openshift-network-diagnostics/network-check-target-jp2cr" Oct 30 20:38:59.365021 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:59.364928 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0915cfe8-4606-4f92-877b-ebe7f52f4187-metrics-certs\") pod \"network-metrics-daemon-pdcfk\" (UID: \"0915cfe8-4606-4f92-877b-ebe7f52f4187\") " pod="openshift-multus/network-metrics-daemon-pdcfk" Oct 30 20:38:59.366947 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:59.366926 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0915cfe8-4606-4f92-877b-ebe7f52f4187-metrics-certs\") pod \"network-metrics-daemon-pdcfk\" (UID: \"0915cfe8-4606-4f92-877b-ebe7f52f4187\") " pod="openshift-multus/network-metrics-daemon-pdcfk" Oct 30 20:38:59.367207 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:59.367185 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6hvqv\" (UniqueName: \"kubernetes.io/projected/0dde7eae-1962-4559-89e9-0c67b1165914-kube-api-access-6hvqv\") pod \"network-check-target-jp2cr\" (UID: \"0dde7eae-1962-4559-89e9-0c67b1165914\") " pod="openshift-network-diagnostics/network-check-target-jp2cr" Oct 30 20:38:59.417024 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:59.416990 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:38:59.543989 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:59.543933 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jp2cr" Oct 30 20:38:59.559066 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:59.559046 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pdcfk" Oct 30 20:38:59.561182 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:59.561170 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-857r2" Oct 30 20:38:59.626695 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:59.626656 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:38:59.778740 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:59.778714 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:38:59.862803 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:59.862739 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-pdcfk"] Oct 30 20:38:59.865953 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:59.865867 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-857r2"] Oct 30 20:38:59.866800 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:59.866780 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-jp2cr"] Oct 30 20:38:59.867075 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:59.867050 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0915cfe8_4606_4f92_877b_ebe7f52f4187.slice/crio-674027bf57e670c920fd19b8e877f7dbbb717cdf3f0206ded7eb16c50d06f5d4 WatchSource:0}: Error finding container 674027bf57e670c920fd19b8e877f7dbbb717cdf3f0206ded7eb16c50d06f5d4: Status 404 returned error can't find the container with id 674027bf57e670c920fd19b8e877f7dbbb717cdf3f0206ded7eb16c50d06f5d4 Oct 30 20:38:59.868223 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:59.868198 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf21cf19_4475_41c3_9153_44d8fb329cdb.slice/crio-67bacfc15712c8b72b34fba7f6724c4b91949a42ff24fc59c1abdc86761f4710 WatchSource:0}: Error finding container 67bacfc15712c8b72b34fba7f6724c4b91949a42ff24fc59c1abdc86761f4710: Status 404 returned error can't find the container with id 67bacfc15712c8b72b34fba7f6724c4b91949a42ff24fc59c1abdc86761f4710 Oct 30 20:38:59.869331 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:38:59.869305 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0dde7eae_1962_4559_89e9_0c67b1165914.slice/crio-7f09df490a9f357a975df928f931adc4a68f4d6d7a4c712186b84a51a5d8eafd WatchSource:0}: Error finding container 7f09df490a9f357a975df928f931adc4a68f4d6d7a4c712186b84a51a5d8eafd: Status 404 returned error can't find the container with id 7f09df490a9f357a975df928f931adc4a68f4d6d7a4c712186b84a51a5d8eafd Oct 30 20:38:59.983525 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:59.983476 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-pdcfk" event={"ID":"0915cfe8-4606-4f92-877b-ebe7f52f4187","Type":"ContainerStarted","Data":"674027bf57e670c920fd19b8e877f7dbbb717cdf3f0206ded7eb16c50d06f5d4"} Oct 30 20:38:59.984632 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:59.984608 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-857r2" event={"ID":"bf21cf19-4475-41c3-9153-44d8fb329cdb","Type":"ContainerStarted","Data":"67bacfc15712c8b72b34fba7f6724c4b91949a42ff24fc59c1abdc86761f4710"} Oct 30 20:38:59.985712 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:59.985690 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-jp2cr" event={"ID":"0dde7eae-1962-4559-89e9-0c67b1165914","Type":"ContainerStarted","Data":"7f09df490a9f357a975df928f931adc4a68f4d6d7a4c712186b84a51a5d8eafd"} Oct 30 20:38:59.985894 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:38:59.985878 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-2sccz" Oct 30 20:39:00.813264 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:39:00.813113 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:39:00.999141 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:39:00.999090 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-8984b54b7-skmcd"] Oct 30 20:39:01.007728 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:39:01.007694 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:39:01.026231 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:39:01.026204 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-8984b54b7-skmcd"] Oct 30 20:39:01.026377 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:39:01.026306 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-8984b54b7-skmcd" Oct 30 20:39:01.033323 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:39:01.033280 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Oct 30 20:39:01.033461 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:39:01.033327 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Oct 30 20:39:01.033814 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:39:01.033578 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Oct 30 20:39:01.033814 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:39:01.033674 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-bv2n9\"" Oct 30 20:39:01.040931 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:39:01.040900 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Oct 30 20:39:01.154579 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:39:01.154309 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:39:01.178919 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:39:01.178444 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/17af283e-b12c-4ed3-abf4-7a4944d1c4f2-trusted-ca\") pod \"image-registry-8984b54b7-skmcd\" (UID: \"17af283e-b12c-4ed3-abf4-7a4944d1c4f2\") " pod="openshift-image-registry/image-registry-8984b54b7-skmcd" Oct 30 20:39:01.178919 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:39:01.178487 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/17af283e-b12c-4ed3-abf4-7a4944d1c4f2-bound-sa-token\") pod \"image-registry-8984b54b7-skmcd\" (UID: \"17af283e-b12c-4ed3-abf4-7a4944d1c4f2\") " pod="openshift-image-registry/image-registry-8984b54b7-skmcd" Oct 30 20:39:01.178919 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:39:01.178545 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/17af283e-b12c-4ed3-abf4-7a4944d1c4f2-ca-trust-extracted\") pod \"image-registry-8984b54b7-skmcd\" (UID: \"17af283e-b12c-4ed3-abf4-7a4944d1c4f2\") " pod="openshift-image-registry/image-registry-8984b54b7-skmcd" Oct 30 20:39:01.178919 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:39:01.178582 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/17af283e-b12c-4ed3-abf4-7a4944d1c4f2-registry-certificates\") pod \"image-registry-8984b54b7-skmcd\" (UID: \"17af283e-b12c-4ed3-abf4-7a4944d1c4f2\") " pod="openshift-image-registry/image-registry-8984b54b7-skmcd" Oct 30 20:39:01.178919 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:39:01.178641 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/17af283e-b12c-4ed3-abf4-7a4944d1c4f2-registry-tls\") pod \"image-registry-8984b54b7-skmcd\" (UID: \"17af283e-b12c-4ed3-abf4-7a4944d1c4f2\") " pod="openshift-image-registry/image-registry-8984b54b7-skmcd" Oct 30 20:39:01.178919 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:39:01.178672 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/17af283e-b12c-4ed3-abf4-7a4944d1c4f2-installation-pull-secrets\") pod \"image-registry-8984b54b7-skmcd\" (UID: \"17af283e-b12c-4ed3-abf4-7a4944d1c4f2\") " pod="openshift-image-registry/image-registry-8984b54b7-skmcd" Oct 30 20:39:01.178919 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:39:01.178698 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cdwk9\" (UniqueName: \"kubernetes.io/projected/17af283e-b12c-4ed3-abf4-7a4944d1c4f2-kube-api-access-cdwk9\") pod \"image-registry-8984b54b7-skmcd\" (UID: \"17af283e-b12c-4ed3-abf4-7a4944d1c4f2\") " pod="openshift-image-registry/image-registry-8984b54b7-skmcd" Oct 30 20:39:01.178919 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:39:01.178737 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/17af283e-b12c-4ed3-abf4-7a4944d1c4f2-image-registry-private-configuration\") pod \"image-registry-8984b54b7-skmcd\" (UID: \"17af283e-b12c-4ed3-abf4-7a4944d1c4f2\") " pod="openshift-image-registry/image-registry-8984b54b7-skmcd" Oct 30 20:39:01.280542 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:39:01.279929 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/17af283e-b12c-4ed3-abf4-7a4944d1c4f2-ca-trust-extracted\") pod \"image-registry-8984b54b7-skmcd\" (UID: \"17af283e-b12c-4ed3-abf4-7a4944d1c4f2\") " pod="openshift-image-registry/image-registry-8984b54b7-skmcd" Oct 30 20:39:01.280542 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:39:01.279980 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/17af283e-b12c-4ed3-abf4-7a4944d1c4f2-registry-certificates\") pod \"image-registry-8984b54b7-skmcd\" (UID: \"17af283e-b12c-4ed3-abf4-7a4944d1c4f2\") " pod="openshift-image-registry/image-registry-8984b54b7-skmcd" Oct 30 20:39:01.280542 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:39:01.280023 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/17af283e-b12c-4ed3-abf4-7a4944d1c4f2-registry-tls\") pod \"image-registry-8984b54b7-skmcd\" (UID: \"17af283e-b12c-4ed3-abf4-7a4944d1c4f2\") " pod="openshift-image-registry/image-registry-8984b54b7-skmcd" Oct 30 20:39:01.280542 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:39:01.280052 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/17af283e-b12c-4ed3-abf4-7a4944d1c4f2-installation-pull-secrets\") pod \"image-registry-8984b54b7-skmcd\" (UID: \"17af283e-b12c-4ed3-abf4-7a4944d1c4f2\") " pod="openshift-image-registry/image-registry-8984b54b7-skmcd" Oct 30 20:39:01.280542 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:39:01.280076 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cdwk9\" (UniqueName: \"kubernetes.io/projected/17af283e-b12c-4ed3-abf4-7a4944d1c4f2-kube-api-access-cdwk9\") pod \"image-registry-8984b54b7-skmcd\" (UID: \"17af283e-b12c-4ed3-abf4-7a4944d1c4f2\") " pod="openshift-image-registry/image-registry-8984b54b7-skmcd" Oct 30 20:39:01.280542 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:39:01.280106 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/17af283e-b12c-4ed3-abf4-7a4944d1c4f2-image-registry-private-configuration\") pod \"image-registry-8984b54b7-skmcd\" (UID: \"17af283e-b12c-4ed3-abf4-7a4944d1c4f2\") " pod="openshift-image-registry/image-registry-8984b54b7-skmcd" Oct 30 20:39:01.280542 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:39:01.280137 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/17af283e-b12c-4ed3-abf4-7a4944d1c4f2-trusted-ca\") pod \"image-registry-8984b54b7-skmcd\" (UID: \"17af283e-b12c-4ed3-abf4-7a4944d1c4f2\") " pod="openshift-image-registry/image-registry-8984b54b7-skmcd" Oct 30 20:39:01.280542 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:39:01.280155 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/17af283e-b12c-4ed3-abf4-7a4944d1c4f2-bound-sa-token\") pod \"image-registry-8984b54b7-skmcd\" (UID: \"17af283e-b12c-4ed3-abf4-7a4944d1c4f2\") " pod="openshift-image-registry/image-registry-8984b54b7-skmcd" Oct 30 20:39:01.280921 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:39:01.280782 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/17af283e-b12c-4ed3-abf4-7a4944d1c4f2-ca-trust-extracted\") pod \"image-registry-8984b54b7-skmcd\" (UID: \"17af283e-b12c-4ed3-abf4-7a4944d1c4f2\") " pod="openshift-image-registry/image-registry-8984b54b7-skmcd" Oct 30 20:39:01.281504 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:39:01.281478 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/17af283e-b12c-4ed3-abf4-7a4944d1c4f2-registry-certificates\") pod \"image-registry-8984b54b7-skmcd\" (UID: \"17af283e-b12c-4ed3-abf4-7a4944d1c4f2\") " pod="openshift-image-registry/image-registry-8984b54b7-skmcd" Oct 30 20:39:01.282420 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:39:01.282395 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/17af283e-b12c-4ed3-abf4-7a4944d1c4f2-trusted-ca\") pod \"image-registry-8984b54b7-skmcd\" (UID: \"17af283e-b12c-4ed3-abf4-7a4944d1c4f2\") " pod="openshift-image-registry/image-registry-8984b54b7-skmcd" Oct 30 20:39:01.284488 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:39:01.284459 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/17af283e-b12c-4ed3-abf4-7a4944d1c4f2-installation-pull-secrets\") pod \"image-registry-8984b54b7-skmcd\" (UID: \"17af283e-b12c-4ed3-abf4-7a4944d1c4f2\") " pod="openshift-image-registry/image-registry-8984b54b7-skmcd" Oct 30 20:39:01.284785 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:39:01.284765 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/17af283e-b12c-4ed3-abf4-7a4944d1c4f2-image-registry-private-configuration\") pod \"image-registry-8984b54b7-skmcd\" (UID: \"17af283e-b12c-4ed3-abf4-7a4944d1c4f2\") " pod="openshift-image-registry/image-registry-8984b54b7-skmcd" Oct 30 20:39:01.284906 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:39:01.284883 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/17af283e-b12c-4ed3-abf4-7a4944d1c4f2-registry-tls\") pod \"image-registry-8984b54b7-skmcd\" (UID: \"17af283e-b12c-4ed3-abf4-7a4944d1c4f2\") " pod="openshift-image-registry/image-registry-8984b54b7-skmcd" Oct 30 20:39:01.293892 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:39:01.293836 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/17af283e-b12c-4ed3-abf4-7a4944d1c4f2-bound-sa-token\") pod \"image-registry-8984b54b7-skmcd\" (UID: \"17af283e-b12c-4ed3-abf4-7a4944d1c4f2\") " pod="openshift-image-registry/image-registry-8984b54b7-skmcd" Oct 30 20:39:01.294796 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:39:01.294751 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cdwk9\" (UniqueName: \"kubernetes.io/projected/17af283e-b12c-4ed3-abf4-7a4944d1c4f2-kube-api-access-cdwk9\") pod \"image-registry-8984b54b7-skmcd\" (UID: \"17af283e-b12c-4ed3-abf4-7a4944d1c4f2\") " pod="openshift-image-registry/image-registry-8984b54b7-skmcd" Oct 30 20:39:01.295838 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:39:01.295813 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:39:01.351348 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:39:01.351318 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-8984b54b7-skmcd" Oct 30 20:39:01.453971 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:39:01.453942 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:39:01.615591 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:39:01.615501 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:39:01.652927 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:39:01.652845 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-8984b54b7-skmcd"] Oct 30 20:39:01.658096 ip-10-0-133-192 kubenswrapper[2578]: W1030 20:39:01.658059 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17af283e_b12c_4ed3_abf4_7a4944d1c4f2.slice/crio-15fedc521c61d80e330f2664466d0af696032c8d112c6aada7dec92eb05ea09b WatchSource:0}: Error finding container 15fedc521c61d80e330f2664466d0af696032c8d112c6aada7dec92eb05ea09b: Status 404 returned error can't find the container with id 15fedc521c61d80e330f2664466d0af696032c8d112c6aada7dec92eb05ea09b Oct 30 20:39:01.785600 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:39:01.785568 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:39:01.998050 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:39:01.997868 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-pdcfk" event={"ID":"0915cfe8-4606-4f92-877b-ebe7f52f4187","Type":"ContainerStarted","Data":"ae8939ec94ee39183abf6c425f1de9095d4958e86b3fe5b2e4f6592d71ccb919"} Oct 30 20:39:01.998050 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:39:01.997935 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-pdcfk" event={"ID":"0915cfe8-4606-4f92-877b-ebe7f52f4187","Type":"ContainerStarted","Data":"e382600f4b27c0d1421f1e7b0a8ff6f8237bc3460940a7c7012ff4761f63d31b"} Oct 30 20:39:01.999935 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:39:01.999903 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-8984b54b7-skmcd" event={"ID":"17af283e-b12c-4ed3-abf4-7a4944d1c4f2","Type":"ContainerStarted","Data":"adf414d5f5c4937152b921b373dda3f6e14c8597e5339c3d34e145a8ee35edfa"} Oct 30 20:39:02.000067 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:39:01.999956 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-8984b54b7-skmcd" event={"ID":"17af283e-b12c-4ed3-abf4-7a4944d1c4f2","Type":"ContainerStarted","Data":"15fedc521c61d80e330f2664466d0af696032c8d112c6aada7dec92eb05ea09b"} Oct 30 20:39:02.000449 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:39:02.000430 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-8984b54b7-skmcd" Oct 30 20:39:02.020276 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:39:02.020227 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-pdcfk" podStartSLOduration=34.273595421 podStartE2EDuration="36.02021492s" totalImagesPullingTime="1.746619499s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:38:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:39:02.019145857 +0000 UTC m=+36.889974415" watchObservedRunningTime="2026-10-30 20:39:02.02021492 +0000 UTC m=+36.891043479" Oct 30 20:39:02.046787 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:39:02.046735 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/image-registry-8984b54b7-skmcd" podStartSLOduration=2.046718181 podStartE2EDuration="2.046718181s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:39:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:39:02.045062688 +0000 UTC m=+36.915891249" watchObservedRunningTime="2026-10-30 20:39:02.046718181 +0000 UTC m=+36.917546740" Oct 30 20:39:02.075205 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:39:02.075171 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:39:02.226241 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:39:02.226213 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:39:02.291799 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:39:02.291225 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c1a9e89f-5d46-4414-b0e0-7adff658510e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hwzc6\" (UID: \"c1a9e89f-5d46-4414-b0e0-7adff658510e\") " pod="openshift-insights/insights-runtime-extractor-hwzc6" Oct 30 20:39:02.291799 ip-10-0-133-192 kubenswrapper[2578]: E1030 20:39:02.291383 2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 20:39:02.291799 ip-10-0-133-192 kubenswrapper[2578]: E1030 20:39:02.291437 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c1a9e89f-5d46-4414-b0e0-7adff658510e-insights-runtime-extractor-tls podName:c1a9e89f-5d46-4414-b0e0-7adff658510e nodeName:}" failed. No retries permitted until 2026-10-30 20:39:10.291423572 +0000 UTC m=+45.162252109 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c1a9e89f-5d46-4414-b0e0-7adff658510e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hwzc6" (UID: "c1a9e89f-5d46-4414-b0e0-7adff658510e") : secret "insights-runtime-extractor-tls" not found Oct 30 20:39:02.827523 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:39:02.827469 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:39:02.982322 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:39:02.982288 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:39:03.147749 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:39:03.147668 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:39:03.290206 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:39:03.290171 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:39:03.430454 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:39:03.430374 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:39:03.642571 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:39:03.642533 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:39:04.008710 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:39:04.008624 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-jp2cr" event={"ID":"0dde7eae-1962-4559-89e9-0c67b1165914","Type":"ContainerStarted","Data":"11ffee4f53385d07eca6e084559fde86b1b1a37d4fc9b763e83e488eb97cdcc5"} Oct 30 20:39:04.714832 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:39:04.714791 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:39:04.857858 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:39:04.857812 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:39:05.011219 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:39:05.011154 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-jp2cr" Oct 30 20:39:05.035120 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:39:05.035071 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:39:05.263423 ip-10-0-133-192 kubenswrapper[2578]: I1030 20:39:05.263334 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:39:05.385593 ip-10-0-133-192 systemd[1]: Stopping Kubernetes Kubelet... Oct 30 20:39:05.406016 ip-10-0-133-192 systemd[1]: kubelet.service: Deactivated successfully. Oct 30 20:39:05.406262 ip-10-0-133-192 systemd[1]: Stopped Kubernetes Kubelet. Oct 30 20:39:05.406352 ip-10-0-133-192 systemd[1]: kubelet.service: Consumed 2.332s CPU time. Oct 30 20:39:05.421118 ip-10-0-133-192 systemd[1]: Starting Kubernetes Kubelet... Oct 30 20:39:05.526444 ip-10-0-133-192 kubenswrapper[4611]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 20:39:05.526444 ip-10-0-133-192 kubenswrapper[4611]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 30 20:39:05.526444 ip-10-0-133-192 kubenswrapper[4611]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 20:39:05.526444 ip-10-0-133-192 kubenswrapper[4611]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 20:39:05.528485 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.528413 4611 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 20:39:05.528485 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.528485 4611 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 20:39:05.528569 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.528488 4611 feature_gate.go:386] unrecognized feature gate: Example Oct 30 20:39:05.528569 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.528492 4611 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 20:39:05.528569 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.528495 4611 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 20:39:05.528569 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.528498 4611 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 20:39:05.528569 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.528502 4611 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 20:39:05.528569 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.528518 4611 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 20:39:05.528569 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.528521 4611 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 20:39:05.528569 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.528524 4611 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 20:39:05.528569 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.528526 4611 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 20:39:05.528569 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.528529 4611 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 20:39:05.528569 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.528532 4611 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 20:39:05.528569 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.528534 4611 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 20:39:05.528569 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.528537 4611 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 20:39:05.528569 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.528539 4611 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 20:39:05.528569 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.528546 4611 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 20:39:05.528569 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.528549 4611 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 20:39:05.528569 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.528551 4611 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 20:39:05.528569 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.528554 4611 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 20:39:05.528569 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.528557 4611 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 20:39:05.528569 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.528559 4611 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 20:39:05.529047 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.528562 4611 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 20:39:05.529047 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.528564 4611 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 20:39:05.529047 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.528567 4611 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 20:39:05.529047 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.528570 4611 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 20:39:05.529047 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.528572 4611 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 20:39:05.529047 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.528575 4611 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 20:39:05.529047 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.528578 4611 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 20:39:05.529047 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.528582 4611 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 20:39:05.529047 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.528587 4611 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 20:39:05.529047 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.528590 4611 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 20:39:05.529047 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.528593 4611 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 20:39:05.529047 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.528596 4611 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 20:39:05.529047 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.528599 4611 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 20:39:05.529047 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.528601 4611 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 20:39:05.529047 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.528604 4611 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 20:39:05.529047 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.528606 4611 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 20:39:05.529047 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.528609 4611 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 20:39:05.529047 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.528612 4611 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 20:39:05.529047 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.528616 4611 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 20:39:05.529520 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.528620 4611 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 20:39:05.529520 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.528624 4611 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 20:39:05.529520 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.528627 4611 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 20:39:05.529520 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.528632 4611 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 20:39:05.529520 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.528636 4611 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 20:39:05.529520 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.528640 4611 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 20:39:05.529520 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.528645 4611 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 20:39:05.529520 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.528649 4611 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 20:39:05.529520 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.528652 4611 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 20:39:05.529520 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.528655 4611 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 20:39:05.529520 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.528657 4611 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 20:39:05.529520 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.528660 4611 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 20:39:05.529520 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.528663 4611 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 20:39:05.529520 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.528665 4611 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 20:39:05.529520 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.528669 4611 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 20:39:05.529520 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.528672 4611 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 20:39:05.529520 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.528674 4611 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 20:39:05.529520 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.528677 4611 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 20:39:05.529520 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.528680 4611 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 20:39:05.529520 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.528683 4611 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 20:39:05.530010 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.528685 4611 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 20:39:05.530010 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.528688 4611 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 20:39:05.530010 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.528690 4611 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 20:39:05.530010 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.528693 4611 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 20:39:05.530010 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.528695 4611 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 20:39:05.530010 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.528698 4611 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 20:39:05.530010 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.528700 4611 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 20:39:05.530010 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.528703 4611 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 20:39:05.530010 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.528705 4611 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 20:39:05.530010 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.528708 4611 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 20:39:05.530010 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.528710 4611 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 20:39:05.530010 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.528714 4611 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 20:39:05.530010 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.528716 4611 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 20:39:05.530010 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.528719 4611 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 20:39:05.530010 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.528721 4611 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 20:39:05.530010 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.528725 4611 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 20:39:05.530010 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.528728 4611 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 20:39:05.530010 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.528731 4611 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 20:39:05.530010 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.528733 4611 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 20:39:05.530010 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.528736 4611 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 20:39:05.530496 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.528739 4611 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 20:39:05.530496 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.528742 4611 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 20:39:05.530496 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.528745 4611 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 20:39:05.530496 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.528747 4611 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 20:39:05.530496 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.528750 4611 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 20:39:05.530496 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.528753 4611 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 20:39:05.530496 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.528755 4611 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 20:39:05.530496 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.528758 4611 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 20:39:05.530496 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.528761 4611 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 20:39:05.530496 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.528763 4611 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 20:39:05.530496 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.528766 4611 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 20:39:05.530496 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.528768 4611 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 20:39:05.530496 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.528771 4611 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 20:39:05.530496 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.528773 4611 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 20:39:05.530496 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.528776 4611 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 20:39:05.530496 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.528779 4611 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 20:39:05.530496 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.528781 4611 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 20:39:05.530496 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.528785 4611 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 20:39:05.530496 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.528789 4611 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 20:39:05.530974 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.528792 4611 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 20:39:05.530974 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.528795 4611 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 20:39:05.530974 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.528798 4611 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 20:39:05.530974 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.528801 4611 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 20:39:05.530974 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.528804 4611 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 20:39:05.530974 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.528806 4611 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 20:39:05.530974 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.529561 4611 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 20:39:05.530974 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.529569 4611 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 20:39:05.530974 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.529572 4611 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 20:39:05.530974 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.529575 4611 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 20:39:05.530974 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.529579 4611 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 20:39:05.530974 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.529583 4611 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 20:39:05.530974 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.529587 4611 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 20:39:05.530974 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.529591 4611 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 20:39:05.530974 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.529594 4611 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 20:39:05.530974 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.529597 4611 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 20:39:05.530974 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.529599 4611 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 20:39:05.530974 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.529602 4611 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 20:39:05.530974 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.529605 4611 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 20:39:05.531448 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.529607 4611 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 20:39:05.531448 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.529610 4611 feature_gate.go:386] unrecognized feature gate: Example Oct 30 20:39:05.531448 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.529613 4611 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 20:39:05.531448 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.529615 4611 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 20:39:05.531448 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.529618 4611 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 20:39:05.531448 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.529620 4611 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 20:39:05.531448 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.529623 4611 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 20:39:05.531448 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.529626 4611 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 20:39:05.531448 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.529628 4611 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 20:39:05.531448 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.529631 4611 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 20:39:05.531448 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.529633 4611 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 20:39:05.531448 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.529636 4611 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 20:39:05.531448 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.529638 4611 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 20:39:05.531448 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.529641 4611 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 20:39:05.531448 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.529644 4611 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 20:39:05.531448 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.529646 4611 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 20:39:05.531448 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.529650 4611 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 20:39:05.531448 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.529654 4611 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 20:39:05.531448 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.529657 4611 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 20:39:05.531448 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.529659 4611 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 20:39:05.532050 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.529662 4611 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 20:39:05.532050 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.529665 4611 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 20:39:05.532050 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.529667 4611 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 20:39:05.532050 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.529669 4611 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 20:39:05.532050 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.529672 4611 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 20:39:05.532050 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.529675 4611 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 20:39:05.532050 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.529678 4611 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 20:39:05.532050 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.529682 4611 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 20:39:05.532050 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.529684 4611 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 20:39:05.532050 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.529687 4611 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 20:39:05.532050 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.529690 4611 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 20:39:05.532050 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.529693 4611 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 20:39:05.532050 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.529695 4611 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 20:39:05.532050 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.529698 4611 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 20:39:05.532050 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.529701 4611 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 20:39:05.532050 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.529703 4611 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 20:39:05.532050 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.529706 4611 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 20:39:05.532050 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.529708 4611 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 20:39:05.532050 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.529711 4611 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 20:39:05.532050 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.529713 4611 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 20:39:05.532567 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.529716 4611 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 20:39:05.532567 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.529719 4611 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 20:39:05.532567 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.529721 4611 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 20:39:05.532567 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.529724 4611 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 20:39:05.532567 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.529727 4611 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 20:39:05.532567 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.529729 4611 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 20:39:05.532567 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.529732 4611 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 20:39:05.532567 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.529734 4611 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 20:39:05.532567 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.529737 4611 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 20:39:05.532567 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.529739 4611 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 20:39:05.532567 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.529742 4611 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 20:39:05.532567 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.529744 4611 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 20:39:05.532567 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.529747 4611 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 20:39:05.532567 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.529749 4611 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 20:39:05.532567 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.529756 4611 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 20:39:05.532567 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.529759 4611 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 20:39:05.532567 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.529762 4611 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 20:39:05.532567 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.529764 4611 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 20:39:05.532567 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.529767 4611 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 20:39:05.532567 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.529769 4611 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 20:39:05.533054 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.529772 4611 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 20:39:05.533054 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.529775 4611 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 20:39:05.533054 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.529778 4611 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 20:39:05.533054 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.529781 4611 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 20:39:05.533054 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.529783 4611 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 20:39:05.533054 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.529786 4611 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 20:39:05.533054 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.529788 4611 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 20:39:05.533054 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.529791 4611 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 20:39:05.533054 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.529793 4611 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 20:39:05.533054 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.529796 4611 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 20:39:05.533054 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.529800 4611 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 20:39:05.533054 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.529802 4611 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 20:39:05.533054 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.529805 4611 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 20:39:05.533054 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.529808 4611 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 20:39:05.533054 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.529810 4611 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 20:39:05.533054 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.529812 4611 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 20:39:05.533054 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.529815 4611 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 20:39:05.533054 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.529817 4611 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 20:39:05.533054 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.529820 4611 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 20:39:05.533524 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.529822 4611 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 20:39:05.533524 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.529825 4611 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 20:39:05.533524 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.529827 4611 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 20:39:05.533524 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.529830 4611 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 20:39:05.533524 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.529832 4611 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 20:39:05.533524 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.529835 4611 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 20:39:05.533524 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.529837 4611 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 20:39:05.533524 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.529841 4611 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 20:39:05.533524 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.529844 4611 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 20:39:05.533524 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.529847 4611 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 20:39:05.533524 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.529849 4611 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 20:39:05.533524 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.529852 4611 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 20:39:05.533524 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.529855 4611 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 20:39:05.533524 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.529858 4611 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 20:39:05.533524 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.530358 4611 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 20:39:05.533524 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.530364 4611 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 20:39:05.533524 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.530368 4611 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 20:39:05.533524 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.530371 4611 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 20:39:05.533524 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.530375 4611 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 20:39:05.533524 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.530377 4611 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 20:39:05.534117 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.530380 4611 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 20:39:05.534117 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.530383 4611 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 20:39:05.534117 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.530386 4611 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 20:39:05.534117 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.530388 4611 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 20:39:05.534117 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.530391 4611 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 20:39:05.534117 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.530394 4611 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 20:39:05.534117 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.530396 4611 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 20:39:05.534117 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.530399 4611 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 20:39:05.534117 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.530401 4611 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 20:39:05.534117 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.530404 4611 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 20:39:05.534117 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.530407 4611 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 20:39:05.534117 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.530409 4611 feature_gate.go:386] unrecognized feature gate: Example Oct 30 20:39:05.534117 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.530412 4611 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 20:39:05.534117 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.530415 4611 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 20:39:05.534117 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.530417 4611 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 20:39:05.534117 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.530420 4611 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 20:39:05.534117 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.530423 4611 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 20:39:05.534117 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.530425 4611 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 20:39:05.534117 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.530428 4611 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 20:39:05.534117 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.530433 4611 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 20:39:05.534664 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.530435 4611 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 20:39:05.534664 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.530438 4611 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 20:39:05.534664 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.530441 4611 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 20:39:05.534664 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.530444 4611 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 20:39:05.534664 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.530447 4611 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 20:39:05.534664 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.530449 4611 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 20:39:05.534664 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.530452 4611 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 20:39:05.534664 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.530455 4611 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 20:39:05.534664 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.530457 4611 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 20:39:05.534664 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.530460 4611 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 20:39:05.534664 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.530463 4611 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 20:39:05.534664 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.530465 4611 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 20:39:05.534664 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.530468 4611 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 20:39:05.534664 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.530471 4611 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 20:39:05.534664 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.530473 4611 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 20:39:05.534664 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.530476 4611 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 20:39:05.534664 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.530478 4611 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 20:39:05.534664 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.530481 4611 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 20:39:05.534664 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.530483 4611 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 20:39:05.534664 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.530486 4611 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 20:39:05.535152 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.530488 4611 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 20:39:05.535152 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.530491 4611 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 20:39:05.535152 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.530494 4611 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 20:39:05.535152 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.530496 4611 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 20:39:05.535152 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.530499 4611 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 20:39:05.535152 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.530501 4611 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 20:39:05.535152 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.530518 4611 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 20:39:05.535152 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.530521 4611 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 20:39:05.535152 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.530524 4611 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 20:39:05.535152 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.530526 4611 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 20:39:05.535152 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.530529 4611 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 20:39:05.535152 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.530533 4611 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 20:39:05.535152 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.530536 4611 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 20:39:05.535152 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.530538 4611 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 20:39:05.535152 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.530542 4611 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 20:39:05.535152 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.530546 4611 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 20:39:05.535152 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.530550 4611 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 20:39:05.535152 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.530555 4611 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 20:39:05.535152 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.530559 4611 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 20:39:05.535152 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.530565 4611 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 20:39:05.535686 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.530571 4611 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 20:39:05.535686 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.530576 4611 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 20:39:05.535686 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.530580 4611 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 20:39:05.535686 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.530583 4611 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 20:39:05.535686 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.530585 4611 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 20:39:05.535686 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.530588 4611 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 20:39:05.535686 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.530591 4611 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 20:39:05.535686 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.530595 4611 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 20:39:05.535686 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.530598 4611 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 20:39:05.535686 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.530601 4611 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 20:39:05.535686 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.530604 4611 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 20:39:05.535686 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.530607 4611 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 20:39:05.535686 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.530610 4611 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 20:39:05.535686 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.530612 4611 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 20:39:05.535686 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.530615 4611 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 20:39:05.535686 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.530618 4611 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 20:39:05.535686 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.530620 4611 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 20:39:05.535686 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.530623 4611 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 20:39:05.535686 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.530625 4611 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 20:39:05.535686 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.530628 4611 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 20:39:05.536173 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.530630 4611 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 20:39:05.536173 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.530632 4611 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 20:39:05.536173 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.530635 4611 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 20:39:05.536173 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.530639 4611 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 20:39:05.536173 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.530641 4611 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 20:39:05.536173 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.530644 4611 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 20:39:05.536173 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.530646 4611 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 20:39:05.536173 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.530649 4611 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 20:39:05.536173 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.530651 4611 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 20:39:05.536173 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.530654 4611 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 20:39:05.536173 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.530656 4611 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 20:39:05.536173 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.530659 4611 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 20:39:05.536173 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.530662 4611 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 20:39:05.536173 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.530665 4611 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 20:39:05.536173 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.530668 4611 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 20:39:05.536173 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.530671 4611 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 20:39:05.536173 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.530674 4611 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 20:39:05.536173 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.530677 4611 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 20:39:05.536173 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.530680 4611 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 20:39:05.536659 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.530682 4611 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 20:39:05.536659 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.530774 4611 flags.go:64] FLAG: --address="0.0.0.0" Oct 30 20:39:05.536659 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.530781 4611 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 30 20:39:05.536659 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.530788 4611 flags.go:64] FLAG: --anonymous-auth="true" Oct 30 20:39:05.536659 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.530793 4611 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 30 20:39:05.536659 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.530797 4611 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 30 20:39:05.536659 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.530800 4611 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 30 20:39:05.536659 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.530804 4611 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 30 20:39:05.536659 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.530808 4611 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 30 20:39:05.536659 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.530812 4611 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 30 20:39:05.536659 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.530815 4611 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 30 20:39:05.536659 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.530818 4611 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 30 20:39:05.536659 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.530822 4611 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 30 20:39:05.536659 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.530825 4611 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 30 20:39:05.536659 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.530828 4611 flags.go:64] FLAG: --cgroup-root="" Oct 30 20:39:05.536659 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.530830 4611 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 30 20:39:05.536659 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.530835 4611 flags.go:64] FLAG: --client-ca-file="" Oct 30 20:39:05.536659 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.530838 4611 flags.go:64] FLAG: --cloud-provider="external" Oct 30 20:39:05.536659 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.530841 4611 flags.go:64] FLAG: --cluster-dns="[]" Oct 30 20:39:05.536659 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.530845 4611 flags.go:64] FLAG: --cluster-domain="" Oct 30 20:39:05.536659 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.530848 4611 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 30 20:39:05.536659 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.530851 4611 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 30 20:39:05.536659 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.530854 4611 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 30 20:39:05.536659 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.530858 4611 flags.go:64] FLAG: --container-log-max-files="5" Oct 30 20:39:05.537255 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.530861 4611 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 30 20:39:05.537255 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.530864 4611 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 30 20:39:05.537255 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.530868 4611 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 30 20:39:05.537255 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.530872 4611 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 30 20:39:05.537255 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.530876 4611 flags.go:64] FLAG: --contention-profiling="false" Oct 30 20:39:05.537255 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.530879 4611 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 30 20:39:05.537255 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.530882 4611 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 30 20:39:05.537255 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.530886 4611 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 30 20:39:05.537255 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.530888 4611 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 30 20:39:05.537255 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.530893 4611 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 30 20:39:05.537255 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.530896 4611 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 30 20:39:05.537255 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.530899 4611 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 30 20:39:05.537255 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.530902 4611 flags.go:64] FLAG: --enable-load-reader="false" Oct 30 20:39:05.537255 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.530905 4611 flags.go:64] FLAG: --enable-server="true" Oct 30 20:39:05.537255 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.530908 4611 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 30 20:39:05.537255 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.530912 4611 flags.go:64] FLAG: --event-burst="100" Oct 30 20:39:05.537255 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.530916 4611 flags.go:64] FLAG: --event-qps="50" Oct 30 20:39:05.537255 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.530919 4611 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 30 20:39:05.537255 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.530922 4611 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 30 20:39:05.537255 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.530925 4611 flags.go:64] FLAG: --eviction-hard="" Oct 30 20:39:05.537255 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.530929 4611 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 30 20:39:05.537255 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.530932 4611 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 30 20:39:05.537255 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.530935 4611 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 30 20:39:05.537255 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.530940 4611 flags.go:64] FLAG: --eviction-soft="" Oct 30 20:39:05.537255 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.530947 4611 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 30 20:39:05.537940 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.530952 4611 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 30 20:39:05.537940 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.530956 4611 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 30 20:39:05.537940 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.530960 4611 flags.go:64] FLAG: --experimental-mounter-path="" Oct 30 20:39:05.537940 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.530965 4611 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 30 20:39:05.537940 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.530970 4611 flags.go:64] FLAG: --fail-swap-on="true" Oct 30 20:39:05.537940 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.530974 4611 flags.go:64] FLAG: --feature-gates="" Oct 30 20:39:05.537940 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.530980 4611 flags.go:64] FLAG: --file-check-frequency="20s" Oct 30 20:39:05.537940 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.530985 4611 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 30 20:39:05.537940 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.530990 4611 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 30 20:39:05.537940 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.530995 4611 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 30 20:39:05.537940 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.531000 4611 flags.go:64] FLAG: --healthz-port="10248" Oct 30 20:39:05.537940 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.531006 4611 flags.go:64] FLAG: --help="false" Oct 30 20:39:05.537940 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.531010 4611 flags.go:64] FLAG: --hostname-override="ip-10-0-133-192.us-east-2.compute.internal" Oct 30 20:39:05.537940 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.531015 4611 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 30 20:39:05.537940 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.531020 4611 flags.go:64] FLAG: --http-check-frequency="20s" Oct 30 20:39:05.537940 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.531024 4611 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 30 20:39:05.537940 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.531029 4611 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 30 20:39:05.537940 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.531035 4611 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 30 20:39:05.537940 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.531040 4611 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 30 20:39:05.537940 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.531044 4611 flags.go:64] FLAG: --image-service-endpoint="" Oct 30 20:39:05.537940 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.531049 4611 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 30 20:39:05.537940 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.531053 4611 flags.go:64] FLAG: --kube-api-burst="100" Oct 30 20:39:05.537940 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.531057 4611 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 30 20:39:05.538500 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.531060 4611 flags.go:64] FLAG: --kube-api-qps="50" Oct 30 20:39:05.538500 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.531063 4611 flags.go:64] FLAG: --kube-reserved="" Oct 30 20:39:05.538500 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.531066 4611 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 30 20:39:05.538500 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.531069 4611 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 30 20:39:05.538500 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.531072 4611 flags.go:64] FLAG: --kubelet-cgroups="" Oct 30 20:39:05.538500 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.531075 4611 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 30 20:39:05.538500 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.531078 4611 flags.go:64] FLAG: --lock-file="" Oct 30 20:39:05.538500 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.531081 4611 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 30 20:39:05.538500 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.531084 4611 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 30 20:39:05.538500 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.531088 4611 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 30 20:39:05.538500 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.531097 4611 flags.go:64] FLAG: --log-json-split-stream="false" Oct 30 20:39:05.538500 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.531101 4611 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 30 20:39:05.538500 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.531106 4611 flags.go:64] FLAG: --log-text-split-stream="false" Oct 30 20:39:05.538500 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.531110 4611 flags.go:64] FLAG: --logging-format="text" Oct 30 20:39:05.538500 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.531115 4611 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 30 20:39:05.538500 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.531120 4611 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 30 20:39:05.538500 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.531124 4611 flags.go:64] FLAG: --manifest-url="" Oct 30 20:39:05.538500 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.531129 4611 flags.go:64] FLAG: --manifest-url-header="" Oct 30 20:39:05.538500 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.531136 4611 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 30 20:39:05.538500 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.531141 4611 flags.go:64] FLAG: --max-open-files="1000000" Oct 30 20:39:05.538500 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.531149 4611 flags.go:64] FLAG: --max-pods="110" Oct 30 20:39:05.538500 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.531154 4611 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 30 20:39:05.538500 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.531158 4611 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 30 20:39:05.538500 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.531163 4611 flags.go:64] FLAG: --memory-manager-policy="None" Oct 30 20:39:05.538500 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.531167 4611 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 30 20:39:05.538500 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.531172 4611 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 30 20:39:05.539172 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.531176 4611 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 30 20:39:05.539172 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.531180 4611 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 30 20:39:05.539172 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.531191 4611 flags.go:64] FLAG: --node-status-max-images="50" Oct 30 20:39:05.539172 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.531195 4611 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 30 20:39:05.539172 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.531200 4611 flags.go:64] FLAG: --oom-score-adj="-999" Oct 30 20:39:05.539172 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.531204 4611 flags.go:64] FLAG: --pod-cidr="" Oct 30 20:39:05.539172 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.531209 4611 flags.go:64] FLAG: --pod-manifest-path="" Oct 30 20:39:05.539172 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.531213 4611 flags.go:64] FLAG: --pod-max-pids="-1" Oct 30 20:39:05.539172 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.531217 4611 flags.go:64] FLAG: --pods-per-core="0" Oct 30 20:39:05.539172 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.531222 4611 flags.go:64] FLAG: --port="10250" Oct 30 20:39:05.539172 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.531226 4611 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 30 20:39:05.539172 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.531231 4611 flags.go:64] FLAG: --provider-id="aws:///us-east-2a/i-01b1d486861e78080" Oct 30 20:39:05.539172 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.531236 4611 flags.go:64] FLAG: --qos-reserved="" Oct 30 20:39:05.539172 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.531240 4611 flags.go:64] FLAG: --read-only-port="10255" Oct 30 20:39:05.539172 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.531245 4611 flags.go:64] FLAG: --register-node="true" Oct 30 20:39:05.539172 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.531249 4611 flags.go:64] FLAG: --register-with-taints="" Oct 30 20:39:05.539172 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.531255 4611 flags.go:64] FLAG: --registry-burst="10" Oct 30 20:39:05.539172 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.531259 4611 flags.go:64] FLAG: --registry-qps="5" Oct 30 20:39:05.539172 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.531263 4611 flags.go:64] FLAG: --reserved-cpus="" Oct 30 20:39:05.539172 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.531267 4611 flags.go:64] FLAG: --reserved-memory="" Oct 30 20:39:05.539172 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.531272 4611 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 30 20:39:05.539172 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.531277 4611 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 30 20:39:05.539172 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.531282 4611 flags.go:64] FLAG: --rotate-certificates="false" Oct 30 20:39:05.539172 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.531286 4611 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 30 20:39:05.539172 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.531290 4611 flags.go:64] FLAG: --runonce="false" Oct 30 20:39:05.539172 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.531295 4611 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 30 20:39:05.539825 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.531304 4611 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 30 20:39:05.539825 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.531310 4611 flags.go:64] FLAG: --seccomp-default="false" Oct 30 20:39:05.539825 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.531315 4611 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 30 20:39:05.539825 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.531319 4611 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 30 20:39:05.539825 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.531324 4611 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 30 20:39:05.539825 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.531329 4611 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 30 20:39:05.539825 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.531333 4611 flags.go:64] FLAG: --storage-driver-password="root" Oct 30 20:39:05.539825 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.531338 4611 flags.go:64] FLAG: --storage-driver-secure="false" Oct 30 20:39:05.539825 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.531342 4611 flags.go:64] FLAG: --storage-driver-table="stats" Oct 30 20:39:05.539825 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.531346 4611 flags.go:64] FLAG: --storage-driver-user="root" Oct 30 20:39:05.539825 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.531350 4611 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 30 20:39:05.539825 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.531355 4611 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 30 20:39:05.539825 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.531359 4611 flags.go:64] FLAG: --system-cgroups="" Oct 30 20:39:05.539825 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.531363 4611 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 30 20:39:05.539825 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.531370 4611 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 30 20:39:05.539825 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.531375 4611 flags.go:64] FLAG: --tls-cert-file="" Oct 30 20:39:05.539825 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.531379 4611 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 30 20:39:05.539825 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.531385 4611 flags.go:64] FLAG: --tls-min-version="" Oct 30 20:39:05.539825 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.531389 4611 flags.go:64] FLAG: --tls-private-key-file="" Oct 30 20:39:05.539825 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.531393 4611 flags.go:64] FLAG: --topology-manager-policy="none" Oct 30 20:39:05.539825 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.531397 4611 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 30 20:39:05.539825 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.531402 4611 flags.go:64] FLAG: --topology-manager-scope="container" Oct 30 20:39:05.539825 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.531407 4611 flags.go:64] FLAG: --v="2" Oct 30 20:39:05.539825 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.531413 4611 flags.go:64] FLAG: --version="false" Oct 30 20:39:05.539825 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.531418 4611 flags.go:64] FLAG: --vmodule="" Oct 30 20:39:05.540417 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.531423 4611 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 30 20:39:05.540417 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.531428 4611 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 30 20:39:05.540417 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.531603 4611 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 20:39:05.540417 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.531608 4611 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 20:39:05.540417 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.531617 4611 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 20:39:05.540417 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.531621 4611 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 20:39:05.540417 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.531624 4611 feature_gate.go:386] unrecognized feature gate: Example Oct 30 20:39:05.540417 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.531627 4611 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 20:39:05.540417 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.531630 4611 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 20:39:05.540417 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.531635 4611 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 20:39:05.540417 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.531637 4611 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 20:39:05.540417 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.531640 4611 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 20:39:05.540417 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.531643 4611 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 20:39:05.540417 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.531645 4611 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 20:39:05.540417 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.531648 4611 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 20:39:05.540417 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.531650 4611 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 20:39:05.540417 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.531653 4611 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 20:39:05.540417 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.531655 4611 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 20:39:05.540417 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.531658 4611 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 20:39:05.540417 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.531660 4611 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 20:39:05.540901 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.531663 4611 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 20:39:05.540901 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.531665 4611 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 20:39:05.540901 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.531668 4611 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 20:39:05.540901 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.531671 4611 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 20:39:05.540901 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.531674 4611 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 20:39:05.540901 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.531676 4611 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 20:39:05.540901 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.531679 4611 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 20:39:05.540901 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.531681 4611 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 20:39:05.540901 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.531684 4611 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 20:39:05.540901 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.531687 4611 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 20:39:05.540901 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.531689 4611 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 20:39:05.540901 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.531692 4611 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 20:39:05.540901 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.531694 4611 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 20:39:05.540901 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.531696 4611 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 20:39:05.540901 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.531699 4611 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 20:39:05.540901 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.531702 4611 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 20:39:05.540901 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.531704 4611 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 20:39:05.540901 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.531706 4611 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 20:39:05.540901 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.531709 4611 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 20:39:05.540901 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.531712 4611 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 20:39:05.541639 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.531715 4611 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 20:39:05.541639 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.531717 4611 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 20:39:05.541639 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.531724 4611 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 20:39:05.541639 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.531728 4611 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 20:39:05.541639 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.531731 4611 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 20:39:05.541639 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.531734 4611 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 20:39:05.541639 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.531736 4611 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 20:39:05.541639 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.531739 4611 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 20:39:05.541639 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.531741 4611 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 20:39:05.541639 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.531744 4611 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 20:39:05.541639 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.531746 4611 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 20:39:05.541639 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.531749 4611 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 20:39:05.541639 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.531753 4611 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 20:39:05.541639 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.531756 4611 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 20:39:05.541639 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.531758 4611 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 20:39:05.541639 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.531761 4611 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 20:39:05.541639 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.531763 4611 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 20:39:05.541639 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.531766 4611 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 20:39:05.541639 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.531769 4611 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 20:39:05.541639 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.531771 4611 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 20:39:05.542397 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.531774 4611 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 20:39:05.542397 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.531777 4611 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 20:39:05.542397 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.531780 4611 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 20:39:05.542397 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.531782 4611 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 20:39:05.542397 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.531785 4611 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 20:39:05.542397 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.531787 4611 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 20:39:05.542397 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.531790 4611 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 20:39:05.542397 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.531792 4611 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 20:39:05.542397 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.531794 4611 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 20:39:05.542397 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.531797 4611 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 20:39:05.542397 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.531800 4611 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 20:39:05.542397 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.531803 4611 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 20:39:05.542397 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.531805 4611 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 20:39:05.542397 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.531808 4611 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 20:39:05.542397 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.531810 4611 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 20:39:05.542397 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.531815 4611 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 20:39:05.542397 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.531820 4611 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 20:39:05.542397 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.531824 4611 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 20:39:05.542930 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.531828 4611 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 20:39:05.542930 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.531831 4611 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 20:39:05.542930 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.531834 4611 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 20:39:05.542930 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.531836 4611 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 20:39:05.542930 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.531839 4611 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 20:39:05.542930 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.531841 4611 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 20:39:05.542930 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.531845 4611 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 20:39:05.542930 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.531848 4611 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 20:39:05.542930 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.531851 4611 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 20:39:05.542930 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.531853 4611 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 20:39:05.542930 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.531856 4611 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 20:39:05.542930 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.531858 4611 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 20:39:05.542930 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.531861 4611 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 20:39:05.542930 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.531863 4611 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 20:39:05.542930 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.531866 4611 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 20:39:05.542930 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.531869 4611 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 20:39:05.542930 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.531871 4611 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 20:39:05.542930 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.531874 4611 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 20:39:05.542930 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.531877 4611 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 20:39:05.542930 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.531880 4611 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 20:39:05.543624 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.531882 4611 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 20:39:05.543624 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.531885 4611 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 20:39:05.543624 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.531888 4611 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 20:39:05.543624 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.531890 4611 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 20:39:05.543624 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.531893 4611 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 20:39:05.543624 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.531896 4611 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 20:39:05.543624 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.531898 4611 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 20:39:05.543624 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.531901 4611 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 20:39:05.543624 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.531903 4611 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 20:39:05.543624 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.531906 4611 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 20:39:05.543624 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.531934 4611 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 20:39:05.543624 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.531970 4611 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 20:39:05.543624 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.531975 4611 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 20:39:05.543624 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.531978 4611 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 20:39:05.543624 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.531981 4611 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 20:39:05.543624 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.531984 4611 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 20:39:05.544071 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.531987 4611 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 20:39:05.544071 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.531990 4611 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 20:39:05.544071 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.531993 4611 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 20:39:05.544071 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.531996 4611 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 20:39:05.544071 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.531999 4611 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 20:39:05.544071 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.532002 4611 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 20:39:05.544071 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.532005 4611 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 20:39:05.544071 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.532008 4611 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 20:39:05.544071 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.532010 4611 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 20:39:05.544071 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.532013 4611 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 20:39:05.544071 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.532015 4611 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 20:39:05.544071 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.532018 4611 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 20:39:05.544071 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.532021 4611 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 20:39:05.544071 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.532023 4611 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 20:39:05.544071 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.532026 4611 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 20:39:05.544071 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.532029 4611 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 20:39:05.544071 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.532031 4611 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 20:39:05.544071 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.532034 4611 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 20:39:05.544071 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.532036 4611 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 20:39:05.544071 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.532039 4611 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 20:39:05.544644 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.532041 4611 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 20:39:05.544644 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.532044 4611 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 20:39:05.544644 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.532046 4611 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 20:39:05.544644 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.532049 4611 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 20:39:05.544644 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.532052 4611 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 20:39:05.544644 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.532055 4611 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 20:39:05.544644 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.532057 4611 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 20:39:05.544644 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.532060 4611 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 20:39:05.544644 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.532064 4611 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 20:39:05.544644 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.532069 4611 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 20:39:05.544644 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.532072 4611 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 20:39:05.544644 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.532076 4611 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 20:39:05.544644 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.532079 4611 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 20:39:05.544644 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.532082 4611 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 20:39:05.544644 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.532085 4611 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 20:39:05.544644 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.532089 4611 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 20:39:05.544644 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.532092 4611 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 20:39:05.544644 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.532095 4611 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 20:39:05.545234 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.532098 4611 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 20:39:05.545234 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.532101 4611 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 20:39:05.545234 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.532104 4611 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 20:39:05.545234 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.532106 4611 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 20:39:05.545234 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.532109 4611 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 20:39:05.545234 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.532112 4611 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 20:39:05.545234 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.532114 4611 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 20:39:05.545234 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.532117 4611 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 20:39:05.545234 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.532119 4611 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 20:39:05.545234 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.532122 4611 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 20:39:05.545234 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.532125 4611 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 20:39:05.545234 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.532128 4611 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 20:39:05.545234 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.532132 4611 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 20:39:05.545234 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.532134 4611 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 20:39:05.545234 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.532137 4611 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 20:39:05.545234 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.532139 4611 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 20:39:05.545234 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.532142 4611 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 20:39:05.545234 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.532144 4611 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 20:39:05.545234 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.532147 4611 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 20:39:05.545234 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.532150 4611 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 20:39:05.545997 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.532152 4611 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 20:39:05.545997 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.532155 4611 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 20:39:05.545997 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.532158 4611 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 20:39:05.545997 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.532160 4611 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 20:39:05.545997 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.532163 4611 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 20:39:05.545997 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.532166 4611 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 20:39:05.545997 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.532168 4611 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 20:39:05.545997 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.532171 4611 feature_gate.go:386] unrecognized feature gate: Example Oct 30 20:39:05.545997 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.532174 4611 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 20:39:05.545997 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.532177 4611 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 20:39:05.545997 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.532180 4611 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 20:39:05.545997 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.532183 4611 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 20:39:05.545997 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.532186 4611 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 20:39:05.545997 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.532189 4611 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 20:39:05.545997 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.532192 4611 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 20:39:05.545997 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.532194 4611 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 20:39:05.545997 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.532198 4611 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 20:39:05.545997 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.532202 4611 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 20:39:05.545997 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.532206 4611 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 20:39:05.545997 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.532210 4611 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 20:39:05.546553 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.532213 4611 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 20:39:05.546553 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.532217 4611 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 20:39:05.546553 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.532221 4611 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 20:39:05.546553 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.532226 4611 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 20:39:05.546553 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.532230 4611 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 20:39:05.546553 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.532234 4611 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 20:39:05.546553 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.532237 4611 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 20:39:05.546553 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.532239 4611 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 20:39:05.546553 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.532242 4611 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 20:39:05.546553 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.532245 4611 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 20:39:05.546553 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.532248 4611 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 20:39:05.546553 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.532251 4611 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 20:39:05.546553 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.532253 4611 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 20:39:05.546553 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.532256 4611 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 20:39:05.546553 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.532259 4611 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 20:39:05.546553 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.532262 4611 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 20:39:05.546553 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.532265 4611 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 20:39:05.546553 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.532267 4611 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 20:39:05.546553 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.532270 4611 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 20:39:05.546553 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.532272 4611 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 20:39:05.547039 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.532276 4611 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 20:39:05.547039 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.532280 4611 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 20:39:05.547039 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.532283 4611 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 20:39:05.547039 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.536134 4611 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 30 20:39:05.547039 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.536150 4611 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 30 20:39:05.547039 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.536166 4611 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 30 20:39:05.547039 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.536172 4611 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 30 20:39:05.547039 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.536236 4611 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 20:39:05.547039 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.536241 4611 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 20:39:05.547039 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.536244 4611 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 20:39:05.547039 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.536247 4611 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 20:39:05.547039 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.536250 4611 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 20:39:05.547039 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.536253 4611 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 20:39:05.547039 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.536256 4611 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 20:39:05.547039 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.536259 4611 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 20:39:05.547039 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.536262 4611 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 20:39:05.547039 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.536264 4611 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 20:39:05.547039 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.536267 4611 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 20:39:05.547039 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.536270 4611 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 20:39:05.547492 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.536272 4611 feature_gate.go:386] unrecognized feature gate: Example Oct 30 20:39:05.547492 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.536275 4611 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 20:39:05.547492 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.536278 4611 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 20:39:05.547492 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.536281 4611 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 20:39:05.547492 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.536283 4611 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 20:39:05.547492 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.536286 4611 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 20:39:05.547492 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.536288 4611 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 20:39:05.547492 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.536291 4611 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 20:39:05.547492 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.536294 4611 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 20:39:05.547492 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.536297 4611 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 20:39:05.547492 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.536300 4611 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 20:39:05.547492 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.536302 4611 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 20:39:05.547492 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.536305 4611 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 20:39:05.547492 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.536308 4611 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 20:39:05.547492 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.536310 4611 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 20:39:05.547492 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.536313 4611 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 20:39:05.547492 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.536316 4611 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 20:39:05.547492 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.536319 4611 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 20:39:05.547492 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.536322 4611 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 20:39:05.547492 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.536325 4611 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 20:39:05.548033 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.536327 4611 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 20:39:05.548033 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.536330 4611 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 20:39:05.548033 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.536332 4611 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 20:39:05.548033 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.536335 4611 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 20:39:05.548033 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.536338 4611 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 20:39:05.548033 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.536340 4611 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 20:39:05.548033 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.536343 4611 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 20:39:05.548033 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.536345 4611 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 20:39:05.548033 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.536348 4611 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 20:39:05.548033 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.536350 4611 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 20:39:05.548033 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.536354 4611 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 20:39:05.548033 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.536358 4611 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 20:39:05.548033 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.536361 4611 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 20:39:05.548033 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.536364 4611 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 20:39:05.548033 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.536367 4611 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 20:39:05.548033 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.536370 4611 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 20:39:05.548033 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.536373 4611 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 20:39:05.548033 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.536375 4611 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 20:39:05.548033 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.536378 4611 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 20:39:05.548730 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.536381 4611 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 20:39:05.548730 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.536383 4611 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 20:39:05.548730 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.536386 4611 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 20:39:05.548730 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.536388 4611 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 20:39:05.548730 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.536391 4611 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 20:39:05.548730 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.536394 4611 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 20:39:05.548730 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.536397 4611 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 20:39:05.548730 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.536399 4611 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 20:39:05.548730 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.536402 4611 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 20:39:05.548730 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.536404 4611 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 20:39:05.548730 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.536408 4611 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 20:39:05.548730 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.536413 4611 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 20:39:05.548730 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.536417 4611 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 20:39:05.548730 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.536421 4611 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 20:39:05.548730 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.536424 4611 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 20:39:05.548730 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.536426 4611 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 20:39:05.548730 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.536429 4611 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 20:39:05.548730 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.536432 4611 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 20:39:05.548730 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.536434 4611 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 20:39:05.548730 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.536437 4611 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 20:39:05.549361 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.536439 4611 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 20:39:05.549361 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.536442 4611 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 20:39:05.549361 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.536445 4611 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 20:39:05.549361 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.536447 4611 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 20:39:05.549361 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.536450 4611 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 20:39:05.549361 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.536452 4611 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 20:39:05.549361 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.536455 4611 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 20:39:05.549361 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.536457 4611 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 20:39:05.549361 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.536460 4611 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 20:39:05.549361 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.536462 4611 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 20:39:05.549361 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.536465 4611 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 20:39:05.549361 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.536467 4611 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 20:39:05.549361 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.536472 4611 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 20:39:05.549361 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.536476 4611 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 20:39:05.549361 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.536479 4611 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 20:39:05.549361 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.536482 4611 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 20:39:05.549361 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.536485 4611 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 20:39:05.549361 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.536489 4611 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 20:39:05.549361 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.536492 4611 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 20:39:05.549361 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.536494 4611 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 20:39:05.549987 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.536497 4611 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 20:39:05.549987 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.536500 4611 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 20:39:05.549987 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.536502 4611 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 20:39:05.549987 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.536521 4611 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 20:39:05.549987 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.536526 4611 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 20:39:05.549987 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.536528 4611 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 20:39:05.549987 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.536531 4611 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 20:39:05.549987 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.536534 4611 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 20:39:05.549987 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.536537 4611 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 20:39:05.549987 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.536539 4611 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 20:39:05.549987 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.536542 4611 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 20:39:05.549987 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.536545 4611 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 20:39:05.549987 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.536547 4611 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 20:39:05.549987 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.536550 4611 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 20:39:05.549987 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.536553 4611 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 20:39:05.549987 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.536579 4611 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 20:39:05.550382 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.536771 4611 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 20:39:05.550382 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.536777 4611 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 20:39:05.550382 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.536781 4611 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 20:39:05.550382 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.536784 4611 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 20:39:05.550382 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.536787 4611 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 20:39:05.550382 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.536789 4611 feature_gate.go:386] unrecognized feature gate: Example Oct 30 20:39:05.550382 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.536792 4611 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 20:39:05.550382 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.536795 4611 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 20:39:05.550382 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.536797 4611 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 20:39:05.550382 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.536800 4611 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 20:39:05.550382 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.536802 4611 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 20:39:05.550382 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.536805 4611 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 20:39:05.550382 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.536807 4611 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 20:39:05.550382 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.536810 4611 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 20:39:05.550382 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.536813 4611 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 20:39:05.550382 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.536815 4611 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 20:39:05.550382 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.536818 4611 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 20:39:05.550382 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.536821 4611 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 20:39:05.550382 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.536824 4611 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 20:39:05.550859 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.536826 4611 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 20:39:05.550859 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.536829 4611 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 20:39:05.550859 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.536837 4611 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 20:39:05.550859 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.536840 4611 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 20:39:05.550859 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.536842 4611 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 20:39:05.550859 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.536845 4611 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 20:39:05.550859 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.536847 4611 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 20:39:05.550859 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.536850 4611 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 20:39:05.550859 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.536852 4611 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 20:39:05.550859 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.536855 4611 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 20:39:05.550859 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.536857 4611 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 20:39:05.550859 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.536859 4611 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 20:39:05.550859 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.536862 4611 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 20:39:05.550859 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.536864 4611 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 20:39:05.550859 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.536867 4611 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 20:39:05.550859 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.536870 4611 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 20:39:05.550859 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.536872 4611 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 20:39:05.550859 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.536875 4611 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 20:39:05.550859 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.536877 4611 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 20:39:05.550859 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.536880 4611 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 20:39:05.551376 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.536882 4611 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 20:39:05.551376 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.536885 4611 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 20:39:05.551376 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.536887 4611 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 20:39:05.551376 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.536890 4611 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 20:39:05.551376 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.536893 4611 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 20:39:05.551376 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.536895 4611 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 20:39:05.551376 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.536898 4611 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 20:39:05.551376 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.536900 4611 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 20:39:05.551376 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.536903 4611 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 20:39:05.551376 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.536911 4611 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 20:39:05.551376 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.536914 4611 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 20:39:05.551376 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.536916 4611 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 20:39:05.551376 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.536919 4611 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 20:39:05.551376 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.536938 4611 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 20:39:05.551376 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.536942 4611 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 20:39:05.551376 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.536951 4611 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 20:39:05.551376 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.536954 4611 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 20:39:05.551376 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.536957 4611 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 20:39:05.551376 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.536959 4611 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 20:39:05.551849 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.536962 4611 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 20:39:05.551849 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.536964 4611 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 20:39:05.551849 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.536967 4611 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 20:39:05.551849 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.536970 4611 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 20:39:05.551849 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.536972 4611 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 20:39:05.551849 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.536975 4611 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 20:39:05.551849 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.536978 4611 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 20:39:05.551849 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.536981 4611 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 20:39:05.551849 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.536983 4611 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 20:39:05.551849 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.536986 4611 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 20:39:05.551849 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.536988 4611 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 20:39:05.551849 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.536991 4611 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 20:39:05.551849 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.536993 4611 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 20:39:05.551849 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.536996 4611 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 20:39:05.551849 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.536998 4611 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 20:39:05.551849 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.537001 4611 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 20:39:05.551849 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.537003 4611 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 20:39:05.551849 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.537010 4611 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 20:39:05.551849 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.537013 4611 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 20:39:05.551849 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.537015 4611 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 20:39:05.552336 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.537017 4611 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 20:39:05.552336 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.537020 4611 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 20:39:05.552336 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.537022 4611 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 20:39:05.552336 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.537025 4611 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 20:39:05.552336 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.537028 4611 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 20:39:05.552336 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.537030 4611 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 20:39:05.552336 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.537033 4611 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 20:39:05.552336 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.537036 4611 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 20:39:05.552336 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.537038 4611 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 20:39:05.552336 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.537047 4611 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 20:39:05.552336 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.537049 4611 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 20:39:05.552336 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.537054 4611 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 20:39:05.552336 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.537057 4611 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 20:39:05.552336 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.537060 4611 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 20:39:05.552336 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.537063 4611 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 20:39:05.552336 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.537065 4611 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 20:39:05.552336 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.537068 4611 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 20:39:05.552336 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.537070 4611 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 20:39:05.552336 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.537073 4611 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 20:39:05.552336 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.537075 4611 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 20:39:05.552825 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.537078 4611 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 20:39:05.552825 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.537080 4611 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 20:39:05.552825 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.537083 4611 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 20:39:05.552825 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.537085 4611 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 20:39:05.552825 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.537088 4611 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 20:39:05.552825 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.537090 4611 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 20:39:05.552825 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.537093 4611 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 20:39:05.552825 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.537095 4611 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 20:39:05.552825 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.537119 4611 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 20:39:05.552825 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.537156 4611 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 20:39:05.552825 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.537162 4611 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 20:39:05.552825 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.537166 4611 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 20:39:05.552825 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.537170 4611 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 20:39:05.552825 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.537173 4611 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 20:39:05.552825 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.537176 4611 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 20:39:05.552825 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.537178 4611 feature_gate.go:386] unrecognized feature gate: Example Oct 30 20:39:05.553208 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.537181 4611 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 20:39:05.553208 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.537184 4611 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 20:39:05.553208 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.537186 4611 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 20:39:05.553208 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.537190 4611 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 20:39:05.553208 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.537193 4611 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 20:39:05.553208 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.537195 4611 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 20:39:05.553208 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.537205 4611 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 20:39:05.553208 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.537208 4611 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 20:39:05.553208 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.537211 4611 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 20:39:05.553208 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.537213 4611 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 20:39:05.553208 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.537216 4611 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 20:39:05.553208 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.537219 4611 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 20:39:05.553208 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.537221 4611 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 20:39:05.553208 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.537224 4611 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 20:39:05.553208 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.537226 4611 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 20:39:05.553208 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.537229 4611 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 20:39:05.553208 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.537231 4611 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 20:39:05.553208 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.537234 4611 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 20:39:05.553208 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.537236 4611 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 20:39:05.553777 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.537239 4611 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 20:39:05.553777 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.537241 4611 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 20:39:05.553777 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.537244 4611 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 20:39:05.553777 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.537247 4611 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 20:39:05.553777 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.537249 4611 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 20:39:05.553777 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.537252 4611 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 20:39:05.553777 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.537254 4611 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 20:39:05.553777 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.537259 4611 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 20:39:05.553777 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.537261 4611 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 20:39:05.553777 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.537264 4611 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 20:39:05.553777 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.537267 4611 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 20:39:05.553777 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.537270 4611 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 20:39:05.553777 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.537272 4611 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 20:39:05.553777 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.537275 4611 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 20:39:05.553777 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.537278 4611 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 20:39:05.553777 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.537280 4611 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 20:39:05.553777 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.537284 4611 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 20:39:05.553777 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.537286 4611 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 20:39:05.553777 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.537289 4611 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 20:39:05.553777 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.537292 4611 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 20:39:05.554262 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.537300 4611 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 20:39:05.554262 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.537302 4611 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 20:39:05.554262 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.537305 4611 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 20:39:05.554262 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.537308 4611 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 20:39:05.554262 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.537310 4611 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 20:39:05.554262 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.537313 4611 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 20:39:05.554262 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.537315 4611 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 20:39:05.554262 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.537317 4611 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 20:39:05.554262 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.537320 4611 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 20:39:05.554262 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.537323 4611 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 20:39:05.554262 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.537325 4611 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 20:39:05.554262 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.537328 4611 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 20:39:05.554262 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.537330 4611 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 20:39:05.554262 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.537333 4611 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 20:39:05.554262 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.537335 4611 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 20:39:05.554262 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.537338 4611 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 20:39:05.554262 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.537341 4611 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 20:39:05.554262 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.537343 4611 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 20:39:05.554262 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.537347 4611 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 20:39:05.554861 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.537350 4611 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 20:39:05.554861 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.537353 4611 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 20:39:05.554861 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.537355 4611 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 20:39:05.554861 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.537358 4611 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 20:39:05.554861 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.537360 4611 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 20:39:05.554861 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.537363 4611 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 20:39:05.554861 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.537365 4611 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 20:39:05.554861 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.537368 4611 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 20:39:05.554861 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.537371 4611 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 20:39:05.554861 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.537374 4611 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 20:39:05.554861 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.537376 4611 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 20:39:05.554861 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.537379 4611 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 20:39:05.554861 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.537381 4611 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 20:39:05.554861 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.537384 4611 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 20:39:05.554861 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.537392 4611 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 20:39:05.554861 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.537395 4611 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 20:39:05.554861 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.537398 4611 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 20:39:05.554861 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.537400 4611 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 20:39:05.554861 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.537403 4611 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 20:39:05.554861 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.537405 4611 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 20:39:05.555776 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.537408 4611 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 20:39:05.555776 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.537410 4611 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 20:39:05.555776 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.537413 4611 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 20:39:05.555776 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.537415 4611 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 20:39:05.555776 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.537417 4611 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 20:39:05.555776 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.537421 4611 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 20:39:05.555776 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.537425 4611 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 20:39:05.555776 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.537427 4611 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 20:39:05.555776 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.537430 4611 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 20:39:05.555776 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.537432 4611 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 20:39:05.555776 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.537435 4611 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 20:39:05.555776 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.537437 4611 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 20:39:05.555776 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.537440 4611 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 20:39:05.555776 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.537442 4611 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 20:39:05.555776 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.537445 4611 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 20:39:05.555776 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.537447 4611 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 20:39:05.555776 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.537450 4611 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 20:39:05.555776 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.537452 4611 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 20:39:05.555776 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.537455 4611 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 20:39:05.555776 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.537457 4611 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 20:39:05.556712 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:05.537461 4611 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 20:39:05.556712 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.537533 4611 server.go:957] "Client rotation is on, will bootstrap in background" Oct 30 20:39:05.556712 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.538499 4611 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Oct 30 20:39:05.556712 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.538561 4611 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Oct 30 20:39:05.556712 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.538756 4611 server.go:1014] "Starting client certificate rotation" Oct 30 20:39:05.556712 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.538890 4611 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 20:39:05.556712 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.538931 4611 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 20:33:25 +0000 UTC" deadline="2028-06-21 21:18:00.57553494 +0000 UTC" Oct 30 20:39:05.556712 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.538994 4611 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14400h38m55.03654439s" Oct 30 20:39:05.556712 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.539268 4611 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 20:39:05.556712 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.540442 4611 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 20:39:05.556712 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.543250 4611 log.go:25] "Validated CRI v1 runtime API" Oct 30 20:39:05.556712 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.544868 4611 log.go:25] "Validated CRI v1 image API" Oct 30 20:39:05.556712 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.545686 4611 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 30 20:39:05.556712 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.547303 4611 fs.go:135] Filesystem UUIDs: map[039b0299-bc26-4617-bc85-4278d8a25752:/dev/nvme0n1p4 42cbd088-c4f9-4973-972e-0963ada6e718:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2] Oct 30 20:39:05.557325 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.547325 4611 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/010c147ffee86dab26403a897c7b0869149d45cdcb3e80147d42ff89dbd53f0c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/010c147ffee86dab26403a897c7b0869149d45cdcb3e80147d42ff89dbd53f0c/userdata/shm major:0 minor:86 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/15fedc521c61d80e330f2664466d0af696032c8d112c6aada7dec92eb05ea09b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/15fedc521c61d80e330f2664466d0af696032c8d112c6aada7dec92eb05ea09b/userdata/shm major:0 minor:249 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/160c0397f4741ec3a44de2d9a323a0a3713a7ab5c7ba1ac1f92e56c2b4711663/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/160c0397f4741ec3a44de2d9a323a0a3713a7ab5c7ba1ac1f92e56c2b4711663/userdata/shm major:0 minor:49 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1989faae8b908dfcf68e0f0d36993b7d6dca11487e1a4da908baed6982c99bd0/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1989faae8b908dfcf68e0f0d36993b7d6dca11487e1a4da908baed6982c99bd0/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3f323fafdad03f14abde257617593b193fdb5f4224c3a96a9af7ef7171ce52e2/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3f323fafdad03f14abde257617593b193fdb5f4224c3a96a9af7ef7171ce52e2/userdata/shm major:0 minor:74 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/45629083268acdae1d03c5389cac435087f01f6bf584b04d7ea7163c7cb3286a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/45629083268acdae1d03c5389cac435087f01f6bf584b04d7ea7163c7cb3286a/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5dceb587cfe62bf5917086004d513955231e7c975d9ab6c60bcd2e597d055e2c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5dceb587cfe62bf5917086004d513955231e7c975d9ab6c60bcd2e597d055e2c/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/674027bf57e670c920fd19b8e877f7dbbb717cdf3f0206ded7eb16c50d06f5d4/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/674027bf57e670c920fd19b8e877f7dbbb717cdf3f0206ded7eb16c50d06f5d4/userdata/shm major:0 minor:233 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/67bacfc15712c8b72b34fba7f6724c4b91949a42ff24fc59c1abdc86761f4710/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/67bacfc15712c8b72b34fba7f6724c4b91949a42ff24fc59c1abdc86761f4710/userdata/shm major:0 minor:236 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7f09df490a9f357a975df928f931adc4a68f4d6d7a4c712186b84a51a5d8eafd/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7f09df490a9f357a975df928f931adc4a68f4d6d7a4c712186b84a51a5d8eafd/userdata/shm major:0 minor:232 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8dcc88a7fd48996068bc3d76be510d02ef2d218a81f7bc76d6722c39bd915e66/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8dcc88a7fd48996068bc3d76be510d02ef2d218a81f7bc76d6722c39bd915e66/userdata/shm major:0 minor:70 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b26584b16a622d31abdf0ac29e363b7db4853bdaf37ec6c855ea80444314bbf5/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b26584b16a622d31abdf0ac29e363b7db4853bdaf37ec6c855ea80444314bbf5/userdata/shm major:0 minor:203 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e050f81a233aef1a82b2b11133bdd960ffe8aa369417dd2ec80ee56b7f3943d3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e050f81a233aef1a82b2b11133bdd960ffe8aa369417dd2ec80ee56b7f3943d3/userdata/shm major:0 minor:82 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ef4fe99de9fe62a9de95ae83fb4b6dccc0ea8e7edb7c3c4ec2475d6d4e3740e3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ef4fe99de9fe62a9de95ae83fb4b6dccc0ea8e7edb7c3c4ec2475d6d4e3740e3/userdata/shm major:0 minor:208 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f51c59026b3d0d64b59aa49ecb7828bc8a105f9afe7bc7cd031ba55b03628fcd/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f51c59026b3d0d64b59aa49ecb7828bc8a105f9afe7bc7cd031ba55b03628fcd/userdata/shm major:0 minor:78 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/fb0f2c227bb739438aaf12d4ffc31d80eab2c7f4b5776fd30ffc16448932514e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/fb0f2c227bb739438aaf12d4ffc31d80eab2c7f4b5776fd30ffc16448932514e/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0915cfe8-4606-4f92-877b-ebe7f52f4187/volumes/kubernetes.io~projected/kube-api-access-mjbz2:{mountpoint:/var/lib/kubelet/pods/0915cfe8-4606-4f92-877b-ebe7f52f4187/volumes/kubernetes.io~projected/kube-api-access-mjbz2 major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0915cfe8-4606-4f92-877b-ebe7f52f4187/volumes/kubernetes.io~secret/metrics-certs:{mountpoint:/var/lib/kubelet/pods/0915cfe8-4606-4f92-877b-ebe7f52f4187/volumes/kubernetes.io~secret/metrics-certs major:0 minor:230 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0aa2bb04-0552-45da-a54f-7e5064cf3113/volumes/kubernetes.io~projected/kube-api-access-hbqzh:{mountpoint:/var/lib/kubelet/pods/0aa2bb04-0552-45da-a54f-7e5064cf3113/volumes/kubernetes.io~projected/kube-api-access-hbqzh major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0aa2bb04-0552-45da-a54f-7e5064cf3113/volumes/kubernetes.io~secret/metrics-tls:{mountpoint:/var/lib/kubelet/pods/0aa2bb04-0552-45da-a54f-7e5064cf3113/volumes/kubernetes.io~secret/metrics-tls major:0 minor:207 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0dde7eae-1962-4559-89e9-0c67b1165914/volumes/kubernetes.io~projected/kube-api-access-6hvqv:{mountpoint:/var/lib/kubelet/pods/0dde7eae-1962-4559-89e9-0c67b1165914/volumes/kubernetes.io~projected/kube-api-access-6hvqv major:0 minor:231 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/17af283e-b12c-4ed3-abf4-7a4944d1c4f2/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/17af283e-b12c-4ed3-abf4-7a4944d1c4f2/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:247 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/17af283e-b12c-4ed3-abf4-7a4944d1c4f2/volumes/kubernetes.io~projected/kube-api-access-cdwk9:{mountpoint:/var/lib/kubelet/pods/17af283e-b12c-4ed3-abf4-7a4944d1c4f2/volumes/kubernetes.io~projected/kube-api-access-cdwk9 major:0 minor:248 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/17af283e-b12c-4ed3-abf4-7a4944d1c4f2/volumes/kubernetes.io~projected/registry-tls:{mountpoint:/var/lib/kubelet/pods/17af283e-b12c-4ed3-abf4-7a4944d1c4f2/volumes/kubernetes.io~projected/registry-tls major:0 minor:246 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/17af283e-b12c-4ed3-abf4-7a4944d1c4f2/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/17af283e-b12c-4ed3-abf4-7a4944d1c4f2/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:245 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/17af283e-b12c-4ed3-abf4-7a4944d1c4f2/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/17af283e-b12c-4ed3-abf4-7a4944d1c4f2/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:244 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/35d51f79-80a9-4f9d-970e-6a35a34976a2/volumes/kubernetes.io~projected/kube-api-access-4ntjf:{mountpoint:/var/lib/kubelet/pods/35d51f79-80a9-4f9d-970e-6a35a34976a2/volumes/kubernetes.io~projected/kube-api-access-4ntjf major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6b88d0a3-97c0-4adf-97a7-2e05d5cef781/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/6b88d0a3-97c0-4adf-97a7-2e05d5cef781/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6b88d0a3-97c0-4adf-97a7-2e05d5cef781/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/6b88d0a3-97c0-4adf-97a7-2e05d5cef781/volumes/kubernetes.io~empty-dir/tmp major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6b88d0a3-97c0-4adf-97a7-2e05d5cef781/volumes/kubernetes.io~projected/kube-api-access-mslf8:{mountpoint:/var/lib/kubelet/pods/6b88d0a3-97c0-4adf-97a7-2e05d5cef781/volumes/kubernetes.io~projected/kube-api-access-mslf8 major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6c3ba176-c9a5-4b6c-ae06-5593f68cf510/volumes/kubernetes.io~projected/kube-api-access-qhljq:{mountpoint:/var/lib/kubelet/pods/6c3ba176-c9a5-4b6c-ae06-5593f68cf510/volumes/kubernetes.io~projected/kube-api-access-qhljq major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6c3ba176-c9a5-4b6c-ae06-5593f68cf510/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/6c3ba176-c9a5-4b6c-ae06-5593f68cf510/volumes/kubernetes.io~secret/cert major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7e2574c1-b59c-453c-9501-25ec13a1ccad/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/7e2574c1-b59c-453c-9501-25ec13a1ccad/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7e2574c1-b59c-453c-9501-25ec13a1ccad/volumes/kubernetes.io~projected/kube-api-access-jk5kq:{mountpoint:/var/lib/kubelet/pods/7e2574c1-b59c-453c-9501-25ec13a1ccad/volumes/kubernetes.io~projected/kube-api-access-jk5kq major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7e2574c1-b59c-453c-9501-25ec13a1ccad/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/7e2574c1-b59c-453c-9501-25ec13a1ccad/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/acf9d9fb-1ae4-40ca-a83e-420a9fe40a01/volumes/kubernetes.io~projected/kube-api-access-szhgd:{mountpoint:/var/lib/kubelet/pods/acf9d9fb-1ae4-40ca-a83e-420a9fe40a01/volumes/kubernetes.io~projected/kube-api-access-szhgd major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bf21cf19-4475-41c3-9153-44d8fb329cdb/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/bf21cf19-4475-41c3-9153-44d8fb329cdb/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:226 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c1a9e89f-5d46-4414-b0e0-7adff658510e/volumes/kubernetes.io~projected/kube-api-access-9d557:{mountpoint:/var/lib/kubelet/pods/c1a9e89f-5d46-4414-b0e0-7adff658510e/volumes/kubernetes.io~projected/kube-api-access-9d557 major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c9257033-53bf-40fc-a1e9-67fd5901678b/volumes/kubernetes.io~projected/kube-api-access-9jszj:{mountpoint:/var/lib/kubelet/pods/c9257033-53bf-40fc-a1e9-67fd5901678b/volumes/kubernetes.io~projected/kube-api-access-9jszj major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/da50258d-eef3-4976-8e22-be0e936b9993/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/da50258d-eef3-4976-8e22-be0e936b9993/volumes/kubernetes.io~secret/agent-certs major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e277a3d9-3fe9-4454-9420-28afea76f745/volumes/kubernetes.io~projected/kube-api-access-bg7g6:{mountpoint:/var/lib/kubelet/pods/e277a3d9-3fe9-4454-9420-28afea76f745/volumes/kubernetes.io~projected/kube-api-access-bg7g6 major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ea0e353b-8cb1-4759-9f47-03f788f256d3/volumes/kubernetes.io~projected/kube-api-access-rjj44:{mountpoint:/var/lib/kubelet/pods/ea0e353b-8cb1-4759-9f47-03f788f256d3/volumes/kubernetes.io~projected/kube-api-access-rjj44 major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f5f694f2-6f18-409b-b777-2dd7a0eb12b2/volumes/kubernetes.io~projected/kube-api-access-cv9nx:{mountpoint:/var/lib/kubelet/pods/f5f694f2-6f18-409b-b777-2dd7a0eb12b2/volumes/kubernetes.io~projected/kube-api-access-cv9nx major:0 minor:65 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/2b5b1ea7b4b4d6310d42dd27be8a76f02d31bdbd32f8aff0e21007c96eecad2b/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/32782768768c4e0b23fecb4bf8285ea34b681d6bf85988466e6dfb5c79ed31b1/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/5d8cc91da2f3a6e4ec458a9f1260ad0f95f96b65200edb960790d6972c9ccbd1/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/ec86637614171b7dc95d97ed8f2b1050d10c1e5ab05a197c018de83bbe382ea1/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/6b931aafaa5adf4d92b55a44625f790df6d323e5b663030ed9419c5a8242fa32/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/0fc4928862eb1ebedff0a95c72f2b858d141b0d4537f67b7e4c91ccd2867252e/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/a611ce4ee41b230388f2a9376ae24fae41b8008dd6423fa3705544e903d9de67/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/293e6edb042bdb3304afc4bac97b642a0b795cf7065d08e86229dc91b4a80621/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/e6916a96a4b4f8090674adc3004db7311b48e3727bcc442d1caa698e97c999a5/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-137:{mountpoint:/var/lib/containers/storage/overlay/b8a1f3480ce858d77f8451820baf61943292e094391248fbfbab525ef812eb3a/merged major:0 minor:137 fsType:overlay blockSize:0} overlay_0-142:{mountpoint:/var/lib/containers/storage/overlay/e13b348dfd555670b8a40ef06ea6d6acb20d9c34926dab0c96d6665d6395ee29/merged major:0 minor:142 fsType:overlay blockSize:0} overlay_0-143:{mountpoint:/var/lib/containers/storage/overlay/f6aa3e43b16b4a741419622b8cd88428e1f7780810874138cf0f7e9a62fdd95b/merged major:0 minor:143 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/94c3a456bbac98dba02bdace99547aab5af261655783e5d269509c45096b5916/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/0d39e98766e66157f44f7d83691060bb90d46ba00640e44cb336bec21d7f28f1/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/1b2487dba018cff036bee6c3b8b78fb6fb1331e107937afea1617ffbe501ceac/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/6caa8b1f1bc4b886aedf2288b2f9d54c415da31431732f9d61f0ed30f92f6bcc/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/760956daf0f3dd22b7379f039c17ceb88f84f536029e59ff8917b28a981f83f9/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/8e581410bd2d74c4be94c7d6c54682548f144c5c1e924bcc107e8b1b86249b7b/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/3b60e3f0254b7143b71f22a698f6fc5865fcbcdfb0dc89a7c26ebdbd286df410/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/4219a3ae7a8826a6392c6c931882bbeb4e440e5cdff6a15b22de518927057ea1/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/9efd3a643ccc12b62adf4753997e9b56f1df41579be2fb690ac2aacfd9ce369c/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-205:{mountpoint:/var/lib/containers/storage/overlay/872d0de525e2642101051f2e0a2fc8e1e18af9b9ab92b8084ff9e96b866d8c82/merged major:0 minor:205 fsType:overlay blockSize:0} overlay_0-210:{mountpoint:/var/lib/containers/storage/overlay/b18c42375c34c6afd99573cde37fe79abc07c0d7f8e0d546ececbf21d0d588bc/merged major:0 minor:210 fsType:overlay blockSize:0} overlay_0-212:{mountpoint:/var/lib/containers/storage/overlay/ba7f95f681389fa62d0526d0f7cb1d23a215cf46359275a27703ca432d420a23/merged major:0 minor:212 fsType:overlay blockSize:0} overlay_0-214:{mountpoint:/var/lib/containers/storage/overlay/a35f37944dc2e0a37a6d5ea3c49fd88879fe4a017c98a0a158a83c4f32ecf7e9/merged major:0 minor:214 fsType:overlay blockSize:0} overlay_0-220:{mountpoint:/var/lib/containers/storage/overlay/2de1529401ae4a605b3bf7d94263632b7ea9810d43a4d6daf3ec75b88f86a0a6/merged major:0 minor:220 fsType:overlay blockSize:0} overlay_0-238:{mountpoint:/var/lib/containers/storage/overlay/1df7dea27a7b8d03c1fa557e680069e73deef17b541c358084734e32472a0e1b/merged major:0 minor:238 fsType:overlay blockSize:0} overlay_0-240:{mountpoint:/var/lib/containers/storage/overlay/6b760ab17b3896adeaf9978f7fe7f0d44822b095a5895d1a74a4eb7d8f3d9593/merged major:0 minor:240 fsType:overlay blockSize:0} overlay_0-242:{mountpoint:/var/lib/containers/storage/overlay/addbcc4936f0c8fcfd61fbe95a4b3e22d078be55ea99c18d61012e41a58802ac/merged major:0 minor:242 fsType:overlay blockSize:0} overlay_0-251:{mountpoint:/var/lib/containers/storage/overlay/c91bb0b951ee4ca8afc9833510e5df5b0452a2a71a3a16217d08795ab88eec93/merged major:0 minor:251 fsType:overlay blockSize:0} overlay_0-253:{mountpoint:/var/lib/containers/storage/overlay/6f4503b1609611479388247024bb4a8af8c06275c7b6ebeec7f6c10fdc50c14b/merged major:0 minor:253 fsType:overlay blockSize:0} overlay_0-255:{mountpoint:/var/lib/containers/storage/overlay/1d9e6db7b9c756887b466779af0a5de884510ee83b39a91d01a7cc0c2c465c08/merged major:0 minor:255 fsType:overlay blockSize:0} overlay_0-257:{mountpoint:/var/lib/containers/storage/overlay/bff116fa7987f77405d7f31c8e039ac7ceaf6224a1ecab16edf42be34d5aa586/merged major:0 minor:257 fsType:overlay blockSize:0} overlay_0-267:{mountpoint:/var/lib/containers/storage/overlay/c38e99838eb7397b5df6aaa0d070d812ad10d0d03d962f64c55bfdd56c78526b/merged major:0 minor:267 fsType:overlay blockSize:0} overlay_0-272:{mountpoint:/var/lib/containers/storage/overlay/3e625b558f7fad07ac1a49fc30a49216d052a21b7ba899a76e73d8ce0ae219c0/merged major:0 minor:272 fsType:overlay blockSize:0} overlay_0-50:{mountpoint:/var/lib/containers/storage/overlay/389b53a3813604d6348451f513e614fe73234d2dadeb145221965b9d937b9b69/merged major:0 minor:50 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/5ab8ae9e3a19b0021e1045e3745cf93154f366864fd21f4ebfb05a4cb785b94a/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-68:{mountpoint:/var/lib/containers/storage/overlay/a2a248381f1c9b8ebb81cef31d2fa60e4c85979488a9b1aec004fc3a2b4d4671/merged major:0 minor:68 fsType:overlay blockSize:0} overlay_0-72:{mountpoint:/var/lib/containers/storage/overlay/1ae4d11881faf45dab68cfcbcedb60739f45be235aaed0b3d8847cc17b91e38e/merged major:0 minor:72 fsType:overlay blockSize:0} overlay_0-76:{mountpoint:/var/lib/containers/storage/overlay/d697817822d5f645ac556643e37b3dd52f031895f7725e4e8d9e0f8b0fc56599/merged major:0 minor:76 fsType:overlay blockSize:0} overlay_0-80:{mountpoint:/var/lib/containers/storage/overlay/0416a0f7eaf8cacdb0ff10d97599abc180297c2d406bf688e88ba596ba0547b3/merged major:0 minor:80 fsType:overlay blockSize:0} overlay_0-84:{mountpoint:/var/lib/containers/storage/overlay/82f523cb77c866c270a946fc8717fe561513ba1f85f1860b1c3ee534b206b51d/merged major:0 minor:84 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/d080151e7d07d9316a187ae0bf3e1e4022708245ec495c59514054a9e8340218/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/64a907023686c635d2c24fc3b415107e495b4904673dd45a28a1720d4e10cf60/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/ef7bd2551994ea4bd0520dddfba9541d106598f0b1f7be848db022a2dd1a6a41/merged major:0 minor:96 fsType:overlay blockSize:0}] Oct 30 20:39:05.557614 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.553443 4611 manager.go:217] Machine: {Timestamp:2026-10-30 20:39:05.553007981 +0000 UTC m=+0.087034600 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100322 MemoryCapacity:33164333056 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec283823ff5dcaf4490b53bdc3778ed4 SystemUUID:ec283823-ff5d-caf4-490b-53bdc3778ed4 BootID:5cd9162a-cf97-4c4e-b2c3-f0cc131338ae Filesystems:[{Device:/var/lib/kubelet/pods/6b88d0a3-97c0-4adf-97a7-2e05d5cef781/volumes/kubernetes.io~projected/kube-api-access-mslf8 DeviceMajor:0 DeviceMinor:58 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b26584b16a622d31abdf0ac29e363b7db4853bdaf37ec6c855ea80444314bbf5/userdata/shm DeviceMajor:0 DeviceMinor:203 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/17af283e-b12c-4ed3-abf4-7a4944d1c4f2/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:247 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-272 DeviceMajor:0 DeviceMinor:272 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/fb0f2c227bb739438aaf12d4ffc31d80eab2c7f4b5776fd30ffc16448932514e/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-210 DeviceMajor:0 DeviceMinor:210 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7f09df490a9f357a975df928f931adc4a68f4d6d7a4c712186b84a51a5d8eafd/userdata/shm DeviceMajor:0 DeviceMinor:232 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/674027bf57e670c920fd19b8e877f7dbbb717cdf3f0206ded7eb16c50d06f5d4/userdata/shm DeviceMajor:0 DeviceMinor:233 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7e2574c1-b59c-453c-9501-25ec13a1ccad/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:60 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-205 DeviceMajor:0 DeviceMinor:205 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ef4fe99de9fe62a9de95ae83fb4b6dccc0ea8e7edb7c3c4ec2475d6d4e3740e3/userdata/shm DeviceMajor:0 DeviceMinor:208 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-253 DeviceMajor:0 DeviceMinor:253 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f5f694f2-6f18-409b-b777-2dd7a0eb12b2/volumes/kubernetes.io~projected/kube-api-access-cv9nx DeviceMajor:0 DeviceMinor:65 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582164480 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-50 DeviceMajor:0 DeviceMinor:50 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e277a3d9-3fe9-4454-9420-28afea76f745/volumes/kubernetes.io~projected/kube-api-access-bg7g6 DeviceMajor:0 DeviceMinor:61 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/acf9d9fb-1ae4-40ca-a83e-420a9fe40a01/volumes/kubernetes.io~projected/kube-api-access-szhgd DeviceMajor:0 DeviceMinor:62 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-68 DeviceMajor:0 DeviceMinor:68 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1989faae8b908dfcf68e0f0d36993b7d6dca11487e1a4da908baed6982c99bd0/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-143 DeviceMajor:0 DeviceMinor:143 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/da50258d-eef3-4976-8e22-be0e936b9993/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:59 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-80 DeviceMajor:0 DeviceMinor:80 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7e2574c1-b59c-453c-9501-25ec13a1ccad/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/0aa2bb04-0552-45da-a54f-7e5064cf3113/volumes/kubernetes.io~projected/kube-api-access-hbqzh DeviceMajor:0 DeviceMinor:201 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/6c3ba176-c9a5-4b6c-ae06-5593f68cf510/volumes/kubernetes.io~projected/kube-api-access-qhljq DeviceMajor:0 DeviceMinor:202 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-220 DeviceMajor:0 DeviceMinor:220 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f51c59026b3d0d64b59aa49ecb7828bc8a105f9afe7bc7cd031ba55b03628fcd/userdata/shm DeviceMajor:0 DeviceMinor:78 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5dceb587cfe62bf5917086004d513955231e7c975d9ab6c60bcd2e597d055e2c/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-137 DeviceMajor:0 DeviceMinor:137 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0aa2bb04-0552-45da-a54f-7e5064cf3113/volumes/kubernetes.io~secret/metrics-tls DeviceMajor:0 DeviceMinor:207 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-214 DeviceMajor:0 DeviceMinor:214 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-240 DeviceMajor:0 DeviceMinor:240 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-72 DeviceMajor:0 DeviceMinor:72 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/6c3ba176-c9a5-4b6c-ae06-5593f68cf510/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:45 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/bf21cf19-4475-41c3-9153-44d8fb329cdb/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:226 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/17af283e-b12c-4ed3-abf4-7a4944d1c4f2/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:245 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-76 DeviceMajor:0 DeviceMinor:76 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7e2574c1-b59c-453c-9501-25ec13a1ccad/volumes/kubernetes.io~projected/kube-api-access-jk5kq DeviceMajor:0 DeviceMinor:64 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/17af283e-b12c-4ed3-abf4-7a4944d1c4f2/volumes/kubernetes.io~projected/kube-api-access-cdwk9 DeviceMajor:0 DeviceMinor:248 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/15fedc521c61d80e330f2664466d0af696032c8d112c6aada7dec92eb05ea09b/userdata/shm DeviceMajor:0 DeviceMinor:249 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-251 DeviceMajor:0 DeviceMinor:251 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:/var/lib/kubelet/pods/0915cfe8-4606-4f92-877b-ebe7f52f4187/volumes/kubernetes.io~projected/kube-api-access-mjbz2 DeviceMajor:0 DeviceMinor:66 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-142 DeviceMajor:0 DeviceMinor:142 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/6b88d0a3-97c0-4adf-97a7-2e05d5cef781/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:55 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-257 DeviceMajor:0 DeviceMinor:257 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-267 DeviceMajor:0 DeviceMinor:267 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c9257033-53bf-40fc-a1e9-67fd5901678b/volumes/kubernetes.io~projected/kube-api-access-9jszj DeviceMajor:0 DeviceMinor:57 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-84 DeviceMajor:0 DeviceMinor:84 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/010c147ffee86dab26403a897c7b0869149d45cdcb3e80147d42ff89dbd53f0c/userdata/shm DeviceMajor:0 DeviceMinor:86 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c1a9e89f-5d46-4414-b0e0-7adff658510e/volumes/kubernetes.io~projected/kube-api-access-9d557 DeviceMajor:0 DeviceMinor:197 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-212 DeviceMajor:0 DeviceMinor:212 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0915cfe8-4606-4f92-877b-ebe7f52f4187/volumes/kubernetes.io~secret/metrics-certs DeviceMajor:0 DeviceMinor:230 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/0dde7eae-1962-4559-89e9-0c67b1165914/volumes/kubernetes.io~projected/kube-api-access-6hvqv DeviceMajor:0 DeviceMinor:231 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/45629083268acdae1d03c5389cac435087f01f6bf584b04d7ea7163c7cb3286a/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3f323fafdad03f14abde257617593b193fdb5f4224c3a96a9af7ef7171ce52e2/userdata/shm DeviceMajor:0 DeviceMinor:74 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-238 DeviceMajor:0 DeviceMinor:238 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-242 DeviceMajor:0 DeviceMinor:242 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8dcc88a7fd48996068bc3d76be510d02ef2d218a81f7bc76d6722c39bd915e66/userdata/shm DeviceMajor:0 DeviceMinor:70 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/run/containers/storage/overlay-containers/160c0397f4741ec3a44de2d9a323a0a3713a7ab5c7ba1ac1f92e56c2b4711663/userdata/shm DeviceMajor:0 DeviceMinor:49 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/35d51f79-80a9-4f9d-970e-6a35a34976a2/volumes/kubernetes.io~projected/kube-api-access-4ntjf DeviceMajor:0 DeviceMinor:67 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e050f81a233aef1a82b2b11133bdd960ffe8aa369417dd2ec80ee56b7f3943d3/userdata/shm DeviceMajor:0 DeviceMinor:82 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/var/lib/kubelet/pods/ea0e353b-8cb1-4759-9f47-03f788f256d3/volumes/kubernetes.io~projected/kube-api-access-rjj44 DeviceMajor:0 DeviceMinor:63 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/6b88d0a3-97c0-4adf-97a7-2e05d5cef781/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:56 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/67bacfc15712c8b72b34fba7f6724c4b91949a42ff24fc59c1abdc86761f4710/userdata/shm DeviceMajor:0 DeviceMinor:236 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/17af283e-b12c-4ed3-abf4-7a4944d1c4f2/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:244 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/17af283e-b12c-4ed3-abf4-7a4944d1c4f2/volumes/kubernetes.io~projected/registry-tls DeviceMajor:0 DeviceMinor:246 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-255 DeviceMajor:0 DeviceMinor:255 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:15fedc521c61d80 MacAddress:d2:c4:c0:c9:f3:77 Speed:10000 Mtu:8901} {Name:674027bf57e670c MacAddress:8e:d6:63:4d:bd:ad Speed:10000 Mtu:8901} {Name:67bacfc15712c8b MacAddress:6e:5c:a3:1d:a3:02 Speed:10000 Mtu:8901} {Name:7f09df490a9f357 MacAddress:d6:ea:8a:f5:b3:1d Speed:10000 Mtu:8901} {Name:b26584b16a622d3 MacAddress:32:3f:5d:a2:4a:bc Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:ea:cf:72:b1 Speed:0 Mtu:9001} {Name:br-int MacAddress:a6:a6:84:6f:ef:5a Speed:0 Mtu:8901} {Name:ef4fe99de9fe62a MacAddress:8a:b9:95:71:48:28 Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:ff:ea:cf:72:b1 Speed:0 Mtu:9001} {Name:genev_sys_6081 MacAddress:ca:02:69:12:30:89 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:85:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:66:eb:9d:3a:12:e6 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164333056 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 30 20:39:05.557614 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.553722 4611 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 30 20:39:05.557614 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.553762 4611 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 30 20:39:05.557614 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.554023 4611 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 30 20:39:05.557614 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.554056 4611 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-133-192.us-east-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 30 20:39:05.557614 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.554179 4611 topology_manager.go:143] "Creating topology manager with none policy" Oct 30 20:39:05.557614 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.554187 4611 container_manager_linux.go:311] "Creating device plugin manager" Oct 30 20:39:05.557614 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.554200 4611 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 20:39:05.557614 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.554212 4611 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 20:39:05.557614 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.554295 4611 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 30 20:39:05.557614 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.554448 4611 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 30 20:39:05.557614 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.554574 4611 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 30 20:39:05.557614 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.554608 4611 kubelet.go:493] "Attempting to sync node with API server" Oct 30 20:39:05.557614 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.554623 4611 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 30 20:39:05.557614 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.554637 4611 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 30 20:39:05.557614 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.554645 4611 kubelet.go:405] "Adding apiserver pod source" Oct 30 20:39:05.557614 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.554653 4611 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 30 20:39:05.557614 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.555576 4611 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 30 20:39:05.557614 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.555585 4611 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 20:39:05.557614 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.555996 4611 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 30 20:39:05.558659 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.558379 4611 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 30 20:39:05.558659 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.558396 4611 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 20:39:05.558659 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.558533 4611 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Oct 30 20:39:05.558974 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.558823 4611 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 30 20:39:05.558974 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.558869 4611 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 30 20:39:05.559544 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.559526 4611 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 30 20:39:05.559901 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.559888 4611 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 30 20:39:05.559939 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.559906 4611 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 30 20:39:05.559939 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.559917 4611 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 30 20:39:05.559939 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.559928 4611 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 30 20:39:05.559939 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.559937 4611 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 30 20:39:05.560049 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.559947 4611 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 30 20:39:05.560049 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.559955 4611 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 30 20:39:05.560049 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.559966 4611 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 30 20:39:05.560049 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.559975 4611 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 30 20:39:05.560049 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.559988 4611 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 30 20:39:05.560049 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.560001 4611 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 30 20:39:05.560049 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.560031 4611 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 30 20:39:05.560049 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.560040 4611 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 30 20:39:05.563267 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.563251 4611 server.go:1263] "Started kubelet" Oct 30 20:39:05.563901 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.563323 4611 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 30 20:39:05.563901 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.563347 4611 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 30 20:39:05.563901 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.563417 4611 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 30 20:39:05.563901 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.563824 4611 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 30 20:39:05.563793 ip-10-0-133-192 systemd[1]: Started Kubernetes Kubelet. Oct 30 20:39:05.564700 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.564663 4611 server.go:317] "Adding debug handlers to kubelet server" Oct 30 20:39:05.565700 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.565684 4611 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 30 20:39:05.566325 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.566169 4611 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 30 20:39:05.566325 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.566220 4611 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 20:33:27 +0000 UTC" deadline="2028-05-27 10:21:57.493850202 +0000 UTC" Oct 30 20:39:05.566325 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.566245 4611 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13789h42m51.927607595s" Oct 30 20:39:05.566325 ip-10-0-133-192 kubenswrapper[4611]: E1030 20:39:05.566271 4611 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-192.us-east-2.compute.internal\" not found" Oct 30 20:39:05.566325 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.566298 4611 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 30 20:39:05.566325 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.566303 4611 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 30 20:39:05.566679 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.566532 4611 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 30 20:39:05.567444 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.567423 4611 factory.go:55] Registering systemd factory Oct 30 20:39:05.567539 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.567447 4611 factory.go:223] Registration of the systemd container factory successfully Oct 30 20:39:05.569620 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.568818 4611 factory.go:153] Registering CRI-O factory Oct 30 20:39:05.569620 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.568839 4611 factory.go:223] Registration of the crio container factory successfully Oct 30 20:39:05.569620 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.569426 4611 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 30 20:39:05.569620 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.569488 4611 factory.go:103] Registering Raw factory Oct 30 20:39:05.569620 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.569528 4611 manager.go:1199] Started watching for new ooms in manager Oct 30 20:39:05.569913 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.569894 4611 manager.go:319] Starting recovery of all containers Oct 30 20:39:05.571182 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.571157 4611 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 20:39:05.571966 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.571943 4611 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 20:39:05.572076 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.572061 4611 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 20:39:05.572322 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.568924 4611 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7e2574c1-b59c-453c-9501-25ec13a1ccad" volumeName="kubernetes.io/configmap/7e2574c1-b59c-453c-9501-25ec13a1ccad-ovnkube-script-lib" seLinuxMountContext="" Oct 30 20:39:05.572396 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.572346 4611 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7e2574c1-b59c-453c-9501-25ec13a1ccad" volumeName="kubernetes.io/configmap/7e2574c1-b59c-453c-9501-25ec13a1ccad-ovnkube-config" seLinuxMountContext="" Oct 30 20:39:05.572396 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.572373 4611 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="da50258d-eef3-4976-8e22-be0e936b9993" volumeName="kubernetes.io/configmap/da50258d-eef3-4976-8e22-be0e936b9993-konnectivity-ca" seLinuxMountContext="" Oct 30 20:39:05.572651 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.572629 4611 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ea0e353b-8cb1-4759-9f47-03f788f256d3" volumeName="kubernetes.io/configmap/ea0e353b-8cb1-4759-9f47-03f788f256d3-whereabouts-flatfile-configmap" seLinuxMountContext="" Oct 30 20:39:05.572722 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.572655 4611 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f5f694f2-6f18-409b-b777-2dd7a0eb12b2" volumeName="kubernetes.io/projected/f5f694f2-6f18-409b-b777-2dd7a0eb12b2-kube-api-access-cv9nx" seLinuxMountContext="" Oct 30 20:39:05.572722 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.572675 4611 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0aa2bb04-0552-45da-a54f-7e5064cf3113" volumeName="kubernetes.io/configmap/0aa2bb04-0552-45da-a54f-7e5064cf3113-config-volume" seLinuxMountContext="" Oct 30 20:39:05.572722 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.572688 4611 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6b88d0a3-97c0-4adf-97a7-2e05d5cef781" volumeName="kubernetes.io/empty-dir/6b88d0a3-97c0-4adf-97a7-2e05d5cef781-tmp" seLinuxMountContext="" Oct 30 20:39:05.572722 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.572701 4611 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="35d51f79-80a9-4f9d-970e-6a35a34976a2" volumeName="kubernetes.io/configmap/35d51f79-80a9-4f9d-970e-6a35a34976a2-serviceca" seLinuxMountContext="" Oct 30 20:39:05.572722 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.572721 4611 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c9257033-53bf-40fc-a1e9-67fd5901678b" volumeName="kubernetes.io/empty-dir/c9257033-53bf-40fc-a1e9-67fd5901678b-tmp-dir" seLinuxMountContext="" Oct 30 20:39:05.572957 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.572734 4611 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ea0e353b-8cb1-4759-9f47-03f788f256d3" volumeName="kubernetes.io/configmap/ea0e353b-8cb1-4759-9f47-03f788f256d3-cni-binary-copy" seLinuxMountContext="" Oct 30 20:39:05.573155 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.573043 4611 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="17af283e-b12c-4ed3-abf4-7a4944d1c4f2" volumeName="kubernetes.io/configmap/17af283e-b12c-4ed3-abf4-7a4944d1c4f2-registry-certificates" seLinuxMountContext="" Oct 30 20:39:05.573155 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.573076 4611 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="17af283e-b12c-4ed3-abf4-7a4944d1c4f2" volumeName="kubernetes.io/projected/17af283e-b12c-4ed3-abf4-7a4944d1c4f2-kube-api-access-cdwk9" seLinuxMountContext="" Oct 30 20:39:05.573155 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.573093 4611 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7e2574c1-b59c-453c-9501-25ec13a1ccad" volumeName="kubernetes.io/secret/7e2574c1-b59c-453c-9501-25ec13a1ccad-ovn-node-metrics-cert" seLinuxMountContext="" Oct 30 20:39:05.573155 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.573112 4611 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bf21cf19-4475-41c3-9153-44d8fb329cdb" volumeName="kubernetes.io/secret/bf21cf19-4475-41c3-9153-44d8fb329cdb-original-pull-secret" seLinuxMountContext="" Oct 30 20:39:05.573616 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.573439 4611 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c1a9e89f-5d46-4414-b0e0-7adff658510e" volumeName="kubernetes.io/configmap/c1a9e89f-5d46-4414-b0e0-7adff658510e-kube-rbac-proxy-cm" seLinuxMountContext="" Oct 30 20:39:05.573616 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.573490 4611 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c1a9e89f-5d46-4414-b0e0-7adff658510e" volumeName="kubernetes.io/empty-dir/c1a9e89f-5d46-4414-b0e0-7adff658510e-data-volume" seLinuxMountContext="" Oct 30 20:39:05.573616 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.573532 4611 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e277a3d9-3fe9-4454-9420-28afea76f745" volumeName="kubernetes.io/configmap/e277a3d9-3fe9-4454-9420-28afea76f745-multus-daemon-config" seLinuxMountContext="" Oct 30 20:39:05.573616 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.573555 4611 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ea0e353b-8cb1-4759-9f47-03f788f256d3" volumeName="kubernetes.io/configmap/ea0e353b-8cb1-4759-9f47-03f788f256d3-cni-sysctl-allowlist" seLinuxMountContext="" Oct 30 20:39:05.573616 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.573572 4611 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0915cfe8-4606-4f92-877b-ebe7f52f4187" volumeName="kubernetes.io/secret/0915cfe8-4606-4f92-877b-ebe7f52f4187-metrics-certs" seLinuxMountContext="" Oct 30 20:39:05.574345 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.573588 4611 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0dde7eae-1962-4559-89e9-0c67b1165914" volumeName="kubernetes.io/projected/0dde7eae-1962-4559-89e9-0c67b1165914-kube-api-access-6hvqv" seLinuxMountContext="" Oct 30 20:39:05.574345 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.573971 4611 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="35d51f79-80a9-4f9d-970e-6a35a34976a2" volumeName="kubernetes.io/projected/35d51f79-80a9-4f9d-970e-6a35a34976a2-kube-api-access-4ntjf" seLinuxMountContext="" Oct 30 20:39:05.574345 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.573984 4611 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c1a9e89f-5d46-4414-b0e0-7adff658510e" volumeName="kubernetes.io/projected/c1a9e89f-5d46-4414-b0e0-7adff658510e-kube-api-access-9d557" seLinuxMountContext="" Oct 30 20:39:05.574345 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.574004 4611 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ea0e353b-8cb1-4759-9f47-03f788f256d3" volumeName="kubernetes.io/projected/ea0e353b-8cb1-4759-9f47-03f788f256d3-kube-api-access-rjj44" seLinuxMountContext="" Oct 30 20:39:05.574345 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.574017 4611 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0915cfe8-4606-4f92-877b-ebe7f52f4187" volumeName="kubernetes.io/projected/0915cfe8-4606-4f92-877b-ebe7f52f4187-kube-api-access-mjbz2" seLinuxMountContext="" Oct 30 20:39:05.574345 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.574033 4611 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="17af283e-b12c-4ed3-abf4-7a4944d1c4f2" volumeName="kubernetes.io/projected/17af283e-b12c-4ed3-abf4-7a4944d1c4f2-registry-tls" seLinuxMountContext="" Oct 30 20:39:05.574345 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.574045 4611 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="17af283e-b12c-4ed3-abf4-7a4944d1c4f2" volumeName="kubernetes.io/secret/17af283e-b12c-4ed3-abf4-7a4944d1c4f2-image-registry-private-configuration" seLinuxMountContext="" Oct 30 20:39:05.574345 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.574060 4611 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7e2574c1-b59c-453c-9501-25ec13a1ccad" volumeName="kubernetes.io/projected/7e2574c1-b59c-453c-9501-25ec13a1ccad-kube-api-access-jk5kq" seLinuxMountContext="" Oct 30 20:39:05.574345 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.574076 4611 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c9257033-53bf-40fc-a1e9-67fd5901678b" volumeName="kubernetes.io/projected/c9257033-53bf-40fc-a1e9-67fd5901678b-kube-api-access-9jszj" seLinuxMountContext="" Oct 30 20:39:05.574345 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.574104 4611 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0aa2bb04-0552-45da-a54f-7e5064cf3113" volumeName="kubernetes.io/projected/0aa2bb04-0552-45da-a54f-7e5064cf3113-kube-api-access-hbqzh" seLinuxMountContext="" Oct 30 20:39:05.574345 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.574117 4611 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="17af283e-b12c-4ed3-abf4-7a4944d1c4f2" volumeName="kubernetes.io/configmap/17af283e-b12c-4ed3-abf4-7a4944d1c4f2-trusted-ca" seLinuxMountContext="" Oct 30 20:39:05.574345 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.574128 4611 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="17af283e-b12c-4ed3-abf4-7a4944d1c4f2" volumeName="kubernetes.io/projected/17af283e-b12c-4ed3-abf4-7a4944d1c4f2-bound-sa-token" seLinuxMountContext="" Oct 30 20:39:05.574345 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.574145 4611 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6c3ba176-c9a5-4b6c-ae06-5593f68cf510" volumeName="kubernetes.io/projected/6c3ba176-c9a5-4b6c-ae06-5593f68cf510-kube-api-access-qhljq" seLinuxMountContext="" Oct 30 20:39:05.574345 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.574159 4611 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6c3ba176-c9a5-4b6c-ae06-5593f68cf510" volumeName="kubernetes.io/secret/6c3ba176-c9a5-4b6c-ae06-5593f68cf510-cert" seLinuxMountContext="" Oct 30 20:39:05.574345 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.574176 4611 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="acf9d9fb-1ae4-40ca-a83e-420a9fe40a01" volumeName="kubernetes.io/projected/acf9d9fb-1ae4-40ca-a83e-420a9fe40a01-kube-api-access-szhgd" seLinuxMountContext="" Oct 30 20:39:05.574345 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.574191 4611 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e277a3d9-3fe9-4454-9420-28afea76f745" volumeName="kubernetes.io/configmap/e277a3d9-3fe9-4454-9420-28afea76f745-cni-binary-copy" seLinuxMountContext="" Oct 30 20:39:05.574345 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.574204 4611 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="acf9d9fb-1ae4-40ca-a83e-420a9fe40a01" volumeName="kubernetes.io/configmap/acf9d9fb-1ae4-40ca-a83e-420a9fe40a01-iptables-alerter-script" seLinuxMountContext="" Oct 30 20:39:05.574345 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.574221 4611 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0aa2bb04-0552-45da-a54f-7e5064cf3113" volumeName="kubernetes.io/empty-dir/0aa2bb04-0552-45da-a54f-7e5064cf3113-tmp-dir" seLinuxMountContext="" Oct 30 20:39:05.574345 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.574234 4611 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="17af283e-b12c-4ed3-abf4-7a4944d1c4f2" volumeName="kubernetes.io/empty-dir/17af283e-b12c-4ed3-abf4-7a4944d1c4f2-ca-trust-extracted" seLinuxMountContext="" Oct 30 20:39:05.574345 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.574251 4611 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="17af283e-b12c-4ed3-abf4-7a4944d1c4f2" volumeName="kubernetes.io/secret/17af283e-b12c-4ed3-abf4-7a4944d1c4f2-installation-pull-secrets" seLinuxMountContext="" Oct 30 20:39:05.574345 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.574264 4611 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6b88d0a3-97c0-4adf-97a7-2e05d5cef781" volumeName="kubernetes.io/empty-dir/6b88d0a3-97c0-4adf-97a7-2e05d5cef781-etc-tuned" seLinuxMountContext="" Oct 30 20:39:05.574345 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.574278 4611 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7e2574c1-b59c-453c-9501-25ec13a1ccad" volumeName="kubernetes.io/configmap/7e2574c1-b59c-453c-9501-25ec13a1ccad-env-overrides" seLinuxMountContext="" Oct 30 20:39:05.574345 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.574295 4611 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="da50258d-eef3-4976-8e22-be0e936b9993" volumeName="kubernetes.io/secret/da50258d-eef3-4976-8e22-be0e936b9993-agent-certs" seLinuxMountContext="" Oct 30 20:39:05.574345 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.574308 4611 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e277a3d9-3fe9-4454-9420-28afea76f745" volumeName="kubernetes.io/projected/e277a3d9-3fe9-4454-9420-28afea76f745-kube-api-access-bg7g6" seLinuxMountContext="" Oct 30 20:39:05.575557 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.574325 4611 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0aa2bb04-0552-45da-a54f-7e5064cf3113" volumeName="kubernetes.io/secret/0aa2bb04-0552-45da-a54f-7e5064cf3113-metrics-tls" seLinuxMountContext="" Oct 30 20:39:05.576403 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.575565 4611 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6b88d0a3-97c0-4adf-97a7-2e05d5cef781" volumeName="kubernetes.io/projected/6b88d0a3-97c0-4adf-97a7-2e05d5cef781-kube-api-access-mslf8" seLinuxMountContext="" Oct 30 20:39:05.576403 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.576403 4611 reconstruct.go:97] "Volume reconstruction finished" Oct 30 20:39:05.576403 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.576409 4611 reconciler.go:29] "Reconciler: start to sync state" Oct 30 20:39:05.580405 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.580160 4611 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Oct 30 20:39:05.581645 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.581573 4611 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 30 20:39:05.586749 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.586729 4611 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7w5wz_7e2574c1-b59c-453c-9501-25ec13a1ccad/ovn-acl-logging/0.log" Oct 30 20:39:05.587240 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.587219 4611 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7w5wz_7e2574c1-b59c-453c-9501-25ec13a1ccad/ovn-acl-logging/0.log" Oct 30 20:39:05.598149 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.598123 4611 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 30 20:39:05.598149 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.598151 4611 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 30 20:39:05.598267 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.598171 4611 kubelet.go:2521] "Starting kubelet main sync loop" Oct 30 20:39:05.598267 ip-10-0-133-192 kubenswrapper[4611]: E1030 20:39:05.598211 4611 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Oct 30 20:39:05.601576 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.601554 4611 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 20:39:05.617233 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.617200 4611 generic.go:356] "Generic (PLEG): container finished" podID="ea0e353b-8cb1-4759-9f47-03f788f256d3" containerID="9d9de1dac3c841a5c65f0990eb6038317a82d39b037aeb1700ab7401e474df06" exitCode=0 Oct 30 20:39:05.617233 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.617232 4611 generic.go:356] "Generic (PLEG): container finished" podID="ea0e353b-8cb1-4759-9f47-03f788f256d3" containerID="356f44059a64aae0d8e96477590362d61a9f5e8ff5231740ca428714df820265" exitCode=0 Oct 30 20:39:05.617341 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.617245 4611 generic.go:356] "Generic (PLEG): container finished" podID="ea0e353b-8cb1-4759-9f47-03f788f256d3" containerID="b68ac9cc1f72f49d199521fc0937d68ad3790a9dd89d9b1f724816fa2de873c7" exitCode=0 Oct 30 20:39:05.617341 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.617252 4611 generic.go:356] "Generic (PLEG): container finished" podID="ea0e353b-8cb1-4759-9f47-03f788f256d3" containerID="f0cbfc41c587c7da1088a5340745e9d8d0aa4d404ae1b997b301149751763a6b" exitCode=0 Oct 30 20:39:05.617341 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.617257 4611 generic.go:356] "Generic (PLEG): container finished" podID="ea0e353b-8cb1-4759-9f47-03f788f256d3" containerID="4c9ade534cca4a562c797d122466d8a245afc6209b7c9d92bca07553602c797b" exitCode=0 Oct 30 20:39:05.617341 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.617262 4611 generic.go:356] "Generic (PLEG): container finished" podID="ea0e353b-8cb1-4759-9f47-03f788f256d3" containerID="c7fcd96fde6e28a72528d2556b5f3039d7e5ce747b8b21b2cf8ca64769055262" exitCode=0 Oct 30 20:39:05.629546 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.629524 4611 generic.go:356] "Generic (PLEG): container finished" podID="a211a81db7289ccbd1f6896c27ff51c5" containerID="16291fa955d1e42357e24beb74fbb648db543686399f9a79bdcf58dcd749e9bb" exitCode=0 Oct 30 20:39:05.633280 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.633262 4611 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7w5wz_7e2574c1-b59c-453c-9501-25ec13a1ccad/ovn-acl-logging/0.log" Oct 30 20:39:05.633651 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.633626 4611 generic.go:356] "Generic (PLEG): container finished" podID="7e2574c1-b59c-453c-9501-25ec13a1ccad" containerID="18eb59f6bb86ae195c78e6f8c8bad822a23d47fd7a6e91c9379af2f71aea686c" exitCode=1 Oct 30 20:39:05.642027 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.642010 4611 manager.go:324] Recovery completed Oct 30 20:39:05.657356 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.657338 4611 cpu_manager.go:226] "Starting" policy="none" Oct 30 20:39:05.657356 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.657350 4611 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 30 20:39:05.657480 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.657366 4611 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 30 20:39:05.657480 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.657468 4611 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Oct 30 20:39:05.657602 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.657478 4611 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Oct 30 20:39:05.657602 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.657494 4611 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Oct 30 20:39:05.657602 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.657499 4611 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Oct 30 20:39:05.657602 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.657504 4611 policy_none.go:50] "Start" Oct 30 20:39:05.657602 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.657531 4611 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 30 20:39:05.657602 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.657542 4611 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 30 20:39:05.657765 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.657653 4611 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Oct 30 20:39:05.657765 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.657660 4611 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Oct 30 20:39:05.657765 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.657665 4611 policy_none.go:44] "Start" Oct 30 20:39:05.661522 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.661484 4611 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 30 20:39:05.664830 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.664810 4611 manager.go:341] "Starting Device Plugin manager" Oct 30 20:39:05.665259 ip-10-0-133-192 kubenswrapper[4611]: E1030 20:39:05.664839 4611 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 30 20:39:05.665328 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.665262 4611 server.go:87] "Starting device plugin registration server" Oct 30 20:39:05.665553 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.665542 4611 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 30 20:39:05.665616 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.665555 4611 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 30 20:39:05.665663 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.665649 4611 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 30 20:39:05.665784 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.665753 4611 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 30 20:39:05.665784 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.665789 4611 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 30 20:39:05.665784 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.665796 4611 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 30 20:39:05.665982 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.665843 4611 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T20:39:05.665779003Z","UUID":"449f7a28-8c18-4567-942d-bbb2189d5f2c","Handler":null,"Name":"","Endpoint":""} Oct 30 20:39:05.667148 ip-10-0-133-192 kubenswrapper[4611]: E1030 20:39:05.667122 4611 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 30 20:39:05.667382 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.667362 4611 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 30 20:39:05.667456 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.667434 4611 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 30 20:39:05.699178 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.699129 4611 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-133-192.us-east-2.compute.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-192.us-east-2.compute.internal"] Oct 30 20:39:05.699595 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.699562 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-192.us-east-2.compute.internal" event={"ID":"10e325ce488448e0259da3eb69cd60ba","Type":"ContainerStarted","Data":"fa27a51d81fd9981d0e02bd8f149f5d9570bdf79dea62f8ce876c8bbf05eb483"} Oct 30 20:39:05.699690 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.699603 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-192.us-east-2.compute.internal" event={"ID":"10e325ce488448e0259da3eb69cd60ba","Type":"ContainerStarted","Data":"fb0f2c227bb739438aaf12d4ffc31d80eab2c7f4b5776fd30ffc16448932514e"} Oct 30 20:39:05.699739 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.699688 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-192.us-east-2.compute.internal" event={"ID":"a211a81db7289ccbd1f6896c27ff51c5","Type":"ContainerStarted","Data":"54bf3937147dd87f3afb0aac3fe6c6a72f66f808c1e4ea40958793c00ebdacbb"} Oct 30 20:39:05.699739 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.699709 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-192.us-east-2.compute.internal" event={"ID":"a211a81db7289ccbd1f6896c27ff51c5","Type":"ContainerDied","Data":"16291fa955d1e42357e24beb74fbb648db543686399f9a79bdcf58dcd749e9bb"} Oct 30 20:39:05.699739 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.699728 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-192.us-east-2.compute.internal" event={"ID":"a211a81db7289ccbd1f6896c27ff51c5","Type":"ContainerStarted","Data":"160c0397f4741ec3a44de2d9a323a0a3713a7ab5c7ba1ac1f92e56c2b4711663"} Oct 30 20:39:05.699859 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.699728 4611 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-192.us-east-2.compute.internal" Oct 30 20:39:05.699901 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.699862 4611 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-192.us-east-2.compute.internal" Oct 30 20:39:05.711502 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.711483 4611 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 20:39:05.711580 ip-10-0-133-192 kubenswrapper[4611]: E1030 20:39:05.711532 4611 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-133-192.us-east-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-192.us-east-2.compute.internal" Oct 30 20:39:05.711855 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.711843 4611 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 20:39:05.711894 ip-10-0-133-192 kubenswrapper[4611]: E1030 20:39:05.711873 4611 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-133-192.us-east-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-192.us-east-2.compute.internal" Oct 30 20:39:05.766048 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.766024 4611 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 20:39:05.767198 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.767182 4611 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-192.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 30 20:39:05.767290 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.767213 4611 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-192.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 30 20:39:05.767290 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.767228 4611 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-192.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 30 20:39:05.767290 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.767267 4611 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-133-192.us-east-2.compute.internal" Oct 30 20:39:05.780374 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.780328 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/10e325ce488448e0259da3eb69cd60ba-config\") pod \"kube-apiserver-proxy-ip-10-0-133-192.us-east-2.compute.internal\" (UID: \"10e325ce488448e0259da3eb69cd60ba\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-192.us-east-2.compute.internal" Oct 30 20:39:05.780374 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.780352 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a211a81db7289ccbd1f6896c27ff51c5-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-192.us-east-2.compute.internal\" (UID: \"a211a81db7289ccbd1f6896c27ff51c5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-192.us-east-2.compute.internal" Oct 30 20:39:05.780374 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.780367 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a211a81db7289ccbd1f6896c27ff51c5-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-192.us-east-2.compute.internal\" (UID: \"a211a81db7289ccbd1f6896c27ff51c5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-192.us-east-2.compute.internal" Oct 30 20:39:05.780702 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.780688 4611 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-133-192.us-east-2.compute.internal" Oct 30 20:39:05.780742 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.780732 4611 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-133-192.us-east-2.compute.internal" Oct 30 20:39:05.881365 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.881340 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/10e325ce488448e0259da3eb69cd60ba-config\") pod \"kube-apiserver-proxy-ip-10-0-133-192.us-east-2.compute.internal\" (UID: \"10e325ce488448e0259da3eb69cd60ba\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-192.us-east-2.compute.internal" Oct 30 20:39:05.881365 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.881366 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a211a81db7289ccbd1f6896c27ff51c5-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-192.us-east-2.compute.internal\" (UID: \"a211a81db7289ccbd1f6896c27ff51c5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-192.us-east-2.compute.internal" Oct 30 20:39:05.881504 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.881383 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a211a81db7289ccbd1f6896c27ff51c5-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-192.us-east-2.compute.internal\" (UID: \"a211a81db7289ccbd1f6896c27ff51c5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-192.us-east-2.compute.internal" Oct 30 20:39:05.881504 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.881431 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a211a81db7289ccbd1f6896c27ff51c5-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-192.us-east-2.compute.internal\" (UID: \"a211a81db7289ccbd1f6896c27ff51c5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-192.us-east-2.compute.internal" Oct 30 20:39:05.881504 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.881443 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/10e325ce488448e0259da3eb69cd60ba-config\") pod \"kube-apiserver-proxy-ip-10-0-133-192.us-east-2.compute.internal\" (UID: \"10e325ce488448e0259da3eb69cd60ba\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-192.us-east-2.compute.internal" Oct 30 20:39:05.881504 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:05.881447 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a211a81db7289ccbd1f6896c27ff51c5-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-192.us-east-2.compute.internal\" (UID: \"a211a81db7289ccbd1f6896c27ff51c5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-192.us-east-2.compute.internal" Oct 30 20:39:06.555393 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.555347 4611 apiserver.go:52] "Watching apiserver" Oct 30 20:39:06.563022 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.563001 4611 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 30 20:39:06.563412 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.563396 4611 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/node-resolver-lc599","openshift-image-registry/image-registry-8984b54b7-skmcd","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t7h2z","openshift-multus/multus-additional-cni-plugins-nxmjg","openshift-cluster-node-tuning-operator/tuned-2qgck","openshift-network-diagnostics/network-check-target-jp2cr","openshift-ingress-canary/ingress-canary-gmq97","openshift-insights/insights-runtime-extractor-hwzc6","openshift-network-operator/iptables-alerter-kn2t9","kube-system/konnectivity-agent-jfxnj","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-192.us-east-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-133-192.us-east-2.compute.internal","openshift-multus/network-metrics-daemon-pdcfk","openshift-multus/multus-c4fvc","openshift-ovn-kubernetes/ovnkube-node-7w5wz","openshift-image-registry/node-ca-hsv4j","openshift-dns/dns-default-2sccz","kube-system/global-pull-secret-syncer-857r2"] Oct 30 20:39:06.568868 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.568838 4611 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 30 20:39:06.569103 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.569079 4611 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-4gdbd\"" Oct 30 20:39:06.569381 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.569336 4611 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-b7wk5\"" Oct 30 20:39:06.569572 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.569389 4611 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 20:39:06.569692 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.569631 4611 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 30 20:39:06.569935 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.569885 4611 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 30 20:39:06.570204 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.570186 4611 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 30 20:39:06.570289 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.570220 4611 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-g4v6z\"" Oct 30 20:39:06.570289 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.570221 4611 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-8b74p\"" Oct 30 20:39:06.570289 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.570238 4611 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 30 20:39:06.570289 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.570249 4611 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 30 20:39:06.570289 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.570270 4611 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-kszdw\"" Oct 30 20:39:06.570593 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.570313 4611 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 30 20:39:06.570593 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.570329 4611 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 30 20:39:06.570593 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.570341 4611 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 30 20:39:06.570593 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.570354 4611 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-ndwvd\"" Oct 30 20:39:06.570593 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.570383 4611 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 30 20:39:06.570593 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.570412 4611 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 30 20:39:06.570593 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.570417 4611 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 30 20:39:06.570593 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.570446 4611 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 30 20:39:06.570593 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.570395 4611 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 30 20:39:06.570593 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.570484 4611 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 30 20:39:06.570593 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.570484 4611 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 30 20:39:06.570593 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.570552 4611 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-d5kr7\"" Oct 30 20:39:06.571140 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.570621 4611 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 30 20:39:06.571140 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.570676 4611 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 30 20:39:06.571140 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.570675 4611 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 30 20:39:06.571140 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.570760 4611 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 30 20:39:06.571140 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.570816 4611 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 30 20:39:06.571140 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.570891 4611 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 30 20:39:06.571337 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.571194 4611 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 30 20:39:06.571337 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.571310 4611 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-hwzc6" Oct 30 20:39:06.571550 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.571528 4611 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-w89q8\"" Oct 30 20:39:06.571694 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.571679 4611 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 30 20:39:06.571769 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.571740 4611 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 30 20:39:06.571829 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.571782 4611 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 30 20:39:06.571829 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.571790 4611 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-7chw8\"" Oct 30 20:39:06.571923 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.571857 4611 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-hh9qp\"" Oct 30 20:39:06.572695 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.572678 4611 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 30 20:39:06.572695 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.572692 4611 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 30 20:39:06.573022 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.573002 4611 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-4khcx\"" Oct 30 20:39:06.573172 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.573155 4611 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 30 20:39:06.573494 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.573477 4611 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-5p7gr\"" Oct 30 20:39:06.573566 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.573482 4611 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 20:39:06.573566 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.573551 4611 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 30 20:39:06.574542 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.574529 4611 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 30 20:39:06.574602 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.574556 4611 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 30 20:39:06.574602 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.574564 4611 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-2twjb\"" Oct 30 20:39:06.575009 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.574993 4611 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 30 20:39:06.575063 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.574983 4611 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 30 20:39:06.575223 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.575209 4611 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-pl5jd\"" Oct 30 20:39:06.575223 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.575215 4611 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 30 20:39:06.575606 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.575592 4611 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 30 20:39:06.576349 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.576336 4611 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 30 20:39:06.576796 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.576781 4611 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Oct 30 20:39:06.576861 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.576809 4611 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 30 20:39:06.576909 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.576873 4611 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Oct 30 20:39:06.576909 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.576895 4611 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-bv2n9\"" Oct 30 20:39:06.577002 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.576923 4611 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-cr4sq\"" Oct 30 20:39:06.577112 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.577054 4611 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Oct 30 20:39:06.578375 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.578357 4611 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Oct 30 20:39:06.584959 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.584943 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/7e2574c1-b59c-453c-9501-25ec13a1ccad-host-kubelet\") pod \"ovnkube-node-7w5wz\" (UID: \"7e2574c1-b59c-453c-9501-25ec13a1ccad\") " pod="openshift-ovn-kubernetes/ovnkube-node-7w5wz" Oct 30 20:39:06.585009 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.584967 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/7e2574c1-b59c-453c-9501-25ec13a1ccad-host-cni-netd\") pod \"ovnkube-node-7w5wz\" (UID: \"7e2574c1-b59c-453c-9501-25ec13a1ccad\") " pod="openshift-ovn-kubernetes/ovnkube-node-7w5wz" Oct 30 20:39:06.585009 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.584983 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e277a3d9-3fe9-4454-9420-28afea76f745-cnibin\") pod \"multus-c4fvc\" (UID: \"e277a3d9-3fe9-4454-9420-28afea76f745\") " pod="openshift-multus/multus-c4fvc" Oct 30 20:39:06.585009 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.584998 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f5f694f2-6f18-409b-b777-2dd7a0eb12b2-registration-dir\") pod \"aws-ebs-csi-driver-node-t7h2z\" (UID: \"f5f694f2-6f18-409b-b777-2dd7a0eb12b2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t7h2z" Oct 30 20:39:06.585131 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.585018 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/6b88d0a3-97c0-4adf-97a7-2e05d5cef781-etc-sysctl-d\") pod \"tuned-2qgck\" (UID: \"6b88d0a3-97c0-4adf-97a7-2e05d5cef781\") " pod="openshift-cluster-node-tuning-operator/tuned-2qgck" Oct 30 20:39:06.585131 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.585051 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/6b88d0a3-97c0-4adf-97a7-2e05d5cef781-etc-systemd\") pod \"tuned-2qgck\" (UID: \"6b88d0a3-97c0-4adf-97a7-2e05d5cef781\") " pod="openshift-cluster-node-tuning-operator/tuned-2qgck" Oct 30 20:39:06.585131 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.585091 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/bf21cf19-4475-41c3-9153-44d8fb329cdb-dbus\") pod \"global-pull-secret-syncer-857r2\" (UID: \"bf21cf19-4475-41c3-9153-44d8fb329cdb\") " pod="kube-system/global-pull-secret-syncer-857r2" Oct 30 20:39:06.585131 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.585124 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/35d51f79-80a9-4f9d-970e-6a35a34976a2-serviceca\") pod \"node-ca-hsv4j\" (UID: \"35d51f79-80a9-4f9d-970e-6a35a34976a2\") " pod="openshift-image-registry/node-ca-hsv4j" Oct 30 20:39:06.585281 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.585253 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e277a3d9-3fe9-4454-9420-28afea76f745-host-var-lib-kubelet\") pod \"multus-c4fvc\" (UID: \"e277a3d9-3fe9-4454-9420-28afea76f745\") " pod="openshift-multus/multus-c4fvc" Oct 30 20:39:06.585335 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.585293 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/e277a3d9-3fe9-4454-9420-28afea76f745-multus-daemon-config\") pod \"multus-c4fvc\" (UID: \"e277a3d9-3fe9-4454-9420-28afea76f745\") " pod="openshift-multus/multus-c4fvc" Oct 30 20:39:06.585335 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.585327 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ea0e353b-8cb1-4759-9f47-03f788f256d3-system-cni-dir\") pod \"multus-additional-cni-plugins-nxmjg\" (UID: \"ea0e353b-8cb1-4759-9f47-03f788f256d3\") " pod="openshift-multus/multus-additional-cni-plugins-nxmjg" Oct 30 20:39:06.585420 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.585358 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ea0e353b-8cb1-4759-9f47-03f788f256d3-cni-binary-copy\") pod \"multus-additional-cni-plugins-nxmjg\" (UID: \"ea0e353b-8cb1-4759-9f47-03f788f256d3\") " pod="openshift-multus/multus-additional-cni-plugins-nxmjg" Oct 30 20:39:06.585420 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.585414 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jk5kq\" (UniqueName: \"kubernetes.io/projected/7e2574c1-b59c-453c-9501-25ec13a1ccad-kube-api-access-jk5kq\") pod \"ovnkube-node-7w5wz\" (UID: \"7e2574c1-b59c-453c-9501-25ec13a1ccad\") " pod="openshift-ovn-kubernetes/ovnkube-node-7w5wz" Oct 30 20:39:06.585486 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.585439 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7e2574c1-b59c-453c-9501-25ec13a1ccad-run-openvswitch\") pod \"ovnkube-node-7w5wz\" (UID: \"7e2574c1-b59c-453c-9501-25ec13a1ccad\") " pod="openshift-ovn-kubernetes/ovnkube-node-7w5wz" Oct 30 20:39:06.585486 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.585454 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/7e2574c1-b59c-453c-9501-25ec13a1ccad-env-overrides\") pod \"ovnkube-node-7w5wz\" (UID: \"7e2574c1-b59c-453c-9501-25ec13a1ccad\") " pod="openshift-ovn-kubernetes/ovnkube-node-7w5wz" Oct 30 20:39:06.585486 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.585480 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f5f694f2-6f18-409b-b777-2dd7a0eb12b2-socket-dir\") pod \"aws-ebs-csi-driver-node-t7h2z\" (UID: \"f5f694f2-6f18-409b-b777-2dd7a0eb12b2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t7h2z" Oct 30 20:39:06.585649 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.585539 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/6b88d0a3-97c0-4adf-97a7-2e05d5cef781-etc-tuned\") pod \"tuned-2qgck\" (UID: \"6b88d0a3-97c0-4adf-97a7-2e05d5cef781\") " pod="openshift-cluster-node-tuning-operator/tuned-2qgck" Oct 30 20:39:06.585649 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.585626 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/7e2574c1-b59c-453c-9501-25ec13a1ccad-ovn-node-metrics-cert\") pod \"ovnkube-node-7w5wz\" (UID: \"7e2574c1-b59c-453c-9501-25ec13a1ccad\") " pod="openshift-ovn-kubernetes/ovnkube-node-7w5wz" Oct 30 20:39:06.585649 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.585633 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/6b88d0a3-97c0-4adf-97a7-2e05d5cef781-etc-tuned\") pod \"tuned-2qgck\" (UID: \"6b88d0a3-97c0-4adf-97a7-2e05d5cef781\") " pod="openshift-cluster-node-tuning-operator/tuned-2qgck" Oct 30 20:39:06.585779 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.585673 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-szhgd\" (UniqueName: \"kubernetes.io/projected/acf9d9fb-1ae4-40ca-a83e-420a9fe40a01-kube-api-access-szhgd\") pod \"iptables-alerter-kn2t9\" (UID: \"acf9d9fb-1ae4-40ca-a83e-420a9fe40a01\") " pod="openshift-network-operator/iptables-alerter-kn2t9" Oct 30 20:39:06.585779 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.585768 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/e277a3d9-3fe9-4454-9420-28afea76f745-multus-daemon-config\") pod \"multus-c4fvc\" (UID: \"e277a3d9-3fe9-4454-9420-28afea76f745\") " pod="openshift-multus/multus-c4fvc" Oct 30 20:39:06.585873 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.585795 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/7e2574c1-b59c-453c-9501-25ec13a1ccad-env-overrides\") pod \"ovnkube-node-7w5wz\" (UID: \"7e2574c1-b59c-453c-9501-25ec13a1ccad\") " pod="openshift-ovn-kubernetes/ovnkube-node-7w5wz" Oct 30 20:39:06.585873 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.585831 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/6b88d0a3-97c0-4adf-97a7-2e05d5cef781-etc-modprobe-d\") pod \"tuned-2qgck\" (UID: \"6b88d0a3-97c0-4adf-97a7-2e05d5cef781\") " pod="openshift-cluster-node-tuning-operator/tuned-2qgck" Oct 30 20:39:06.585873 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.585834 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/35d51f79-80a9-4f9d-970e-6a35a34976a2-serviceca\") pod \"node-ca-hsv4j\" (UID: \"35d51f79-80a9-4f9d-970e-6a35a34976a2\") " pod="openshift-image-registry/node-ca-hsv4j" Oct 30 20:39:06.586016 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.585876 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6b88d0a3-97c0-4adf-97a7-2e05d5cef781-sys\") pod \"tuned-2qgck\" (UID: \"6b88d0a3-97c0-4adf-97a7-2e05d5cef781\") " pod="openshift-cluster-node-tuning-operator/tuned-2qgck" Oct 30 20:39:06.586016 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.585896 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ea0e353b-8cb1-4759-9f47-03f788f256d3-cni-binary-copy\") pod \"multus-additional-cni-plugins-nxmjg\" (UID: \"ea0e353b-8cb1-4759-9f47-03f788f256d3\") " pod="openshift-multus/multus-additional-cni-plugins-nxmjg" Oct 30 20:39:06.586016 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.585912 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7e2574c1-b59c-453c-9501-25ec13a1ccad-etc-openvswitch\") pod \"ovnkube-node-7w5wz\" (UID: \"7e2574c1-b59c-453c-9501-25ec13a1ccad\") " pod="openshift-ovn-kubernetes/ovnkube-node-7w5wz" Oct 30 20:39:06.586016 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.585966 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e277a3d9-3fe9-4454-9420-28afea76f745-host-var-lib-cni-multus\") pod \"multus-c4fvc\" (UID: \"e277a3d9-3fe9-4454-9420-28afea76f745\") " pod="openshift-multus/multus-c4fvc" Oct 30 20:39:06.586016 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.585992 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/6b88d0a3-97c0-4adf-97a7-2e05d5cef781-etc-sysctl-conf\") pod \"tuned-2qgck\" (UID: \"6b88d0a3-97c0-4adf-97a7-2e05d5cef781\") " pod="openshift-cluster-node-tuning-operator/tuned-2qgck" Oct 30 20:39:06.586194 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.586020 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6b88d0a3-97c0-4adf-97a7-2e05d5cef781-lib-modules\") pod \"tuned-2qgck\" (UID: \"6b88d0a3-97c0-4adf-97a7-2e05d5cef781\") " pod="openshift-cluster-node-tuning-operator/tuned-2qgck" Oct 30 20:39:06.586194 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.586048 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6b88d0a3-97c0-4adf-97a7-2e05d5cef781-host\") pod \"tuned-2qgck\" (UID: \"6b88d0a3-97c0-4adf-97a7-2e05d5cef781\") " pod="openshift-cluster-node-tuning-operator/tuned-2qgck" Oct 30 20:39:06.586194 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.586080 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/7e2574c1-b59c-453c-9501-25ec13a1ccad-run-ovn\") pod \"ovnkube-node-7w5wz\" (UID: \"7e2574c1-b59c-453c-9501-25ec13a1ccad\") " pod="openshift-ovn-kubernetes/ovnkube-node-7w5wz" Oct 30 20:39:06.586194 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.586093 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/7e2574c1-b59c-453c-9501-25ec13a1ccad-ovn-node-metrics-cert\") pod \"ovnkube-node-7w5wz\" (UID: \"7e2574c1-b59c-453c-9501-25ec13a1ccad\") " pod="openshift-ovn-kubernetes/ovnkube-node-7w5wz" Oct 30 20:39:06.586194 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.586102 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7e2574c1-b59c-453c-9501-25ec13a1ccad-host-run-ovn-kubernetes\") pod \"ovnkube-node-7w5wz\" (UID: \"7e2574c1-b59c-453c-9501-25ec13a1ccad\") " pod="openshift-ovn-kubernetes/ovnkube-node-7w5wz" Oct 30 20:39:06.586194 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.586123 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/acf9d9fb-1ae4-40ca-a83e-420a9fe40a01-iptables-alerter-script\") pod \"iptables-alerter-kn2t9\" (UID: \"acf9d9fb-1ae4-40ca-a83e-420a9fe40a01\") " pod="openshift-network-operator/iptables-alerter-kn2t9" Oct 30 20:39:06.586194 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.586151 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e277a3d9-3fe9-4454-9420-28afea76f745-host-run-k8s-cni-cncf-io\") pod \"multus-c4fvc\" (UID: \"e277a3d9-3fe9-4454-9420-28afea76f745\") " pod="openshift-multus/multus-c4fvc" Oct 30 20:39:06.586470 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.586196 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e277a3d9-3fe9-4454-9420-28afea76f745-host-run-netns\") pod \"multus-c4fvc\" (UID: \"e277a3d9-3fe9-4454-9420-28afea76f745\") " pod="openshift-multus/multus-c4fvc" Oct 30 20:39:06.586470 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.586214 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f5f694f2-6f18-409b-b777-2dd7a0eb12b2-device-dir\") pod \"aws-ebs-csi-driver-node-t7h2z\" (UID: \"f5f694f2-6f18-409b-b777-2dd7a0eb12b2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t7h2z" Oct 30 20:39:06.586470 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.586232 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rjj44\" (UniqueName: \"kubernetes.io/projected/ea0e353b-8cb1-4759-9f47-03f788f256d3-kube-api-access-rjj44\") pod \"multus-additional-cni-plugins-nxmjg\" (UID: \"ea0e353b-8cb1-4759-9f47-03f788f256d3\") " pod="openshift-multus/multus-additional-cni-plugins-nxmjg" Oct 30 20:39:06.586470 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.586285 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6b88d0a3-97c0-4adf-97a7-2e05d5cef781-etc-kubernetes\") pod \"tuned-2qgck\" (UID: \"6b88d0a3-97c0-4adf-97a7-2e05d5cef781\") " pod="openshift-cluster-node-tuning-operator/tuned-2qgck" Oct 30 20:39:06.586470 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.586315 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/acf9d9fb-1ae4-40ca-a83e-420a9fe40a01-iptables-alerter-script\") pod \"iptables-alerter-kn2t9\" (UID: \"acf9d9fb-1ae4-40ca-a83e-420a9fe40a01\") " pod="openshift-network-operator/iptables-alerter-kn2t9" Oct 30 20:39:06.586470 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.586323 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/7e2574c1-b59c-453c-9501-25ec13a1ccad-ovnkube-config\") pod \"ovnkube-node-7w5wz\" (UID: \"7e2574c1-b59c-453c-9501-25ec13a1ccad\") " pod="openshift-ovn-kubernetes/ovnkube-node-7w5wz" Oct 30 20:39:06.586470 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.586356 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6b88d0a3-97c0-4adf-97a7-2e05d5cef781-tmp\") pod \"tuned-2qgck\" (UID: \"6b88d0a3-97c0-4adf-97a7-2e05d5cef781\") " pod="openshift-cluster-node-tuning-operator/tuned-2qgck" Oct 30 20:39:06.586470 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.586373 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6hvqv\" (UniqueName: \"kubernetes.io/projected/0dde7eae-1962-4559-89e9-0c67b1165914-kube-api-access-6hvqv\") pod \"network-check-target-jp2cr\" (UID: \"0dde7eae-1962-4559-89e9-0c67b1165914\") " pod="openshift-network-diagnostics/network-check-target-jp2cr" Oct 30 20:39:06.586470 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.586442 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6b88d0a3-97c0-4adf-97a7-2e05d5cef781-tmp\") pod \"tuned-2qgck\" (UID: \"6b88d0a3-97c0-4adf-97a7-2e05d5cef781\") " pod="openshift-cluster-node-tuning-operator/tuned-2qgck" Oct 30 20:39:06.586470 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.586465 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/bf21cf19-4475-41c3-9153-44d8fb329cdb-kubelet-config\") pod \"global-pull-secret-syncer-857r2\" (UID: \"bf21cf19-4475-41c3-9153-44d8fb329cdb\") " pod="kube-system/global-pull-secret-syncer-857r2" Oct 30 20:39:06.586852 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.586503 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cv9nx\" (UniqueName: \"kubernetes.io/projected/f5f694f2-6f18-409b-b777-2dd7a0eb12b2-kube-api-access-cv9nx\") pod \"aws-ebs-csi-driver-node-t7h2z\" (UID: \"f5f694f2-6f18-409b-b777-2dd7a0eb12b2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t7h2z" Oct 30 20:39:06.586852 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.586598 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/7e2574c1-b59c-453c-9501-25ec13a1ccad-ovnkube-config\") pod \"ovnkube-node-7w5wz\" (UID: \"7e2574c1-b59c-453c-9501-25ec13a1ccad\") " pod="openshift-ovn-kubernetes/ovnkube-node-7w5wz" Oct 30 20:39:06.586852 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.586628 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ea0e353b-8cb1-4759-9f47-03f788f256d3-tuning-conf-dir\") pod \"multus-additional-cni-plugins-nxmjg\" (UID: \"ea0e353b-8cb1-4759-9f47-03f788f256d3\") " pod="openshift-multus/multus-additional-cni-plugins-nxmjg" Oct 30 20:39:06.586852 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.586664 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/acf9d9fb-1ae4-40ca-a83e-420a9fe40a01-host-slash\") pod \"iptables-alerter-kn2t9\" (UID: \"acf9d9fb-1ae4-40ca-a83e-420a9fe40a01\") " pod="openshift-network-operator/iptables-alerter-kn2t9" Oct 30 20:39:06.586852 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.586696 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bg7g6\" (UniqueName: \"kubernetes.io/projected/e277a3d9-3fe9-4454-9420-28afea76f745-kube-api-access-bg7g6\") pod \"multus-c4fvc\" (UID: \"e277a3d9-3fe9-4454-9420-28afea76f745\") " pod="openshift-multus/multus-c4fvc" Oct 30 20:39:06.586852 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.586722 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/ea0e353b-8cb1-4759-9f47-03f788f256d3-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-nxmjg\" (UID: \"ea0e353b-8cb1-4759-9f47-03f788f256d3\") " pod="openshift-multus/multus-additional-cni-plugins-nxmjg" Oct 30 20:39:06.586852 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.586764 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mslf8\" (UniqueName: \"kubernetes.io/projected/6b88d0a3-97c0-4adf-97a7-2e05d5cef781-kube-api-access-mslf8\") pod \"tuned-2qgck\" (UID: \"6b88d0a3-97c0-4adf-97a7-2e05d5cef781\") " pod="openshift-cluster-node-tuning-operator/tuned-2qgck" Oct 30 20:39:06.586852 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.586789 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bf21cf19-4475-41c3-9153-44d8fb329cdb-original-pull-secret\") pod \"global-pull-secret-syncer-857r2\" (UID: \"bf21cf19-4475-41c3-9153-44d8fb329cdb\") " pod="kube-system/global-pull-secret-syncer-857r2" Oct 30 20:39:06.587165 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.586954 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/7e2574c1-b59c-453c-9501-25ec13a1ccad-log-socket\") pod \"ovnkube-node-7w5wz\" (UID: \"7e2574c1-b59c-453c-9501-25ec13a1ccad\") " pod="openshift-ovn-kubernetes/ovnkube-node-7w5wz" Oct 30 20:39:06.587165 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.586979 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bf21cf19-4475-41c3-9153-44d8fb329cdb-original-pull-secret\") pod \"global-pull-secret-syncer-857r2\" (UID: \"bf21cf19-4475-41c3-9153-44d8fb329cdb\") " pod="kube-system/global-pull-secret-syncer-857r2" Oct 30 20:39:06.587165 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.587001 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e277a3d9-3fe9-4454-9420-28afea76f745-multus-socket-dir-parent\") pod \"multus-c4fvc\" (UID: \"e277a3d9-3fe9-4454-9420-28afea76f745\") " pod="openshift-multus/multus-c4fvc" Oct 30 20:39:06.587165 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.586997 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/ea0e353b-8cb1-4759-9f47-03f788f256d3-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-nxmjg\" (UID: \"ea0e353b-8cb1-4759-9f47-03f788f256d3\") " pod="openshift-multus/multus-additional-cni-plugins-nxmjg" Oct 30 20:39:06.587165 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.587037 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e277a3d9-3fe9-4454-9420-28afea76f745-host-run-multus-certs\") pod \"multus-c4fvc\" (UID: \"e277a3d9-3fe9-4454-9420-28afea76f745\") " pod="openshift-multus/multus-c4fvc" Oct 30 20:39:06.587165 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.587066 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/6b88d0a3-97c0-4adf-97a7-2e05d5cef781-run\") pod \"tuned-2qgck\" (UID: \"6b88d0a3-97c0-4adf-97a7-2e05d5cef781\") " pod="openshift-cluster-node-tuning-operator/tuned-2qgck" Oct 30 20:39:06.587165 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.587100 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ea0e353b-8cb1-4759-9f47-03f788f256d3-cnibin\") pod \"multus-additional-cni-plugins-nxmjg\" (UID: \"ea0e353b-8cb1-4759-9f47-03f788f256d3\") " pod="openshift-multus/multus-additional-cni-plugins-nxmjg" Oct 30 20:39:06.587165 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.587122 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/ea0e353b-8cb1-4759-9f47-03f788f256d3-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-nxmjg\" (UID: \"ea0e353b-8cb1-4759-9f47-03f788f256d3\") " pod="openshift-multus/multus-additional-cni-plugins-nxmjg" Oct 30 20:39:06.587165 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.587140 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/7e2574c1-b59c-453c-9501-25ec13a1ccad-systemd-units\") pod \"ovnkube-node-7w5wz\" (UID: \"7e2574c1-b59c-453c-9501-25ec13a1ccad\") " pod="openshift-ovn-kubernetes/ovnkube-node-7w5wz" Oct 30 20:39:06.587165 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.587154 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7e2574c1-b59c-453c-9501-25ec13a1ccad-host-run-netns\") pod \"ovnkube-node-7w5wz\" (UID: \"7e2574c1-b59c-453c-9501-25ec13a1ccad\") " pod="openshift-ovn-kubernetes/ovnkube-node-7w5wz" Oct 30 20:39:06.587165 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.587168 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e277a3d9-3fe9-4454-9420-28afea76f745-etc-kubernetes\") pod \"multus-c4fvc\" (UID: \"e277a3d9-3fe9-4454-9420-28afea76f745\") " pod="openshift-multus/multus-c4fvc" Oct 30 20:39:06.587651 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.587190 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f5f694f2-6f18-409b-b777-2dd7a0eb12b2-kubelet-dir\") pod \"aws-ebs-csi-driver-node-t7h2z\" (UID: \"f5f694f2-6f18-409b-b777-2dd7a0eb12b2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t7h2z" Oct 30 20:39:06.587651 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.587205 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/6b88d0a3-97c0-4adf-97a7-2e05d5cef781-etc-sysconfig\") pod \"tuned-2qgck\" (UID: \"6b88d0a3-97c0-4adf-97a7-2e05d5cef781\") " pod="openshift-cluster-node-tuning-operator/tuned-2qgck" Oct 30 20:39:06.587651 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.587234 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e277a3d9-3fe9-4454-9420-28afea76f745-host-var-lib-cni-bin\") pod \"multus-c4fvc\" (UID: \"e277a3d9-3fe9-4454-9420-28afea76f745\") " pod="openshift-multus/multus-c4fvc" Oct 30 20:39:06.587651 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.587269 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f5f694f2-6f18-409b-b777-2dd7a0eb12b2-etc-selinux\") pod \"aws-ebs-csi-driver-node-t7h2z\" (UID: \"f5f694f2-6f18-409b-b777-2dd7a0eb12b2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t7h2z" Oct 30 20:39:06.587651 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.587299 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f5f694f2-6f18-409b-b777-2dd7a0eb12b2-sys-fs\") pod \"aws-ebs-csi-driver-node-t7h2z\" (UID: \"f5f694f2-6f18-409b-b777-2dd7a0eb12b2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t7h2z" Oct 30 20:39:06.587651 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.587317 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/7e2574c1-b59c-453c-9501-25ec13a1ccad-node-log\") pod \"ovnkube-node-7w5wz\" (UID: \"7e2574c1-b59c-453c-9501-25ec13a1ccad\") " pod="openshift-ovn-kubernetes/ovnkube-node-7w5wz" Oct 30 20:39:06.587651 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.587345 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e277a3d9-3fe9-4454-9420-28afea76f745-system-cni-dir\") pod \"multus-c4fvc\" (UID: \"e277a3d9-3fe9-4454-9420-28afea76f745\") " pod="openshift-multus/multus-c4fvc" Oct 30 20:39:06.587651 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.587372 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e277a3d9-3fe9-4454-9420-28afea76f745-cni-binary-copy\") pod \"multus-c4fvc\" (UID: \"e277a3d9-3fe9-4454-9420-28afea76f745\") " pod="openshift-multus/multus-c4fvc" Oct 30 20:39:06.587651 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.587384 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/ea0e353b-8cb1-4759-9f47-03f788f256d3-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-nxmjg\" (UID: \"ea0e353b-8cb1-4759-9f47-03f788f256d3\") " pod="openshift-multus/multus-additional-cni-plugins-nxmjg" Oct 30 20:39:06.587651 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.587388 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/35d51f79-80a9-4f9d-970e-6a35a34976a2-host\") pod \"node-ca-hsv4j\" (UID: \"35d51f79-80a9-4f9d-970e-6a35a34976a2\") " pod="openshift-image-registry/node-ca-hsv4j" Oct 30 20:39:06.587651 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.587432 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4ntjf\" (UniqueName: \"kubernetes.io/projected/35d51f79-80a9-4f9d-970e-6a35a34976a2-kube-api-access-4ntjf\") pod \"node-ca-hsv4j\" (UID: \"35d51f79-80a9-4f9d-970e-6a35a34976a2\") " pod="openshift-image-registry/node-ca-hsv4j" Oct 30 20:39:06.587651 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.587470 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7e2574c1-b59c-453c-9501-25ec13a1ccad-host-slash\") pod \"ovnkube-node-7w5wz\" (UID: \"7e2574c1-b59c-453c-9501-25ec13a1ccad\") " pod="openshift-ovn-kubernetes/ovnkube-node-7w5wz" Oct 30 20:39:06.587651 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.587503 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7e2574c1-b59c-453c-9501-25ec13a1ccad-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-7w5wz\" (UID: \"7e2574c1-b59c-453c-9501-25ec13a1ccad\") " pod="openshift-ovn-kubernetes/ovnkube-node-7w5wz" Oct 30 20:39:06.587651 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.587537 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e277a3d9-3fe9-4454-9420-28afea76f745-multus-cni-dir\") pod \"multus-c4fvc\" (UID: \"e277a3d9-3fe9-4454-9420-28afea76f745\") " pod="openshift-multus/multus-c4fvc" Oct 30 20:39:06.587651 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.587542 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e277a3d9-3fe9-4454-9420-28afea76f745-cni-binary-copy\") pod \"multus-c4fvc\" (UID: \"e277a3d9-3fe9-4454-9420-28afea76f745\") " pod="openshift-multus/multus-c4fvc" Oct 30 20:39:06.587651 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.587560 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e277a3d9-3fe9-4454-9420-28afea76f745-multus-conf-dir\") pod \"multus-c4fvc\" (UID: \"e277a3d9-3fe9-4454-9420-28afea76f745\") " pod="openshift-multus/multus-c4fvc" Oct 30 20:39:06.587651 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.587604 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7e2574c1-b59c-453c-9501-25ec13a1ccad-var-lib-openvswitch\") pod \"ovnkube-node-7w5wz\" (UID: \"7e2574c1-b59c-453c-9501-25ec13a1ccad\") " pod="openshift-ovn-kubernetes/ovnkube-node-7w5wz" Oct 30 20:39:06.587651 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.587642 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/7e2574c1-b59c-453c-9501-25ec13a1ccad-ovnkube-script-lib\") pod \"ovnkube-node-7w5wz\" (UID: \"7e2574c1-b59c-453c-9501-25ec13a1ccad\") " pod="openshift-ovn-kubernetes/ovnkube-node-7w5wz" Oct 30 20:39:06.588240 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.587704 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/7e2574c1-b59c-453c-9501-25ec13a1ccad-run-systemd\") pod \"ovnkube-node-7w5wz\" (UID: \"7e2574c1-b59c-453c-9501-25ec13a1ccad\") " pod="openshift-ovn-kubernetes/ovnkube-node-7w5wz" Oct 30 20:39:06.588240 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.587728 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7e2574c1-b59c-453c-9501-25ec13a1ccad-host-cni-bin\") pod \"ovnkube-node-7w5wz\" (UID: \"7e2574c1-b59c-453c-9501-25ec13a1ccad\") " pod="openshift-ovn-kubernetes/ovnkube-node-7w5wz" Oct 30 20:39:06.588240 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.587742 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e277a3d9-3fe9-4454-9420-28afea76f745-hostroot\") pod \"multus-c4fvc\" (UID: \"e277a3d9-3fe9-4454-9420-28afea76f745\") " pod="openshift-multus/multus-c4fvc" Oct 30 20:39:06.588240 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.587786 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6b88d0a3-97c0-4adf-97a7-2e05d5cef781-var-lib-kubelet\") pod \"tuned-2qgck\" (UID: \"6b88d0a3-97c0-4adf-97a7-2e05d5cef781\") " pod="openshift-cluster-node-tuning-operator/tuned-2qgck" Oct 30 20:39:06.588240 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.587878 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/7e2574c1-b59c-453c-9501-25ec13a1ccad-ovnkube-script-lib\") pod \"ovnkube-node-7w5wz\" (UID: \"7e2574c1-b59c-453c-9501-25ec13a1ccad\") " pod="openshift-ovn-kubernetes/ovnkube-node-7w5wz" Oct 30 20:39:06.597458 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.597435 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bg7g6\" (UniqueName: \"kubernetes.io/projected/e277a3d9-3fe9-4454-9420-28afea76f745-kube-api-access-bg7g6\") pod \"multus-c4fvc\" (UID: \"e277a3d9-3fe9-4454-9420-28afea76f745\") " pod="openshift-multus/multus-c4fvc" Oct 30 20:39:06.597582 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.597564 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-szhgd\" (UniqueName: \"kubernetes.io/projected/acf9d9fb-1ae4-40ca-a83e-420a9fe40a01-kube-api-access-szhgd\") pod \"iptables-alerter-kn2t9\" (UID: \"acf9d9fb-1ae4-40ca-a83e-420a9fe40a01\") " pod="openshift-network-operator/iptables-alerter-kn2t9" Oct 30 20:39:06.597640 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.597621 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cv9nx\" (UniqueName: \"kubernetes.io/projected/f5f694f2-6f18-409b-b777-2dd7a0eb12b2-kube-api-access-cv9nx\") pod \"aws-ebs-csi-driver-node-t7h2z\" (UID: \"f5f694f2-6f18-409b-b777-2dd7a0eb12b2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t7h2z" Oct 30 20:39:06.597723 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.597710 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4ntjf\" (UniqueName: \"kubernetes.io/projected/35d51f79-80a9-4f9d-970e-6a35a34976a2-kube-api-access-4ntjf\") pod \"node-ca-hsv4j\" (UID: \"35d51f79-80a9-4f9d-970e-6a35a34976a2\") " pod="openshift-image-registry/node-ca-hsv4j" Oct 30 20:39:06.597784 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.597770 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mslf8\" (UniqueName: \"kubernetes.io/projected/6b88d0a3-97c0-4adf-97a7-2e05d5cef781-kube-api-access-mslf8\") pod \"tuned-2qgck\" (UID: \"6b88d0a3-97c0-4adf-97a7-2e05d5cef781\") " pod="openshift-cluster-node-tuning-operator/tuned-2qgck" Oct 30 20:39:06.598237 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.598221 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jk5kq\" (UniqueName: \"kubernetes.io/projected/7e2574c1-b59c-453c-9501-25ec13a1ccad-kube-api-access-jk5kq\") pod \"ovnkube-node-7w5wz\" (UID: \"7e2574c1-b59c-453c-9501-25ec13a1ccad\") " pod="openshift-ovn-kubernetes/ovnkube-node-7w5wz" Oct 30 20:39:06.598379 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.598364 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rjj44\" (UniqueName: \"kubernetes.io/projected/ea0e353b-8cb1-4759-9f47-03f788f256d3-kube-api-access-rjj44\") pod \"multus-additional-cni-plugins-nxmjg\" (UID: \"ea0e353b-8cb1-4759-9f47-03f788f256d3\") " pod="openshift-multus/multus-additional-cni-plugins-nxmjg" Oct 30 20:39:06.598697 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.598681 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6hvqv\" (UniqueName: \"kubernetes.io/projected/0dde7eae-1962-4559-89e9-0c67b1165914-kube-api-access-6hvqv\") pod \"network-check-target-jp2cr\" (UID: \"0dde7eae-1962-4559-89e9-0c67b1165914\") " pod="openshift-network-diagnostics/network-check-target-jp2cr" Oct 30 20:39:06.638407 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.638388 4611 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-192.us-east-2.compute.internal" Oct 30 20:39:06.638504 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.638490 4611 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-192.us-east-2.compute.internal" Oct 30 20:39:06.649833 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.649816 4611 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 20:39:06.649905 ip-10-0-133-192 kubenswrapper[4611]: E1030 20:39:06.649861 4611 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-133-192.us-east-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-192.us-east-2.compute.internal" Oct 30 20:39:06.650337 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.650324 4611 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 20:39:06.650385 ip-10-0-133-192 kubenswrapper[4611]: E1030 20:39:06.650351 4611 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-133-192.us-east-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-192.us-east-2.compute.internal" Oct 30 20:39:06.667711 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.667692 4611 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 30 20:39:06.688799 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.688778 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9jszj\" (UniqueName: \"kubernetes.io/projected/c9257033-53bf-40fc-a1e9-67fd5901678b-kube-api-access-9jszj\") pod \"node-resolver-lc599\" (UID: \"c9257033-53bf-40fc-a1e9-67fd5901678b\") " pod="openshift-dns/node-resolver-lc599" Oct 30 20:39:06.688864 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.688805 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/17af283e-b12c-4ed3-abf4-7a4944d1c4f2-ca-trust-extracted\") pod \"image-registry-8984b54b7-skmcd\" (UID: \"17af283e-b12c-4ed3-abf4-7a4944d1c4f2\") " pod="openshift-image-registry/image-registry-8984b54b7-skmcd" Oct 30 20:39:06.688864 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.688820 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/7e2574c1-b59c-453c-9501-25ec13a1ccad-node-log\") pod \"ovnkube-node-7w5wz\" (UID: \"7e2574c1-b59c-453c-9501-25ec13a1ccad\") " pod="openshift-ovn-kubernetes/ovnkube-node-7w5wz" Oct 30 20:39:06.688864 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.688840 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e277a3d9-3fe9-4454-9420-28afea76f745-system-cni-dir\") pod \"multus-c4fvc\" (UID: \"e277a3d9-3fe9-4454-9420-28afea76f745\") " pod="openshift-multus/multus-c4fvc" Oct 30 20:39:06.688864 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.688862 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/35d51f79-80a9-4f9d-970e-6a35a34976a2-host\") pod \"node-ca-hsv4j\" (UID: \"35d51f79-80a9-4f9d-970e-6a35a34976a2\") " pod="openshift-image-registry/node-ca-hsv4j" Oct 30 20:39:06.689037 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.688885 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0915cfe8-4606-4f92-877b-ebe7f52f4187-metrics-certs\") pod \"network-metrics-daemon-pdcfk\" (UID: \"0915cfe8-4606-4f92-877b-ebe7f52f4187\") " pod="openshift-multus/network-metrics-daemon-pdcfk" Oct 30 20:39:06.689037 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.688890 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/7e2574c1-b59c-453c-9501-25ec13a1ccad-node-log\") pod \"ovnkube-node-7w5wz\" (UID: \"7e2574c1-b59c-453c-9501-25ec13a1ccad\") " pod="openshift-ovn-kubernetes/ovnkube-node-7w5wz" Oct 30 20:39:06.689037 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.688914 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/17af283e-b12c-4ed3-abf4-7a4944d1c4f2-ca-trust-extracted\") pod \"image-registry-8984b54b7-skmcd\" (UID: \"17af283e-b12c-4ed3-abf4-7a4944d1c4f2\") " pod="openshift-image-registry/image-registry-8984b54b7-skmcd" Oct 30 20:39:06.689037 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.688926 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e277a3d9-3fe9-4454-9420-28afea76f745-system-cni-dir\") pod \"multus-c4fvc\" (UID: \"e277a3d9-3fe9-4454-9420-28afea76f745\") " pod="openshift-multus/multus-c4fvc" Oct 30 20:39:06.689037 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.688940 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/35d51f79-80a9-4f9d-970e-6a35a34976a2-host\") pod \"node-ca-hsv4j\" (UID: \"35d51f79-80a9-4f9d-970e-6a35a34976a2\") " pod="openshift-image-registry/node-ca-hsv4j" Oct 30 20:39:06.689037 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.688969 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/da50258d-eef3-4976-8e22-be0e936b9993-agent-certs\") pod \"konnectivity-agent-jfxnj\" (UID: \"da50258d-eef3-4976-8e22-be0e936b9993\") " pod="kube-system/konnectivity-agent-jfxnj" Oct 30 20:39:06.689037 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.689002 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/da50258d-eef3-4976-8e22-be0e936b9993-konnectivity-ca\") pod \"konnectivity-agent-jfxnj\" (UID: \"da50258d-eef3-4976-8e22-be0e936b9993\") " pod="kube-system/konnectivity-agent-jfxnj" Oct 30 20:39:06.689037 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.689027 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7e2574c1-b59c-453c-9501-25ec13a1ccad-host-slash\") pod \"ovnkube-node-7w5wz\" (UID: \"7e2574c1-b59c-453c-9501-25ec13a1ccad\") " pod="openshift-ovn-kubernetes/ovnkube-node-7w5wz" Oct 30 20:39:06.689303 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.689054 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7e2574c1-b59c-453c-9501-25ec13a1ccad-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-7w5wz\" (UID: \"7e2574c1-b59c-453c-9501-25ec13a1ccad\") " pod="openshift-ovn-kubernetes/ovnkube-node-7w5wz" Oct 30 20:39:06.689303 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.689077 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e277a3d9-3fe9-4454-9420-28afea76f745-multus-cni-dir\") pod \"multus-c4fvc\" (UID: \"e277a3d9-3fe9-4454-9420-28afea76f745\") " pod="openshift-multus/multus-c4fvc" Oct 30 20:39:06.689303 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.689080 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7e2574c1-b59c-453c-9501-25ec13a1ccad-host-slash\") pod \"ovnkube-node-7w5wz\" (UID: \"7e2574c1-b59c-453c-9501-25ec13a1ccad\") " pod="openshift-ovn-kubernetes/ovnkube-node-7w5wz" Oct 30 20:39:06.689303 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.689100 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e277a3d9-3fe9-4454-9420-28afea76f745-multus-conf-dir\") pod \"multus-c4fvc\" (UID: \"e277a3d9-3fe9-4454-9420-28afea76f745\") " pod="openshift-multus/multus-c4fvc" Oct 30 20:39:06.689303 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.689102 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0915cfe8-4606-4f92-877b-ebe7f52f4187-metrics-certs\") pod \"network-metrics-daemon-pdcfk\" (UID: \"0915cfe8-4606-4f92-877b-ebe7f52f4187\") " pod="openshift-multus/network-metrics-daemon-pdcfk" Oct 30 20:39:06.689303 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.689124 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/17af283e-b12c-4ed3-abf4-7a4944d1c4f2-registry-tls\") pod \"image-registry-8984b54b7-skmcd\" (UID: \"17af283e-b12c-4ed3-abf4-7a4944d1c4f2\") " pod="openshift-image-registry/image-registry-8984b54b7-skmcd" Oct 30 20:39:06.689303 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.689134 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7e2574c1-b59c-453c-9501-25ec13a1ccad-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-7w5wz\" (UID: \"7e2574c1-b59c-453c-9501-25ec13a1ccad\") " pod="openshift-ovn-kubernetes/ovnkube-node-7w5wz" Oct 30 20:39:06.689303 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.689150 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cdwk9\" (UniqueName: \"kubernetes.io/projected/17af283e-b12c-4ed3-abf4-7a4944d1c4f2-kube-api-access-cdwk9\") pod \"image-registry-8984b54b7-skmcd\" (UID: \"17af283e-b12c-4ed3-abf4-7a4944d1c4f2\") " pod="openshift-image-registry/image-registry-8984b54b7-skmcd" Oct 30 20:39:06.689303 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.689152 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e277a3d9-3fe9-4454-9420-28afea76f745-multus-cni-dir\") pod \"multus-c4fvc\" (UID: \"e277a3d9-3fe9-4454-9420-28afea76f745\") " pod="openshift-multus/multus-c4fvc" Oct 30 20:39:06.689303 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.689172 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e277a3d9-3fe9-4454-9420-28afea76f745-multus-conf-dir\") pod \"multus-c4fvc\" (UID: \"e277a3d9-3fe9-4454-9420-28afea76f745\") " pod="openshift-multus/multus-c4fvc" Oct 30 20:39:06.689303 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.689175 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7e2574c1-b59c-453c-9501-25ec13a1ccad-var-lib-openvswitch\") pod \"ovnkube-node-7w5wz\" (UID: \"7e2574c1-b59c-453c-9501-25ec13a1ccad\") " pod="openshift-ovn-kubernetes/ovnkube-node-7w5wz" Oct 30 20:39:06.689303 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.689213 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7e2574c1-b59c-453c-9501-25ec13a1ccad-var-lib-openvswitch\") pod \"ovnkube-node-7w5wz\" (UID: \"7e2574c1-b59c-453c-9501-25ec13a1ccad\") " pod="openshift-ovn-kubernetes/ovnkube-node-7w5wz" Oct 30 20:39:06.689303 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.689229 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/17af283e-b12c-4ed3-abf4-7a4944d1c4f2-bound-sa-token\") pod \"image-registry-8984b54b7-skmcd\" (UID: \"17af283e-b12c-4ed3-abf4-7a4944d1c4f2\") " pod="openshift-image-registry/image-registry-8984b54b7-skmcd" Oct 30 20:39:06.689303 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.689232 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/da50258d-eef3-4976-8e22-be0e936b9993-agent-certs\") pod \"konnectivity-agent-jfxnj\" (UID: \"da50258d-eef3-4976-8e22-be0e936b9993\") " pod="kube-system/konnectivity-agent-jfxnj" Oct 30 20:39:06.689303 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.689254 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/7e2574c1-b59c-453c-9501-25ec13a1ccad-run-systemd\") pod \"ovnkube-node-7w5wz\" (UID: \"7e2574c1-b59c-453c-9501-25ec13a1ccad\") " pod="openshift-ovn-kubernetes/ovnkube-node-7w5wz" Oct 30 20:39:06.689303 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.689278 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7e2574c1-b59c-453c-9501-25ec13a1ccad-host-cni-bin\") pod \"ovnkube-node-7w5wz\" (UID: \"7e2574c1-b59c-453c-9501-25ec13a1ccad\") " pod="openshift-ovn-kubernetes/ovnkube-node-7w5wz" Oct 30 20:39:06.689303 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.689300 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e277a3d9-3fe9-4454-9420-28afea76f745-hostroot\") pod \"multus-c4fvc\" (UID: \"e277a3d9-3fe9-4454-9420-28afea76f745\") " pod="openshift-multus/multus-c4fvc" Oct 30 20:39:06.689303 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.689251 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/da50258d-eef3-4976-8e22-be0e936b9993-konnectivity-ca\") pod \"konnectivity-agent-jfxnj\" (UID: \"da50258d-eef3-4976-8e22-be0e936b9993\") " pod="kube-system/konnectivity-agent-jfxnj" Oct 30 20:39:06.689872 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.689321 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6b88d0a3-97c0-4adf-97a7-2e05d5cef781-var-lib-kubelet\") pod \"tuned-2qgck\" (UID: \"6b88d0a3-97c0-4adf-97a7-2e05d5cef781\") " pod="openshift-cluster-node-tuning-operator/tuned-2qgck" Oct 30 20:39:06.689872 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.689353 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hbqzh\" (UniqueName: \"kubernetes.io/projected/0aa2bb04-0552-45da-a54f-7e5064cf3113-kube-api-access-hbqzh\") pod \"dns-default-2sccz\" (UID: \"0aa2bb04-0552-45da-a54f-7e5064cf3113\") " pod="openshift-dns/dns-default-2sccz" Oct 30 20:39:06.689872 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.689364 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/7e2574c1-b59c-453c-9501-25ec13a1ccad-run-systemd\") pod \"ovnkube-node-7w5wz\" (UID: \"7e2574c1-b59c-453c-9501-25ec13a1ccad\") " pod="openshift-ovn-kubernetes/ovnkube-node-7w5wz" Oct 30 20:39:06.689872 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.689372 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e277a3d9-3fe9-4454-9420-28afea76f745-hostroot\") pod \"multus-c4fvc\" (UID: \"e277a3d9-3fe9-4454-9420-28afea76f745\") " pod="openshift-multus/multus-c4fvc" Oct 30 20:39:06.689872 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.689375 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/7e2574c1-b59c-453c-9501-25ec13a1ccad-host-kubelet\") pod \"ovnkube-node-7w5wz\" (UID: \"7e2574c1-b59c-453c-9501-25ec13a1ccad\") " pod="openshift-ovn-kubernetes/ovnkube-node-7w5wz" Oct 30 20:39:06.689872 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.689364 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7e2574c1-b59c-453c-9501-25ec13a1ccad-host-cni-bin\") pod \"ovnkube-node-7w5wz\" (UID: \"7e2574c1-b59c-453c-9501-25ec13a1ccad\") " pod="openshift-ovn-kubernetes/ovnkube-node-7w5wz" Oct 30 20:39:06.689872 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.689411 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6b88d0a3-97c0-4adf-97a7-2e05d5cef781-var-lib-kubelet\") pod \"tuned-2qgck\" (UID: \"6b88d0a3-97c0-4adf-97a7-2e05d5cef781\") " pod="openshift-cluster-node-tuning-operator/tuned-2qgck" Oct 30 20:39:06.689872 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.689364 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/17af283e-b12c-4ed3-abf4-7a4944d1c4f2-registry-tls\") pod \"image-registry-8984b54b7-skmcd\" (UID: \"17af283e-b12c-4ed3-abf4-7a4944d1c4f2\") " pod="openshift-image-registry/image-registry-8984b54b7-skmcd" Oct 30 20:39:06.689872 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.689428 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/7e2574c1-b59c-453c-9501-25ec13a1ccad-host-kubelet\") pod \"ovnkube-node-7w5wz\" (UID: \"7e2574c1-b59c-453c-9501-25ec13a1ccad\") " pod="openshift-ovn-kubernetes/ovnkube-node-7w5wz" Oct 30 20:39:06.689872 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.689436 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/7e2574c1-b59c-453c-9501-25ec13a1ccad-host-cni-netd\") pod \"ovnkube-node-7w5wz\" (UID: \"7e2574c1-b59c-453c-9501-25ec13a1ccad\") " pod="openshift-ovn-kubernetes/ovnkube-node-7w5wz" Oct 30 20:39:06.689872 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.689468 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/7e2574c1-b59c-453c-9501-25ec13a1ccad-host-cni-netd\") pod \"ovnkube-node-7w5wz\" (UID: \"7e2574c1-b59c-453c-9501-25ec13a1ccad\") " pod="openshift-ovn-kubernetes/ovnkube-node-7w5wz" Oct 30 20:39:06.689872 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.689496 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e277a3d9-3fe9-4454-9420-28afea76f745-cnibin\") pod \"multus-c4fvc\" (UID: \"e277a3d9-3fe9-4454-9420-28afea76f745\") " pod="openshift-multus/multus-c4fvc" Oct 30 20:39:06.689872 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.689540 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f5f694f2-6f18-409b-b777-2dd7a0eb12b2-registration-dir\") pod \"aws-ebs-csi-driver-node-t7h2z\" (UID: \"f5f694f2-6f18-409b-b777-2dd7a0eb12b2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t7h2z" Oct 30 20:39:06.689872 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.689564 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/6b88d0a3-97c0-4adf-97a7-2e05d5cef781-etc-sysctl-d\") pod \"tuned-2qgck\" (UID: \"6b88d0a3-97c0-4adf-97a7-2e05d5cef781\") " pod="openshift-cluster-node-tuning-operator/tuned-2qgck" Oct 30 20:39:06.689872 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.689585 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/6b88d0a3-97c0-4adf-97a7-2e05d5cef781-etc-systemd\") pod \"tuned-2qgck\" (UID: \"6b88d0a3-97c0-4adf-97a7-2e05d5cef781\") " pod="openshift-cluster-node-tuning-operator/tuned-2qgck" Oct 30 20:39:06.689872 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.689608 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/bf21cf19-4475-41c3-9153-44d8fb329cdb-dbus\") pod \"global-pull-secret-syncer-857r2\" (UID: \"bf21cf19-4475-41c3-9153-44d8fb329cdb\") " pod="kube-system/global-pull-secret-syncer-857r2" Oct 30 20:39:06.689872 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.689625 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f5f694f2-6f18-409b-b777-2dd7a0eb12b2-registration-dir\") pod \"aws-ebs-csi-driver-node-t7h2z\" (UID: \"f5f694f2-6f18-409b-b777-2dd7a0eb12b2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t7h2z" Oct 30 20:39:06.689872 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.689662 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e277a3d9-3fe9-4454-9420-28afea76f745-cnibin\") pod \"multus-c4fvc\" (UID: \"e277a3d9-3fe9-4454-9420-28afea76f745\") " pod="openshift-multus/multus-c4fvc" Oct 30 20:39:06.689872 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.689668 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/bf21cf19-4475-41c3-9153-44d8fb329cdb-dbus\") pod \"global-pull-secret-syncer-857r2\" (UID: \"bf21cf19-4475-41c3-9153-44d8fb329cdb\") " pod="kube-system/global-pull-secret-syncer-857r2" Oct 30 20:39:06.689872 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.689673 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/6b88d0a3-97c0-4adf-97a7-2e05d5cef781-etc-systemd\") pod \"tuned-2qgck\" (UID: \"6b88d0a3-97c0-4adf-97a7-2e05d5cef781\") " pod="openshift-cluster-node-tuning-operator/tuned-2qgck" Oct 30 20:39:06.689872 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.689691 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e277a3d9-3fe9-4454-9420-28afea76f745-host-var-lib-kubelet\") pod \"multus-c4fvc\" (UID: \"e277a3d9-3fe9-4454-9420-28afea76f745\") " pod="openshift-multus/multus-c4fvc" Oct 30 20:39:06.689872 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.689711 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ea0e353b-8cb1-4759-9f47-03f788f256d3-system-cni-dir\") pod \"multus-additional-cni-plugins-nxmjg\" (UID: \"ea0e353b-8cb1-4759-9f47-03f788f256d3\") " pod="openshift-multus/multus-additional-cni-plugins-nxmjg" Oct 30 20:39:06.689872 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.689714 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/6b88d0a3-97c0-4adf-97a7-2e05d5cef781-etc-sysctl-d\") pod \"tuned-2qgck\" (UID: \"6b88d0a3-97c0-4adf-97a7-2e05d5cef781\") " pod="openshift-cluster-node-tuning-operator/tuned-2qgck" Oct 30 20:39:06.689872 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.689727 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/17af283e-b12c-4ed3-abf4-7a4944d1c4f2-trusted-ca\") pod \"image-registry-8984b54b7-skmcd\" (UID: \"17af283e-b12c-4ed3-abf4-7a4944d1c4f2\") " pod="openshift-image-registry/image-registry-8984b54b7-skmcd" Oct 30 20:39:06.689872 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.689748 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/17af283e-b12c-4ed3-abf4-7a4944d1c4f2-installation-pull-secrets\") pod \"image-registry-8984b54b7-skmcd\" (UID: \"17af283e-b12c-4ed3-abf4-7a4944d1c4f2\") " pod="openshift-image-registry/image-registry-8984b54b7-skmcd" Oct 30 20:39:06.689872 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.689774 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e277a3d9-3fe9-4454-9420-28afea76f745-host-var-lib-kubelet\") pod \"multus-c4fvc\" (UID: \"e277a3d9-3fe9-4454-9420-28afea76f745\") " pod="openshift-multus/multus-c4fvc" Oct 30 20:39:06.689872 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.689788 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7e2574c1-b59c-453c-9501-25ec13a1ccad-run-openvswitch\") pod \"ovnkube-node-7w5wz\" (UID: \"7e2574c1-b59c-453c-9501-25ec13a1ccad\") " pod="openshift-ovn-kubernetes/ovnkube-node-7w5wz" Oct 30 20:39:06.689872 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.689832 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f5f694f2-6f18-409b-b777-2dd7a0eb12b2-socket-dir\") pod \"aws-ebs-csi-driver-node-t7h2z\" (UID: \"f5f694f2-6f18-409b-b777-2dd7a0eb12b2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t7h2z" Oct 30 20:39:06.689872 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.689856 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mjbz2\" (UniqueName: \"kubernetes.io/projected/0915cfe8-4606-4f92-877b-ebe7f52f4187-kube-api-access-mjbz2\") pod \"network-metrics-daemon-pdcfk\" (UID: \"0915cfe8-4606-4f92-877b-ebe7f52f4187\") " pod="openshift-multus/network-metrics-daemon-pdcfk" Oct 30 20:39:06.689872 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.689865 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7e2574c1-b59c-453c-9501-25ec13a1ccad-run-openvswitch\") pod \"ovnkube-node-7w5wz\" (UID: \"7e2574c1-b59c-453c-9501-25ec13a1ccad\") " pod="openshift-ovn-kubernetes/ovnkube-node-7w5wz" Oct 30 20:39:06.691077 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.689917 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f5f694f2-6f18-409b-b777-2dd7a0eb12b2-socket-dir\") pod \"aws-ebs-csi-driver-node-t7h2z\" (UID: \"f5f694f2-6f18-409b-b777-2dd7a0eb12b2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t7h2z" Oct 30 20:39:06.691077 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.689943 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/17af283e-b12c-4ed3-abf4-7a4944d1c4f2-installation-pull-secrets\") pod \"image-registry-8984b54b7-skmcd\" (UID: \"17af283e-b12c-4ed3-abf4-7a4944d1c4f2\") " pod="openshift-image-registry/image-registry-8984b54b7-skmcd" Oct 30 20:39:06.691077 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.690010 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/6b88d0a3-97c0-4adf-97a7-2e05d5cef781-etc-modprobe-d\") pod \"tuned-2qgck\" (UID: \"6b88d0a3-97c0-4adf-97a7-2e05d5cef781\") " pod="openshift-cluster-node-tuning-operator/tuned-2qgck" Oct 30 20:39:06.691077 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.690064 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6b88d0a3-97c0-4adf-97a7-2e05d5cef781-sys\") pod \"tuned-2qgck\" (UID: \"6b88d0a3-97c0-4adf-97a7-2e05d5cef781\") " pod="openshift-cluster-node-tuning-operator/tuned-2qgck" Oct 30 20:39:06.691077 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.690071 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ea0e353b-8cb1-4759-9f47-03f788f256d3-system-cni-dir\") pod \"multus-additional-cni-plugins-nxmjg\" (UID: \"ea0e353b-8cb1-4759-9f47-03f788f256d3\") " pod="openshift-multus/multus-additional-cni-plugins-nxmjg" Oct 30 20:39:06.691077 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.690094 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c1a9e89f-5d46-4414-b0e0-7adff658510e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hwzc6\" (UID: \"c1a9e89f-5d46-4414-b0e0-7adff658510e\") " pod="openshift-insights/insights-runtime-extractor-hwzc6" Oct 30 20:39:06.691077 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.690105 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/17af283e-b12c-4ed3-abf4-7a4944d1c4f2-trusted-ca\") pod \"image-registry-8984b54b7-skmcd\" (UID: \"17af283e-b12c-4ed3-abf4-7a4944d1c4f2\") " pod="openshift-image-registry/image-registry-8984b54b7-skmcd" Oct 30 20:39:06.691077 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.690119 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c9257033-53bf-40fc-a1e9-67fd5901678b-hosts-file\") pod \"node-resolver-lc599\" (UID: \"c9257033-53bf-40fc-a1e9-67fd5901678b\") " pod="openshift-dns/node-resolver-lc599" Oct 30 20:39:06.691077 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.690121 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6b88d0a3-97c0-4adf-97a7-2e05d5cef781-sys\") pod \"tuned-2qgck\" (UID: \"6b88d0a3-97c0-4adf-97a7-2e05d5cef781\") " pod="openshift-cluster-node-tuning-operator/tuned-2qgck" Oct 30 20:39:06.691077 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.690144 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/17af283e-b12c-4ed3-abf4-7a4944d1c4f2-image-registry-private-configuration\") pod \"image-registry-8984b54b7-skmcd\" (UID: \"17af283e-b12c-4ed3-abf4-7a4944d1c4f2\") " pod="openshift-image-registry/image-registry-8984b54b7-skmcd" Oct 30 20:39:06.691077 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.690156 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/6b88d0a3-97c0-4adf-97a7-2e05d5cef781-etc-modprobe-d\") pod \"tuned-2qgck\" (UID: \"6b88d0a3-97c0-4adf-97a7-2e05d5cef781\") " pod="openshift-cluster-node-tuning-operator/tuned-2qgck" Oct 30 20:39:06.691077 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.690170 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7e2574c1-b59c-453c-9501-25ec13a1ccad-etc-openvswitch\") pod \"ovnkube-node-7w5wz\" (UID: \"7e2574c1-b59c-453c-9501-25ec13a1ccad\") " pod="openshift-ovn-kubernetes/ovnkube-node-7w5wz" Oct 30 20:39:06.691077 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.690206 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7e2574c1-b59c-453c-9501-25ec13a1ccad-etc-openvswitch\") pod \"ovnkube-node-7w5wz\" (UID: \"7e2574c1-b59c-453c-9501-25ec13a1ccad\") " pod="openshift-ovn-kubernetes/ovnkube-node-7w5wz" Oct 30 20:39:06.691077 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.690235 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e277a3d9-3fe9-4454-9420-28afea76f745-host-var-lib-cni-multus\") pod \"multus-c4fvc\" (UID: \"e277a3d9-3fe9-4454-9420-28afea76f745\") " pod="openshift-multus/multus-c4fvc" Oct 30 20:39:06.691077 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.690269 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/6b88d0a3-97c0-4adf-97a7-2e05d5cef781-etc-sysctl-conf\") pod \"tuned-2qgck\" (UID: \"6b88d0a3-97c0-4adf-97a7-2e05d5cef781\") " pod="openshift-cluster-node-tuning-operator/tuned-2qgck" Oct 30 20:39:06.691077 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.690278 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c1a9e89f-5d46-4414-b0e0-7adff658510e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hwzc6\" (UID: \"c1a9e89f-5d46-4414-b0e0-7adff658510e\") " pod="openshift-insights/insights-runtime-extractor-hwzc6" Oct 30 20:39:06.691077 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.690293 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6b88d0a3-97c0-4adf-97a7-2e05d5cef781-lib-modules\") pod \"tuned-2qgck\" (UID: \"6b88d0a3-97c0-4adf-97a7-2e05d5cef781\") " pod="openshift-cluster-node-tuning-operator/tuned-2qgck" Oct 30 20:39:06.691077 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.690302 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e277a3d9-3fe9-4454-9420-28afea76f745-host-var-lib-cni-multus\") pod \"multus-c4fvc\" (UID: \"e277a3d9-3fe9-4454-9420-28afea76f745\") " pod="openshift-multus/multus-c4fvc" Oct 30 20:39:06.691077 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.690308 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/17af283e-b12c-4ed3-abf4-7a4944d1c4f2-image-registry-private-configuration\") pod \"image-registry-8984b54b7-skmcd\" (UID: \"17af283e-b12c-4ed3-abf4-7a4944d1c4f2\") " pod="openshift-image-registry/image-registry-8984b54b7-skmcd" Oct 30 20:39:06.691077 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.690314 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6b88d0a3-97c0-4adf-97a7-2e05d5cef781-host\") pod \"tuned-2qgck\" (UID: \"6b88d0a3-97c0-4adf-97a7-2e05d5cef781\") " pod="openshift-cluster-node-tuning-operator/tuned-2qgck" Oct 30 20:39:06.691077 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.690347 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c1a9e89f-5d46-4414-b0e0-7adff658510e-crio-socket\") pod \"insights-runtime-extractor-hwzc6\" (UID: \"c1a9e89f-5d46-4414-b0e0-7adff658510e\") " pod="openshift-insights/insights-runtime-extractor-hwzc6" Oct 30 20:39:06.691077 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.690350 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6b88d0a3-97c0-4adf-97a7-2e05d5cef781-host\") pod \"tuned-2qgck\" (UID: \"6b88d0a3-97c0-4adf-97a7-2e05d5cef781\") " pod="openshift-cluster-node-tuning-operator/tuned-2qgck" Oct 30 20:39:06.691077 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.690364 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/7e2574c1-b59c-453c-9501-25ec13a1ccad-run-ovn\") pod \"ovnkube-node-7w5wz\" (UID: \"7e2574c1-b59c-453c-9501-25ec13a1ccad\") " pod="openshift-ovn-kubernetes/ovnkube-node-7w5wz" Oct 30 20:39:06.691077 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.690376 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6b88d0a3-97c0-4adf-97a7-2e05d5cef781-lib-modules\") pod \"tuned-2qgck\" (UID: \"6b88d0a3-97c0-4adf-97a7-2e05d5cef781\") " pod="openshift-cluster-node-tuning-operator/tuned-2qgck" Oct 30 20:39:06.691077 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.690379 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/6b88d0a3-97c0-4adf-97a7-2e05d5cef781-etc-sysctl-conf\") pod \"tuned-2qgck\" (UID: \"6b88d0a3-97c0-4adf-97a7-2e05d5cef781\") " pod="openshift-cluster-node-tuning-operator/tuned-2qgck" Oct 30 20:39:06.691077 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.690412 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/7e2574c1-b59c-453c-9501-25ec13a1ccad-run-ovn\") pod \"ovnkube-node-7w5wz\" (UID: \"7e2574c1-b59c-453c-9501-25ec13a1ccad\") " pod="openshift-ovn-kubernetes/ovnkube-node-7w5wz" Oct 30 20:39:06.691077 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.690429 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7e2574c1-b59c-453c-9501-25ec13a1ccad-host-run-ovn-kubernetes\") pod \"ovnkube-node-7w5wz\" (UID: \"7e2574c1-b59c-453c-9501-25ec13a1ccad\") " pod="openshift-ovn-kubernetes/ovnkube-node-7w5wz" Oct 30 20:39:06.691077 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.690445 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e277a3d9-3fe9-4454-9420-28afea76f745-host-run-k8s-cni-cncf-io\") pod \"multus-c4fvc\" (UID: \"e277a3d9-3fe9-4454-9420-28afea76f745\") " pod="openshift-multus/multus-c4fvc" Oct 30 20:39:06.691077 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.690460 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e277a3d9-3fe9-4454-9420-28afea76f745-host-run-netns\") pod \"multus-c4fvc\" (UID: \"e277a3d9-3fe9-4454-9420-28afea76f745\") " pod="openshift-multus/multus-c4fvc" Oct 30 20:39:06.691077 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.690463 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7e2574c1-b59c-453c-9501-25ec13a1ccad-host-run-ovn-kubernetes\") pod \"ovnkube-node-7w5wz\" (UID: \"7e2574c1-b59c-453c-9501-25ec13a1ccad\") " pod="openshift-ovn-kubernetes/ovnkube-node-7w5wz" Oct 30 20:39:06.691077 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.690473 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f5f694f2-6f18-409b-b777-2dd7a0eb12b2-device-dir\") pod \"aws-ebs-csi-driver-node-t7h2z\" (UID: \"f5f694f2-6f18-409b-b777-2dd7a0eb12b2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t7h2z" Oct 30 20:39:06.691077 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.690489 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6b88d0a3-97c0-4adf-97a7-2e05d5cef781-etc-kubernetes\") pod \"tuned-2qgck\" (UID: \"6b88d0a3-97c0-4adf-97a7-2e05d5cef781\") " pod="openshift-cluster-node-tuning-operator/tuned-2qgck" Oct 30 20:39:06.691077 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.690534 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/bf21cf19-4475-41c3-9153-44d8fb329cdb-kubelet-config\") pod \"global-pull-secret-syncer-857r2\" (UID: \"bf21cf19-4475-41c3-9153-44d8fb329cdb\") " pod="kube-system/global-pull-secret-syncer-857r2" Oct 30 20:39:06.691077 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.690548 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e277a3d9-3fe9-4454-9420-28afea76f745-host-run-netns\") pod \"multus-c4fvc\" (UID: \"e277a3d9-3fe9-4454-9420-28afea76f745\") " pod="openshift-multus/multus-c4fvc" Oct 30 20:39:06.691077 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.690560 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0aa2bb04-0552-45da-a54f-7e5064cf3113-metrics-tls\") pod \"dns-default-2sccz\" (UID: \"0aa2bb04-0552-45da-a54f-7e5064cf3113\") " pod="openshift-dns/dns-default-2sccz" Oct 30 20:39:06.691077 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.690554 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e277a3d9-3fe9-4454-9420-28afea76f745-host-run-k8s-cni-cncf-io\") pod \"multus-c4fvc\" (UID: \"e277a3d9-3fe9-4454-9420-28afea76f745\") " pod="openshift-multus/multus-c4fvc" Oct 30 20:39:06.691077 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.690591 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6b88d0a3-97c0-4adf-97a7-2e05d5cef781-etc-kubernetes\") pod \"tuned-2qgck\" (UID: \"6b88d0a3-97c0-4adf-97a7-2e05d5cef781\") " pod="openshift-cluster-node-tuning-operator/tuned-2qgck" Oct 30 20:39:06.691077 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.690601 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/17af283e-b12c-4ed3-abf4-7a4944d1c4f2-registry-certificates\") pod \"image-registry-8984b54b7-skmcd\" (UID: \"17af283e-b12c-4ed3-abf4-7a4944d1c4f2\") " pod="openshift-image-registry/image-registry-8984b54b7-skmcd" Oct 30 20:39:06.691077 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.690617 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/bf21cf19-4475-41c3-9153-44d8fb329cdb-kubelet-config\") pod \"global-pull-secret-syncer-857r2\" (UID: \"bf21cf19-4475-41c3-9153-44d8fb329cdb\") " pod="kube-system/global-pull-secret-syncer-857r2" Oct 30 20:39:06.691077 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.690560 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f5f694f2-6f18-409b-b777-2dd7a0eb12b2-device-dir\") pod \"aws-ebs-csi-driver-node-t7h2z\" (UID: \"f5f694f2-6f18-409b-b777-2dd7a0eb12b2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t7h2z" Oct 30 20:39:06.691077 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.690638 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ea0e353b-8cb1-4759-9f47-03f788f256d3-tuning-conf-dir\") pod \"multus-additional-cni-plugins-nxmjg\" (UID: \"ea0e353b-8cb1-4759-9f47-03f788f256d3\") " pod="openshift-multus/multus-additional-cni-plugins-nxmjg" Oct 30 20:39:06.691077 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.690665 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qhljq\" (UniqueName: \"kubernetes.io/projected/6c3ba176-c9a5-4b6c-ae06-5593f68cf510-kube-api-access-qhljq\") pod \"ingress-canary-gmq97\" (UID: \"6c3ba176-c9a5-4b6c-ae06-5593f68cf510\") " pod="openshift-ingress-canary/ingress-canary-gmq97" Oct 30 20:39:06.691077 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.690689 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/acf9d9fb-1ae4-40ca-a83e-420a9fe40a01-host-slash\") pod \"iptables-alerter-kn2t9\" (UID: \"acf9d9fb-1ae4-40ca-a83e-420a9fe40a01\") " pod="openshift-network-operator/iptables-alerter-kn2t9" Oct 30 20:39:06.691077 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.690715 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/c1a9e89f-5d46-4414-b0e0-7adff658510e-data-volume\") pod \"insights-runtime-extractor-hwzc6\" (UID: \"c1a9e89f-5d46-4414-b0e0-7adff658510e\") " pod="openshift-insights/insights-runtime-extractor-hwzc6" Oct 30 20:39:06.691077 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.690721 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0aa2bb04-0552-45da-a54f-7e5064cf3113-metrics-tls\") pod \"dns-default-2sccz\" (UID: \"0aa2bb04-0552-45da-a54f-7e5064cf3113\") " pod="openshift-dns/dns-default-2sccz" Oct 30 20:39:06.691077 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.690742 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6c3ba176-c9a5-4b6c-ae06-5593f68cf510-cert\") pod \"ingress-canary-gmq97\" (UID: \"6c3ba176-c9a5-4b6c-ae06-5593f68cf510\") " pod="openshift-ingress-canary/ingress-canary-gmq97" Oct 30 20:39:06.691077 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.690759 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/acf9d9fb-1ae4-40ca-a83e-420a9fe40a01-host-slash\") pod \"iptables-alerter-kn2t9\" (UID: \"acf9d9fb-1ae4-40ca-a83e-420a9fe40a01\") " pod="openshift-network-operator/iptables-alerter-kn2t9" Oct 30 20:39:06.691077 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.690766 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0aa2bb04-0552-45da-a54f-7e5064cf3113-tmp-dir\") pod \"dns-default-2sccz\" (UID: \"0aa2bb04-0552-45da-a54f-7e5064cf3113\") " pod="openshift-dns/dns-default-2sccz" Oct 30 20:39:06.691077 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.690790 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/7e2574c1-b59c-453c-9501-25ec13a1ccad-log-socket\") pod \"ovnkube-node-7w5wz\" (UID: \"7e2574c1-b59c-453c-9501-25ec13a1ccad\") " pod="openshift-ovn-kubernetes/ovnkube-node-7w5wz" Oct 30 20:39:06.691077 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.690807 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/c1a9e89f-5d46-4414-b0e0-7adff658510e-data-volume\") pod \"insights-runtime-extractor-hwzc6\" (UID: \"c1a9e89f-5d46-4414-b0e0-7adff658510e\") " pod="openshift-insights/insights-runtime-extractor-hwzc6" Oct 30 20:39:06.691077 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.690796 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/17af283e-b12c-4ed3-abf4-7a4944d1c4f2-registry-certificates\") pod \"image-registry-8984b54b7-skmcd\" (UID: \"17af283e-b12c-4ed3-abf4-7a4944d1c4f2\") " pod="openshift-image-registry/image-registry-8984b54b7-skmcd" Oct 30 20:39:06.691077 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.690837 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e277a3d9-3fe9-4454-9420-28afea76f745-multus-socket-dir-parent\") pod \"multus-c4fvc\" (UID: \"e277a3d9-3fe9-4454-9420-28afea76f745\") " pod="openshift-multus/multus-c4fvc" Oct 30 20:39:06.691077 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.690848 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/7e2574c1-b59c-453c-9501-25ec13a1ccad-log-socket\") pod \"ovnkube-node-7w5wz\" (UID: \"7e2574c1-b59c-453c-9501-25ec13a1ccad\") " pod="openshift-ovn-kubernetes/ovnkube-node-7w5wz" Oct 30 20:39:06.691077 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.690854 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ea0e353b-8cb1-4759-9f47-03f788f256d3-tuning-conf-dir\") pod \"multus-additional-cni-plugins-nxmjg\" (UID: \"ea0e353b-8cb1-4759-9f47-03f788f256d3\") " pod="openshift-multus/multus-additional-cni-plugins-nxmjg" Oct 30 20:39:06.691077 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.690865 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e277a3d9-3fe9-4454-9420-28afea76f745-host-run-multus-certs\") pod \"multus-c4fvc\" (UID: \"e277a3d9-3fe9-4454-9420-28afea76f745\") " pod="openshift-multus/multus-c4fvc" Oct 30 20:39:06.691077 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.690886 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0aa2bb04-0552-45da-a54f-7e5064cf3113-tmp-dir\") pod \"dns-default-2sccz\" (UID: \"0aa2bb04-0552-45da-a54f-7e5064cf3113\") " pod="openshift-dns/dns-default-2sccz" Oct 30 20:39:06.691077 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.690889 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/6b88d0a3-97c0-4adf-97a7-2e05d5cef781-run\") pod \"tuned-2qgck\" (UID: \"6b88d0a3-97c0-4adf-97a7-2e05d5cef781\") " pod="openshift-cluster-node-tuning-operator/tuned-2qgck" Oct 30 20:39:06.691077 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.690912 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e277a3d9-3fe9-4454-9420-28afea76f745-multus-socket-dir-parent\") pod \"multus-c4fvc\" (UID: \"e277a3d9-3fe9-4454-9420-28afea76f745\") " pod="openshift-multus/multus-c4fvc" Oct 30 20:39:06.691077 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.690912 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e277a3d9-3fe9-4454-9420-28afea76f745-host-run-multus-certs\") pod \"multus-c4fvc\" (UID: \"e277a3d9-3fe9-4454-9420-28afea76f745\") " pod="openshift-multus/multus-c4fvc" Oct 30 20:39:06.691077 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.690927 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9d557\" (UniqueName: \"kubernetes.io/projected/c1a9e89f-5d46-4414-b0e0-7adff658510e-kube-api-access-9d557\") pod \"insights-runtime-extractor-hwzc6\" (UID: \"c1a9e89f-5d46-4414-b0e0-7adff658510e\") " pod="openshift-insights/insights-runtime-extractor-hwzc6" Oct 30 20:39:06.691077 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.690925 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/6b88d0a3-97c0-4adf-97a7-2e05d5cef781-run\") pod \"tuned-2qgck\" (UID: \"6b88d0a3-97c0-4adf-97a7-2e05d5cef781\") " pod="openshift-cluster-node-tuning-operator/tuned-2qgck" Oct 30 20:39:06.691077 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.690959 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ea0e353b-8cb1-4759-9f47-03f788f256d3-cnibin\") pod \"multus-additional-cni-plugins-nxmjg\" (UID: \"ea0e353b-8cb1-4759-9f47-03f788f256d3\") " pod="openshift-multus/multus-additional-cni-plugins-nxmjg" Oct 30 20:39:06.691077 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.690968 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6c3ba176-c9a5-4b6c-ae06-5593f68cf510-cert\") pod \"ingress-canary-gmq97\" (UID: \"6c3ba176-c9a5-4b6c-ae06-5593f68cf510\") " pod="openshift-ingress-canary/ingress-canary-gmq97" Oct 30 20:39:06.691077 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.690981 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/7e2574c1-b59c-453c-9501-25ec13a1ccad-systemd-units\") pod \"ovnkube-node-7w5wz\" (UID: \"7e2574c1-b59c-453c-9501-25ec13a1ccad\") " pod="openshift-ovn-kubernetes/ovnkube-node-7w5wz" Oct 30 20:39:06.691077 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.691013 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7e2574c1-b59c-453c-9501-25ec13a1ccad-host-run-netns\") pod \"ovnkube-node-7w5wz\" (UID: \"7e2574c1-b59c-453c-9501-25ec13a1ccad\") " pod="openshift-ovn-kubernetes/ovnkube-node-7w5wz" Oct 30 20:39:06.691077 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.691018 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ea0e353b-8cb1-4759-9f47-03f788f256d3-cnibin\") pod \"multus-additional-cni-plugins-nxmjg\" (UID: \"ea0e353b-8cb1-4759-9f47-03f788f256d3\") " pod="openshift-multus/multus-additional-cni-plugins-nxmjg" Oct 30 20:39:06.691077 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.691022 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/7e2574c1-b59c-453c-9501-25ec13a1ccad-systemd-units\") pod \"ovnkube-node-7w5wz\" (UID: \"7e2574c1-b59c-453c-9501-25ec13a1ccad\") " pod="openshift-ovn-kubernetes/ovnkube-node-7w5wz" Oct 30 20:39:06.691077 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.691041 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e277a3d9-3fe9-4454-9420-28afea76f745-etc-kubernetes\") pod \"multus-c4fvc\" (UID: \"e277a3d9-3fe9-4454-9420-28afea76f745\") " pod="openshift-multus/multus-c4fvc" Oct 30 20:39:06.691077 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.691056 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7e2574c1-b59c-453c-9501-25ec13a1ccad-host-run-netns\") pod \"ovnkube-node-7w5wz\" (UID: \"7e2574c1-b59c-453c-9501-25ec13a1ccad\") " pod="openshift-ovn-kubernetes/ovnkube-node-7w5wz" Oct 30 20:39:06.691077 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.691064 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f5f694f2-6f18-409b-b777-2dd7a0eb12b2-kubelet-dir\") pod \"aws-ebs-csi-driver-node-t7h2z\" (UID: \"f5f694f2-6f18-409b-b777-2dd7a0eb12b2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t7h2z" Oct 30 20:39:06.691077 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.691087 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/6b88d0a3-97c0-4adf-97a7-2e05d5cef781-etc-sysconfig\") pod \"tuned-2qgck\" (UID: \"6b88d0a3-97c0-4adf-97a7-2e05d5cef781\") " pod="openshift-cluster-node-tuning-operator/tuned-2qgck" Oct 30 20:39:06.691077 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.691093 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e277a3d9-3fe9-4454-9420-28afea76f745-etc-kubernetes\") pod \"multus-c4fvc\" (UID: \"e277a3d9-3fe9-4454-9420-28afea76f745\") " pod="openshift-multus/multus-c4fvc" Oct 30 20:39:06.691077 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.691105 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f5f694f2-6f18-409b-b777-2dd7a0eb12b2-kubelet-dir\") pod \"aws-ebs-csi-driver-node-t7h2z\" (UID: \"f5f694f2-6f18-409b-b777-2dd7a0eb12b2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t7h2z" Oct 30 20:39:06.691077 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.691110 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/0aa2bb04-0552-45da-a54f-7e5064cf3113-config-volume\") pod \"dns-default-2sccz\" (UID: \"0aa2bb04-0552-45da-a54f-7e5064cf3113\") " pod="openshift-dns/dns-default-2sccz" Oct 30 20:39:06.693113 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.691137 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e277a3d9-3fe9-4454-9420-28afea76f745-host-var-lib-cni-bin\") pod \"multus-c4fvc\" (UID: \"e277a3d9-3fe9-4454-9420-28afea76f745\") " pod="openshift-multus/multus-c4fvc" Oct 30 20:39:06.693113 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.691140 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/6b88d0a3-97c0-4adf-97a7-2e05d5cef781-etc-sysconfig\") pod \"tuned-2qgck\" (UID: \"6b88d0a3-97c0-4adf-97a7-2e05d5cef781\") " pod="openshift-cluster-node-tuning-operator/tuned-2qgck" Oct 30 20:39:06.693113 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.691161 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f5f694f2-6f18-409b-b777-2dd7a0eb12b2-etc-selinux\") pod \"aws-ebs-csi-driver-node-t7h2z\" (UID: \"f5f694f2-6f18-409b-b777-2dd7a0eb12b2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t7h2z" Oct 30 20:39:06.693113 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.691178 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f5f694f2-6f18-409b-b777-2dd7a0eb12b2-sys-fs\") pod \"aws-ebs-csi-driver-node-t7h2z\" (UID: \"f5f694f2-6f18-409b-b777-2dd7a0eb12b2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t7h2z" Oct 30 20:39:06.693113 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.691199 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c1a9e89f-5d46-4414-b0e0-7adff658510e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hwzc6\" (UID: \"c1a9e89f-5d46-4414-b0e0-7adff658510e\") " pod="openshift-insights/insights-runtime-extractor-hwzc6" Oct 30 20:39:06.693113 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.691209 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e277a3d9-3fe9-4454-9420-28afea76f745-host-var-lib-cni-bin\") pod \"multus-c4fvc\" (UID: \"e277a3d9-3fe9-4454-9420-28afea76f745\") " pod="openshift-multus/multus-c4fvc" Oct 30 20:39:06.693113 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.691225 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c9257033-53bf-40fc-a1e9-67fd5901678b-tmp-dir\") pod \"node-resolver-lc599\" (UID: \"c9257033-53bf-40fc-a1e9-67fd5901678b\") " pod="openshift-dns/node-resolver-lc599" Oct 30 20:39:06.693113 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.691230 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f5f694f2-6f18-409b-b777-2dd7a0eb12b2-etc-selinux\") pod \"aws-ebs-csi-driver-node-t7h2z\" (UID: \"f5f694f2-6f18-409b-b777-2dd7a0eb12b2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t7h2z" Oct 30 20:39:06.693113 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.691242 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f5f694f2-6f18-409b-b777-2dd7a0eb12b2-sys-fs\") pod \"aws-ebs-csi-driver-node-t7h2z\" (UID: \"f5f694f2-6f18-409b-b777-2dd7a0eb12b2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t7h2z" Oct 30 20:39:06.693113 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.691304 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c9257033-53bf-40fc-a1e9-67fd5901678b-tmp-dir\") pod \"node-resolver-lc599\" (UID: \"c9257033-53bf-40fc-a1e9-67fd5901678b\") " pod="openshift-dns/node-resolver-lc599" Oct 30 20:39:06.693113 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.691347 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/0aa2bb04-0552-45da-a54f-7e5064cf3113-config-volume\") pod \"dns-default-2sccz\" (UID: \"0aa2bb04-0552-45da-a54f-7e5064cf3113\") " pod="openshift-dns/dns-default-2sccz" Oct 30 20:39:06.702867 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.702843 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9jszj\" (UniqueName: \"kubernetes.io/projected/c9257033-53bf-40fc-a1e9-67fd5901678b-kube-api-access-9jszj\") pod \"node-resolver-lc599\" (UID: \"c9257033-53bf-40fc-a1e9-67fd5901678b\") " pod="openshift-dns/node-resolver-lc599" Oct 30 20:39:06.702967 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.702919 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mjbz2\" (UniqueName: \"kubernetes.io/projected/0915cfe8-4606-4f92-877b-ebe7f52f4187-kube-api-access-mjbz2\") pod \"network-metrics-daemon-pdcfk\" (UID: \"0915cfe8-4606-4f92-877b-ebe7f52f4187\") " pod="openshift-multus/network-metrics-daemon-pdcfk" Oct 30 20:39:06.703026 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.702980 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qhljq\" (UniqueName: \"kubernetes.io/projected/6c3ba176-c9a5-4b6c-ae06-5593f68cf510-kube-api-access-qhljq\") pod \"ingress-canary-gmq97\" (UID: \"6c3ba176-c9a5-4b6c-ae06-5593f68cf510\") " pod="openshift-ingress-canary/ingress-canary-gmq97" Oct 30 20:39:06.703096 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.703078 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cdwk9\" (UniqueName: \"kubernetes.io/projected/17af283e-b12c-4ed3-abf4-7a4944d1c4f2-kube-api-access-cdwk9\") pod \"image-registry-8984b54b7-skmcd\" (UID: \"17af283e-b12c-4ed3-abf4-7a4944d1c4f2\") " pod="openshift-image-registry/image-registry-8984b54b7-skmcd" Oct 30 20:39:06.703217 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.703187 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9d557\" (UniqueName: \"kubernetes.io/projected/c1a9e89f-5d46-4414-b0e0-7adff658510e-kube-api-access-9d557\") pod \"insights-runtime-extractor-hwzc6\" (UID: \"c1a9e89f-5d46-4414-b0e0-7adff658510e\") " pod="openshift-insights/insights-runtime-extractor-hwzc6" Oct 30 20:39:06.707786 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.707770 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hbqzh\" (UniqueName: \"kubernetes.io/projected/0aa2bb04-0552-45da-a54f-7e5064cf3113-kube-api-access-hbqzh\") pod \"dns-default-2sccz\" (UID: \"0aa2bb04-0552-45da-a54f-7e5064cf3113\") " pod="openshift-dns/dns-default-2sccz" Oct 30 20:39:06.708355 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.708341 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/17af283e-b12c-4ed3-abf4-7a4944d1c4f2-bound-sa-token\") pod \"image-registry-8984b54b7-skmcd\" (UID: \"17af283e-b12c-4ed3-abf4-7a4944d1c4f2\") " pod="openshift-image-registry/image-registry-8984b54b7-skmcd" Oct 30 20:39:06.791800 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.791779 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c1a9e89f-5d46-4414-b0e0-7adff658510e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hwzc6\" (UID: \"c1a9e89f-5d46-4414-b0e0-7adff658510e\") " pod="openshift-insights/insights-runtime-extractor-hwzc6" Oct 30 20:39:06.791917 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.791820 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c9257033-53bf-40fc-a1e9-67fd5901678b-hosts-file\") pod \"node-resolver-lc599\" (UID: \"c9257033-53bf-40fc-a1e9-67fd5901678b\") " pod="openshift-dns/node-resolver-lc599" Oct 30 20:39:06.791917 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.791843 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c1a9e89f-5d46-4414-b0e0-7adff658510e-crio-socket\") pod \"insights-runtime-extractor-hwzc6\" (UID: \"c1a9e89f-5d46-4414-b0e0-7adff658510e\") " pod="openshift-insights/insights-runtime-extractor-hwzc6" Oct 30 20:39:06.791917 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.791906 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c1a9e89f-5d46-4414-b0e0-7adff658510e-crio-socket\") pod \"insights-runtime-extractor-hwzc6\" (UID: \"c1a9e89f-5d46-4414-b0e0-7adff658510e\") " pod="openshift-insights/insights-runtime-extractor-hwzc6" Oct 30 20:39:06.792020 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.791916 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c9257033-53bf-40fc-a1e9-67fd5901678b-hosts-file\") pod \"node-resolver-lc599\" (UID: \"c9257033-53bf-40fc-a1e9-67fd5901678b\") " pod="openshift-dns/node-resolver-lc599" Oct 30 20:39:06.792160 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.792143 4611 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 30 20:39:06.794075 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.794058 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c1a9e89f-5d46-4414-b0e0-7adff658510e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hwzc6\" (UID: \"c1a9e89f-5d46-4414-b0e0-7adff658510e\") " pod="openshift-insights/insights-runtime-extractor-hwzc6" Oct 30 20:39:06.871833 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:06.871786 4611 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-hwzc6" Oct 30 20:39:07.024766 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:07.024739 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-7w5wz" Oct 30 20:39:07.119042 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:07.119018 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-2sccz" Oct 30 20:39:07.172882 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:07.172823 4611 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-hwzc6"] Oct 30 20:39:07.177649 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:07.177624 4611 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1a9e89f_5d46_4414_b0e0_7adff658510e.slice/crio-4d50f15bd73b57a9f0925d7ef0cf7e6a6b449cff73eafb3197bdfef992834204 WatchSource:0}: Error finding container 4d50f15bd73b57a9f0925d7ef0cf7e6a6b449cff73eafb3197bdfef992834204: Status 404 returned error can't find the container with id 4d50f15bd73b57a9f0925d7ef0cf7e6a6b449cff73eafb3197bdfef992834204 Oct 30 20:39:07.273109 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:07.273090 4611 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 20:39:07.641733 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:07.641697 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-hwzc6" event={"ID":"c1a9e89f-5d46-4414-b0e0-7adff658510e","Type":"ContainerStarted","Data":"2ecd1b3836263b0d7bb78fc960d77aa5679444553066155e8489a5c22bc7e6ac"} Oct 30 20:39:07.642186 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:07.641771 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-hwzc6" event={"ID":"c1a9e89f-5d46-4414-b0e0-7adff658510e","Type":"ContainerStarted","Data":"4d50f15bd73b57a9f0925d7ef0cf7e6a6b449cff73eafb3197bdfef992834204"} Oct 30 20:39:07.694755 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:07.694729 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-7w5wz" Oct 30 20:39:08.648986 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:08.648954 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-hwzc6" event={"ID":"c1a9e89f-5d46-4414-b0e0-7adff658510e","Type":"ContainerStarted","Data":"f58d4100ef567196180c67b2095329959e05d8df2904fcd34002c7d587d8b320"} Oct 30 20:39:08.649383 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:08.648993 4611 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 20:39:10.644523 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:10.644488 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-2sccz" Oct 30 20:39:10.656087 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:10.656060 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-hwzc6" event={"ID":"c1a9e89f-5d46-4414-b0e0-7adff658510e","Type":"ContainerStarted","Data":"496ee897d0d80772f1750bd390053c77e08eb20bf39100a1f2c24beb31c87939"} Oct 30 20:39:13.925717 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:13.925690 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-7w5wz" Oct 30 20:39:14.501399 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:14.501363 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:39:15.148219 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:15.148193 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-jp2cr" Oct 30 20:39:15.529077 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:15.529016 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-8984b54b7-skmcd" Oct 30 20:39:20.869942 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:20.869905 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:39:21.025958 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:21.025925 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:39:21.161406 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:21.161329 4611 patch_prober.go:29] interesting pod/image-registry-8984b54b7-skmcd container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Oct 30 20:39:21.161561 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:21.161426 4611 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-8984b54b7-skmcd" podUID="17af283e-b12c-4ed3-abf4-7a4944d1c4f2" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 20:39:22.781746 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:22.781709 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:39:23.796695 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:23.796665 4611 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-fgskx"] Oct 30 20:39:23.798335 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:23.798319 4611 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-fgskx" Oct 30 20:39:23.805294 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:23.805273 4611 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Oct 30 20:39:23.805400 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:23.805300 4611 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-ch86z\"" Oct 30 20:39:23.809342 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:23.809323 4611 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-fgskx"] Oct 30 20:39:23.837653 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:23.837625 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:39:23.892036 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:23.892010 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/312370de-5cc7-438b-b70a-49b8dad611ec-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-fgskx\" (UID: \"312370de-5cc7-438b-b70a-49b8dad611ec\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-fgskx" Oct 30 20:39:23.980614 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:23.980587 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:39:23.992598 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:23.992576 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/312370de-5cc7-438b-b70a-49b8dad611ec-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-fgskx\" (UID: \"312370de-5cc7-438b-b70a-49b8dad611ec\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-fgskx" Oct 30 20:39:23.994710 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:23.994691 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/312370de-5cc7-438b-b70a-49b8dad611ec-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-fgskx\" (UID: \"312370de-5cc7-438b-b70a-49b8dad611ec\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-fgskx" Oct 30 20:39:24.108054 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:24.108033 4611 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-fgskx" Oct 30 20:39:24.375771 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:24.375693 4611 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-fgskx"] Oct 30 20:39:24.380412 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:24.380383 4611 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod312370de_5cc7_438b_b70a_49b8dad611ec.slice/crio-210fa269ee8551c69f548b54a38ef1c249ffc7bcfd30d6a293fa74f49b49de73 WatchSource:0}: Error finding container 210fa269ee8551c69f548b54a38ef1c249ffc7bcfd30d6a293fa74f49b49de73: Status 404 returned error can't find the container with id 210fa269ee8551c69f548b54a38ef1c249ffc7bcfd30d6a293fa74f49b49de73 Oct 30 20:39:24.649501 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:24.649444 4611 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-8984b54b7-skmcd_17af283e-b12c-4ed3-abf4-7a4944d1c4f2/registry/0.log" Oct 30 20:39:24.695948 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:24.695915 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-fgskx" event={"ID":"312370de-5cc7-438b-b70a-49b8dad611ec","Type":"ContainerStarted","Data":"210fa269ee8551c69f548b54a38ef1c249ffc7bcfd30d6a293fa74f49b49de73"} Oct 30 20:39:25.386125 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:25.386101 4611 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-hsv4j_35d51f79-80a9-4f9d-970e-6a35a34976a2/node-ca/0.log" Oct 30 20:39:25.533466 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:25.533436 4611 patch_prober.go:29] interesting pod/image-registry-8984b54b7-skmcd container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Oct 30 20:39:25.533626 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:25.533485 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-8984b54b7-skmcd" podUID="17af283e-b12c-4ed3-abf4-7a4944d1c4f2" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 20:39:25.699837 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:25.699767 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-fgskx" event={"ID":"312370de-5cc7-438b-b70a-49b8dad611ec","Type":"ContainerStarted","Data":"8b2639c4c98bef89214f10cd68fc0547bde392970d41b68921e5422e9addd03d"} Oct 30 20:39:25.699989 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:25.699974 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-fgskx" Oct 30 20:39:25.704574 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:25.704550 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-fgskx" Oct 30 20:39:25.715806 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:25.715762 4611 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-fgskx" podStartSLOduration=1.725240677 podStartE2EDuration="2.715752832s" totalImagesPullingTime="990.512155ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:39:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:39:25.715330377 +0000 UTC m=+20.249356994" watchObservedRunningTime="2026-10-30 20:39:25.715752832 +0000 UTC m=+20.249779448" Oct 30 20:39:25.790607 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:25.790557 4611 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-gmq97_6c3ba176-c9a5-4b6c-ae06-5593f68cf510/serve-healthcheck-canary/0.log" Oct 30 20:39:25.924560 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:25.924534 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-jfxnj" Oct 30 20:39:26.137229 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:26.137194 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:39:26.278709 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:26.278687 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:39:26.479205 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:26.479121 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:39:28.760338 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:28.760305 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:39:28.919989 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:28.919958 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:39:29.068659 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:29.068585 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:39:30.289107 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:30.289076 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:39:30.511079 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:30.511045 4611 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-d4qf4"] Oct 30 20:39:30.524480 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:30.524450 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:39:30.527667 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:30.527648 4611 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-d4qf4" Oct 30 20:39:30.568413 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:30.568358 4611 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Oct 30 20:39:30.568494 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:30.568414 4611 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Oct 30 20:39:30.568494 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:30.568435 4611 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-dctv6\"" Oct 30 20:39:30.592065 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:30.591995 4611 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 20:39:30.592206 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:30.592163 4611 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Oct 30 20:39:30.592206 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:30.592187 4611 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 30 20:39:30.592295 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:30.592212 4611 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Oct 30 20:39:30.631569 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:30.631546 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/7a680d75-deea-4997-b813-1a144a1a9435-node-exporter-textfile\") pod \"node-exporter-d4qf4\" (UID: \"7a680d75-deea-4997-b813-1a144a1a9435\") " pod="openshift-monitoring/node-exporter-d4qf4" Oct 30 20:39:30.631656 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:30.631575 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/7a680d75-deea-4997-b813-1a144a1a9435-node-exporter-wtmp\") pod \"node-exporter-d4qf4\" (UID: \"7a680d75-deea-4997-b813-1a144a1a9435\") " pod="openshift-monitoring/node-exporter-d4qf4" Oct 30 20:39:30.631656 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:30.631597 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/7a680d75-deea-4997-b813-1a144a1a9435-node-exporter-accelerators-collector-config\") pod \"node-exporter-d4qf4\" (UID: \"7a680d75-deea-4997-b813-1a144a1a9435\") " pod="openshift-monitoring/node-exporter-d4qf4" Oct 30 20:39:30.631656 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:30.631615 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qpsh8\" (UniqueName: \"kubernetes.io/projected/7a680d75-deea-4997-b813-1a144a1a9435-kube-api-access-qpsh8\") pod \"node-exporter-d4qf4\" (UID: \"7a680d75-deea-4997-b813-1a144a1a9435\") " pod="openshift-monitoring/node-exporter-d4qf4" Oct 30 20:39:30.631656 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:30.631632 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/7a680d75-deea-4997-b813-1a144a1a9435-root\") pod \"node-exporter-d4qf4\" (UID: \"7a680d75-deea-4997-b813-1a144a1a9435\") " pod="openshift-monitoring/node-exporter-d4qf4" Oct 30 20:39:30.631785 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:30.631764 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7a680d75-deea-4997-b813-1a144a1a9435-sys\") pod \"node-exporter-d4qf4\" (UID: \"7a680d75-deea-4997-b813-1a144a1a9435\") " pod="openshift-monitoring/node-exporter-d4qf4" Oct 30 20:39:30.631817 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:30.631786 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7a680d75-deea-4997-b813-1a144a1a9435-metrics-client-ca\") pod \"node-exporter-d4qf4\" (UID: \"7a680d75-deea-4997-b813-1a144a1a9435\") " pod="openshift-monitoring/node-exporter-d4qf4" Oct 30 20:39:30.631881 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:30.631859 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/7a680d75-deea-4997-b813-1a144a1a9435-node-exporter-tls\") pod \"node-exporter-d4qf4\" (UID: \"7a680d75-deea-4997-b813-1a144a1a9435\") " pod="openshift-monitoring/node-exporter-d4qf4" Oct 30 20:39:30.631919 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:30.631910 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7a680d75-deea-4997-b813-1a144a1a9435-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-d4qf4\" (UID: \"7a680d75-deea-4997-b813-1a144a1a9435\") " pod="openshift-monitoring/node-exporter-d4qf4" Oct 30 20:39:30.673285 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:30.673259 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:39:30.733108 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:30.733083 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/7a680d75-deea-4997-b813-1a144a1a9435-node-exporter-accelerators-collector-config\") pod \"node-exporter-d4qf4\" (UID: \"7a680d75-deea-4997-b813-1a144a1a9435\") " pod="openshift-monitoring/node-exporter-d4qf4" Oct 30 20:39:30.733235 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:30.733110 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qpsh8\" (UniqueName: \"kubernetes.io/projected/7a680d75-deea-4997-b813-1a144a1a9435-kube-api-access-qpsh8\") pod \"node-exporter-d4qf4\" (UID: \"7a680d75-deea-4997-b813-1a144a1a9435\") " pod="openshift-monitoring/node-exporter-d4qf4" Oct 30 20:39:30.733235 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:30.733127 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/7a680d75-deea-4997-b813-1a144a1a9435-root\") pod \"node-exporter-d4qf4\" (UID: \"7a680d75-deea-4997-b813-1a144a1a9435\") " pod="openshift-monitoring/node-exporter-d4qf4" Oct 30 20:39:30.733235 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:30.733150 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7a680d75-deea-4997-b813-1a144a1a9435-sys\") pod \"node-exporter-d4qf4\" (UID: \"7a680d75-deea-4997-b813-1a144a1a9435\") " pod="openshift-monitoring/node-exporter-d4qf4" Oct 30 20:39:30.733235 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:30.733166 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7a680d75-deea-4997-b813-1a144a1a9435-metrics-client-ca\") pod \"node-exporter-d4qf4\" (UID: \"7a680d75-deea-4997-b813-1a144a1a9435\") " pod="openshift-monitoring/node-exporter-d4qf4" Oct 30 20:39:30.733235 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:30.733185 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/7a680d75-deea-4997-b813-1a144a1a9435-node-exporter-tls\") pod \"node-exporter-d4qf4\" (UID: \"7a680d75-deea-4997-b813-1a144a1a9435\") " pod="openshift-monitoring/node-exporter-d4qf4" Oct 30 20:39:30.733235 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:30.733213 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7a680d75-deea-4997-b813-1a144a1a9435-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-d4qf4\" (UID: \"7a680d75-deea-4997-b813-1a144a1a9435\") " pod="openshift-monitoring/node-exporter-d4qf4" Oct 30 20:39:30.733235 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:30.733225 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/7a680d75-deea-4997-b813-1a144a1a9435-root\") pod \"node-exporter-d4qf4\" (UID: \"7a680d75-deea-4997-b813-1a144a1a9435\") " pod="openshift-monitoring/node-exporter-d4qf4" Oct 30 20:39:30.733562 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:30.733240 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7a680d75-deea-4997-b813-1a144a1a9435-sys\") pod \"node-exporter-d4qf4\" (UID: \"7a680d75-deea-4997-b813-1a144a1a9435\") " pod="openshift-monitoring/node-exporter-d4qf4" Oct 30 20:39:30.733562 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:30.733248 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/7a680d75-deea-4997-b813-1a144a1a9435-node-exporter-textfile\") pod \"node-exporter-d4qf4\" (UID: \"7a680d75-deea-4997-b813-1a144a1a9435\") " pod="openshift-monitoring/node-exporter-d4qf4" Oct 30 20:39:30.733562 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:30.733296 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/7a680d75-deea-4997-b813-1a144a1a9435-node-exporter-wtmp\") pod \"node-exporter-d4qf4\" (UID: \"7a680d75-deea-4997-b813-1a144a1a9435\") " pod="openshift-monitoring/node-exporter-d4qf4" Oct 30 20:39:30.733562 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:30.733471 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/7a680d75-deea-4997-b813-1a144a1a9435-node-exporter-wtmp\") pod \"node-exporter-d4qf4\" (UID: \"7a680d75-deea-4997-b813-1a144a1a9435\") " pod="openshift-monitoring/node-exporter-d4qf4" Oct 30 20:39:30.733680 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:30.733631 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/7a680d75-deea-4997-b813-1a144a1a9435-node-exporter-textfile\") pod \"node-exporter-d4qf4\" (UID: \"7a680d75-deea-4997-b813-1a144a1a9435\") " pod="openshift-monitoring/node-exporter-d4qf4" Oct 30 20:39:30.733807 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:30.733771 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/7a680d75-deea-4997-b813-1a144a1a9435-node-exporter-accelerators-collector-config\") pod \"node-exporter-d4qf4\" (UID: \"7a680d75-deea-4997-b813-1a144a1a9435\") " pod="openshift-monitoring/node-exporter-d4qf4" Oct 30 20:39:30.733907 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:30.733807 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7a680d75-deea-4997-b813-1a144a1a9435-metrics-client-ca\") pod \"node-exporter-d4qf4\" (UID: \"7a680d75-deea-4997-b813-1a144a1a9435\") " pod="openshift-monitoring/node-exporter-d4qf4" Oct 30 20:39:30.735654 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:30.735622 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7a680d75-deea-4997-b813-1a144a1a9435-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-d4qf4\" (UID: \"7a680d75-deea-4997-b813-1a144a1a9435\") " pod="openshift-monitoring/node-exporter-d4qf4" Oct 30 20:39:30.735727 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:30.735628 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/7a680d75-deea-4997-b813-1a144a1a9435-node-exporter-tls\") pod \"node-exporter-d4qf4\" (UID: \"7a680d75-deea-4997-b813-1a144a1a9435\") " pod="openshift-monitoring/node-exporter-d4qf4" Oct 30 20:39:30.744382 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:30.744364 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qpsh8\" (UniqueName: \"kubernetes.io/projected/7a680d75-deea-4997-b813-1a144a1a9435-kube-api-access-qpsh8\") pod \"node-exporter-d4qf4\" (UID: \"7a680d75-deea-4997-b813-1a144a1a9435\") " pod="openshift-monitoring/node-exporter-d4qf4" Oct 30 20:39:30.816889 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:30.816855 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:39:30.871821 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:30.871798 4611 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-d4qf4" Oct 30 20:39:30.880984 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:30.880964 4611 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a680d75_deea_4997_b813_1a144a1a9435.slice/crio-e3b8721548d73edf3f3c325125af4103d2ad3d6f883475b04f23c6d2469e0395 WatchSource:0}: Error finding container e3b8721548d73edf3f3c325125af4103d2ad3d6f883475b04f23c6d2469e0395: Status 404 returned error can't find the container with id e3b8721548d73edf3f3c325125af4103d2ad3d6f883475b04f23c6d2469e0395 Oct 30 20:39:30.965749 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:30.965724 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:39:31.106629 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:31.106601 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:39:31.161222 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:31.161173 4611 patch_prober.go:29] interesting pod/image-registry-8984b54b7-skmcd container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Oct 30 20:39:31.161222 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:31.161213 4611 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-8984b54b7-skmcd" podUID="17af283e-b12c-4ed3-abf4-7a4944d1c4f2" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 20:39:31.487473 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:31.487401 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:39:31.531149 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:31.531116 4611 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 20:39:31.533646 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:31.533621 4611 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 30 20:39:31.538425 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:31.538290 4611 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Oct 30 20:39:31.538425 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:31.538338 4611 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Oct 30 20:39:31.538425 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:31.538298 4611 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Oct 30 20:39:31.538425 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:31.538360 4611 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Oct 30 20:39:31.538425 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:31.538302 4611 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Oct 30 20:39:31.538425 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:31.538298 4611 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Oct 30 20:39:31.538425 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:31.538386 4611 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Oct 30 20:39:31.538425 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:31.538414 4611 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Oct 30 20:39:31.538763 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:31.538451 4611 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-pcdbr\"" Oct 30 20:39:31.538763 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:31.538469 4611 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Oct 30 20:39:31.555643 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:31.555614 4611 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 20:39:31.632516 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:31.632464 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:39:31.639437 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:31.639409 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/60f9f978-737b-4279-927d-25f18934a44e-web-config\") pod \"alertmanager-main-0\" (UID: \"60f9f978-737b-4279-927d-25f18934a44e\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 20:39:31.639560 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:31.639457 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/60f9f978-737b-4279-927d-25f18934a44e-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"60f9f978-737b-4279-927d-25f18934a44e\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 20:39:31.639560 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:31.639482 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/60f9f978-737b-4279-927d-25f18934a44e-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"60f9f978-737b-4279-927d-25f18934a44e\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 20:39:31.639656 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:31.639603 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/60f9f978-737b-4279-927d-25f18934a44e-config-volume\") pod \"alertmanager-main-0\" (UID: \"60f9f978-737b-4279-927d-25f18934a44e\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 20:39:31.639656 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:31.639633 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/60f9f978-737b-4279-927d-25f18934a44e-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"60f9f978-737b-4279-927d-25f18934a44e\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 20:39:31.639756 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:31.639663 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/60f9f978-737b-4279-927d-25f18934a44e-tls-assets\") pod \"alertmanager-main-0\" (UID: \"60f9f978-737b-4279-927d-25f18934a44e\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 20:39:31.639790 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:31.639772 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/60f9f978-737b-4279-927d-25f18934a44e-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"60f9f978-737b-4279-927d-25f18934a44e\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 20:39:31.639835 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:31.639799 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/60f9f978-737b-4279-927d-25f18934a44e-config-out\") pod \"alertmanager-main-0\" (UID: \"60f9f978-737b-4279-927d-25f18934a44e\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 20:39:31.639835 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:31.639825 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t7vr9\" (UniqueName: \"kubernetes.io/projected/60f9f978-737b-4279-927d-25f18934a44e-kube-api-access-t7vr9\") pod \"alertmanager-main-0\" (UID: \"60f9f978-737b-4279-927d-25f18934a44e\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 20:39:31.639922 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:31.639866 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/60f9f978-737b-4279-927d-25f18934a44e-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"60f9f978-737b-4279-927d-25f18934a44e\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 20:39:31.639922 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:31.639902 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/60f9f978-737b-4279-927d-25f18934a44e-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"60f9f978-737b-4279-927d-25f18934a44e\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 20:39:31.640014 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:31.639930 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/60f9f978-737b-4279-927d-25f18934a44e-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"60f9f978-737b-4279-927d-25f18934a44e\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 20:39:31.640058 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:31.640008 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/60f9f978-737b-4279-927d-25f18934a44e-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"60f9f978-737b-4279-927d-25f18934a44e\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 20:39:31.716434 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:31.716399 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-d4qf4" event={"ID":"7a680d75-deea-4997-b813-1a144a1a9435","Type":"ContainerStarted","Data":"e3b8721548d73edf3f3c325125af4103d2ad3d6f883475b04f23c6d2469e0395"} Oct 30 20:39:31.740874 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:31.740807 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/60f9f978-737b-4279-927d-25f18934a44e-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"60f9f978-737b-4279-927d-25f18934a44e\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 20:39:31.740874 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:31.740849 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/60f9f978-737b-4279-927d-25f18934a44e-config-out\") pod \"alertmanager-main-0\" (UID: \"60f9f978-737b-4279-927d-25f18934a44e\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 20:39:31.741051 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:31.740888 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t7vr9\" (UniqueName: \"kubernetes.io/projected/60f9f978-737b-4279-927d-25f18934a44e-kube-api-access-t7vr9\") pod \"alertmanager-main-0\" (UID: \"60f9f978-737b-4279-927d-25f18934a44e\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 20:39:31.741051 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:31.740918 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/60f9f978-737b-4279-927d-25f18934a44e-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"60f9f978-737b-4279-927d-25f18934a44e\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 20:39:31.741051 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:31.740946 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/60f9f978-737b-4279-927d-25f18934a44e-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"60f9f978-737b-4279-927d-25f18934a44e\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 20:39:31.741051 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:31.740979 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/60f9f978-737b-4279-927d-25f18934a44e-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"60f9f978-737b-4279-927d-25f18934a44e\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 20:39:31.741051 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:31.741017 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/60f9f978-737b-4279-927d-25f18934a44e-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"60f9f978-737b-4279-927d-25f18934a44e\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 20:39:31.741051 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:31.741045 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/60f9f978-737b-4279-927d-25f18934a44e-web-config\") pod \"alertmanager-main-0\" (UID: \"60f9f978-737b-4279-927d-25f18934a44e\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 20:39:31.741351 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:31.741247 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/60f9f978-737b-4279-927d-25f18934a44e-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"60f9f978-737b-4279-927d-25f18934a44e\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 20:39:31.741351 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:31.741297 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/60f9f978-737b-4279-927d-25f18934a44e-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"60f9f978-737b-4279-927d-25f18934a44e\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 20:39:31.741351 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:31.741342 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/60f9f978-737b-4279-927d-25f18934a44e-config-volume\") pod \"alertmanager-main-0\" (UID: \"60f9f978-737b-4279-927d-25f18934a44e\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 20:39:31.741577 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:31.741367 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/60f9f978-737b-4279-927d-25f18934a44e-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"60f9f978-737b-4279-927d-25f18934a44e\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 20:39:31.741577 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:31.741392 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/60f9f978-737b-4279-927d-25f18934a44e-tls-assets\") pod \"alertmanager-main-0\" (UID: \"60f9f978-737b-4279-927d-25f18934a44e\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 20:39:31.741889 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:31.741868 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/60f9f978-737b-4279-927d-25f18934a44e-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"60f9f978-737b-4279-927d-25f18934a44e\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 20:39:31.742258 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:31.742038 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/60f9f978-737b-4279-927d-25f18934a44e-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"60f9f978-737b-4279-927d-25f18934a44e\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 20:39:31.742258 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:31.742123 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/60f9f978-737b-4279-927d-25f18934a44e-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"60f9f978-737b-4279-927d-25f18934a44e\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 20:39:31.743837 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:31.743807 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/60f9f978-737b-4279-927d-25f18934a44e-web-config\") pod \"alertmanager-main-0\" (UID: \"60f9f978-737b-4279-927d-25f18934a44e\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 20:39:31.743933 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:31.743865 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/60f9f978-737b-4279-927d-25f18934a44e-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"60f9f978-737b-4279-927d-25f18934a44e\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 20:39:31.744497 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:31.744456 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/60f9f978-737b-4279-927d-25f18934a44e-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"60f9f978-737b-4279-927d-25f18934a44e\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 20:39:31.744642 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:31.744604 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/60f9f978-737b-4279-927d-25f18934a44e-config-out\") pod \"alertmanager-main-0\" (UID: \"60f9f978-737b-4279-927d-25f18934a44e\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 20:39:31.745077 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:31.744732 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/60f9f978-737b-4279-927d-25f18934a44e-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"60f9f978-737b-4279-927d-25f18934a44e\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 20:39:31.745077 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:31.744917 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/60f9f978-737b-4279-927d-25f18934a44e-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"60f9f978-737b-4279-927d-25f18934a44e\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 20:39:31.745077 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:31.745053 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/60f9f978-737b-4279-927d-25f18934a44e-config-volume\") pod \"alertmanager-main-0\" (UID: \"60f9f978-737b-4279-927d-25f18934a44e\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 20:39:31.745571 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:31.745552 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/60f9f978-737b-4279-927d-25f18934a44e-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"60f9f978-737b-4279-927d-25f18934a44e\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 20:39:31.745661 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:31.745610 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/60f9f978-737b-4279-927d-25f18934a44e-tls-assets\") pod \"alertmanager-main-0\" (UID: \"60f9f978-737b-4279-927d-25f18934a44e\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 20:39:31.752478 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:31.752460 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t7vr9\" (UniqueName: \"kubernetes.io/projected/60f9f978-737b-4279-927d-25f18934a44e-kube-api-access-t7vr9\") pod \"alertmanager-main-0\" (UID: \"60f9f978-737b-4279-927d-25f18934a44e\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 20:39:31.850716 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:31.850686 4611 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 30 20:39:32.135947 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:32.135918 4611 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 20:39:32.138592 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:32.138563 4611 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60f9f978_737b_4279_927d_25f18934a44e.slice/crio-04a6ad0e4010eac913715f39a9857f19415b668ebc01bf0dfa6c48d2f00bdade WatchSource:0}: Error finding container 04a6ad0e4010eac913715f39a9857f19415b668ebc01bf0dfa6c48d2f00bdade: Status 404 returned error can't find the container with id 04a6ad0e4010eac913715f39a9857f19415b668ebc01bf0dfa6c48d2f00bdade Oct 30 20:39:32.538224 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:32.538157 4611 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-549c68bbc-hg8cl"] Oct 30 20:39:32.540928 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:32.540907 4611 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-549c68bbc-hg8cl" Oct 30 20:39:32.554016 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:32.553823 4611 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Oct 30 20:39:32.554016 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:32.553879 4611 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-casaeb65jqouu\"" Oct 30 20:39:32.554016 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:32.553918 4611 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-qllwd\"" Oct 30 20:39:32.554016 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:32.553924 4611 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Oct 30 20:39:32.554016 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:32.553965 4611 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Oct 30 20:39:32.554016 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:32.553967 4611 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Oct 30 20:39:32.554016 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:32.554001 4611 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Oct 30 20:39:32.560120 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:32.560091 4611 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-549c68bbc-hg8cl"] Oct 30 20:39:32.646867 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:32.646832 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/27d1aa75-f4b0-4223-a1d3-548b8b3374d3-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-549c68bbc-hg8cl\" (UID: \"27d1aa75-f4b0-4223-a1d3-548b8b3374d3\") " pod="openshift-monitoring/thanos-querier-549c68bbc-hg8cl" Oct 30 20:39:32.646867 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:32.646879 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/27d1aa75-f4b0-4223-a1d3-548b8b3374d3-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-549c68bbc-hg8cl\" (UID: \"27d1aa75-f4b0-4223-a1d3-548b8b3374d3\") " pod="openshift-monitoring/thanos-querier-549c68bbc-hg8cl" Oct 30 20:39:32.647072 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:32.646909 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/27d1aa75-f4b0-4223-a1d3-548b8b3374d3-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-549c68bbc-hg8cl\" (UID: \"27d1aa75-f4b0-4223-a1d3-548b8b3374d3\") " pod="openshift-monitoring/thanos-querier-549c68bbc-hg8cl" Oct 30 20:39:32.647072 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:32.647053 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/27d1aa75-f4b0-4223-a1d3-548b8b3374d3-secret-grpc-tls\") pod \"thanos-querier-549c68bbc-hg8cl\" (UID: \"27d1aa75-f4b0-4223-a1d3-548b8b3374d3\") " pod="openshift-monitoring/thanos-querier-549c68bbc-hg8cl" Oct 30 20:39:32.647149 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:32.647081 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/27d1aa75-f4b0-4223-a1d3-548b8b3374d3-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-549c68bbc-hg8cl\" (UID: \"27d1aa75-f4b0-4223-a1d3-548b8b3374d3\") " pod="openshift-monitoring/thanos-querier-549c68bbc-hg8cl" Oct 30 20:39:32.647149 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:32.647112 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/27d1aa75-f4b0-4223-a1d3-548b8b3374d3-secret-thanos-querier-tls\") pod \"thanos-querier-549c68bbc-hg8cl\" (UID: \"27d1aa75-f4b0-4223-a1d3-548b8b3374d3\") " pod="openshift-monitoring/thanos-querier-549c68bbc-hg8cl" Oct 30 20:39:32.647210 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:32.647145 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5rggj\" (UniqueName: \"kubernetes.io/projected/27d1aa75-f4b0-4223-a1d3-548b8b3374d3-kube-api-access-5rggj\") pod \"thanos-querier-549c68bbc-hg8cl\" (UID: \"27d1aa75-f4b0-4223-a1d3-548b8b3374d3\") " pod="openshift-monitoring/thanos-querier-549c68bbc-hg8cl" Oct 30 20:39:32.647210 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:32.647191 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/27d1aa75-f4b0-4223-a1d3-548b8b3374d3-metrics-client-ca\") pod \"thanos-querier-549c68bbc-hg8cl\" (UID: \"27d1aa75-f4b0-4223-a1d3-548b8b3374d3\") " pod="openshift-monitoring/thanos-querier-549c68bbc-hg8cl" Oct 30 20:39:32.647271 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:32.647216 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/27d1aa75-f4b0-4223-a1d3-548b8b3374d3-tmp\") pod \"thanos-querier-549c68bbc-hg8cl\" (UID: \"27d1aa75-f4b0-4223-a1d3-548b8b3374d3\") " pod="openshift-monitoring/thanos-querier-549c68bbc-hg8cl" Oct 30 20:39:32.721694 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:32.721667 4611 generic.go:356] "Generic (PLEG): container finished" podID="7a680d75-deea-4997-b813-1a144a1a9435" containerID="a013e0f79883022fc4b36d9ac867780e30452b5df14a1454459f27efdd23b6f2" exitCode=0 Oct 30 20:39:32.721863 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:32.721755 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-d4qf4" event={"ID":"7a680d75-deea-4997-b813-1a144a1a9435","Type":"ContainerDied","Data":"a013e0f79883022fc4b36d9ac867780e30452b5df14a1454459f27efdd23b6f2"} Oct 30 20:39:32.723168 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:32.723132 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"60f9f978-737b-4279-927d-25f18934a44e","Type":"ContainerStarted","Data":"04a6ad0e4010eac913715f39a9857f19415b668ebc01bf0dfa6c48d2f00bdade"} Oct 30 20:39:32.747561 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:32.747531 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/27d1aa75-f4b0-4223-a1d3-548b8b3374d3-secret-thanos-querier-tls\") pod \"thanos-querier-549c68bbc-hg8cl\" (UID: \"27d1aa75-f4b0-4223-a1d3-548b8b3374d3\") " pod="openshift-monitoring/thanos-querier-549c68bbc-hg8cl" Oct 30 20:39:32.747703 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:32.747567 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5rggj\" (UniqueName: \"kubernetes.io/projected/27d1aa75-f4b0-4223-a1d3-548b8b3374d3-kube-api-access-5rggj\") pod \"thanos-querier-549c68bbc-hg8cl\" (UID: \"27d1aa75-f4b0-4223-a1d3-548b8b3374d3\") " pod="openshift-monitoring/thanos-querier-549c68bbc-hg8cl" Oct 30 20:39:32.747703 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:32.747593 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/27d1aa75-f4b0-4223-a1d3-548b8b3374d3-metrics-client-ca\") pod \"thanos-querier-549c68bbc-hg8cl\" (UID: \"27d1aa75-f4b0-4223-a1d3-548b8b3374d3\") " pod="openshift-monitoring/thanos-querier-549c68bbc-hg8cl" Oct 30 20:39:32.747703 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:32.747623 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/27d1aa75-f4b0-4223-a1d3-548b8b3374d3-tmp\") pod \"thanos-querier-549c68bbc-hg8cl\" (UID: \"27d1aa75-f4b0-4223-a1d3-548b8b3374d3\") " pod="openshift-monitoring/thanos-querier-549c68bbc-hg8cl" Oct 30 20:39:32.747703 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:32.747689 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/27d1aa75-f4b0-4223-a1d3-548b8b3374d3-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-549c68bbc-hg8cl\" (UID: \"27d1aa75-f4b0-4223-a1d3-548b8b3374d3\") " pod="openshift-monitoring/thanos-querier-549c68bbc-hg8cl" Oct 30 20:39:32.748596 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:32.747718 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/27d1aa75-f4b0-4223-a1d3-548b8b3374d3-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-549c68bbc-hg8cl\" (UID: \"27d1aa75-f4b0-4223-a1d3-548b8b3374d3\") " pod="openshift-monitoring/thanos-querier-549c68bbc-hg8cl" Oct 30 20:39:32.748596 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:32.747749 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/27d1aa75-f4b0-4223-a1d3-548b8b3374d3-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-549c68bbc-hg8cl\" (UID: \"27d1aa75-f4b0-4223-a1d3-548b8b3374d3\") " pod="openshift-monitoring/thanos-querier-549c68bbc-hg8cl" Oct 30 20:39:32.748596 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:32.747781 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/27d1aa75-f4b0-4223-a1d3-548b8b3374d3-secret-grpc-tls\") pod \"thanos-querier-549c68bbc-hg8cl\" (UID: \"27d1aa75-f4b0-4223-a1d3-548b8b3374d3\") " pod="openshift-monitoring/thanos-querier-549c68bbc-hg8cl" Oct 30 20:39:32.748596 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:32.747809 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/27d1aa75-f4b0-4223-a1d3-548b8b3374d3-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-549c68bbc-hg8cl\" (UID: \"27d1aa75-f4b0-4223-a1d3-548b8b3374d3\") " pod="openshift-monitoring/thanos-querier-549c68bbc-hg8cl" Oct 30 20:39:32.752781 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:32.748670 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/27d1aa75-f4b0-4223-a1d3-548b8b3374d3-metrics-client-ca\") pod \"thanos-querier-549c68bbc-hg8cl\" (UID: \"27d1aa75-f4b0-4223-a1d3-548b8b3374d3\") " pod="openshift-monitoring/thanos-querier-549c68bbc-hg8cl" Oct 30 20:39:32.752781 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:32.749308 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/27d1aa75-f4b0-4223-a1d3-548b8b3374d3-tmp\") pod \"thanos-querier-549c68bbc-hg8cl\" (UID: \"27d1aa75-f4b0-4223-a1d3-548b8b3374d3\") " pod="openshift-monitoring/thanos-querier-549c68bbc-hg8cl" Oct 30 20:39:32.752781 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:32.750740 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/27d1aa75-f4b0-4223-a1d3-548b8b3374d3-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-549c68bbc-hg8cl\" (UID: \"27d1aa75-f4b0-4223-a1d3-548b8b3374d3\") " pod="openshift-monitoring/thanos-querier-549c68bbc-hg8cl" Oct 30 20:39:32.752781 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:32.751197 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/27d1aa75-f4b0-4223-a1d3-548b8b3374d3-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-549c68bbc-hg8cl\" (UID: \"27d1aa75-f4b0-4223-a1d3-548b8b3374d3\") " pod="openshift-monitoring/thanos-querier-549c68bbc-hg8cl" Oct 30 20:39:32.752781 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:32.751445 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/27d1aa75-f4b0-4223-a1d3-548b8b3374d3-secret-thanos-querier-tls\") pod \"thanos-querier-549c68bbc-hg8cl\" (UID: \"27d1aa75-f4b0-4223-a1d3-548b8b3374d3\") " pod="openshift-monitoring/thanos-querier-549c68bbc-hg8cl" Oct 30 20:39:32.752982 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:32.752948 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/27d1aa75-f4b0-4223-a1d3-548b8b3374d3-secret-grpc-tls\") pod \"thanos-querier-549c68bbc-hg8cl\" (UID: \"27d1aa75-f4b0-4223-a1d3-548b8b3374d3\") " pod="openshift-monitoring/thanos-querier-549c68bbc-hg8cl" Oct 30 20:39:32.753236 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:32.753216 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/27d1aa75-f4b0-4223-a1d3-548b8b3374d3-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-549c68bbc-hg8cl\" (UID: \"27d1aa75-f4b0-4223-a1d3-548b8b3374d3\") " pod="openshift-monitoring/thanos-querier-549c68bbc-hg8cl" Oct 30 20:39:32.753313 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:32.753266 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/27d1aa75-f4b0-4223-a1d3-548b8b3374d3-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-549c68bbc-hg8cl\" (UID: \"27d1aa75-f4b0-4223-a1d3-548b8b3374d3\") " pod="openshift-monitoring/thanos-querier-549c68bbc-hg8cl" Oct 30 20:39:32.760695 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:32.760669 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5rggj\" (UniqueName: \"kubernetes.io/projected/27d1aa75-f4b0-4223-a1d3-548b8b3374d3-kube-api-access-5rggj\") pod \"thanos-querier-549c68bbc-hg8cl\" (UID: \"27d1aa75-f4b0-4223-a1d3-548b8b3374d3\") " pod="openshift-monitoring/thanos-querier-549c68bbc-hg8cl" Oct 30 20:39:32.857628 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:32.857604 4611 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-549c68bbc-hg8cl" Oct 30 20:39:33.156845 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:33.156744 4611 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-549c68bbc-hg8cl"] Oct 30 20:39:33.195300 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:33.195264 4611 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27d1aa75_f4b0_4223_a1d3_548b8b3374d3.slice/crio-1993b9839a56c3db1c794da697d40aba9dac40c02ad228f8155857b9c4616a5f WatchSource:0}: Error finding container 1993b9839a56c3db1c794da697d40aba9dac40c02ad228f8155857b9c4616a5f: Status 404 returned error can't find the container with id 1993b9839a56c3db1c794da697d40aba9dac40c02ad228f8155857b9c4616a5f Oct 30 20:39:33.471419 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:33.471372 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:39:33.612081 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:33.612051 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:39:33.729073 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:33.729000 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-d4qf4" event={"ID":"7a680d75-deea-4997-b813-1a144a1a9435","Type":"ContainerStarted","Data":"c6b8869c28a0864e153a5a07ee1cccc9c56ac05671d8b93b7f9433a3bc5f9fad"} Oct 30 20:39:33.729073 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:33.729062 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-d4qf4" event={"ID":"7a680d75-deea-4997-b813-1a144a1a9435","Type":"ContainerStarted","Data":"d290936d06e2a09fcb203bec5d3c4a0ed8c0b40a0ae6b17b18594c76f14b4737"} Oct 30 20:39:33.730426 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:33.730398 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-549c68bbc-hg8cl" event={"ID":"27d1aa75-f4b0-4223-a1d3-548b8b3374d3","Type":"ContainerStarted","Data":"1993b9839a56c3db1c794da697d40aba9dac40c02ad228f8155857b9c4616a5f"} Oct 30 20:39:33.732013 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:33.731989 4611 generic.go:356] "Generic (PLEG): container finished" podID="60f9f978-737b-4279-927d-25f18934a44e" containerID="11d12b7b64f786ce05d475a22d175128b43566a4a574b021609b54c310875319" exitCode=0 Oct 30 20:39:33.732133 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:33.732017 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"60f9f978-737b-4279-927d-25f18934a44e","Type":"ContainerDied","Data":"11d12b7b64f786ce05d475a22d175128b43566a4a574b021609b54c310875319"} Oct 30 20:39:33.784991 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:33.784940 4611 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-d4qf4" podStartSLOduration=2.843376944 podStartE2EDuration="3.784925067s" totalImagesPullingTime="874.724828ms" totalInitContainerRuntime="66.823295ms" isStatefulPod=true podCreationTimestamp="2026-10-30 20:39:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:39:33.750372901 +0000 UTC m=+28.284399518" watchObservedRunningTime="2026-10-30 20:39:33.784925067 +0000 UTC m=+28.318951677" Oct 30 20:39:33.919355 ip-10-0-133-192 kubenswrapper[4611]: E1030 20:39:33.919312 4611 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hwzc6" containerName="extractor" Oct 30 20:39:35.151261 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:35.151236 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-jp2cr" Oct 30 20:39:35.247021 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:35.246987 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:39:35.447163 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:35.447132 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:39:35.533416 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:35.533391 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-8984b54b7-skmcd" Oct 30 20:39:35.610619 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:35.610577 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:39:35.740683 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:35.740651 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-549c68bbc-hg8cl" event={"ID":"27d1aa75-f4b0-4223-a1d3-548b8b3374d3","Type":"ContainerStarted","Data":"d246240edfd06aaeb52a7a1d4c978ae5a336f99bb3ddfbcd6bec29e3e2781844"} Oct 30 20:39:35.740834 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:35.740711 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-549c68bbc-hg8cl" event={"ID":"27d1aa75-f4b0-4223-a1d3-548b8b3374d3","Type":"ContainerStarted","Data":"228110b2f263f038a7a35f010fbfba132240199c438ebf4562d49ceeb684cb6e"} Oct 30 20:39:35.740834 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:35.740750 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-549c68bbc-hg8cl" event={"ID":"27d1aa75-f4b0-4223-a1d3-548b8b3374d3","Type":"ContainerStarted","Data":"318c3188913f07c1f4e5ae30c03ea3a281b2d8030dd304da1ebc77e23c60732c"} Oct 30 20:39:35.743655 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:35.743621 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"60f9f978-737b-4279-927d-25f18934a44e","Type":"ContainerStarted","Data":"ee350b5ab878d95e791e98ce1212f08c5ecc82128f4835efad634a77244e38cb"} Oct 30 20:39:35.743763 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:35.743681 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"60f9f978-737b-4279-927d-25f18934a44e","Type":"ContainerStarted","Data":"457204ac3591bc71feebd9ca375c2ec2e77e255e1ddc339f6302c22a787fea29"} Oct 30 20:39:35.743763 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:35.743733 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"60f9f978-737b-4279-927d-25f18934a44e","Type":"ContainerStarted","Data":"db46bb292b29cee7e347c34c47f3a13ffcd9b206c8bc54a2ee11564097fa5e7c"} Oct 30 20:39:35.743832 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:35.743763 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"60f9f978-737b-4279-927d-25f18934a44e","Type":"ContainerStarted","Data":"8862aa2f6b166bfaa0f82605eea54c6e3f7d1b1929f9dfc3d27c6150eeb13672"} Oct 30 20:39:35.743832 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:35.743786 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"60f9f978-737b-4279-927d-25f18934a44e","Type":"ContainerStarted","Data":"8fe5b6206f3963c00215018db89c750157d4a8d73a5c42abf6ba630b6905c626"} Oct 30 20:39:35.946559 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:35.946475 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:39:36.098387 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:36.098360 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:39:36.275802 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:36.275773 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:39:36.750243 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:36.750207 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-549c68bbc-hg8cl" event={"ID":"27d1aa75-f4b0-4223-a1d3-548b8b3374d3","Type":"ContainerStarted","Data":"a1a79cd9d523043c498e96a7b08fba0e6cc1dd4ca0fbd8d2ce1e6c83e2576360"} Oct 30 20:39:36.750409 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:36.750310 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-549c68bbc-hg8cl" event={"ID":"27d1aa75-f4b0-4223-a1d3-548b8b3374d3","Type":"ContainerStarted","Data":"5f93b1f2f9ef3899264895ee52122f1ebff5544a9367fc5da0b6e28fa0649ffc"} Oct 30 20:39:36.750409 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:36.750349 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-549c68bbc-hg8cl" Oct 30 20:39:36.750409 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:36.750385 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-549c68bbc-hg8cl" event={"ID":"27d1aa75-f4b0-4223-a1d3-548b8b3374d3","Type":"ContainerStarted","Data":"c51e50f87e421a63e7f4dc0d4f39688a18cbd345ba610e94cdbea10b9aeda9ab"} Oct 30 20:39:36.753862 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:36.753833 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"60f9f978-737b-4279-927d-25f18934a44e","Type":"ContainerStarted","Data":"d6533f84e1a198b7a022405fae9cc6c68db9f4a8c75cebac583749fff94fedf0"} Oct 30 20:39:36.775620 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:36.775558 4611 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-549c68bbc-hg8cl" podStartSLOduration=2.08911861 podStartE2EDuration="4.775543494s" totalImagesPullingTime="2.686424884s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:39:32 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:39:36.774870545 +0000 UTC m=+31.308897179" watchObservedRunningTime="2026-10-30 20:39:36.775543494 +0000 UTC m=+31.309570110" Oct 30 20:39:36.816894 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:36.816816 4611 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.781923437 podStartE2EDuration="5.816806334s" totalImagesPullingTime="3.023987161s" totalInitContainerRuntime="10.895736ms" isStatefulPod=false podCreationTimestamp="2026-10-30 20:39:31 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:39:36.816426643 +0000 UTC m=+31.350453258" watchObservedRunningTime="2026-10-30 20:39:36.816806334 +0000 UTC m=+31.350832950" Oct 30 20:39:36.864776 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:36.864749 4611 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 20:39:36.883309 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:36.883287 4611 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 20:39:36.895814 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:36.895793 4611 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 20:39:36.896835 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:36.896814 4611 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Oct 30 20:39:36.897012 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:36.896994 4611 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Oct 30 20:39:36.897069 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:36.896994 4611 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Oct 30 20:39:36.897106 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:36.897064 4611 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Oct 30 20:39:36.897173 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:36.897158 4611 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Oct 30 20:39:36.897221 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:36.897186 4611 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Oct 30 20:39:36.897221 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:36.897203 4611 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Oct 30 20:39:36.897284 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:36.897237 4611 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Oct 30 20:39:36.897284 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:36.897250 4611 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-7fopgrptfcdd5\"" Oct 30 20:39:36.897284 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:36.897192 4611 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Oct 30 20:39:36.897467 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:36.897452 4611 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Oct 30 20:39:36.897523 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:36.897489 4611 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Oct 30 20:39:36.897888 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:36.897870 4611 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-xtgt6\"" Oct 30 20:39:36.899188 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:36.899157 4611 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Oct 30 20:39:36.899481 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:36.899460 4611 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Oct 30 20:39:36.899669 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:36.899654 4611 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Oct 30 20:39:36.979577 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:36.979549 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:39:36.987108 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:36.987087 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c00ce45d-b9cf-4f43-a7ae-469d405ee218-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"c00ce45d-b9cf-4f43-a7ae-469d405ee218\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 20:39:36.987204 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:36.987114 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c00ce45d-b9cf-4f43-a7ae-469d405ee218-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"c00ce45d-b9cf-4f43-a7ae-469d405ee218\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 20:39:36.987204 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:36.987133 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/c00ce45d-b9cf-4f43-a7ae-469d405ee218-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"c00ce45d-b9cf-4f43-a7ae-469d405ee218\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 20:39:36.987204 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:36.987152 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/c00ce45d-b9cf-4f43-a7ae-469d405ee218-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"c00ce45d-b9cf-4f43-a7ae-469d405ee218\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 20:39:36.987317 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:36.987228 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/c00ce45d-b9cf-4f43-a7ae-469d405ee218-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"c00ce45d-b9cf-4f43-a7ae-469d405ee218\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 20:39:36.987317 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:36.987279 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c00ce45d-b9cf-4f43-a7ae-469d405ee218-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"c00ce45d-b9cf-4f43-a7ae-469d405ee218\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 20:39:36.987317 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:36.987303 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/c00ce45d-b9cf-4f43-a7ae-469d405ee218-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"c00ce45d-b9cf-4f43-a7ae-469d405ee218\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 20:39:36.987417 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:36.987338 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/c00ce45d-b9cf-4f43-a7ae-469d405ee218-config\") pod \"prometheus-k8s-0\" (UID: \"c00ce45d-b9cf-4f43-a7ae-469d405ee218\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 20:39:36.987417 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:36.987353 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/c00ce45d-b9cf-4f43-a7ae-469d405ee218-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"c00ce45d-b9cf-4f43-a7ae-469d405ee218\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 20:39:36.987417 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:36.987370 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c00ce45d-b9cf-4f43-a7ae-469d405ee218-config-out\") pod \"prometheus-k8s-0\" (UID: \"c00ce45d-b9cf-4f43-a7ae-469d405ee218\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 20:39:36.987417 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:36.987401 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xwfmh\" (UniqueName: \"kubernetes.io/projected/c00ce45d-b9cf-4f43-a7ae-469d405ee218-kube-api-access-xwfmh\") pod \"prometheus-k8s-0\" (UID: \"c00ce45d-b9cf-4f43-a7ae-469d405ee218\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 20:39:36.987584 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:36.987439 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/c00ce45d-b9cf-4f43-a7ae-469d405ee218-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"c00ce45d-b9cf-4f43-a7ae-469d405ee218\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 20:39:36.987584 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:36.987471 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/c00ce45d-b9cf-4f43-a7ae-469d405ee218-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"c00ce45d-b9cf-4f43-a7ae-469d405ee218\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 20:39:36.987584 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:36.987542 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/c00ce45d-b9cf-4f43-a7ae-469d405ee218-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"c00ce45d-b9cf-4f43-a7ae-469d405ee218\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 20:39:36.987676 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:36.987624 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c00ce45d-b9cf-4f43-a7ae-469d405ee218-web-config\") pod \"prometheus-k8s-0\" (UID: \"c00ce45d-b9cf-4f43-a7ae-469d405ee218\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 20:39:36.987676 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:36.987659 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c00ce45d-b9cf-4f43-a7ae-469d405ee218-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"c00ce45d-b9cf-4f43-a7ae-469d405ee218\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 20:39:36.987770 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:36.987687 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/c00ce45d-b9cf-4f43-a7ae-469d405ee218-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"c00ce45d-b9cf-4f43-a7ae-469d405ee218\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 20:39:36.987827 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:36.987790 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c00ce45d-b9cf-4f43-a7ae-469d405ee218-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c00ce45d-b9cf-4f43-a7ae-469d405ee218\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 20:39:36.987876 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:36.987826 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c00ce45d-b9cf-4f43-a7ae-469d405ee218-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c00ce45d-b9cf-4f43-a7ae-469d405ee218\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 20:39:36.987876 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:36.987865 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c00ce45d-b9cf-4f43-a7ae-469d405ee218-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c00ce45d-b9cf-4f43-a7ae-469d405ee218\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 20:39:36.995423 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:36.995409 4611 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Oct 30 20:39:37.088701 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:37.088647 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c00ce45d-b9cf-4f43-a7ae-469d405ee218-web-config\") pod \"prometheus-k8s-0\" (UID: \"c00ce45d-b9cf-4f43-a7ae-469d405ee218\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 20:39:37.088701 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:37.088675 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c00ce45d-b9cf-4f43-a7ae-469d405ee218-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"c00ce45d-b9cf-4f43-a7ae-469d405ee218\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 20:39:37.088701 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:37.088692 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/c00ce45d-b9cf-4f43-a7ae-469d405ee218-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"c00ce45d-b9cf-4f43-a7ae-469d405ee218\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 20:39:37.088911 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:37.088737 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c00ce45d-b9cf-4f43-a7ae-469d405ee218-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c00ce45d-b9cf-4f43-a7ae-469d405ee218\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 20:39:37.088911 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:37.088764 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c00ce45d-b9cf-4f43-a7ae-469d405ee218-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c00ce45d-b9cf-4f43-a7ae-469d405ee218\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 20:39:37.088911 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:37.088786 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c00ce45d-b9cf-4f43-a7ae-469d405ee218-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c00ce45d-b9cf-4f43-a7ae-469d405ee218\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 20:39:37.088911 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:37.088818 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c00ce45d-b9cf-4f43-a7ae-469d405ee218-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"c00ce45d-b9cf-4f43-a7ae-469d405ee218\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 20:39:37.088911 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:37.088842 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c00ce45d-b9cf-4f43-a7ae-469d405ee218-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"c00ce45d-b9cf-4f43-a7ae-469d405ee218\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 20:39:37.090331 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:37.089923 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/c00ce45d-b9cf-4f43-a7ae-469d405ee218-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"c00ce45d-b9cf-4f43-a7ae-469d405ee218\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 20:39:37.090331 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:37.089956 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c00ce45d-b9cf-4f43-a7ae-469d405ee218-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c00ce45d-b9cf-4f43-a7ae-469d405ee218\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 20:39:37.090331 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:37.090024 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c00ce45d-b9cf-4f43-a7ae-469d405ee218-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"c00ce45d-b9cf-4f43-a7ae-469d405ee218\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 20:39:37.090331 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:37.088873 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/c00ce45d-b9cf-4f43-a7ae-469d405ee218-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"c00ce45d-b9cf-4f43-a7ae-469d405ee218\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 20:39:37.090331 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:37.090158 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/c00ce45d-b9cf-4f43-a7ae-469d405ee218-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"c00ce45d-b9cf-4f43-a7ae-469d405ee218\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 20:39:37.090331 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:37.090291 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/c00ce45d-b9cf-4f43-a7ae-469d405ee218-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"c00ce45d-b9cf-4f43-a7ae-469d405ee218\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 20:39:37.090331 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:37.090331 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c00ce45d-b9cf-4f43-a7ae-469d405ee218-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"c00ce45d-b9cf-4f43-a7ae-469d405ee218\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 20:39:37.090743 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:37.090368 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/c00ce45d-b9cf-4f43-a7ae-469d405ee218-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"c00ce45d-b9cf-4f43-a7ae-469d405ee218\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 20:39:37.090743 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:37.090397 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/c00ce45d-b9cf-4f43-a7ae-469d405ee218-config\") pod \"prometheus-k8s-0\" (UID: \"c00ce45d-b9cf-4f43-a7ae-469d405ee218\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 20:39:37.090743 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:37.090431 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/c00ce45d-b9cf-4f43-a7ae-469d405ee218-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"c00ce45d-b9cf-4f43-a7ae-469d405ee218\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 20:39:37.090898 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:37.090750 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c00ce45d-b9cf-4f43-a7ae-469d405ee218-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c00ce45d-b9cf-4f43-a7ae-469d405ee218\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 20:39:37.091955 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:37.091041 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c00ce45d-b9cf-4f43-a7ae-469d405ee218-config-out\") pod \"prometheus-k8s-0\" (UID: \"c00ce45d-b9cf-4f43-a7ae-469d405ee218\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 20:39:37.091955 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:37.091120 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xwfmh\" (UniqueName: \"kubernetes.io/projected/c00ce45d-b9cf-4f43-a7ae-469d405ee218-kube-api-access-xwfmh\") pod \"prometheus-k8s-0\" (UID: \"c00ce45d-b9cf-4f43-a7ae-469d405ee218\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 20:39:37.091955 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:37.091157 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/c00ce45d-b9cf-4f43-a7ae-469d405ee218-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"c00ce45d-b9cf-4f43-a7ae-469d405ee218\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 20:39:37.091955 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:37.091306 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/c00ce45d-b9cf-4f43-a7ae-469d405ee218-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"c00ce45d-b9cf-4f43-a7ae-469d405ee218\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 20:39:37.091955 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:37.091192 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c00ce45d-b9cf-4f43-a7ae-469d405ee218-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c00ce45d-b9cf-4f43-a7ae-469d405ee218\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 20:39:37.091955 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:37.091351 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/c00ce45d-b9cf-4f43-a7ae-469d405ee218-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"c00ce45d-b9cf-4f43-a7ae-469d405ee218\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 20:39:37.091955 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:37.091396 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/c00ce45d-b9cf-4f43-a7ae-469d405ee218-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"c00ce45d-b9cf-4f43-a7ae-469d405ee218\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 20:39:37.093321 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:37.092986 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c00ce45d-b9cf-4f43-a7ae-469d405ee218-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"c00ce45d-b9cf-4f43-a7ae-469d405ee218\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 20:39:37.095035 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:37.094485 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/c00ce45d-b9cf-4f43-a7ae-469d405ee218-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"c00ce45d-b9cf-4f43-a7ae-469d405ee218\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 20:39:37.095035 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:37.094991 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/c00ce45d-b9cf-4f43-a7ae-469d405ee218-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"c00ce45d-b9cf-4f43-a7ae-469d405ee218\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 20:39:37.095198 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:37.095170 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c00ce45d-b9cf-4f43-a7ae-469d405ee218-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"c00ce45d-b9cf-4f43-a7ae-469d405ee218\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 20:39:37.096696 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:37.096668 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/c00ce45d-b9cf-4f43-a7ae-469d405ee218-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"c00ce45d-b9cf-4f43-a7ae-469d405ee218\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 20:39:37.097632 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:37.096898 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c00ce45d-b9cf-4f43-a7ae-469d405ee218-config-out\") pod \"prometheus-k8s-0\" (UID: \"c00ce45d-b9cf-4f43-a7ae-469d405ee218\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 20:39:37.097632 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:37.097057 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c00ce45d-b9cf-4f43-a7ae-469d405ee218-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"c00ce45d-b9cf-4f43-a7ae-469d405ee218\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 20:39:37.097937 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:37.097911 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/c00ce45d-b9cf-4f43-a7ae-469d405ee218-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"c00ce45d-b9cf-4f43-a7ae-469d405ee218\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 20:39:37.099006 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:37.098977 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/c00ce45d-b9cf-4f43-a7ae-469d405ee218-config\") pod \"prometheus-k8s-0\" (UID: \"c00ce45d-b9cf-4f43-a7ae-469d405ee218\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 20:39:37.099073 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:37.099019 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/c00ce45d-b9cf-4f43-a7ae-469d405ee218-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"c00ce45d-b9cf-4f43-a7ae-469d405ee218\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 20:39:37.099073 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:37.099068 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c00ce45d-b9cf-4f43-a7ae-469d405ee218-web-config\") pod \"prometheus-k8s-0\" (UID: \"c00ce45d-b9cf-4f43-a7ae-469d405ee218\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 20:39:37.099470 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:37.099440 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/c00ce45d-b9cf-4f43-a7ae-469d405ee218-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"c00ce45d-b9cf-4f43-a7ae-469d405ee218\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 20:39:37.099590 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:37.099573 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/c00ce45d-b9cf-4f43-a7ae-469d405ee218-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"c00ce45d-b9cf-4f43-a7ae-469d405ee218\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 20:39:37.104775 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:37.104756 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xwfmh\" (UniqueName: \"kubernetes.io/projected/c00ce45d-b9cf-4f43-a7ae-469d405ee218-kube-api-access-xwfmh\") pod \"prometheus-k8s-0\" (UID: \"c00ce45d-b9cf-4f43-a7ae-469d405ee218\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 20:39:37.131263 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:37.131241 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:39:37.302281 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:37.302257 4611 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 20:39:37.617312 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:37.617284 4611 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 20:39:37.620476 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:39:37.620446 4611 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc00ce45d_b9cf_4f43_a7ae_469d405ee218.slice/crio-f855654166e32ab8d996b1cc21ca3d12660c6c66ab62294cb3a9e35c2a1cb994 WatchSource:0}: Error finding container f855654166e32ab8d996b1cc21ca3d12660c6c66ab62294cb3a9e35c2a1cb994: Status 404 returned error can't find the container with id f855654166e32ab8d996b1cc21ca3d12660c6c66ab62294cb3a9e35c2a1cb994 Oct 30 20:39:37.758936 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:37.758911 4611 generic.go:356] "Generic (PLEG): container finished" podID="c00ce45d-b9cf-4f43-a7ae-469d405ee218" containerID="1ef98c6b694a49f92cb7dcf6f66bf657f9ad24fcb0a70964bb07914279067535" exitCode=0 Oct 30 20:39:37.759061 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:37.758990 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c00ce45d-b9cf-4f43-a7ae-469d405ee218","Type":"ContainerDied","Data":"1ef98c6b694a49f92cb7dcf6f66bf657f9ad24fcb0a70964bb07914279067535"} Oct 30 20:39:37.759123 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:37.759067 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c00ce45d-b9cf-4f43-a7ae-469d405ee218","Type":"ContainerStarted","Data":"f855654166e32ab8d996b1cc21ca3d12660c6c66ab62294cb3a9e35c2a1cb994"} Oct 30 20:39:41.776840 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:41.776772 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c00ce45d-b9cf-4f43-a7ae-469d405ee218","Type":"ContainerStarted","Data":"bd780bc3fb65e4d12779165c27a6dfbdbd45920cf73c112ad98a670756e1a4e5"} Oct 30 20:39:41.776840 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:41.776833 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c00ce45d-b9cf-4f43-a7ae-469d405ee218","Type":"ContainerStarted","Data":"84d757c8c869d6915ec69bbacff82881bdbf1d6930d62edabe31443f7ffa090d"} Oct 30 20:39:41.777237 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:41.776859 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c00ce45d-b9cf-4f43-a7ae-469d405ee218","Type":"ContainerStarted","Data":"7541462a2fd6aa2efe5b45e1416c0487a63fc404e5fae807966ebf120d5a02ff"} Oct 30 20:39:41.777237 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:41.776882 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c00ce45d-b9cf-4f43-a7ae-469d405ee218","Type":"ContainerStarted","Data":"498b8f447913baf70893f05429326f82ad6ef7df92bf823401bdb6744fd54f48"} Oct 30 20:39:41.777237 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:41.776903 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c00ce45d-b9cf-4f43-a7ae-469d405ee218","Type":"ContainerStarted","Data":"93f724d20999d506314f28a07321f47391012d60d7c2a8b383e4b73a79fc6d9c"} Oct 30 20:39:41.777237 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:41.776927 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c00ce45d-b9cf-4f43-a7ae-469d405ee218","Type":"ContainerStarted","Data":"07b909a4c6ca8a75b95d21e72ed6f8d1983e7851052e2c7789ff0f19df742b5b"} Oct 30 20:39:41.816881 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:41.816833 4611 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.467253884 podStartE2EDuration="5.816820202s" totalImagesPullingTime="3.336861951s" totalInitContainerRuntime="12.704367ms" isStatefulPod=false podCreationTimestamp="2026-10-30 20:39:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:39:41.815170888 +0000 UTC m=+36.349197505" watchObservedRunningTime="2026-10-30 20:39:41.816820202 +0000 UTC m=+36.350846814" Oct 30 20:39:42.302621 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:42.302591 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 20:39:42.765060 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:42.765036 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-549c68bbc-hg8cl" Oct 30 20:39:43.414436 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:43.414399 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:39:43.559580 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:43.559538 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:39:45.855928 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:45.855895 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:39:46.016520 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:46.016487 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:39:47.739888 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:47.739855 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:39:47.922908 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:47.922831 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:39:48.066866 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:48.066780 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:39:49.557218 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:49.557192 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:39:49.762325 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:49.762296 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:39:52.418925 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:52.418894 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:39:52.682128 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:52.682054 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:39:55.186660 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:55.186623 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:39:55.427430 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:39:55.427396 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:40:02.844350 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:02.844311 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:40:03.001970 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:03.001941 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:40:03.915187 ip-10-0-133-192 kubenswrapper[4611]: E1030 20:40:03.915148 4611 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hwzc6" containerName="extractor" Oct 30 20:40:05.875175 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:05.875139 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:40:06.024724 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:06.024695 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:40:13.034647 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:13.034606 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:40:13.209113 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:13.209079 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:40:20.872435 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:20.872399 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:40:21.026882 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:21.026845 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:40:30.862969 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:30.862938 4611 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 20:40:30.880995 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:30.880966 4611 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 20:40:30.954150 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:30.954122 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:40:30.963517 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:30.963484 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 20:40:31.159675 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:31.159589 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:40:33.914519 ip-10-0-133-192 kubenswrapper[4611]: E1030 20:40:33.914463 4611 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hwzc6" containerName="extractor" Oct 30 20:40:39.743220 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:39.743183 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:40:39.889873 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:39.889845 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:40:40.751359 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:40.751331 4611 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 20:40:40.751867 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:40.751826 4611 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="60f9f978-737b-4279-927d-25f18934a44e" containerName="alertmanager" containerID="cri-o://8fe5b6206f3963c00215018db89c750157d4a8d73a5c42abf6ba630b6905c626" gracePeriod=120 Oct 30 20:40:40.751964 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:40.751855 4611 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="60f9f978-737b-4279-927d-25f18934a44e" containerName="kube-rbac-proxy-web" containerID="cri-o://db46bb292b29cee7e347c34c47f3a13ffcd9b206c8bc54a2ee11564097fa5e7c" gracePeriod=120 Oct 30 20:40:40.751964 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:40.751863 4611 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="60f9f978-737b-4279-927d-25f18934a44e" containerName="config-reloader" containerID="cri-o://8862aa2f6b166bfaa0f82605eea54c6e3f7d1b1929f9dfc3d27c6150eeb13672" gracePeriod=120 Oct 30 20:40:40.751964 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:40.751911 4611 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="60f9f978-737b-4279-927d-25f18934a44e" containerName="prom-label-proxy" containerID="cri-o://d6533f84e1a198b7a022405fae9cc6c68db9f4a8c75cebac583749fff94fedf0" gracePeriod=120 Oct 30 20:40:40.752104 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:40.751942 4611 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="60f9f978-737b-4279-927d-25f18934a44e" containerName="kube-rbac-proxy-metric" containerID="cri-o://ee350b5ab878d95e791e98ce1212f08c5ecc82128f4835efad634a77244e38cb" gracePeriod=120 Oct 30 20:40:40.752181 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:40.751922 4611 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="60f9f978-737b-4279-927d-25f18934a44e" containerName="kube-rbac-proxy" containerID="cri-o://457204ac3591bc71feebd9ca375c2ec2e77e255e1ddc339f6302c22a787fea29" gracePeriod=120 Oct 30 20:40:40.983605 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:40.983580 4611 generic.go:356] "Generic (PLEG): container finished" podID="60f9f978-737b-4279-927d-25f18934a44e" containerID="d6533f84e1a198b7a022405fae9cc6c68db9f4a8c75cebac583749fff94fedf0" exitCode=0 Oct 30 20:40:40.983605 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:40.983601 4611 generic.go:356] "Generic (PLEG): container finished" podID="60f9f978-737b-4279-927d-25f18934a44e" containerID="ee350b5ab878d95e791e98ce1212f08c5ecc82128f4835efad634a77244e38cb" exitCode=0 Oct 30 20:40:40.983605 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:40.983610 4611 generic.go:356] "Generic (PLEG): container finished" podID="60f9f978-737b-4279-927d-25f18934a44e" containerID="457204ac3591bc71feebd9ca375c2ec2e77e255e1ddc339f6302c22a787fea29" exitCode=0 Oct 30 20:40:40.983779 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:40.983616 4611 generic.go:356] "Generic (PLEG): container finished" podID="60f9f978-737b-4279-927d-25f18934a44e" containerID="8862aa2f6b166bfaa0f82605eea54c6e3f7d1b1929f9dfc3d27c6150eeb13672" exitCode=0 Oct 30 20:40:40.983779 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:40.983621 4611 generic.go:356] "Generic (PLEG): container finished" podID="60f9f978-737b-4279-927d-25f18934a44e" containerID="8fe5b6206f3963c00215018db89c750157d4a8d73a5c42abf6ba630b6905c626" exitCode=0 Oct 30 20:40:40.983779 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:40.983652 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"60f9f978-737b-4279-927d-25f18934a44e","Type":"ContainerDied","Data":"d6533f84e1a198b7a022405fae9cc6c68db9f4a8c75cebac583749fff94fedf0"} Oct 30 20:40:40.983779 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:40.983726 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"60f9f978-737b-4279-927d-25f18934a44e","Type":"ContainerDied","Data":"ee350b5ab878d95e791e98ce1212f08c5ecc82128f4835efad634a77244e38cb"} Oct 30 20:40:40.983779 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:40.983759 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"60f9f978-737b-4279-927d-25f18934a44e","Type":"ContainerDied","Data":"457204ac3591bc71feebd9ca375c2ec2e77e255e1ddc339f6302c22a787fea29"} Oct 30 20:40:40.983925 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:40.983785 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"60f9f978-737b-4279-927d-25f18934a44e","Type":"ContainerDied","Data":"8862aa2f6b166bfaa0f82605eea54c6e3f7d1b1929f9dfc3d27c6150eeb13672"} Oct 30 20:40:40.983925 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:40.983808 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"60f9f978-737b-4279-927d-25f18934a44e","Type":"ContainerDied","Data":"8fe5b6206f3963c00215018db89c750157d4a8d73a5c42abf6ba630b6905c626"} Oct 30 20:40:41.990312 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:41.990287 4611 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 30 20:40:41.990635 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:41.990420 4611 generic.go:356] "Generic (PLEG): container finished" podID="60f9f978-737b-4279-927d-25f18934a44e" containerID="db46bb292b29cee7e347c34c47f3a13ffcd9b206c8bc54a2ee11564097fa5e7c" exitCode=0 Oct 30 20:40:41.990635 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:41.990452 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"60f9f978-737b-4279-927d-25f18934a44e","Type":"ContainerDied","Data":"db46bb292b29cee7e347c34c47f3a13ffcd9b206c8bc54a2ee11564097fa5e7c"} Oct 30 20:40:41.990635 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:41.990538 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"60f9f978-737b-4279-927d-25f18934a44e","Type":"ContainerDied","Data":"04a6ad0e4010eac913715f39a9857f19415b668ebc01bf0dfa6c48d2f00bdade"} Oct 30 20:40:41.990635 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:41.990600 4611 scope.go:122] "RemoveContainer" containerID="d6533f84e1a198b7a022405fae9cc6c68db9f4a8c75cebac583749fff94fedf0" Oct 30 20:40:41.997682 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:41.997657 4611 scope.go:122] "RemoveContainer" containerID="ee350b5ab878d95e791e98ce1212f08c5ecc82128f4835efad634a77244e38cb" Oct 30 20:40:42.004013 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:42.003995 4611 scope.go:122] "RemoveContainer" containerID="457204ac3591bc71feebd9ca375c2ec2e77e255e1ddc339f6302c22a787fea29" Oct 30 20:40:42.010240 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:42.010223 4611 scope.go:122] "RemoveContainer" containerID="db46bb292b29cee7e347c34c47f3a13ffcd9b206c8bc54a2ee11564097fa5e7c" Oct 30 20:40:42.016238 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:42.016219 4611 scope.go:122] "RemoveContainer" containerID="8862aa2f6b166bfaa0f82605eea54c6e3f7d1b1929f9dfc3d27c6150eeb13672" Oct 30 20:40:42.022373 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:42.022355 4611 scope.go:122] "RemoveContainer" containerID="8fe5b6206f3963c00215018db89c750157d4a8d73a5c42abf6ba630b6905c626" Oct 30 20:40:42.031650 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:42.031553 4611 scope.go:122] "RemoveContainer" containerID="11d12b7b64f786ce05d475a22d175128b43566a4a574b021609b54c310875319" Oct 30 20:40:42.038926 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:42.038899 4611 scope.go:122] "RemoveContainer" containerID="d6533f84e1a198b7a022405fae9cc6c68db9f4a8c75cebac583749fff94fedf0" Oct 30 20:40:42.039226 ip-10-0-133-192 kubenswrapper[4611]: E1030 20:40:42.039205 4611 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d6533f84e1a198b7a022405fae9cc6c68db9f4a8c75cebac583749fff94fedf0\": container with ID starting with d6533f84e1a198b7a022405fae9cc6c68db9f4a8c75cebac583749fff94fedf0 not found: ID does not exist" containerID="d6533f84e1a198b7a022405fae9cc6c68db9f4a8c75cebac583749fff94fedf0" Oct 30 20:40:42.039308 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:42.039233 4611 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d6533f84e1a198b7a022405fae9cc6c68db9f4a8c75cebac583749fff94fedf0"} err="failed to get container status \"d6533f84e1a198b7a022405fae9cc6c68db9f4a8c75cebac583749fff94fedf0\": rpc error: code = NotFound desc = could not find container \"d6533f84e1a198b7a022405fae9cc6c68db9f4a8c75cebac583749fff94fedf0\": container with ID starting with d6533f84e1a198b7a022405fae9cc6c68db9f4a8c75cebac583749fff94fedf0 not found: ID does not exist" Oct 30 20:40:42.039308 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:42.039264 4611 scope.go:122] "RemoveContainer" containerID="ee350b5ab878d95e791e98ce1212f08c5ecc82128f4835efad634a77244e38cb" Oct 30 20:40:42.039542 ip-10-0-133-192 kubenswrapper[4611]: E1030 20:40:42.039520 4611 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ee350b5ab878d95e791e98ce1212f08c5ecc82128f4835efad634a77244e38cb\": container with ID starting with ee350b5ab878d95e791e98ce1212f08c5ecc82128f4835efad634a77244e38cb not found: ID does not exist" containerID="ee350b5ab878d95e791e98ce1212f08c5ecc82128f4835efad634a77244e38cb" Oct 30 20:40:42.039608 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:42.039548 4611 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee350b5ab878d95e791e98ce1212f08c5ecc82128f4835efad634a77244e38cb"} err="failed to get container status \"ee350b5ab878d95e791e98ce1212f08c5ecc82128f4835efad634a77244e38cb\": rpc error: code = NotFound desc = could not find container \"ee350b5ab878d95e791e98ce1212f08c5ecc82128f4835efad634a77244e38cb\": container with ID starting with ee350b5ab878d95e791e98ce1212f08c5ecc82128f4835efad634a77244e38cb not found: ID does not exist" Oct 30 20:40:42.039608 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:42.039562 4611 scope.go:122] "RemoveContainer" containerID="457204ac3591bc71feebd9ca375c2ec2e77e255e1ddc339f6302c22a787fea29" Oct 30 20:40:42.039830 ip-10-0-133-192 kubenswrapper[4611]: E1030 20:40:42.039811 4611 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"457204ac3591bc71feebd9ca375c2ec2e77e255e1ddc339f6302c22a787fea29\": container with ID starting with 457204ac3591bc71feebd9ca375c2ec2e77e255e1ddc339f6302c22a787fea29 not found: ID does not exist" containerID="457204ac3591bc71feebd9ca375c2ec2e77e255e1ddc339f6302c22a787fea29" Oct 30 20:40:42.039903 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:42.039835 4611 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"457204ac3591bc71feebd9ca375c2ec2e77e255e1ddc339f6302c22a787fea29"} err="failed to get container status \"457204ac3591bc71feebd9ca375c2ec2e77e255e1ddc339f6302c22a787fea29\": rpc error: code = NotFound desc = could not find container \"457204ac3591bc71feebd9ca375c2ec2e77e255e1ddc339f6302c22a787fea29\": container with ID starting with 457204ac3591bc71feebd9ca375c2ec2e77e255e1ddc339f6302c22a787fea29 not found: ID does not exist" Oct 30 20:40:42.039903 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:42.039848 4611 scope.go:122] "RemoveContainer" containerID="db46bb292b29cee7e347c34c47f3a13ffcd9b206c8bc54a2ee11564097fa5e7c" Oct 30 20:40:42.040080 ip-10-0-133-192 kubenswrapper[4611]: E1030 20:40:42.040064 4611 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db46bb292b29cee7e347c34c47f3a13ffcd9b206c8bc54a2ee11564097fa5e7c\": container with ID starting with db46bb292b29cee7e347c34c47f3a13ffcd9b206c8bc54a2ee11564097fa5e7c not found: ID does not exist" containerID="db46bb292b29cee7e347c34c47f3a13ffcd9b206c8bc54a2ee11564097fa5e7c" Oct 30 20:40:42.040146 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:42.040085 4611 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db46bb292b29cee7e347c34c47f3a13ffcd9b206c8bc54a2ee11564097fa5e7c"} err="failed to get container status \"db46bb292b29cee7e347c34c47f3a13ffcd9b206c8bc54a2ee11564097fa5e7c\": rpc error: code = NotFound desc = could not find container \"db46bb292b29cee7e347c34c47f3a13ffcd9b206c8bc54a2ee11564097fa5e7c\": container with ID starting with db46bb292b29cee7e347c34c47f3a13ffcd9b206c8bc54a2ee11564097fa5e7c not found: ID does not exist" Oct 30 20:40:42.040146 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:42.040097 4611 scope.go:122] "RemoveContainer" containerID="8862aa2f6b166bfaa0f82605eea54c6e3f7d1b1929f9dfc3d27c6150eeb13672" Oct 30 20:40:42.040323 ip-10-0-133-192 kubenswrapper[4611]: E1030 20:40:42.040306 4611 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8862aa2f6b166bfaa0f82605eea54c6e3f7d1b1929f9dfc3d27c6150eeb13672\": container with ID starting with 8862aa2f6b166bfaa0f82605eea54c6e3f7d1b1929f9dfc3d27c6150eeb13672 not found: ID does not exist" containerID="8862aa2f6b166bfaa0f82605eea54c6e3f7d1b1929f9dfc3d27c6150eeb13672" Oct 30 20:40:42.040387 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:42.040328 4611 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8862aa2f6b166bfaa0f82605eea54c6e3f7d1b1929f9dfc3d27c6150eeb13672"} err="failed to get container status \"8862aa2f6b166bfaa0f82605eea54c6e3f7d1b1929f9dfc3d27c6150eeb13672\": rpc error: code = NotFound desc = could not find container \"8862aa2f6b166bfaa0f82605eea54c6e3f7d1b1929f9dfc3d27c6150eeb13672\": container with ID starting with 8862aa2f6b166bfaa0f82605eea54c6e3f7d1b1929f9dfc3d27c6150eeb13672 not found: ID does not exist" Oct 30 20:40:42.040387 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:42.040340 4611 scope.go:122] "RemoveContainer" containerID="8fe5b6206f3963c00215018db89c750157d4a8d73a5c42abf6ba630b6905c626" Oct 30 20:40:42.040599 ip-10-0-133-192 kubenswrapper[4611]: E1030 20:40:42.040582 4611 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8fe5b6206f3963c00215018db89c750157d4a8d73a5c42abf6ba630b6905c626\": container with ID starting with 8fe5b6206f3963c00215018db89c750157d4a8d73a5c42abf6ba630b6905c626 not found: ID does not exist" containerID="8fe5b6206f3963c00215018db89c750157d4a8d73a5c42abf6ba630b6905c626" Oct 30 20:40:42.040673 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:42.040604 4611 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8fe5b6206f3963c00215018db89c750157d4a8d73a5c42abf6ba630b6905c626"} err="failed to get container status \"8fe5b6206f3963c00215018db89c750157d4a8d73a5c42abf6ba630b6905c626\": rpc error: code = NotFound desc = could not find container \"8fe5b6206f3963c00215018db89c750157d4a8d73a5c42abf6ba630b6905c626\": container with ID starting with 8fe5b6206f3963c00215018db89c750157d4a8d73a5c42abf6ba630b6905c626 not found: ID does not exist" Oct 30 20:40:42.040673 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:42.040626 4611 scope.go:122] "RemoveContainer" containerID="11d12b7b64f786ce05d475a22d175128b43566a4a574b021609b54c310875319" Oct 30 20:40:42.040906 ip-10-0-133-192 kubenswrapper[4611]: E1030 20:40:42.040885 4611 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"11d12b7b64f786ce05d475a22d175128b43566a4a574b021609b54c310875319\": container with ID starting with 11d12b7b64f786ce05d475a22d175128b43566a4a574b021609b54c310875319 not found: ID does not exist" containerID="11d12b7b64f786ce05d475a22d175128b43566a4a574b021609b54c310875319" Oct 30 20:40:42.040981 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:42.040913 4611 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"11d12b7b64f786ce05d475a22d175128b43566a4a574b021609b54c310875319"} err="failed to get container status \"11d12b7b64f786ce05d475a22d175128b43566a4a574b021609b54c310875319\": rpc error: code = NotFound desc = could not find container \"11d12b7b64f786ce05d475a22d175128b43566a4a574b021609b54c310875319\": container with ID starting with 11d12b7b64f786ce05d475a22d175128b43566a4a574b021609b54c310875319 not found: ID does not exist" Oct 30 20:40:42.057230 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:42.057211 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/60f9f978-737b-4279-927d-25f18934a44e-tls-assets\" (UniqueName: \"kubernetes.io/projected/60f9f978-737b-4279-927d-25f18934a44e-tls-assets\") pod \"60f9f978-737b-4279-927d-25f18934a44e\" (UID: \"60f9f978-737b-4279-927d-25f18934a44e\") " Oct 30 20:40:42.057320 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:42.057244 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60f9f978-737b-4279-927d-25f18934a44e-alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/60f9f978-737b-4279-927d-25f18934a44e-alertmanager-main-db\") pod \"60f9f978-737b-4279-927d-25f18934a44e\" (UID: \"60f9f978-737b-4279-927d-25f18934a44e\") " Oct 30 20:40:42.057320 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:42.057271 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/60f9f978-737b-4279-927d-25f18934a44e-secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/60f9f978-737b-4279-927d-25f18934a44e-secret-alertmanager-kube-rbac-proxy\") pod \"60f9f978-737b-4279-927d-25f18934a44e\" (UID: \"60f9f978-737b-4279-927d-25f18934a44e\") " Oct 30 20:40:42.057320 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:42.057305 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/60f9f978-737b-4279-927d-25f18934a44e-web-config\" (UniqueName: \"kubernetes.io/secret/60f9f978-737b-4279-927d-25f18934a44e-web-config\") pod \"60f9f978-737b-4279-927d-25f18934a44e\" (UID: \"60f9f978-737b-4279-927d-25f18934a44e\") " Oct 30 20:40:42.057505 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:42.057487 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/60f9f978-737b-4279-927d-25f18934a44e-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/60f9f978-737b-4279-927d-25f18934a44e-metrics-client-ca\") pod \"60f9f978-737b-4279-927d-25f18934a44e\" (UID: \"60f9f978-737b-4279-927d-25f18934a44e\") " Oct 30 20:40:42.057653 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:42.057554 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60f9f978-737b-4279-927d-25f18934a44e-config-out\" (UniqueName: \"kubernetes.io/empty-dir/60f9f978-737b-4279-927d-25f18934a44e-config-out\") pod \"60f9f978-737b-4279-927d-25f18934a44e\" (UID: \"60f9f978-737b-4279-927d-25f18934a44e\") " Oct 30 20:40:42.057653 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:42.057568 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60f9f978-737b-4279-927d-25f18934a44e-alertmanager-main-db" pod "60f9f978-737b-4279-927d-25f18934a44e" (UID: "60f9f978-737b-4279-927d-25f18934a44e"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:40:42.057653 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:42.057637 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/60f9f978-737b-4279-927d-25f18934a44e-config-volume\" (UniqueName: \"kubernetes.io/secret/60f9f978-737b-4279-927d-25f18934a44e-config-volume\") pod \"60f9f978-737b-4279-927d-25f18934a44e\" (UID: \"60f9f978-737b-4279-927d-25f18934a44e\") " Oct 30 20:40:42.057796 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:42.057706 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/60f9f978-737b-4279-927d-25f18934a44e-kube-api-access-t7vr9\" (UniqueName: \"kubernetes.io/projected/60f9f978-737b-4279-927d-25f18934a44e-kube-api-access-t7vr9\") pod \"60f9f978-737b-4279-927d-25f18934a44e\" (UID: \"60f9f978-737b-4279-927d-25f18934a44e\") " Oct 30 20:40:42.057796 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:42.057789 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/60f9f978-737b-4279-927d-25f18934a44e-secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/60f9f978-737b-4279-927d-25f18934a44e-secret-alertmanager-kube-rbac-proxy-web\") pod \"60f9f978-737b-4279-927d-25f18934a44e\" (UID: \"60f9f978-737b-4279-927d-25f18934a44e\") " Oct 30 20:40:42.057918 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:42.057820 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/60f9f978-737b-4279-927d-25f18934a44e-cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/60f9f978-737b-4279-927d-25f18934a44e-cluster-tls-config\") pod \"60f9f978-737b-4279-927d-25f18934a44e\" (UID: \"60f9f978-737b-4279-927d-25f18934a44e\") " Oct 30 20:40:42.057918 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:42.057847 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/60f9f978-737b-4279-927d-25f18934a44e-secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/60f9f978-737b-4279-927d-25f18934a44e-secret-alertmanager-main-tls\") pod \"60f9f978-737b-4279-927d-25f18934a44e\" (UID: \"60f9f978-737b-4279-927d-25f18934a44e\") " Oct 30 20:40:42.057918 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:42.057866 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60f9f978-737b-4279-927d-25f18934a44e-metrics-client-ca" pod "60f9f978-737b-4279-927d-25f18934a44e" (UID: "60f9f978-737b-4279-927d-25f18934a44e"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:40:42.057918 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:42.057882 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/60f9f978-737b-4279-927d-25f18934a44e-secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/60f9f978-737b-4279-927d-25f18934a44e-secret-alertmanager-kube-rbac-proxy-metric\") pod \"60f9f978-737b-4279-927d-25f18934a44e\" (UID: \"60f9f978-737b-4279-927d-25f18934a44e\") " Oct 30 20:40:42.057918 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:42.057916 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/60f9f978-737b-4279-927d-25f18934a44e-alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/60f9f978-737b-4279-927d-25f18934a44e-alertmanager-trusted-ca-bundle\") pod \"60f9f978-737b-4279-927d-25f18934a44e\" (UID: \"60f9f978-737b-4279-927d-25f18934a44e\") " Oct 30 20:40:42.058181 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:42.058162 4611 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/60f9f978-737b-4279-927d-25f18934a44e-alertmanager-main-db\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:40:42.058232 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:42.058189 4611 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/60f9f978-737b-4279-927d-25f18934a44e-metrics-client-ca\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:40:42.058608 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:42.058582 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60f9f978-737b-4279-927d-25f18934a44e-alertmanager-trusted-ca-bundle" pod "60f9f978-737b-4279-927d-25f18934a44e" (UID: "60f9f978-737b-4279-927d-25f18934a44e"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:40:42.060683 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:42.059786 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60f9f978-737b-4279-927d-25f18934a44e-tls-assets" pod "60f9f978-737b-4279-927d-25f18934a44e" (UID: "60f9f978-737b-4279-927d-25f18934a44e"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:40:42.060683 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:42.060013 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/60f9f978-737b-4279-927d-25f18934a44e-secret-alertmanager-kube-rbac-proxy" pod "60f9f978-737b-4279-927d-25f18934a44e" (UID: "60f9f978-737b-4279-927d-25f18934a44e"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:40:42.060683 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:42.060093 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60f9f978-737b-4279-927d-25f18934a44e-config-out" pod "60f9f978-737b-4279-927d-25f18934a44e" (UID: "60f9f978-737b-4279-927d-25f18934a44e"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:40:42.061215 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:42.061028 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/60f9f978-737b-4279-927d-25f18934a44e-secret-alertmanager-kube-rbac-proxy-metric" pod "60f9f978-737b-4279-927d-25f18934a44e" (UID: "60f9f978-737b-4279-927d-25f18934a44e"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:40:42.061971 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:42.061942 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60f9f978-737b-4279-927d-25f18934a44e-kube-api-access-t7vr9" pod "60f9f978-737b-4279-927d-25f18934a44e" (UID: "60f9f978-737b-4279-927d-25f18934a44e"). InnerVolumeSpecName "kube-api-access-t7vr9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:40:42.062056 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:42.061997 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/60f9f978-737b-4279-927d-25f18934a44e-secret-alertmanager-main-tls" pod "60f9f978-737b-4279-927d-25f18934a44e" (UID: "60f9f978-737b-4279-927d-25f18934a44e"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:40:42.062554 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:42.062529 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/60f9f978-737b-4279-927d-25f18934a44e-secret-alertmanager-kube-rbac-proxy-web" pod "60f9f978-737b-4279-927d-25f18934a44e" (UID: "60f9f978-737b-4279-927d-25f18934a44e"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:40:42.062833 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:42.062814 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/60f9f978-737b-4279-927d-25f18934a44e-config-volume" pod "60f9f978-737b-4279-927d-25f18934a44e" (UID: "60f9f978-737b-4279-927d-25f18934a44e"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:40:42.065832 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:42.065803 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/60f9f978-737b-4279-927d-25f18934a44e-cluster-tls-config" pod "60f9f978-737b-4279-927d-25f18934a44e" (UID: "60f9f978-737b-4279-927d-25f18934a44e"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:40:42.071126 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:42.071106 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/60f9f978-737b-4279-927d-25f18934a44e-web-config" pod "60f9f978-737b-4279-927d-25f18934a44e" (UID: "60f9f978-737b-4279-927d-25f18934a44e"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:40:42.158630 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:42.158604 4611 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/60f9f978-737b-4279-927d-25f18934a44e-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:40:42.158630 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:42.158627 4611 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/60f9f978-737b-4279-927d-25f18934a44e-tls-assets\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:40:42.158752 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:42.158638 4611 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/60f9f978-737b-4279-927d-25f18934a44e-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:40:42.158752 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:42.158647 4611 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/60f9f978-737b-4279-927d-25f18934a44e-web-config\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:40:42.158752 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:42.158655 4611 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/60f9f978-737b-4279-927d-25f18934a44e-config-out\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:40:42.158752 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:42.158663 4611 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/60f9f978-737b-4279-927d-25f18934a44e-config-volume\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:40:42.158752 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:42.158671 4611 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t7vr9\" (UniqueName: \"kubernetes.io/projected/60f9f978-737b-4279-927d-25f18934a44e-kube-api-access-t7vr9\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:40:42.158752 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:42.158681 4611 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/60f9f978-737b-4279-927d-25f18934a44e-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:40:42.158752 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:42.158689 4611 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/60f9f978-737b-4279-927d-25f18934a44e-cluster-tls-config\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:40:42.158752 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:42.158697 4611 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/60f9f978-737b-4279-927d-25f18934a44e-secret-alertmanager-main-tls\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:40:42.158752 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:42.158706 4611 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/60f9f978-737b-4279-927d-25f18934a44e-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:40:42.994232 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:42.994206 4611 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 30 20:40:43.018155 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:43.018130 4611 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 20:40:43.022689 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:43.022664 4611 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 20:40:43.053165 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:43.053112 4611 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 20:40:43.053412 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:43.053397 4611 cpu_manager.go:414] "containerMap: removing container" podUID="60f9f978-737b-4279-927d-25f18934a44e" containerName="kube-rbac-proxy" Oct 30 20:40:43.053475 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:43.053414 4611 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60f9f978-737b-4279-927d-25f18934a44e" containerName="kube-rbac-proxy" Oct 30 20:40:43.053475 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:43.053423 4611 cpu_manager.go:414] "containerMap: removing container" podUID="60f9f978-737b-4279-927d-25f18934a44e" containerName="kube-rbac-proxy-web" Oct 30 20:40:43.053475 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:43.053428 4611 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60f9f978-737b-4279-927d-25f18934a44e" containerName="kube-rbac-proxy-web" Oct 30 20:40:43.053475 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:43.053441 4611 cpu_manager.go:414] "containerMap: removing container" podUID="60f9f978-737b-4279-927d-25f18934a44e" containerName="init-config-reloader" Oct 30 20:40:43.053475 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:43.053446 4611 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60f9f978-737b-4279-927d-25f18934a44e" containerName="init-config-reloader" Oct 30 20:40:43.053475 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:43.053452 4611 cpu_manager.go:414] "containerMap: removing container" podUID="60f9f978-737b-4279-927d-25f18934a44e" containerName="kube-rbac-proxy-metric" Oct 30 20:40:43.053475 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:43.053458 4611 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60f9f978-737b-4279-927d-25f18934a44e" containerName="kube-rbac-proxy-metric" Oct 30 20:40:43.053475 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:43.053470 4611 cpu_manager.go:414] "containerMap: removing container" podUID="60f9f978-737b-4279-927d-25f18934a44e" containerName="prom-label-proxy" Oct 30 20:40:43.053716 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:43.053478 4611 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60f9f978-737b-4279-927d-25f18934a44e" containerName="prom-label-proxy" Oct 30 20:40:43.053716 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:43.053492 4611 cpu_manager.go:414] "containerMap: removing container" podUID="60f9f978-737b-4279-927d-25f18934a44e" containerName="alertmanager" Oct 30 20:40:43.053716 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:43.053497 4611 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60f9f978-737b-4279-927d-25f18934a44e" containerName="alertmanager" Oct 30 20:40:43.053716 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:43.053529 4611 cpu_manager.go:414] "containerMap: removing container" podUID="60f9f978-737b-4279-927d-25f18934a44e" containerName="config-reloader" Oct 30 20:40:43.053716 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:43.053537 4611 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60f9f978-737b-4279-927d-25f18934a44e" containerName="config-reloader" Oct 30 20:40:43.053716 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:43.053592 4611 memory_manager.go:365] "RemoveStaleState removing state" podUID="60f9f978-737b-4279-927d-25f18934a44e" containerName="alertmanager" Oct 30 20:40:43.053716 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:43.053603 4611 memory_manager.go:365] "RemoveStaleState removing state" podUID="60f9f978-737b-4279-927d-25f18934a44e" containerName="config-reloader" Oct 30 20:40:43.053716 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:43.053610 4611 memory_manager.go:365] "RemoveStaleState removing state" podUID="60f9f978-737b-4279-927d-25f18934a44e" containerName="kube-rbac-proxy" Oct 30 20:40:43.053716 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:43.053615 4611 memory_manager.go:365] "RemoveStaleState removing state" podUID="60f9f978-737b-4279-927d-25f18934a44e" containerName="kube-rbac-proxy-web" Oct 30 20:40:43.053716 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:43.053622 4611 memory_manager.go:365] "RemoveStaleState removing state" podUID="60f9f978-737b-4279-927d-25f18934a44e" containerName="kube-rbac-proxy-metric" Oct 30 20:40:43.053716 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:43.053629 4611 memory_manager.go:365] "RemoveStaleState removing state" podUID="60f9f978-737b-4279-927d-25f18934a44e" containerName="prom-label-proxy" Oct 30 20:40:43.058680 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:43.058664 4611 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 30 20:40:43.062593 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:43.062571 4611 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Oct 30 20:40:43.062925 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:43.062904 4611 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Oct 30 20:40:43.063018 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:43.062969 4611 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Oct 30 20:40:43.063018 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:43.062976 4611 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Oct 30 20:40:43.063104 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:43.062967 4611 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Oct 30 20:40:43.063104 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:43.062928 4611 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Oct 30 20:40:43.063104 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:43.062944 4611 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Oct 30 20:40:43.063104 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:43.063003 4611 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Oct 30 20:40:43.063104 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:43.063069 4611 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-pcdbr\"" Oct 30 20:40:43.064208 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:43.064192 4611 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Oct 30 20:40:43.076782 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:43.076761 4611 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 20:40:43.164493 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:43.164473 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/4014e0f7-c655-4bb8-a3ee-ca5de34e03aa-config-out\") pod \"alertmanager-main-0\" (UID: \"4014e0f7-c655-4bb8-a3ee-ca5de34e03aa\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 20:40:43.164624 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:43.164498 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4014e0f7-c655-4bb8-a3ee-ca5de34e03aa-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"4014e0f7-c655-4bb8-a3ee-ca5de34e03aa\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 20:40:43.164624 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:43.164532 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/4014e0f7-c655-4bb8-a3ee-ca5de34e03aa-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"4014e0f7-c655-4bb8-a3ee-ca5de34e03aa\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 20:40:43.164731 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:43.164665 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/4014e0f7-c655-4bb8-a3ee-ca5de34e03aa-tls-assets\") pod \"alertmanager-main-0\" (UID: \"4014e0f7-c655-4bb8-a3ee-ca5de34e03aa\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 20:40:43.164731 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:43.164704 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4014e0f7-c655-4bb8-a3ee-ca5de34e03aa-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"4014e0f7-c655-4bb8-a3ee-ca5de34e03aa\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 20:40:43.164799 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:43.164770 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4014e0f7-c655-4bb8-a3ee-ca5de34e03aa-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"4014e0f7-c655-4bb8-a3ee-ca5de34e03aa\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 20:40:43.164834 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:43.164807 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/4014e0f7-c655-4bb8-a3ee-ca5de34e03aa-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"4014e0f7-c655-4bb8-a3ee-ca5de34e03aa\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 20:40:43.164834 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:43.164830 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/4014e0f7-c655-4bb8-a3ee-ca5de34e03aa-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"4014e0f7-c655-4bb8-a3ee-ca5de34e03aa\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 20:40:43.164908 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:43.164848 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4014e0f7-c655-4bb8-a3ee-ca5de34e03aa-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"4014e0f7-c655-4bb8-a3ee-ca5de34e03aa\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 20:40:43.164941 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:43.164920 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/4014e0f7-c655-4bb8-a3ee-ca5de34e03aa-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"4014e0f7-c655-4bb8-a3ee-ca5de34e03aa\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 20:40:43.164976 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:43.164947 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/4014e0f7-c655-4bb8-a3ee-ca5de34e03aa-config-volume\") pod \"alertmanager-main-0\" (UID: \"4014e0f7-c655-4bb8-a3ee-ca5de34e03aa\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 20:40:43.164976 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:43.164964 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/4014e0f7-c655-4bb8-a3ee-ca5de34e03aa-web-config\") pod \"alertmanager-main-0\" (UID: \"4014e0f7-c655-4bb8-a3ee-ca5de34e03aa\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 20:40:43.165041 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:43.164988 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w5crp\" (UniqueName: \"kubernetes.io/projected/4014e0f7-c655-4bb8-a3ee-ca5de34e03aa-kube-api-access-w5crp\") pod \"alertmanager-main-0\" (UID: \"4014e0f7-c655-4bb8-a3ee-ca5de34e03aa\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 20:40:43.266215 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:43.266191 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/4014e0f7-c655-4bb8-a3ee-ca5de34e03aa-config-out\") pod \"alertmanager-main-0\" (UID: \"4014e0f7-c655-4bb8-a3ee-ca5de34e03aa\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 20:40:43.266215 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:43.266217 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4014e0f7-c655-4bb8-a3ee-ca5de34e03aa-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"4014e0f7-c655-4bb8-a3ee-ca5de34e03aa\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 20:40:43.266378 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:43.266234 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/4014e0f7-c655-4bb8-a3ee-ca5de34e03aa-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"4014e0f7-c655-4bb8-a3ee-ca5de34e03aa\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 20:40:43.266378 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:43.266259 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/4014e0f7-c655-4bb8-a3ee-ca5de34e03aa-tls-assets\") pod \"alertmanager-main-0\" (UID: \"4014e0f7-c655-4bb8-a3ee-ca5de34e03aa\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 20:40:43.266482 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:43.266386 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4014e0f7-c655-4bb8-a3ee-ca5de34e03aa-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"4014e0f7-c655-4bb8-a3ee-ca5de34e03aa\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 20:40:43.266482 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:43.266423 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4014e0f7-c655-4bb8-a3ee-ca5de34e03aa-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"4014e0f7-c655-4bb8-a3ee-ca5de34e03aa\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 20:40:43.266482 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:43.266464 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/4014e0f7-c655-4bb8-a3ee-ca5de34e03aa-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"4014e0f7-c655-4bb8-a3ee-ca5de34e03aa\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 20:40:43.266672 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:43.266497 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/4014e0f7-c655-4bb8-a3ee-ca5de34e03aa-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"4014e0f7-c655-4bb8-a3ee-ca5de34e03aa\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 20:40:43.267629 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:43.267134 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4014e0f7-c655-4bb8-a3ee-ca5de34e03aa-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"4014e0f7-c655-4bb8-a3ee-ca5de34e03aa\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 20:40:43.267629 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:43.267191 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4014e0f7-c655-4bb8-a3ee-ca5de34e03aa-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"4014e0f7-c655-4bb8-a3ee-ca5de34e03aa\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 20:40:43.267629 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:43.267232 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/4014e0f7-c655-4bb8-a3ee-ca5de34e03aa-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"4014e0f7-c655-4bb8-a3ee-ca5de34e03aa\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 20:40:43.267629 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:43.267261 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/4014e0f7-c655-4bb8-a3ee-ca5de34e03aa-config-volume\") pod \"alertmanager-main-0\" (UID: \"4014e0f7-c655-4bb8-a3ee-ca5de34e03aa\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 20:40:43.267629 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:43.267288 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/4014e0f7-c655-4bb8-a3ee-ca5de34e03aa-web-config\") pod \"alertmanager-main-0\" (UID: \"4014e0f7-c655-4bb8-a3ee-ca5de34e03aa\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 20:40:43.267629 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:43.267317 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4014e0f7-c655-4bb8-a3ee-ca5de34e03aa-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"4014e0f7-c655-4bb8-a3ee-ca5de34e03aa\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 20:40:43.267629 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:43.267596 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/4014e0f7-c655-4bb8-a3ee-ca5de34e03aa-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"4014e0f7-c655-4bb8-a3ee-ca5de34e03aa\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 20:40:43.267629 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:43.267327 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w5crp\" (UniqueName: \"kubernetes.io/projected/4014e0f7-c655-4bb8-a3ee-ca5de34e03aa-kube-api-access-w5crp\") pod \"alertmanager-main-0\" (UID: \"4014e0f7-c655-4bb8-a3ee-ca5de34e03aa\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 20:40:43.269111 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:43.269078 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/4014e0f7-c655-4bb8-a3ee-ca5de34e03aa-config-out\") pod \"alertmanager-main-0\" (UID: \"4014e0f7-c655-4bb8-a3ee-ca5de34e03aa\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 20:40:43.269312 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:43.269282 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/4014e0f7-c655-4bb8-a3ee-ca5de34e03aa-tls-assets\") pod \"alertmanager-main-0\" (UID: \"4014e0f7-c655-4bb8-a3ee-ca5de34e03aa\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 20:40:43.269585 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:43.269561 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/4014e0f7-c655-4bb8-a3ee-ca5de34e03aa-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"4014e0f7-c655-4bb8-a3ee-ca5de34e03aa\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 20:40:43.269957 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:43.269934 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4014e0f7-c655-4bb8-a3ee-ca5de34e03aa-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"4014e0f7-c655-4bb8-a3ee-ca5de34e03aa\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 20:40:43.270048 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:43.269987 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/4014e0f7-c655-4bb8-a3ee-ca5de34e03aa-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"4014e0f7-c655-4bb8-a3ee-ca5de34e03aa\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 20:40:43.270112 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:43.270066 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/4014e0f7-c655-4bb8-a3ee-ca5de34e03aa-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"4014e0f7-c655-4bb8-a3ee-ca5de34e03aa\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 20:40:43.270191 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:43.270174 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/4014e0f7-c655-4bb8-a3ee-ca5de34e03aa-config-volume\") pod \"alertmanager-main-0\" (UID: \"4014e0f7-c655-4bb8-a3ee-ca5de34e03aa\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 20:40:43.270461 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:43.270439 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4014e0f7-c655-4bb8-a3ee-ca5de34e03aa-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"4014e0f7-c655-4bb8-a3ee-ca5de34e03aa\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 20:40:43.271287 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:43.271265 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/4014e0f7-c655-4bb8-a3ee-ca5de34e03aa-web-config\") pod \"alertmanager-main-0\" (UID: \"4014e0f7-c655-4bb8-a3ee-ca5de34e03aa\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 20:40:43.279204 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:43.279183 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w5crp\" (UniqueName: \"kubernetes.io/projected/4014e0f7-c655-4bb8-a3ee-ca5de34e03aa-kube-api-access-w5crp\") pod \"alertmanager-main-0\" (UID: \"4014e0f7-c655-4bb8-a3ee-ca5de34e03aa\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 20:40:43.383095 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:43.383071 4611 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 30 20:40:43.603031 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:43.602989 4611 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="60f9f978-737b-4279-927d-25f18934a44e" path="/var/lib/kubelet/pods/60f9f978-737b-4279-927d-25f18934a44e/volumes" Oct 30 20:40:43.670059 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:43.670028 4611 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 20:40:43.673374 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:40:43.673337 4611 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4014e0f7_c655_4bb8_a3ee_ca5de34e03aa.slice/crio-50b8a21f491be2d0e1b15951690d29afa3f4235ea52d39505514aee74a9cecd9 WatchSource:0}: Error finding container 50b8a21f491be2d0e1b15951690d29afa3f4235ea52d39505514aee74a9cecd9: Status 404 returned error can't find the container with id 50b8a21f491be2d0e1b15951690d29afa3f4235ea52d39505514aee74a9cecd9 Oct 30 20:40:43.998715 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:43.998651 4611 generic.go:356] "Generic (PLEG): container finished" podID="4014e0f7-c655-4bb8-a3ee-ca5de34e03aa" containerID="765d81e49e9dcd166d67b81247090c9c4f57b789072c4715d7c8a8c78dd183c3" exitCode=0 Oct 30 20:40:43.998715 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:43.998704 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4014e0f7-c655-4bb8-a3ee-ca5de34e03aa","Type":"ContainerDied","Data":"765d81e49e9dcd166d67b81247090c9c4f57b789072c4715d7c8a8c78dd183c3"} Oct 30 20:40:43.999124 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:43.998754 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4014e0f7-c655-4bb8-a3ee-ca5de34e03aa","Type":"ContainerStarted","Data":"50b8a21f491be2d0e1b15951690d29afa3f4235ea52d39505514aee74a9cecd9"} Oct 30 20:40:45.004821 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:45.004787 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4014e0f7-c655-4bb8-a3ee-ca5de34e03aa","Type":"ContainerStarted","Data":"222c02a16d70e9a3a73839e0ce61660837e9b230d590813735f584a01c627711"} Oct 30 20:40:45.005229 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:45.004875 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4014e0f7-c655-4bb8-a3ee-ca5de34e03aa","Type":"ContainerStarted","Data":"7c80d207bddc4081a5e0f9cbaf4e10fb63f465ac16dd32acc18aee4790318e32"} Oct 30 20:40:45.005229 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:45.004900 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4014e0f7-c655-4bb8-a3ee-ca5de34e03aa","Type":"ContainerStarted","Data":"12e7538c0436f421cb1a63897691da33b2fd5c2e0fade3ae9cdd5f054406b8a2"} Oct 30 20:40:45.005229 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:45.004924 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4014e0f7-c655-4bb8-a3ee-ca5de34e03aa","Type":"ContainerStarted","Data":"94b7a02971c5453064256f85e7964ec7cf8e9be79cb7a7caa78ddcacd1969bbc"} Oct 30 20:40:45.005229 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:45.004953 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4014e0f7-c655-4bb8-a3ee-ca5de34e03aa","Type":"ContainerStarted","Data":"b097bc48843fba339318df19ad5d0051802548fe09e2ec39fcf55ce5dc4fe276"} Oct 30 20:40:45.005229 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:45.004976 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4014e0f7-c655-4bb8-a3ee-ca5de34e03aa","Type":"ContainerStarted","Data":"c88c690766981b6ab0a6f64098633a1d7b59206eb95527e088f9b9c3b414f8b5"} Oct 30 20:40:45.129761 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:45.129701 4611 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.117723191 podStartE2EDuration="2.129680983s" totalImagesPullingTime="0s" totalInitContainerRuntime="11.957792ms" isStatefulPod=false podCreationTimestamp="2026-10-30 20:40:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:40:45.053686497 +0000 UTC m=+99.587713114" watchObservedRunningTime="2026-10-30 20:40:45.129680983 +0000 UTC m=+99.663707601" Oct 30 20:40:45.130245 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:45.130227 4611 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 20:40:45.130548 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:45.130496 4611 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="c00ce45d-b9cf-4f43-a7ae-469d405ee218" containerName="prometheus" containerID="cri-o://07b909a4c6ca8a75b95d21e72ed6f8d1983e7851052e2c7789ff0f19df742b5b" gracePeriod=600 Oct 30 20:40:45.130698 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:45.130516 4611 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="c00ce45d-b9cf-4f43-a7ae-469d405ee218" containerName="kube-rbac-proxy-thanos" containerID="cri-o://bd780bc3fb65e4d12779165c27a6dfbdbd45920cf73c112ad98a670756e1a4e5" gracePeriod=600 Oct 30 20:40:45.130698 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:45.130556 4611 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="c00ce45d-b9cf-4f43-a7ae-469d405ee218" containerName="kube-rbac-proxy" containerID="cri-o://84d757c8c869d6915ec69bbacff82881bdbf1d6930d62edabe31443f7ffa090d" gracePeriod=600 Oct 30 20:40:45.130698 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:45.130556 4611 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="c00ce45d-b9cf-4f43-a7ae-469d405ee218" containerName="thanos-sidecar" containerID="cri-o://498b8f447913baf70893f05429326f82ad6ef7df92bf823401bdb6744fd54f48" gracePeriod=600 Oct 30 20:40:45.130698 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:45.130552 4611 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="c00ce45d-b9cf-4f43-a7ae-469d405ee218" containerName="config-reloader" containerID="cri-o://93f724d20999d506314f28a07321f47391012d60d7c2a8b383e4b73a79fc6d9c" gracePeriod=600 Oct 30 20:40:45.130698 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:45.130545 4611 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="c00ce45d-b9cf-4f43-a7ae-469d405ee218" containerName="kube-rbac-proxy-web" containerID="cri-o://7541462a2fd6aa2efe5b45e1416c0487a63fc404e5fae807966ebf120d5a02ff" gracePeriod=600 Oct 30 20:40:45.364605 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:45.364581 4611 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 20:40:45.486285 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:45.486250 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c00ce45d-b9cf-4f43-a7ae-469d405ee218-secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/c00ce45d-b9cf-4f43-a7ae-469d405ee218-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"c00ce45d-b9cf-4f43-a7ae-469d405ee218\" (UID: \"c00ce45d-b9cf-4f43-a7ae-469d405ee218\") " Oct 30 20:40:45.486447 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:45.486313 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c00ce45d-b9cf-4f43-a7ae-469d405ee218-configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c00ce45d-b9cf-4f43-a7ae-469d405ee218-configmap-kubelet-serving-ca-bundle\") pod \"c00ce45d-b9cf-4f43-a7ae-469d405ee218\" (UID: \"c00ce45d-b9cf-4f43-a7ae-469d405ee218\") " Oct 30 20:40:45.486447 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:45.486347 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c00ce45d-b9cf-4f43-a7ae-469d405ee218-secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/c00ce45d-b9cf-4f43-a7ae-469d405ee218-secret-metrics-client-certs\") pod \"c00ce45d-b9cf-4f43-a7ae-469d405ee218\" (UID: \"c00ce45d-b9cf-4f43-a7ae-469d405ee218\") " Oct 30 20:40:45.486447 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:45.486379 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c00ce45d-b9cf-4f43-a7ae-469d405ee218-kube-api-access-xwfmh\" (UniqueName: \"kubernetes.io/projected/c00ce45d-b9cf-4f43-a7ae-469d405ee218-kube-api-access-xwfmh\") pod \"c00ce45d-b9cf-4f43-a7ae-469d405ee218\" (UID: \"c00ce45d-b9cf-4f43-a7ae-469d405ee218\") " Oct 30 20:40:45.486447 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:45.486414 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c00ce45d-b9cf-4f43-a7ae-469d405ee218-secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/c00ce45d-b9cf-4f43-a7ae-469d405ee218-secret-prometheus-k8s-tls\") pod \"c00ce45d-b9cf-4f43-a7ae-469d405ee218\" (UID: \"c00ce45d-b9cf-4f43-a7ae-469d405ee218\") " Oct 30 20:40:45.486695 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:45.486449 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c00ce45d-b9cf-4f43-a7ae-469d405ee218-secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c00ce45d-b9cf-4f43-a7ae-469d405ee218-secret-kube-rbac-proxy\") pod \"c00ce45d-b9cf-4f43-a7ae-469d405ee218\" (UID: \"c00ce45d-b9cf-4f43-a7ae-469d405ee218\") " Oct 30 20:40:45.486695 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:45.486480 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c00ce45d-b9cf-4f43-a7ae-469d405ee218-config-out\" (UniqueName: \"kubernetes.io/empty-dir/c00ce45d-b9cf-4f43-a7ae-469d405ee218-config-out\") pod \"c00ce45d-b9cf-4f43-a7ae-469d405ee218\" (UID: \"c00ce45d-b9cf-4f43-a7ae-469d405ee218\") " Oct 30 20:40:45.486808 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:45.486740 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c00ce45d-b9cf-4f43-a7ae-469d405ee218-configmap-kubelet-serving-ca-bundle" pod "c00ce45d-b9cf-4f43-a7ae-469d405ee218" (UID: "c00ce45d-b9cf-4f43-a7ae-469d405ee218"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:40:45.486859 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:45.486843 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c00ce45d-b9cf-4f43-a7ae-469d405ee218-prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/c00ce45d-b9cf-4f43-a7ae-469d405ee218-prometheus-k8s-rulefiles-2\") pod \"c00ce45d-b9cf-4f43-a7ae-469d405ee218\" (UID: \"c00ce45d-b9cf-4f43-a7ae-469d405ee218\") " Oct 30 20:40:45.486914 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:45.486880 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c00ce45d-b9cf-4f43-a7ae-469d405ee218-configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c00ce45d-b9cf-4f43-a7ae-469d405ee218-configmap-serving-certs-ca-bundle\") pod \"c00ce45d-b9cf-4f43-a7ae-469d405ee218\" (UID: \"c00ce45d-b9cf-4f43-a7ae-469d405ee218\") " Oct 30 20:40:45.486967 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:45.486916 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c00ce45d-b9cf-4f43-a7ae-469d405ee218-thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/c00ce45d-b9cf-4f43-a7ae-469d405ee218-thanos-prometheus-http-client-file\") pod \"c00ce45d-b9cf-4f43-a7ae-469d405ee218\" (UID: \"c00ce45d-b9cf-4f43-a7ae-469d405ee218\") " Oct 30 20:40:45.486967 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:45.486952 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c00ce45d-b9cf-4f43-a7ae-469d405ee218-secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/c00ce45d-b9cf-4f43-a7ae-469d405ee218-secret-grpc-tls\") pod \"c00ce45d-b9cf-4f43-a7ae-469d405ee218\" (UID: \"c00ce45d-b9cf-4f43-a7ae-469d405ee218\") " Oct 30 20:40:45.487066 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:45.486982 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c00ce45d-b9cf-4f43-a7ae-469d405ee218-prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/c00ce45d-b9cf-4f43-a7ae-469d405ee218-prometheus-k8s-rulefiles-1\") pod \"c00ce45d-b9cf-4f43-a7ae-469d405ee218\" (UID: \"c00ce45d-b9cf-4f43-a7ae-469d405ee218\") " Oct 30 20:40:45.487271 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:45.487242 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c00ce45d-b9cf-4f43-a7ae-469d405ee218-prometheus-k8s-rulefiles-2" pod "c00ce45d-b9cf-4f43-a7ae-469d405ee218" (UID: "c00ce45d-b9cf-4f43-a7ae-469d405ee218"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:40:45.487364 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:45.487334 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c00ce45d-b9cf-4f43-a7ae-469d405ee218-prometheus-k8s-rulefiles-1" pod "c00ce45d-b9cf-4f43-a7ae-469d405ee218" (UID: "c00ce45d-b9cf-4f43-a7ae-469d405ee218"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:40:45.487469 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:45.487450 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c00ce45d-b9cf-4f43-a7ae-469d405ee218-prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c00ce45d-b9cf-4f43-a7ae-469d405ee218-prometheus-trusted-ca-bundle\") pod \"c00ce45d-b9cf-4f43-a7ae-469d405ee218\" (UID: \"c00ce45d-b9cf-4f43-a7ae-469d405ee218\") " Oct 30 20:40:45.487551 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:45.487504 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c00ce45d-b9cf-4f43-a7ae-469d405ee218-web-config\" (UniqueName: \"kubernetes.io/secret/c00ce45d-b9cf-4f43-a7ae-469d405ee218-web-config\") pod \"c00ce45d-b9cf-4f43-a7ae-469d405ee218\" (UID: \"c00ce45d-b9cf-4f43-a7ae-469d405ee218\") " Oct 30 20:40:45.487614 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:45.487559 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c00ce45d-b9cf-4f43-a7ae-469d405ee218-prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/c00ce45d-b9cf-4f43-a7ae-469d405ee218-prometheus-k8s-db\") pod \"c00ce45d-b9cf-4f43-a7ae-469d405ee218\" (UID: \"c00ce45d-b9cf-4f43-a7ae-469d405ee218\") " Oct 30 20:40:45.487614 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:45.487597 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c00ce45d-b9cf-4f43-a7ae-469d405ee218-prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/c00ce45d-b9cf-4f43-a7ae-469d405ee218-prometheus-k8s-rulefiles-0\") pod \"c00ce45d-b9cf-4f43-a7ae-469d405ee218\" (UID: \"c00ce45d-b9cf-4f43-a7ae-469d405ee218\") " Oct 30 20:40:45.487717 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:45.487632 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c00ce45d-b9cf-4f43-a7ae-469d405ee218-tls-assets\" (UniqueName: \"kubernetes.io/projected/c00ce45d-b9cf-4f43-a7ae-469d405ee218-tls-assets\") pod \"c00ce45d-b9cf-4f43-a7ae-469d405ee218\" (UID: \"c00ce45d-b9cf-4f43-a7ae-469d405ee218\") " Oct 30 20:40:45.487717 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:45.487663 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c00ce45d-b9cf-4f43-a7ae-469d405ee218-secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c00ce45d-b9cf-4f43-a7ae-469d405ee218-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"c00ce45d-b9cf-4f43-a7ae-469d405ee218\" (UID: \"c00ce45d-b9cf-4f43-a7ae-469d405ee218\") " Oct 30 20:40:45.487821 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:45.487751 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c00ce45d-b9cf-4f43-a7ae-469d405ee218-configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c00ce45d-b9cf-4f43-a7ae-469d405ee218-configmap-metrics-client-ca\") pod \"c00ce45d-b9cf-4f43-a7ae-469d405ee218\" (UID: \"c00ce45d-b9cf-4f43-a7ae-469d405ee218\") " Oct 30 20:40:45.487821 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:45.487787 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c00ce45d-b9cf-4f43-a7ae-469d405ee218-config\" (UniqueName: \"kubernetes.io/secret/c00ce45d-b9cf-4f43-a7ae-469d405ee218-config\") pod \"c00ce45d-b9cf-4f43-a7ae-469d405ee218\" (UID: \"c00ce45d-b9cf-4f43-a7ae-469d405ee218\") " Oct 30 20:40:45.488090 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:45.488070 4611 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c00ce45d-b9cf-4f43-a7ae-469d405ee218-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:40:45.488155 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:45.488097 4611 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/c00ce45d-b9cf-4f43-a7ae-469d405ee218-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:40:45.488155 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:45.488111 4611 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/c00ce45d-b9cf-4f43-a7ae-469d405ee218-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:40:45.488260 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:45.488173 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c00ce45d-b9cf-4f43-a7ae-469d405ee218-configmap-serving-certs-ca-bundle" pod "c00ce45d-b9cf-4f43-a7ae-469d405ee218" (UID: "c00ce45d-b9cf-4f43-a7ae-469d405ee218"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:40:45.488636 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:45.488606 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c00ce45d-b9cf-4f43-a7ae-469d405ee218-prometheus-trusted-ca-bundle" pod "c00ce45d-b9cf-4f43-a7ae-469d405ee218" (UID: "c00ce45d-b9cf-4f43-a7ae-469d405ee218"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:40:45.489419 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:45.489394 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c00ce45d-b9cf-4f43-a7ae-469d405ee218-configmap-metrics-client-ca" pod "c00ce45d-b9cf-4f43-a7ae-469d405ee218" (UID: "c00ce45d-b9cf-4f43-a7ae-469d405ee218"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:40:45.489905 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:45.489878 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c00ce45d-b9cf-4f43-a7ae-469d405ee218-kube-api-access-xwfmh" pod "c00ce45d-b9cf-4f43-a7ae-469d405ee218" (UID: "c00ce45d-b9cf-4f43-a7ae-469d405ee218"). InnerVolumeSpecName "kube-api-access-xwfmh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:40:45.490005 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:45.489947 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c00ce45d-b9cf-4f43-a7ae-469d405ee218-secret-prometheus-k8s-tls" pod "c00ce45d-b9cf-4f43-a7ae-469d405ee218" (UID: "c00ce45d-b9cf-4f43-a7ae-469d405ee218"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:40:45.490005 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:45.489949 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c00ce45d-b9cf-4f43-a7ae-469d405ee218-prometheus-k8s-rulefiles-0" pod "c00ce45d-b9cf-4f43-a7ae-469d405ee218" (UID: "c00ce45d-b9cf-4f43-a7ae-469d405ee218"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:40:45.490005 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:45.489998 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c00ce45d-b9cf-4f43-a7ae-469d405ee218-secret-prometheus-k8s-thanos-sidecar-tls" pod "c00ce45d-b9cf-4f43-a7ae-469d405ee218" (UID: "c00ce45d-b9cf-4f43-a7ae-469d405ee218"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:40:45.490153 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:45.490031 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c00ce45d-b9cf-4f43-a7ae-469d405ee218-secret-metrics-client-certs" pod "c00ce45d-b9cf-4f43-a7ae-469d405ee218" (UID: "c00ce45d-b9cf-4f43-a7ae-469d405ee218"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:40:45.490153 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:45.490123 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c00ce45d-b9cf-4f43-a7ae-469d405ee218-prometheus-k8s-db" pod "c00ce45d-b9cf-4f43-a7ae-469d405ee218" (UID: "c00ce45d-b9cf-4f43-a7ae-469d405ee218"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:40:45.490628 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:45.490604 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c00ce45d-b9cf-4f43-a7ae-469d405ee218-secret-kube-rbac-proxy" pod "c00ce45d-b9cf-4f43-a7ae-469d405ee218" (UID: "c00ce45d-b9cf-4f43-a7ae-469d405ee218"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:40:45.490706 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:45.490691 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c00ce45d-b9cf-4f43-a7ae-469d405ee218-config" pod "c00ce45d-b9cf-4f43-a7ae-469d405ee218" (UID: "c00ce45d-b9cf-4f43-a7ae-469d405ee218"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:40:45.490826 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:45.490801 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c00ce45d-b9cf-4f43-a7ae-469d405ee218-thanos-prometheus-http-client-file" pod "c00ce45d-b9cf-4f43-a7ae-469d405ee218" (UID: "c00ce45d-b9cf-4f43-a7ae-469d405ee218"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:40:45.491187 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:45.491161 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c00ce45d-b9cf-4f43-a7ae-469d405ee218-tls-assets" pod "c00ce45d-b9cf-4f43-a7ae-469d405ee218" (UID: "c00ce45d-b9cf-4f43-a7ae-469d405ee218"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:40:45.491488 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:45.491464 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c00ce45d-b9cf-4f43-a7ae-469d405ee218-config-out" pod "c00ce45d-b9cf-4f43-a7ae-469d405ee218" (UID: "c00ce45d-b9cf-4f43-a7ae-469d405ee218"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:40:45.491637 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:45.491611 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c00ce45d-b9cf-4f43-a7ae-469d405ee218-secret-grpc-tls" pod "c00ce45d-b9cf-4f43-a7ae-469d405ee218" (UID: "c00ce45d-b9cf-4f43-a7ae-469d405ee218"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:40:45.491880 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:45.491860 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c00ce45d-b9cf-4f43-a7ae-469d405ee218-secret-prometheus-k8s-kube-rbac-proxy-web" pod "c00ce45d-b9cf-4f43-a7ae-469d405ee218" (UID: "c00ce45d-b9cf-4f43-a7ae-469d405ee218"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:40:45.500369 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:45.500348 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c00ce45d-b9cf-4f43-a7ae-469d405ee218-web-config" pod "c00ce45d-b9cf-4f43-a7ae-469d405ee218" (UID: "c00ce45d-b9cf-4f43-a7ae-469d405ee218"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:40:45.589085 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:45.589029 4611 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/c00ce45d-b9cf-4f43-a7ae-469d405ee218-secret-grpc-tls\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:40:45.589085 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:45.589059 4611 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c00ce45d-b9cf-4f43-a7ae-469d405ee218-prometheus-trusted-ca-bundle\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:40:45.589085 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:45.589069 4611 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c00ce45d-b9cf-4f43-a7ae-469d405ee218-web-config\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:40:45.589085 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:45.589077 4611 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/c00ce45d-b9cf-4f43-a7ae-469d405ee218-prometheus-k8s-db\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:40:45.589085 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:45.589085 4611 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/c00ce45d-b9cf-4f43-a7ae-469d405ee218-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:40:45.589085 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:45.589094 4611 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c00ce45d-b9cf-4f43-a7ae-469d405ee218-tls-assets\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:40:45.589313 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:45.589103 4611 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c00ce45d-b9cf-4f43-a7ae-469d405ee218-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:40:45.589313 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:45.589113 4611 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c00ce45d-b9cf-4f43-a7ae-469d405ee218-configmap-metrics-client-ca\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:40:45.589313 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:45.589122 4611 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/c00ce45d-b9cf-4f43-a7ae-469d405ee218-config\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:40:45.589313 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:45.589131 4611 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/c00ce45d-b9cf-4f43-a7ae-469d405ee218-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:40:45.589313 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:45.589140 4611 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/c00ce45d-b9cf-4f43-a7ae-469d405ee218-secret-metrics-client-certs\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:40:45.589313 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:45.589148 4611 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xwfmh\" (UniqueName: \"kubernetes.io/projected/c00ce45d-b9cf-4f43-a7ae-469d405ee218-kube-api-access-xwfmh\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:40:45.589313 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:45.589157 4611 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/c00ce45d-b9cf-4f43-a7ae-469d405ee218-secret-prometheus-k8s-tls\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:40:45.589313 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:45.589165 4611 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c00ce45d-b9cf-4f43-a7ae-469d405ee218-secret-kube-rbac-proxy\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:40:45.589313 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:45.589173 4611 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c00ce45d-b9cf-4f43-a7ae-469d405ee218-config-out\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:40:45.589313 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:45.589182 4611 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c00ce45d-b9cf-4f43-a7ae-469d405ee218-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:40:45.589313 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:45.589191 4611 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/c00ce45d-b9cf-4f43-a7ae-469d405ee218-thanos-prometheus-http-client-file\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:40:46.011017 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:46.010988 4611 generic.go:356] "Generic (PLEG): container finished" podID="c00ce45d-b9cf-4f43-a7ae-469d405ee218" containerID="bd780bc3fb65e4d12779165c27a6dfbdbd45920cf73c112ad98a670756e1a4e5" exitCode=0 Oct 30 20:40:46.011017 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:46.011014 4611 generic.go:356] "Generic (PLEG): container finished" podID="c00ce45d-b9cf-4f43-a7ae-469d405ee218" containerID="84d757c8c869d6915ec69bbacff82881bdbf1d6930d62edabe31443f7ffa090d" exitCode=0 Oct 30 20:40:46.011353 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:46.011022 4611 generic.go:356] "Generic (PLEG): container finished" podID="c00ce45d-b9cf-4f43-a7ae-469d405ee218" containerID="7541462a2fd6aa2efe5b45e1416c0487a63fc404e5fae807966ebf120d5a02ff" exitCode=0 Oct 30 20:40:46.011353 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:46.011029 4611 generic.go:356] "Generic (PLEG): container finished" podID="c00ce45d-b9cf-4f43-a7ae-469d405ee218" containerID="498b8f447913baf70893f05429326f82ad6ef7df92bf823401bdb6744fd54f48" exitCode=0 Oct 30 20:40:46.011353 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:46.011037 4611 generic.go:356] "Generic (PLEG): container finished" podID="c00ce45d-b9cf-4f43-a7ae-469d405ee218" containerID="93f724d20999d506314f28a07321f47391012d60d7c2a8b383e4b73a79fc6d9c" exitCode=0 Oct 30 20:40:46.011353 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:46.011041 4611 generic.go:356] "Generic (PLEG): container finished" podID="c00ce45d-b9cf-4f43-a7ae-469d405ee218" containerID="07b909a4c6ca8a75b95d21e72ed6f8d1983e7851052e2c7789ff0f19df742b5b" exitCode=0 Oct 30 20:40:46.011353 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:46.011070 4611 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 20:40:46.011353 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:46.011072 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c00ce45d-b9cf-4f43-a7ae-469d405ee218","Type":"ContainerDied","Data":"bd780bc3fb65e4d12779165c27a6dfbdbd45920cf73c112ad98a670756e1a4e5"} Oct 30 20:40:46.011353 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:46.011170 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c00ce45d-b9cf-4f43-a7ae-469d405ee218","Type":"ContainerDied","Data":"84d757c8c869d6915ec69bbacff82881bdbf1d6930d62edabe31443f7ffa090d"} Oct 30 20:40:46.011353 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:46.011199 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c00ce45d-b9cf-4f43-a7ae-469d405ee218","Type":"ContainerDied","Data":"7541462a2fd6aa2efe5b45e1416c0487a63fc404e5fae807966ebf120d5a02ff"} Oct 30 20:40:46.011353 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:46.011230 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c00ce45d-b9cf-4f43-a7ae-469d405ee218","Type":"ContainerDied","Data":"498b8f447913baf70893f05429326f82ad6ef7df92bf823401bdb6744fd54f48"} Oct 30 20:40:46.011353 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:46.011237 4611 scope.go:122] "RemoveContainer" containerID="bd780bc3fb65e4d12779165c27a6dfbdbd45920cf73c112ad98a670756e1a4e5" Oct 30 20:40:46.011353 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:46.011268 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c00ce45d-b9cf-4f43-a7ae-469d405ee218","Type":"ContainerDied","Data":"93f724d20999d506314f28a07321f47391012d60d7c2a8b383e4b73a79fc6d9c"} Oct 30 20:40:46.011353 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:46.011307 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c00ce45d-b9cf-4f43-a7ae-469d405ee218","Type":"ContainerDied","Data":"07b909a4c6ca8a75b95d21e72ed6f8d1983e7851052e2c7789ff0f19df742b5b"} Oct 30 20:40:46.011353 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:46.011345 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c00ce45d-b9cf-4f43-a7ae-469d405ee218","Type":"ContainerDied","Data":"f855654166e32ab8d996b1cc21ca3d12660c6c66ab62294cb3a9e35c2a1cb994"} Oct 30 20:40:46.020747 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:46.020731 4611 scope.go:122] "RemoveContainer" containerID="84d757c8c869d6915ec69bbacff82881bdbf1d6930d62edabe31443f7ffa090d" Oct 30 20:40:46.027256 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:46.027242 4611 scope.go:122] "RemoveContainer" containerID="7541462a2fd6aa2efe5b45e1416c0487a63fc404e5fae807966ebf120d5a02ff" Oct 30 20:40:46.029964 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:46.029942 4611 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 20:40:46.034150 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:46.034129 4611 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 20:40:46.034694 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:46.034678 4611 scope.go:122] "RemoveContainer" containerID="498b8f447913baf70893f05429326f82ad6ef7df92bf823401bdb6744fd54f48" Oct 30 20:40:46.041196 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:46.041181 4611 scope.go:122] "RemoveContainer" containerID="93f724d20999d506314f28a07321f47391012d60d7c2a8b383e4b73a79fc6d9c" Oct 30 20:40:46.047318 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:46.047301 4611 scope.go:122] "RemoveContainer" containerID="07b909a4c6ca8a75b95d21e72ed6f8d1983e7851052e2c7789ff0f19df742b5b" Oct 30 20:40:46.053880 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:46.053865 4611 scope.go:122] "RemoveContainer" containerID="1ef98c6b694a49f92cb7dcf6f66bf657f9ad24fcb0a70964bb07914279067535" Oct 30 20:40:46.060014 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:46.060001 4611 scope.go:122] "RemoveContainer" containerID="bd780bc3fb65e4d12779165c27a6dfbdbd45920cf73c112ad98a670756e1a4e5" Oct 30 20:40:46.060267 ip-10-0-133-192 kubenswrapper[4611]: E1030 20:40:46.060248 4611 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd780bc3fb65e4d12779165c27a6dfbdbd45920cf73c112ad98a670756e1a4e5\": container with ID starting with bd780bc3fb65e4d12779165c27a6dfbdbd45920cf73c112ad98a670756e1a4e5 not found: ID does not exist" containerID="bd780bc3fb65e4d12779165c27a6dfbdbd45920cf73c112ad98a670756e1a4e5" Oct 30 20:40:46.060341 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:46.060280 4611 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd780bc3fb65e4d12779165c27a6dfbdbd45920cf73c112ad98a670756e1a4e5"} err="failed to get container status \"bd780bc3fb65e4d12779165c27a6dfbdbd45920cf73c112ad98a670756e1a4e5\": rpc error: code = NotFound desc = could not find container \"bd780bc3fb65e4d12779165c27a6dfbdbd45920cf73c112ad98a670756e1a4e5\": container with ID starting with bd780bc3fb65e4d12779165c27a6dfbdbd45920cf73c112ad98a670756e1a4e5 not found: ID does not exist" Oct 30 20:40:46.060341 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:46.060304 4611 scope.go:122] "RemoveContainer" containerID="84d757c8c869d6915ec69bbacff82881bdbf1d6930d62edabe31443f7ffa090d" Oct 30 20:40:46.060575 ip-10-0-133-192 kubenswrapper[4611]: E1030 20:40:46.060553 4611 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"84d757c8c869d6915ec69bbacff82881bdbf1d6930d62edabe31443f7ffa090d\": container with ID starting with 84d757c8c869d6915ec69bbacff82881bdbf1d6930d62edabe31443f7ffa090d not found: ID does not exist" containerID="84d757c8c869d6915ec69bbacff82881bdbf1d6930d62edabe31443f7ffa090d" Oct 30 20:40:46.060617 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:46.060584 4611 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84d757c8c869d6915ec69bbacff82881bdbf1d6930d62edabe31443f7ffa090d"} err="failed to get container status \"84d757c8c869d6915ec69bbacff82881bdbf1d6930d62edabe31443f7ffa090d\": rpc error: code = NotFound desc = could not find container \"84d757c8c869d6915ec69bbacff82881bdbf1d6930d62edabe31443f7ffa090d\": container with ID starting with 84d757c8c869d6915ec69bbacff82881bdbf1d6930d62edabe31443f7ffa090d not found: ID does not exist" Oct 30 20:40:46.060617 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:46.060605 4611 scope.go:122] "RemoveContainer" containerID="7541462a2fd6aa2efe5b45e1416c0487a63fc404e5fae807966ebf120d5a02ff" Oct 30 20:40:46.060838 ip-10-0-133-192 kubenswrapper[4611]: E1030 20:40:46.060820 4611 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7541462a2fd6aa2efe5b45e1416c0487a63fc404e5fae807966ebf120d5a02ff\": container with ID starting with 7541462a2fd6aa2efe5b45e1416c0487a63fc404e5fae807966ebf120d5a02ff not found: ID does not exist" containerID="7541462a2fd6aa2efe5b45e1416c0487a63fc404e5fae807966ebf120d5a02ff" Oct 30 20:40:46.060895 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:46.060846 4611 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7541462a2fd6aa2efe5b45e1416c0487a63fc404e5fae807966ebf120d5a02ff"} err="failed to get container status \"7541462a2fd6aa2efe5b45e1416c0487a63fc404e5fae807966ebf120d5a02ff\": rpc error: code = NotFound desc = could not find container \"7541462a2fd6aa2efe5b45e1416c0487a63fc404e5fae807966ebf120d5a02ff\": container with ID starting with 7541462a2fd6aa2efe5b45e1416c0487a63fc404e5fae807966ebf120d5a02ff not found: ID does not exist" Oct 30 20:40:46.060895 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:46.060865 4611 scope.go:122] "RemoveContainer" containerID="498b8f447913baf70893f05429326f82ad6ef7df92bf823401bdb6744fd54f48" Oct 30 20:40:46.061080 ip-10-0-133-192 kubenswrapper[4611]: E1030 20:40:46.061064 4611 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"498b8f447913baf70893f05429326f82ad6ef7df92bf823401bdb6744fd54f48\": container with ID starting with 498b8f447913baf70893f05429326f82ad6ef7df92bf823401bdb6744fd54f48 not found: ID does not exist" containerID="498b8f447913baf70893f05429326f82ad6ef7df92bf823401bdb6744fd54f48" Oct 30 20:40:46.061116 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:46.061084 4611 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"498b8f447913baf70893f05429326f82ad6ef7df92bf823401bdb6744fd54f48"} err="failed to get container status \"498b8f447913baf70893f05429326f82ad6ef7df92bf823401bdb6744fd54f48\": rpc error: code = NotFound desc = could not find container \"498b8f447913baf70893f05429326f82ad6ef7df92bf823401bdb6744fd54f48\": container with ID starting with 498b8f447913baf70893f05429326f82ad6ef7df92bf823401bdb6744fd54f48 not found: ID does not exist" Oct 30 20:40:46.061116 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:46.061096 4611 scope.go:122] "RemoveContainer" containerID="93f724d20999d506314f28a07321f47391012d60d7c2a8b383e4b73a79fc6d9c" Oct 30 20:40:46.061302 ip-10-0-133-192 kubenswrapper[4611]: E1030 20:40:46.061285 4611 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"93f724d20999d506314f28a07321f47391012d60d7c2a8b383e4b73a79fc6d9c\": container with ID starting with 93f724d20999d506314f28a07321f47391012d60d7c2a8b383e4b73a79fc6d9c not found: ID does not exist" containerID="93f724d20999d506314f28a07321f47391012d60d7c2a8b383e4b73a79fc6d9c" Oct 30 20:40:46.061372 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:46.061308 4611 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"93f724d20999d506314f28a07321f47391012d60d7c2a8b383e4b73a79fc6d9c"} err="failed to get container status \"93f724d20999d506314f28a07321f47391012d60d7c2a8b383e4b73a79fc6d9c\": rpc error: code = NotFound desc = could not find container \"93f724d20999d506314f28a07321f47391012d60d7c2a8b383e4b73a79fc6d9c\": container with ID starting with 93f724d20999d506314f28a07321f47391012d60d7c2a8b383e4b73a79fc6d9c not found: ID does not exist" Oct 30 20:40:46.061372 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:46.061325 4611 scope.go:122] "RemoveContainer" containerID="07b909a4c6ca8a75b95d21e72ed6f8d1983e7851052e2c7789ff0f19df742b5b" Oct 30 20:40:46.061554 ip-10-0-133-192 kubenswrapper[4611]: E1030 20:40:46.061538 4611 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"07b909a4c6ca8a75b95d21e72ed6f8d1983e7851052e2c7789ff0f19df742b5b\": container with ID starting with 07b909a4c6ca8a75b95d21e72ed6f8d1983e7851052e2c7789ff0f19df742b5b not found: ID does not exist" containerID="07b909a4c6ca8a75b95d21e72ed6f8d1983e7851052e2c7789ff0f19df742b5b" Oct 30 20:40:46.061595 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:46.061558 4611 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07b909a4c6ca8a75b95d21e72ed6f8d1983e7851052e2c7789ff0f19df742b5b"} err="failed to get container status \"07b909a4c6ca8a75b95d21e72ed6f8d1983e7851052e2c7789ff0f19df742b5b\": rpc error: code = NotFound desc = could not find container \"07b909a4c6ca8a75b95d21e72ed6f8d1983e7851052e2c7789ff0f19df742b5b\": container with ID starting with 07b909a4c6ca8a75b95d21e72ed6f8d1983e7851052e2c7789ff0f19df742b5b not found: ID does not exist" Oct 30 20:40:46.061595 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:46.061572 4611 scope.go:122] "RemoveContainer" containerID="1ef98c6b694a49f92cb7dcf6f66bf657f9ad24fcb0a70964bb07914279067535" Oct 30 20:40:46.061782 ip-10-0-133-192 kubenswrapper[4611]: E1030 20:40:46.061765 4611 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ef98c6b694a49f92cb7dcf6f66bf657f9ad24fcb0a70964bb07914279067535\": container with ID starting with 1ef98c6b694a49f92cb7dcf6f66bf657f9ad24fcb0a70964bb07914279067535 not found: ID does not exist" containerID="1ef98c6b694a49f92cb7dcf6f66bf657f9ad24fcb0a70964bb07914279067535" Oct 30 20:40:46.061831 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:46.061788 4611 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ef98c6b694a49f92cb7dcf6f66bf657f9ad24fcb0a70964bb07914279067535"} err="failed to get container status \"1ef98c6b694a49f92cb7dcf6f66bf657f9ad24fcb0a70964bb07914279067535\": rpc error: code = NotFound desc = could not find container \"1ef98c6b694a49f92cb7dcf6f66bf657f9ad24fcb0a70964bb07914279067535\": container with ID starting with 1ef98c6b694a49f92cb7dcf6f66bf657f9ad24fcb0a70964bb07914279067535 not found: ID does not exist" Oct 30 20:40:46.061831 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:46.061801 4611 scope.go:122] "RemoveContainer" containerID="bd780bc3fb65e4d12779165c27a6dfbdbd45920cf73c112ad98a670756e1a4e5" Oct 30 20:40:46.061999 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:46.061980 4611 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd780bc3fb65e4d12779165c27a6dfbdbd45920cf73c112ad98a670756e1a4e5"} err="failed to get container status \"bd780bc3fb65e4d12779165c27a6dfbdbd45920cf73c112ad98a670756e1a4e5\": rpc error: code = NotFound desc = could not find container \"bd780bc3fb65e4d12779165c27a6dfbdbd45920cf73c112ad98a670756e1a4e5\": container with ID starting with bd780bc3fb65e4d12779165c27a6dfbdbd45920cf73c112ad98a670756e1a4e5 not found: ID does not exist" Oct 30 20:40:46.062065 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:46.061999 4611 scope.go:122] "RemoveContainer" containerID="84d757c8c869d6915ec69bbacff82881bdbf1d6930d62edabe31443f7ffa090d" Oct 30 20:40:46.062201 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:46.062185 4611 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84d757c8c869d6915ec69bbacff82881bdbf1d6930d62edabe31443f7ffa090d"} err="failed to get container status \"84d757c8c869d6915ec69bbacff82881bdbf1d6930d62edabe31443f7ffa090d\": rpc error: code = NotFound desc = could not find container \"84d757c8c869d6915ec69bbacff82881bdbf1d6930d62edabe31443f7ffa090d\": container with ID starting with 84d757c8c869d6915ec69bbacff82881bdbf1d6930d62edabe31443f7ffa090d not found: ID does not exist" Oct 30 20:40:46.062282 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:46.062200 4611 scope.go:122] "RemoveContainer" containerID="7541462a2fd6aa2efe5b45e1416c0487a63fc404e5fae807966ebf120d5a02ff" Oct 30 20:40:46.062416 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:46.062398 4611 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7541462a2fd6aa2efe5b45e1416c0487a63fc404e5fae807966ebf120d5a02ff"} err="failed to get container status \"7541462a2fd6aa2efe5b45e1416c0487a63fc404e5fae807966ebf120d5a02ff\": rpc error: code = NotFound desc = could not find container \"7541462a2fd6aa2efe5b45e1416c0487a63fc404e5fae807966ebf120d5a02ff\": container with ID starting with 7541462a2fd6aa2efe5b45e1416c0487a63fc404e5fae807966ebf120d5a02ff not found: ID does not exist" Oct 30 20:40:46.062484 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:46.062418 4611 scope.go:122] "RemoveContainer" containerID="498b8f447913baf70893f05429326f82ad6ef7df92bf823401bdb6744fd54f48" Oct 30 20:40:46.062648 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:46.062630 4611 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"498b8f447913baf70893f05429326f82ad6ef7df92bf823401bdb6744fd54f48"} err="failed to get container status \"498b8f447913baf70893f05429326f82ad6ef7df92bf823401bdb6744fd54f48\": rpc error: code = NotFound desc = could not find container \"498b8f447913baf70893f05429326f82ad6ef7df92bf823401bdb6744fd54f48\": container with ID starting with 498b8f447913baf70893f05429326f82ad6ef7df92bf823401bdb6744fd54f48 not found: ID does not exist" Oct 30 20:40:46.062697 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:46.062649 4611 scope.go:122] "RemoveContainer" containerID="93f724d20999d506314f28a07321f47391012d60d7c2a8b383e4b73a79fc6d9c" Oct 30 20:40:46.062841 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:46.062824 4611 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"93f724d20999d506314f28a07321f47391012d60d7c2a8b383e4b73a79fc6d9c"} err="failed to get container status \"93f724d20999d506314f28a07321f47391012d60d7c2a8b383e4b73a79fc6d9c\": rpc error: code = NotFound desc = could not find container \"93f724d20999d506314f28a07321f47391012d60d7c2a8b383e4b73a79fc6d9c\": container with ID starting with 93f724d20999d506314f28a07321f47391012d60d7c2a8b383e4b73a79fc6d9c not found: ID does not exist" Oct 30 20:40:46.062910 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:46.062842 4611 scope.go:122] "RemoveContainer" containerID="07b909a4c6ca8a75b95d21e72ed6f8d1983e7851052e2c7789ff0f19df742b5b" Oct 30 20:40:46.063054 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:46.063035 4611 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07b909a4c6ca8a75b95d21e72ed6f8d1983e7851052e2c7789ff0f19df742b5b"} err="failed to get container status \"07b909a4c6ca8a75b95d21e72ed6f8d1983e7851052e2c7789ff0f19df742b5b\": rpc error: code = NotFound desc = could not find container \"07b909a4c6ca8a75b95d21e72ed6f8d1983e7851052e2c7789ff0f19df742b5b\": container with ID starting with 07b909a4c6ca8a75b95d21e72ed6f8d1983e7851052e2c7789ff0f19df742b5b not found: ID does not exist" Oct 30 20:40:46.063095 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:46.063055 4611 scope.go:122] "RemoveContainer" containerID="1ef98c6b694a49f92cb7dcf6f66bf657f9ad24fcb0a70964bb07914279067535" Oct 30 20:40:46.063237 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:46.063219 4611 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ef98c6b694a49f92cb7dcf6f66bf657f9ad24fcb0a70964bb07914279067535"} err="failed to get container status \"1ef98c6b694a49f92cb7dcf6f66bf657f9ad24fcb0a70964bb07914279067535\": rpc error: code = NotFound desc = could not find container \"1ef98c6b694a49f92cb7dcf6f66bf657f9ad24fcb0a70964bb07914279067535\": container with ID starting with 1ef98c6b694a49f92cb7dcf6f66bf657f9ad24fcb0a70964bb07914279067535 not found: ID does not exist" Oct 30 20:40:46.063299 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:46.063238 4611 scope.go:122] "RemoveContainer" containerID="bd780bc3fb65e4d12779165c27a6dfbdbd45920cf73c112ad98a670756e1a4e5" Oct 30 20:40:46.063435 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:46.063415 4611 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd780bc3fb65e4d12779165c27a6dfbdbd45920cf73c112ad98a670756e1a4e5"} err="failed to get container status \"bd780bc3fb65e4d12779165c27a6dfbdbd45920cf73c112ad98a670756e1a4e5\": rpc error: code = NotFound desc = could not find container \"bd780bc3fb65e4d12779165c27a6dfbdbd45920cf73c112ad98a670756e1a4e5\": container with ID starting with bd780bc3fb65e4d12779165c27a6dfbdbd45920cf73c112ad98a670756e1a4e5 not found: ID does not exist" Oct 30 20:40:46.063475 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:46.063436 4611 scope.go:122] "RemoveContainer" containerID="84d757c8c869d6915ec69bbacff82881bdbf1d6930d62edabe31443f7ffa090d" Oct 30 20:40:46.063662 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:46.063643 4611 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84d757c8c869d6915ec69bbacff82881bdbf1d6930d62edabe31443f7ffa090d"} err="failed to get container status \"84d757c8c869d6915ec69bbacff82881bdbf1d6930d62edabe31443f7ffa090d\": rpc error: code = NotFound desc = could not find container \"84d757c8c869d6915ec69bbacff82881bdbf1d6930d62edabe31443f7ffa090d\": container with ID starting with 84d757c8c869d6915ec69bbacff82881bdbf1d6930d62edabe31443f7ffa090d not found: ID does not exist" Oct 30 20:40:46.063721 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:46.063662 4611 scope.go:122] "RemoveContainer" containerID="7541462a2fd6aa2efe5b45e1416c0487a63fc404e5fae807966ebf120d5a02ff" Oct 30 20:40:46.063923 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:46.063906 4611 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7541462a2fd6aa2efe5b45e1416c0487a63fc404e5fae807966ebf120d5a02ff"} err="failed to get container status \"7541462a2fd6aa2efe5b45e1416c0487a63fc404e5fae807966ebf120d5a02ff\": rpc error: code = NotFound desc = could not find container \"7541462a2fd6aa2efe5b45e1416c0487a63fc404e5fae807966ebf120d5a02ff\": container with ID starting with 7541462a2fd6aa2efe5b45e1416c0487a63fc404e5fae807966ebf120d5a02ff not found: ID does not exist" Oct 30 20:40:46.063965 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:46.063923 4611 scope.go:122] "RemoveContainer" containerID="498b8f447913baf70893f05429326f82ad6ef7df92bf823401bdb6744fd54f48" Oct 30 20:40:46.064155 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:46.064137 4611 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"498b8f447913baf70893f05429326f82ad6ef7df92bf823401bdb6744fd54f48"} err="failed to get container status \"498b8f447913baf70893f05429326f82ad6ef7df92bf823401bdb6744fd54f48\": rpc error: code = NotFound desc = could not find container \"498b8f447913baf70893f05429326f82ad6ef7df92bf823401bdb6744fd54f48\": container with ID starting with 498b8f447913baf70893f05429326f82ad6ef7df92bf823401bdb6744fd54f48 not found: ID does not exist" Oct 30 20:40:46.064198 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:46.064156 4611 scope.go:122] "RemoveContainer" containerID="93f724d20999d506314f28a07321f47391012d60d7c2a8b383e4b73a79fc6d9c" Oct 30 20:40:46.064412 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:46.064387 4611 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"93f724d20999d506314f28a07321f47391012d60d7c2a8b383e4b73a79fc6d9c"} err="failed to get container status \"93f724d20999d506314f28a07321f47391012d60d7c2a8b383e4b73a79fc6d9c\": rpc error: code = NotFound desc = could not find container \"93f724d20999d506314f28a07321f47391012d60d7c2a8b383e4b73a79fc6d9c\": container with ID starting with 93f724d20999d506314f28a07321f47391012d60d7c2a8b383e4b73a79fc6d9c not found: ID does not exist" Oct 30 20:40:46.064412 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:46.064409 4611 scope.go:122] "RemoveContainer" containerID="07b909a4c6ca8a75b95d21e72ed6f8d1983e7851052e2c7789ff0f19df742b5b" Oct 30 20:40:46.064718 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:46.064696 4611 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07b909a4c6ca8a75b95d21e72ed6f8d1983e7851052e2c7789ff0f19df742b5b"} err="failed to get container status \"07b909a4c6ca8a75b95d21e72ed6f8d1983e7851052e2c7789ff0f19df742b5b\": rpc error: code = NotFound desc = could not find container \"07b909a4c6ca8a75b95d21e72ed6f8d1983e7851052e2c7789ff0f19df742b5b\": container with ID starting with 07b909a4c6ca8a75b95d21e72ed6f8d1983e7851052e2c7789ff0f19df742b5b not found: ID does not exist" Oct 30 20:40:46.064816 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:46.064720 4611 scope.go:122] "RemoveContainer" containerID="1ef98c6b694a49f92cb7dcf6f66bf657f9ad24fcb0a70964bb07914279067535" Oct 30 20:40:46.065080 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:46.065050 4611 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ef98c6b694a49f92cb7dcf6f66bf657f9ad24fcb0a70964bb07914279067535"} err="failed to get container status \"1ef98c6b694a49f92cb7dcf6f66bf657f9ad24fcb0a70964bb07914279067535\": rpc error: code = NotFound desc = could not find container \"1ef98c6b694a49f92cb7dcf6f66bf657f9ad24fcb0a70964bb07914279067535\": container with ID starting with 1ef98c6b694a49f92cb7dcf6f66bf657f9ad24fcb0a70964bb07914279067535 not found: ID does not exist" Oct 30 20:40:46.065080 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:46.065078 4611 scope.go:122] "RemoveContainer" containerID="bd780bc3fb65e4d12779165c27a6dfbdbd45920cf73c112ad98a670756e1a4e5" Oct 30 20:40:46.065525 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:46.065485 4611 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd780bc3fb65e4d12779165c27a6dfbdbd45920cf73c112ad98a670756e1a4e5"} err="failed to get container status \"bd780bc3fb65e4d12779165c27a6dfbdbd45920cf73c112ad98a670756e1a4e5\": rpc error: code = NotFound desc = could not find container \"bd780bc3fb65e4d12779165c27a6dfbdbd45920cf73c112ad98a670756e1a4e5\": container with ID starting with bd780bc3fb65e4d12779165c27a6dfbdbd45920cf73c112ad98a670756e1a4e5 not found: ID does not exist" Oct 30 20:40:46.065525 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:46.065518 4611 scope.go:122] "RemoveContainer" containerID="84d757c8c869d6915ec69bbacff82881bdbf1d6930d62edabe31443f7ffa090d" Oct 30 20:40:46.065649 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:46.065575 4611 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 20:40:46.065758 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:46.065738 4611 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84d757c8c869d6915ec69bbacff82881bdbf1d6930d62edabe31443f7ffa090d"} err="failed to get container status \"84d757c8c869d6915ec69bbacff82881bdbf1d6930d62edabe31443f7ffa090d\": rpc error: code = NotFound desc = could not find container \"84d757c8c869d6915ec69bbacff82881bdbf1d6930d62edabe31443f7ffa090d\": container with ID starting with 84d757c8c869d6915ec69bbacff82881bdbf1d6930d62edabe31443f7ffa090d not found: ID does not exist" Oct 30 20:40:46.065815 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:46.065757 4611 scope.go:122] "RemoveContainer" containerID="7541462a2fd6aa2efe5b45e1416c0487a63fc404e5fae807966ebf120d5a02ff" Oct 30 20:40:46.065897 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:46.065881 4611 cpu_manager.go:414] "containerMap: removing container" podUID="c00ce45d-b9cf-4f43-a7ae-469d405ee218" containerName="thanos-sidecar" Oct 30 20:40:46.065951 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:46.065901 4611 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c00ce45d-b9cf-4f43-a7ae-469d405ee218" containerName="thanos-sidecar" Oct 30 20:40:46.065951 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:46.065920 4611 cpu_manager.go:414] "containerMap: removing container" podUID="c00ce45d-b9cf-4f43-a7ae-469d405ee218" containerName="prometheus" Oct 30 20:40:46.065951 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:46.065929 4611 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c00ce45d-b9cf-4f43-a7ae-469d405ee218" containerName="prometheus" Oct 30 20:40:46.065951 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:46.065943 4611 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7541462a2fd6aa2efe5b45e1416c0487a63fc404e5fae807966ebf120d5a02ff"} err="failed to get container status \"7541462a2fd6aa2efe5b45e1416c0487a63fc404e5fae807966ebf120d5a02ff\": rpc error: code = NotFound desc = could not find container \"7541462a2fd6aa2efe5b45e1416c0487a63fc404e5fae807966ebf120d5a02ff\": container with ID starting with 7541462a2fd6aa2efe5b45e1416c0487a63fc404e5fae807966ebf120d5a02ff not found: ID does not exist" Oct 30 20:40:46.066118 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:46.065955 4611 cpu_manager.go:414] "containerMap: removing container" podUID="c00ce45d-b9cf-4f43-a7ae-469d405ee218" containerName="kube-rbac-proxy-web" Oct 30 20:40:46.066118 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:46.065971 4611 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c00ce45d-b9cf-4f43-a7ae-469d405ee218" containerName="kube-rbac-proxy-web" Oct 30 20:40:46.066118 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:46.065990 4611 cpu_manager.go:414] "containerMap: removing container" podUID="c00ce45d-b9cf-4f43-a7ae-469d405ee218" containerName="kube-rbac-proxy-thanos" Oct 30 20:40:46.066118 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:46.065998 4611 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c00ce45d-b9cf-4f43-a7ae-469d405ee218" containerName="kube-rbac-proxy-thanos" Oct 30 20:40:46.066118 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:46.066014 4611 cpu_manager.go:414] "containerMap: removing container" podUID="c00ce45d-b9cf-4f43-a7ae-469d405ee218" containerName="kube-rbac-proxy" Oct 30 20:40:46.066118 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:46.066021 4611 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c00ce45d-b9cf-4f43-a7ae-469d405ee218" containerName="kube-rbac-proxy" Oct 30 20:40:46.066118 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:46.065958 4611 scope.go:122] "RemoveContainer" containerID="498b8f447913baf70893f05429326f82ad6ef7df92bf823401bdb6744fd54f48" Oct 30 20:40:46.066118 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:46.066032 4611 cpu_manager.go:414] "containerMap: removing container" podUID="c00ce45d-b9cf-4f43-a7ae-469d405ee218" containerName="init-config-reloader" Oct 30 20:40:46.066118 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:46.066070 4611 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c00ce45d-b9cf-4f43-a7ae-469d405ee218" containerName="init-config-reloader" Oct 30 20:40:46.066118 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:46.066084 4611 cpu_manager.go:414] "containerMap: removing container" podUID="c00ce45d-b9cf-4f43-a7ae-469d405ee218" containerName="config-reloader" Oct 30 20:40:46.066118 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:46.066089 4611 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c00ce45d-b9cf-4f43-a7ae-469d405ee218" containerName="config-reloader" Oct 30 20:40:46.066531 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:46.066153 4611 memory_manager.go:365] "RemoveStaleState removing state" podUID="c00ce45d-b9cf-4f43-a7ae-469d405ee218" containerName="kube-rbac-proxy-web" Oct 30 20:40:46.066531 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:46.066165 4611 memory_manager.go:365] "RemoveStaleState removing state" podUID="c00ce45d-b9cf-4f43-a7ae-469d405ee218" containerName="kube-rbac-proxy-thanos" Oct 30 20:40:46.066531 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:46.066174 4611 memory_manager.go:365] "RemoveStaleState removing state" podUID="c00ce45d-b9cf-4f43-a7ae-469d405ee218" containerName="kube-rbac-proxy" Oct 30 20:40:46.066531 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:46.066184 4611 memory_manager.go:365] "RemoveStaleState removing state" podUID="c00ce45d-b9cf-4f43-a7ae-469d405ee218" containerName="config-reloader" Oct 30 20:40:46.066531 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:46.066192 4611 memory_manager.go:365] "RemoveStaleState removing state" podUID="c00ce45d-b9cf-4f43-a7ae-469d405ee218" containerName="thanos-sidecar" Oct 30 20:40:46.066531 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:46.066198 4611 memory_manager.go:365] "RemoveStaleState removing state" podUID="c00ce45d-b9cf-4f43-a7ae-469d405ee218" containerName="prometheus" Oct 30 20:40:46.066531 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:46.066237 4611 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"498b8f447913baf70893f05429326f82ad6ef7df92bf823401bdb6744fd54f48"} err="failed to get container status \"498b8f447913baf70893f05429326f82ad6ef7df92bf823401bdb6744fd54f48\": rpc error: code = NotFound desc = could not find container \"498b8f447913baf70893f05429326f82ad6ef7df92bf823401bdb6744fd54f48\": container with ID starting with 498b8f447913baf70893f05429326f82ad6ef7df92bf823401bdb6744fd54f48 not found: ID does not exist" Oct 30 20:40:46.066531 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:46.066253 4611 scope.go:122] "RemoveContainer" containerID="93f724d20999d506314f28a07321f47391012d60d7c2a8b383e4b73a79fc6d9c" Oct 30 20:40:46.066531 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:46.066450 4611 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"93f724d20999d506314f28a07321f47391012d60d7c2a8b383e4b73a79fc6d9c"} err="failed to get container status \"93f724d20999d506314f28a07321f47391012d60d7c2a8b383e4b73a79fc6d9c\": rpc error: code = NotFound desc = could not find container \"93f724d20999d506314f28a07321f47391012d60d7c2a8b383e4b73a79fc6d9c\": container with ID starting with 93f724d20999d506314f28a07321f47391012d60d7c2a8b383e4b73a79fc6d9c not found: ID does not exist" Oct 30 20:40:46.066531 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:46.066470 4611 scope.go:122] "RemoveContainer" containerID="07b909a4c6ca8a75b95d21e72ed6f8d1983e7851052e2c7789ff0f19df742b5b" Oct 30 20:40:46.066922 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:46.066694 4611 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07b909a4c6ca8a75b95d21e72ed6f8d1983e7851052e2c7789ff0f19df742b5b"} err="failed to get container status \"07b909a4c6ca8a75b95d21e72ed6f8d1983e7851052e2c7789ff0f19df742b5b\": rpc error: code = NotFound desc = could not find container \"07b909a4c6ca8a75b95d21e72ed6f8d1983e7851052e2c7789ff0f19df742b5b\": container with ID starting with 07b909a4c6ca8a75b95d21e72ed6f8d1983e7851052e2c7789ff0f19df742b5b not found: ID does not exist" Oct 30 20:40:46.066922 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:46.066712 4611 scope.go:122] "RemoveContainer" containerID="1ef98c6b694a49f92cb7dcf6f66bf657f9ad24fcb0a70964bb07914279067535" Oct 30 20:40:46.066993 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:46.066940 4611 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ef98c6b694a49f92cb7dcf6f66bf657f9ad24fcb0a70964bb07914279067535"} err="failed to get container status \"1ef98c6b694a49f92cb7dcf6f66bf657f9ad24fcb0a70964bb07914279067535\": rpc error: code = NotFound desc = could not find container \"1ef98c6b694a49f92cb7dcf6f66bf657f9ad24fcb0a70964bb07914279067535\": container with ID starting with 1ef98c6b694a49f92cb7dcf6f66bf657f9ad24fcb0a70964bb07914279067535 not found: ID does not exist" Oct 30 20:40:46.066993 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:46.066961 4611 scope.go:122] "RemoveContainer" containerID="bd780bc3fb65e4d12779165c27a6dfbdbd45920cf73c112ad98a670756e1a4e5" Oct 30 20:40:46.067197 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:46.067179 4611 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd780bc3fb65e4d12779165c27a6dfbdbd45920cf73c112ad98a670756e1a4e5"} err="failed to get container status \"bd780bc3fb65e4d12779165c27a6dfbdbd45920cf73c112ad98a670756e1a4e5\": rpc error: code = NotFound desc = could not find container \"bd780bc3fb65e4d12779165c27a6dfbdbd45920cf73c112ad98a670756e1a4e5\": container with ID starting with bd780bc3fb65e4d12779165c27a6dfbdbd45920cf73c112ad98a670756e1a4e5 not found: ID does not exist" Oct 30 20:40:46.067268 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:46.067197 4611 scope.go:122] "RemoveContainer" containerID="84d757c8c869d6915ec69bbacff82881bdbf1d6930d62edabe31443f7ffa090d" Oct 30 20:40:46.067422 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:46.067404 4611 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84d757c8c869d6915ec69bbacff82881bdbf1d6930d62edabe31443f7ffa090d"} err="failed to get container status \"84d757c8c869d6915ec69bbacff82881bdbf1d6930d62edabe31443f7ffa090d\": rpc error: code = NotFound desc = could not find container \"84d757c8c869d6915ec69bbacff82881bdbf1d6930d62edabe31443f7ffa090d\": container with ID starting with 84d757c8c869d6915ec69bbacff82881bdbf1d6930d62edabe31443f7ffa090d not found: ID does not exist" Oct 30 20:40:46.067492 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:46.067423 4611 scope.go:122] "RemoveContainer" containerID="7541462a2fd6aa2efe5b45e1416c0487a63fc404e5fae807966ebf120d5a02ff" Oct 30 20:40:46.067643 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:46.067626 4611 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7541462a2fd6aa2efe5b45e1416c0487a63fc404e5fae807966ebf120d5a02ff"} err="failed to get container status \"7541462a2fd6aa2efe5b45e1416c0487a63fc404e5fae807966ebf120d5a02ff\": rpc error: code = NotFound desc = could not find container \"7541462a2fd6aa2efe5b45e1416c0487a63fc404e5fae807966ebf120d5a02ff\": container with ID starting with 7541462a2fd6aa2efe5b45e1416c0487a63fc404e5fae807966ebf120d5a02ff not found: ID does not exist" Oct 30 20:40:46.067689 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:46.067643 4611 scope.go:122] "RemoveContainer" containerID="498b8f447913baf70893f05429326f82ad6ef7df92bf823401bdb6744fd54f48" Oct 30 20:40:46.067850 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:46.067835 4611 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"498b8f447913baf70893f05429326f82ad6ef7df92bf823401bdb6744fd54f48"} err="failed to get container status \"498b8f447913baf70893f05429326f82ad6ef7df92bf823401bdb6744fd54f48\": rpc error: code = NotFound desc = could not find container \"498b8f447913baf70893f05429326f82ad6ef7df92bf823401bdb6744fd54f48\": container with ID starting with 498b8f447913baf70893f05429326f82ad6ef7df92bf823401bdb6744fd54f48 not found: ID does not exist" Oct 30 20:40:46.067896 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:46.067850 4611 scope.go:122] "RemoveContainer" containerID="93f724d20999d506314f28a07321f47391012d60d7c2a8b383e4b73a79fc6d9c" Oct 30 20:40:46.068043 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:46.068026 4611 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"93f724d20999d506314f28a07321f47391012d60d7c2a8b383e4b73a79fc6d9c"} err="failed to get container status \"93f724d20999d506314f28a07321f47391012d60d7c2a8b383e4b73a79fc6d9c\": rpc error: code = NotFound desc = could not find container \"93f724d20999d506314f28a07321f47391012d60d7c2a8b383e4b73a79fc6d9c\": container with ID starting with 93f724d20999d506314f28a07321f47391012d60d7c2a8b383e4b73a79fc6d9c not found: ID does not exist" Oct 30 20:40:46.068111 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:46.068044 4611 scope.go:122] "RemoveContainer" containerID="07b909a4c6ca8a75b95d21e72ed6f8d1983e7851052e2c7789ff0f19df742b5b" Oct 30 20:40:46.068250 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:46.068233 4611 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07b909a4c6ca8a75b95d21e72ed6f8d1983e7851052e2c7789ff0f19df742b5b"} err="failed to get container status \"07b909a4c6ca8a75b95d21e72ed6f8d1983e7851052e2c7789ff0f19df742b5b\": rpc error: code = NotFound desc = could not find container \"07b909a4c6ca8a75b95d21e72ed6f8d1983e7851052e2c7789ff0f19df742b5b\": container with ID starting with 07b909a4c6ca8a75b95d21e72ed6f8d1983e7851052e2c7789ff0f19df742b5b not found: ID does not exist" Oct 30 20:40:46.068295 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:46.068250 4611 scope.go:122] "RemoveContainer" containerID="1ef98c6b694a49f92cb7dcf6f66bf657f9ad24fcb0a70964bb07914279067535" Oct 30 20:40:46.068437 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:46.068418 4611 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ef98c6b694a49f92cb7dcf6f66bf657f9ad24fcb0a70964bb07914279067535"} err="failed to get container status \"1ef98c6b694a49f92cb7dcf6f66bf657f9ad24fcb0a70964bb07914279067535\": rpc error: code = NotFound desc = could not find container \"1ef98c6b694a49f92cb7dcf6f66bf657f9ad24fcb0a70964bb07914279067535\": container with ID starting with 1ef98c6b694a49f92cb7dcf6f66bf657f9ad24fcb0a70964bb07914279067535 not found: ID does not exist" Oct 30 20:40:46.068498 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:46.068438 4611 scope.go:122] "RemoveContainer" containerID="bd780bc3fb65e4d12779165c27a6dfbdbd45920cf73c112ad98a670756e1a4e5" Oct 30 20:40:46.068708 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:46.068686 4611 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd780bc3fb65e4d12779165c27a6dfbdbd45920cf73c112ad98a670756e1a4e5"} err="failed to get container status \"bd780bc3fb65e4d12779165c27a6dfbdbd45920cf73c112ad98a670756e1a4e5\": rpc error: code = NotFound desc = could not find container \"bd780bc3fb65e4d12779165c27a6dfbdbd45920cf73c112ad98a670756e1a4e5\": container with ID starting with bd780bc3fb65e4d12779165c27a6dfbdbd45920cf73c112ad98a670756e1a4e5 not found: ID does not exist" Oct 30 20:40:46.068759 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:46.068711 4611 scope.go:122] "RemoveContainer" containerID="84d757c8c869d6915ec69bbacff82881bdbf1d6930d62edabe31443f7ffa090d" Oct 30 20:40:46.068955 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:46.068935 4611 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84d757c8c869d6915ec69bbacff82881bdbf1d6930d62edabe31443f7ffa090d"} err="failed to get container status \"84d757c8c869d6915ec69bbacff82881bdbf1d6930d62edabe31443f7ffa090d\": rpc error: code = NotFound desc = could not find container \"84d757c8c869d6915ec69bbacff82881bdbf1d6930d62edabe31443f7ffa090d\": container with ID starting with 84d757c8c869d6915ec69bbacff82881bdbf1d6930d62edabe31443f7ffa090d not found: ID does not exist" Oct 30 20:40:46.069014 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:46.068957 4611 scope.go:122] "RemoveContainer" containerID="7541462a2fd6aa2efe5b45e1416c0487a63fc404e5fae807966ebf120d5a02ff" Oct 30 20:40:46.069158 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:46.069143 4611 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7541462a2fd6aa2efe5b45e1416c0487a63fc404e5fae807966ebf120d5a02ff"} err="failed to get container status \"7541462a2fd6aa2efe5b45e1416c0487a63fc404e5fae807966ebf120d5a02ff\": rpc error: code = NotFound desc = could not find container \"7541462a2fd6aa2efe5b45e1416c0487a63fc404e5fae807966ebf120d5a02ff\": container with ID starting with 7541462a2fd6aa2efe5b45e1416c0487a63fc404e5fae807966ebf120d5a02ff not found: ID does not exist" Oct 30 20:40:46.069221 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:46.069159 4611 scope.go:122] "RemoveContainer" containerID="498b8f447913baf70893f05429326f82ad6ef7df92bf823401bdb6744fd54f48" Oct 30 20:40:46.069373 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:46.069356 4611 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"498b8f447913baf70893f05429326f82ad6ef7df92bf823401bdb6744fd54f48"} err="failed to get container status \"498b8f447913baf70893f05429326f82ad6ef7df92bf823401bdb6744fd54f48\": rpc error: code = NotFound desc = could not find container \"498b8f447913baf70893f05429326f82ad6ef7df92bf823401bdb6744fd54f48\": container with ID starting with 498b8f447913baf70893f05429326f82ad6ef7df92bf823401bdb6744fd54f48 not found: ID does not exist" Oct 30 20:40:46.069419 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:46.069374 4611 scope.go:122] "RemoveContainer" containerID="93f724d20999d506314f28a07321f47391012d60d7c2a8b383e4b73a79fc6d9c" Oct 30 20:40:46.069611 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:46.069594 4611 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"93f724d20999d506314f28a07321f47391012d60d7c2a8b383e4b73a79fc6d9c"} err="failed to get container status \"93f724d20999d506314f28a07321f47391012d60d7c2a8b383e4b73a79fc6d9c\": rpc error: code = NotFound desc = could not find container \"93f724d20999d506314f28a07321f47391012d60d7c2a8b383e4b73a79fc6d9c\": container with ID starting with 93f724d20999d506314f28a07321f47391012d60d7c2a8b383e4b73a79fc6d9c not found: ID does not exist" Oct 30 20:40:46.069673 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:46.069611 4611 scope.go:122] "RemoveContainer" containerID="07b909a4c6ca8a75b95d21e72ed6f8d1983e7851052e2c7789ff0f19df742b5b" Oct 30 20:40:46.069805 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:46.069789 4611 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07b909a4c6ca8a75b95d21e72ed6f8d1983e7851052e2c7789ff0f19df742b5b"} err="failed to get container status \"07b909a4c6ca8a75b95d21e72ed6f8d1983e7851052e2c7789ff0f19df742b5b\": rpc error: code = NotFound desc = could not find container \"07b909a4c6ca8a75b95d21e72ed6f8d1983e7851052e2c7789ff0f19df742b5b\": container with ID starting with 07b909a4c6ca8a75b95d21e72ed6f8d1983e7851052e2c7789ff0f19df742b5b not found: ID does not exist" Oct 30 20:40:46.069864 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:46.069806 4611 scope.go:122] "RemoveContainer" containerID="1ef98c6b694a49f92cb7dcf6f66bf657f9ad24fcb0a70964bb07914279067535" Oct 30 20:40:46.070047 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:46.070030 4611 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ef98c6b694a49f92cb7dcf6f66bf657f9ad24fcb0a70964bb07914279067535"} err="failed to get container status \"1ef98c6b694a49f92cb7dcf6f66bf657f9ad24fcb0a70964bb07914279067535\": rpc error: code = NotFound desc = could not find container \"1ef98c6b694a49f92cb7dcf6f66bf657f9ad24fcb0a70964bb07914279067535\": container with ID starting with 1ef98c6b694a49f92cb7dcf6f66bf657f9ad24fcb0a70964bb07914279067535 not found: ID does not exist" Oct 30 20:40:46.071158 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:46.071144 4611 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 20:40:46.074817 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:46.074795 4611 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Oct 30 20:40:46.074817 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:46.074809 4611 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Oct 30 20:40:46.074960 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:46.074817 4611 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-7fopgrptfcdd5\"" Oct 30 20:40:46.075137 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:46.075122 4611 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Oct 30 20:40:46.075403 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:46.075305 4611 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Oct 30 20:40:46.075403 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:46.075320 4611 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Oct 30 20:40:46.075403 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:46.075326 4611 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Oct 30 20:40:46.075403 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:46.075347 4611 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Oct 30 20:40:46.075711 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:46.075462 4611 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Oct 30 20:40:46.075711 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:46.075465 4611 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Oct 30 20:40:46.075711 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:46.075480 4611 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Oct 30 20:40:46.075711 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:46.075485 4611 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Oct 30 20:40:46.075711 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:46.075527 4611 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Oct 30 20:40:46.075711 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:46.075540 4611 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Oct 30 20:40:46.075711 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:46.075692 4611 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-xtgt6\"" Oct 30 20:40:46.077700 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:46.077628 4611 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Oct 30 20:40:46.077700 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:46.077649 4611 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Oct 30 20:40:46.089225 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:46.089205 4611 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 20:40:46.193748 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:46.193721 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5cjls\" (UniqueName: \"kubernetes.io/projected/211b0851-ee7c-4c0a-9595-b60d23d9753d-kube-api-access-5cjls\") pod \"prometheus-k8s-0\" (UID: \"211b0851-ee7c-4c0a-9595-b60d23d9753d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 20:40:46.193842 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:46.193764 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/211b0851-ee7c-4c0a-9595-b60d23d9753d-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"211b0851-ee7c-4c0a-9595-b60d23d9753d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 20:40:46.193842 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:46.193793 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/211b0851-ee7c-4c0a-9595-b60d23d9753d-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"211b0851-ee7c-4c0a-9595-b60d23d9753d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 20:40:46.193927 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:46.193842 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/211b0851-ee7c-4c0a-9595-b60d23d9753d-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"211b0851-ee7c-4c0a-9595-b60d23d9753d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 20:40:46.193927 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:46.193879 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/211b0851-ee7c-4c0a-9595-b60d23d9753d-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"211b0851-ee7c-4c0a-9595-b60d23d9753d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 20:40:46.193927 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:46.193902 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/211b0851-ee7c-4c0a-9595-b60d23d9753d-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"211b0851-ee7c-4c0a-9595-b60d23d9753d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 20:40:46.193927 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:46.193917 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/211b0851-ee7c-4c0a-9595-b60d23d9753d-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"211b0851-ee7c-4c0a-9595-b60d23d9753d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 20:40:46.194044 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:46.193965 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/211b0851-ee7c-4c0a-9595-b60d23d9753d-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"211b0851-ee7c-4c0a-9595-b60d23d9753d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 20:40:46.194044 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:46.193982 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/211b0851-ee7c-4c0a-9595-b60d23d9753d-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"211b0851-ee7c-4c0a-9595-b60d23d9753d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 20:40:46.194044 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:46.194010 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/211b0851-ee7c-4c0a-9595-b60d23d9753d-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"211b0851-ee7c-4c0a-9595-b60d23d9753d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 20:40:46.194044 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:46.194037 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/211b0851-ee7c-4c0a-9595-b60d23d9753d-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"211b0851-ee7c-4c0a-9595-b60d23d9753d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 20:40:46.194157 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:46.194060 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/211b0851-ee7c-4c0a-9595-b60d23d9753d-web-config\") pod \"prometheus-k8s-0\" (UID: \"211b0851-ee7c-4c0a-9595-b60d23d9753d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 20:40:46.194157 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:46.194115 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/211b0851-ee7c-4c0a-9595-b60d23d9753d-config\") pod \"prometheus-k8s-0\" (UID: \"211b0851-ee7c-4c0a-9595-b60d23d9753d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 20:40:46.194157 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:46.194141 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/211b0851-ee7c-4c0a-9595-b60d23d9753d-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"211b0851-ee7c-4c0a-9595-b60d23d9753d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 20:40:46.194254 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:46.194157 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/211b0851-ee7c-4c0a-9595-b60d23d9753d-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"211b0851-ee7c-4c0a-9595-b60d23d9753d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 20:40:46.194254 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:46.194173 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/211b0851-ee7c-4c0a-9595-b60d23d9753d-config-out\") pod \"prometheus-k8s-0\" (UID: \"211b0851-ee7c-4c0a-9595-b60d23d9753d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 20:40:46.194254 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:46.194211 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/211b0851-ee7c-4c0a-9595-b60d23d9753d-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"211b0851-ee7c-4c0a-9595-b60d23d9753d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 20:40:46.194399 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:46.194253 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/211b0851-ee7c-4c0a-9595-b60d23d9753d-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"211b0851-ee7c-4c0a-9595-b60d23d9753d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 20:40:46.194399 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:46.194270 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/211b0851-ee7c-4c0a-9595-b60d23d9753d-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"211b0851-ee7c-4c0a-9595-b60d23d9753d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 20:40:46.194399 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:46.194340 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/211b0851-ee7c-4c0a-9595-b60d23d9753d-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"211b0851-ee7c-4c0a-9595-b60d23d9753d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 20:40:46.295013 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:46.294990 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/211b0851-ee7c-4c0a-9595-b60d23d9753d-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"211b0851-ee7c-4c0a-9595-b60d23d9753d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 20:40:46.295120 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:46.295018 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/211b0851-ee7c-4c0a-9595-b60d23d9753d-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"211b0851-ee7c-4c0a-9595-b60d23d9753d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 20:40:46.295120 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:46.295037 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/211b0851-ee7c-4c0a-9595-b60d23d9753d-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"211b0851-ee7c-4c0a-9595-b60d23d9753d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 20:40:46.295120 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:46.295054 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/211b0851-ee7c-4c0a-9595-b60d23d9753d-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"211b0851-ee7c-4c0a-9595-b60d23d9753d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 20:40:46.295249 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:46.295167 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/211b0851-ee7c-4c0a-9595-b60d23d9753d-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"211b0851-ee7c-4c0a-9595-b60d23d9753d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 20:40:46.295249 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:46.295207 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/211b0851-ee7c-4c0a-9595-b60d23d9753d-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"211b0851-ee7c-4c0a-9595-b60d23d9753d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 20:40:46.295249 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:46.295242 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/211b0851-ee7c-4c0a-9595-b60d23d9753d-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"211b0851-ee7c-4c0a-9595-b60d23d9753d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 20:40:46.295392 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:46.295267 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/211b0851-ee7c-4c0a-9595-b60d23d9753d-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"211b0851-ee7c-4c0a-9595-b60d23d9753d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 20:40:46.295392 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:46.295291 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/211b0851-ee7c-4c0a-9595-b60d23d9753d-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"211b0851-ee7c-4c0a-9595-b60d23d9753d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 20:40:46.295392 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:46.295315 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/211b0851-ee7c-4c0a-9595-b60d23d9753d-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"211b0851-ee7c-4c0a-9595-b60d23d9753d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 20:40:46.295392 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:46.295342 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/211b0851-ee7c-4c0a-9595-b60d23d9753d-web-config\") pod \"prometheus-k8s-0\" (UID: \"211b0851-ee7c-4c0a-9595-b60d23d9753d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 20:40:46.295392 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:46.295365 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/211b0851-ee7c-4c0a-9595-b60d23d9753d-config\") pod \"prometheus-k8s-0\" (UID: \"211b0851-ee7c-4c0a-9595-b60d23d9753d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 20:40:46.295668 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:46.295405 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/211b0851-ee7c-4c0a-9595-b60d23d9753d-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"211b0851-ee7c-4c0a-9595-b60d23d9753d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 20:40:46.295668 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:46.295408 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/211b0851-ee7c-4c0a-9595-b60d23d9753d-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"211b0851-ee7c-4c0a-9595-b60d23d9753d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 20:40:46.295668 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:46.295431 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/211b0851-ee7c-4c0a-9595-b60d23d9753d-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"211b0851-ee7c-4c0a-9595-b60d23d9753d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 20:40:46.295668 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:46.295458 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/211b0851-ee7c-4c0a-9595-b60d23d9753d-config-out\") pod \"prometheus-k8s-0\" (UID: \"211b0851-ee7c-4c0a-9595-b60d23d9753d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 20:40:46.295668 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:46.295485 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/211b0851-ee7c-4c0a-9595-b60d23d9753d-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"211b0851-ee7c-4c0a-9595-b60d23d9753d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 20:40:46.295668 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:46.295537 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/211b0851-ee7c-4c0a-9595-b60d23d9753d-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"211b0851-ee7c-4c0a-9595-b60d23d9753d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 20:40:46.295668 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:46.295563 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/211b0851-ee7c-4c0a-9595-b60d23d9753d-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"211b0851-ee7c-4c0a-9595-b60d23d9753d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 20:40:46.295668 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:46.295593 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/211b0851-ee7c-4c0a-9595-b60d23d9753d-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"211b0851-ee7c-4c0a-9595-b60d23d9753d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 20:40:46.295668 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:46.295621 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5cjls\" (UniqueName: \"kubernetes.io/projected/211b0851-ee7c-4c0a-9595-b60d23d9753d-kube-api-access-5cjls\") pod \"prometheus-k8s-0\" (UID: \"211b0851-ee7c-4c0a-9595-b60d23d9753d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 20:40:46.296154 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:46.295746 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/211b0851-ee7c-4c0a-9595-b60d23d9753d-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"211b0851-ee7c-4c0a-9595-b60d23d9753d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 20:40:46.296154 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:46.295891 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/211b0851-ee7c-4c0a-9595-b60d23d9753d-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"211b0851-ee7c-4c0a-9595-b60d23d9753d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 20:40:46.296154 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:46.296047 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/211b0851-ee7c-4c0a-9595-b60d23d9753d-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"211b0851-ee7c-4c0a-9595-b60d23d9753d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 20:40:46.296358 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:46.296334 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/211b0851-ee7c-4c0a-9595-b60d23d9753d-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"211b0851-ee7c-4c0a-9595-b60d23d9753d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 20:40:46.297002 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:46.296976 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/211b0851-ee7c-4c0a-9595-b60d23d9753d-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"211b0851-ee7c-4c0a-9595-b60d23d9753d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 20:40:46.298270 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:46.297969 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/211b0851-ee7c-4c0a-9595-b60d23d9753d-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"211b0851-ee7c-4c0a-9595-b60d23d9753d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 20:40:46.298270 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:46.298143 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/211b0851-ee7c-4c0a-9595-b60d23d9753d-web-config\") pod \"prometheus-k8s-0\" (UID: \"211b0851-ee7c-4c0a-9595-b60d23d9753d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 20:40:46.298270 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:46.298143 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/211b0851-ee7c-4c0a-9595-b60d23d9753d-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"211b0851-ee7c-4c0a-9595-b60d23d9753d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 20:40:46.298481 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:46.298450 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/211b0851-ee7c-4c0a-9595-b60d23d9753d-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"211b0851-ee7c-4c0a-9595-b60d23d9753d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 20:40:46.299097 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:46.299056 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/211b0851-ee7c-4c0a-9595-b60d23d9753d-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"211b0851-ee7c-4c0a-9595-b60d23d9753d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 20:40:46.299414 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:46.299356 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/211b0851-ee7c-4c0a-9595-b60d23d9753d-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"211b0851-ee7c-4c0a-9595-b60d23d9753d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 20:40:46.299577 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:46.299553 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/211b0851-ee7c-4c0a-9595-b60d23d9753d-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"211b0851-ee7c-4c0a-9595-b60d23d9753d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 20:40:46.299918 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:46.299902 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/211b0851-ee7c-4c0a-9595-b60d23d9753d-config-out\") pod \"prometheus-k8s-0\" (UID: \"211b0851-ee7c-4c0a-9595-b60d23d9753d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 20:40:46.300359 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:46.300338 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/211b0851-ee7c-4c0a-9595-b60d23d9753d-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"211b0851-ee7c-4c0a-9595-b60d23d9753d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 20:40:46.300539 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:46.300500 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/211b0851-ee7c-4c0a-9595-b60d23d9753d-config\") pod \"prometheus-k8s-0\" (UID: \"211b0851-ee7c-4c0a-9595-b60d23d9753d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 20:40:46.300871 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:46.300855 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/211b0851-ee7c-4c0a-9595-b60d23d9753d-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"211b0851-ee7c-4c0a-9595-b60d23d9753d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 20:40:46.301239 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:46.301220 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/211b0851-ee7c-4c0a-9595-b60d23d9753d-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"211b0851-ee7c-4c0a-9595-b60d23d9753d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 20:40:46.301523 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:46.301487 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/211b0851-ee7c-4c0a-9595-b60d23d9753d-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"211b0851-ee7c-4c0a-9595-b60d23d9753d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 20:40:46.307066 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:46.307047 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5cjls\" (UniqueName: \"kubernetes.io/projected/211b0851-ee7c-4c0a-9595-b60d23d9753d-kube-api-access-5cjls\") pod \"prometheus-k8s-0\" (UID: \"211b0851-ee7c-4c0a-9595-b60d23d9753d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 20:40:46.396067 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:46.396044 4611 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 20:40:46.692130 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:46.692099 4611 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 20:40:46.695637 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:40:46.695581 4611 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod211b0851_ee7c_4c0a_9595_b60d23d9753d.slice/crio-8240f0170c0f453eeeb53453fcbba962a06a71b5977bc558a6aa929d47f590d4 WatchSource:0}: Error finding container 8240f0170c0f453eeeb53453fcbba962a06a71b5977bc558a6aa929d47f590d4: Status 404 returned error can't find the container with id 8240f0170c0f453eeeb53453fcbba962a06a71b5977bc558a6aa929d47f590d4 Oct 30 20:40:47.015833 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:47.015772 4611 generic.go:356] "Generic (PLEG): container finished" podID="211b0851-ee7c-4c0a-9595-b60d23d9753d" containerID="fb0da550d0da1d581564190857a2c56916e5988926fa339911b32c16529e20c6" exitCode=0 Oct 30 20:40:47.016188 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:47.015861 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"211b0851-ee7c-4c0a-9595-b60d23d9753d","Type":"ContainerDied","Data":"fb0da550d0da1d581564190857a2c56916e5988926fa339911b32c16529e20c6"} Oct 30 20:40:47.016188 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:47.015942 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"211b0851-ee7c-4c0a-9595-b60d23d9753d","Type":"ContainerStarted","Data":"8240f0170c0f453eeeb53453fcbba962a06a71b5977bc558a6aa929d47f590d4"} Oct 30 20:40:47.603864 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:47.603837 4611 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c00ce45d-b9cf-4f43-a7ae-469d405ee218" path="/var/lib/kubelet/pods/c00ce45d-b9cf-4f43-a7ae-469d405ee218/volumes" Oct 30 20:40:47.797673 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:47.797648 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:40:48.024837 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:48.024760 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"211b0851-ee7c-4c0a-9595-b60d23d9753d","Type":"ContainerStarted","Data":"a160bbb867d2016f34524bac8742ea6bcca62d06010abc1c6f7c7bfa7cbad4b5"} Oct 30 20:40:48.025268 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:48.024835 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"211b0851-ee7c-4c0a-9595-b60d23d9753d","Type":"ContainerStarted","Data":"9d5bae8ad9ac48780e012f4da61cb97b315b34271b70e0ee5d34f4eb0b55d0a5"} Oct 30 20:40:48.025268 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:48.024879 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"211b0851-ee7c-4c0a-9595-b60d23d9753d","Type":"ContainerStarted","Data":"5a10aad4de633105b37c9c864907993145e5c9fdf730bfbee606a568cebc18af"} Oct 30 20:40:48.025268 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:48.024919 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"211b0851-ee7c-4c0a-9595-b60d23d9753d","Type":"ContainerStarted","Data":"8c4c832ae356fd2aef2cd5bc0eb3e522a9386b79ab05d4f2c992ff38fbdd7c56"} Oct 30 20:40:48.025268 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:48.024960 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"211b0851-ee7c-4c0a-9595-b60d23d9753d","Type":"ContainerStarted","Data":"7e389e2c0eb4eb455e5714a968affea50811c7bc9be2b0eb225d084f219d4d46"} Oct 30 20:40:48.025268 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:48.025000 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"211b0851-ee7c-4c0a-9595-b60d23d9753d","Type":"ContainerStarted","Data":"f2f06a3a2ce2cb451830c2aad1428ac124bf333fe0487d4df94d1534d925f351"} Oct 30 20:40:48.060028 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:48.059975 4611 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.044876401 podStartE2EDuration="2.059960231s" totalImagesPullingTime="0s" totalInitContainerRuntime="15.08383ms" isStatefulPod=false podCreationTimestamp="2026-10-30 20:40:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:40:48.058897066 +0000 UTC m=+102.592923681" watchObservedRunningTime="2026-10-30 20:40:48.059960231 +0000 UTC m=+102.593986846" Oct 30 20:40:50.336826 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:50.336788 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:40:50.485187 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:50.485131 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:40:50.809077 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:50.809043 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:40:50.954724 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:50.954696 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:40:51.397112 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:40:51.397081 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 20:41:03.915279 ip-10-0-133-192 kubenswrapper[4611]: E1030 20:41:03.915236 4611 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hwzc6" containerName="extractor" Oct 30 20:41:04.217626 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:41:04.217549 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:41:04.382594 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:41:04.382571 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:41:30.633913 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:41:30.633828 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:41:33.915409 ip-10-0-133-192 kubenswrapper[4611]: E1030 20:41:33.915372 4611 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hwzc6" containerName="extractor" Oct 30 20:41:46.396925 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:41:46.396894 4611 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 20:41:46.411857 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:41:46.411830 4611 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 20:41:47.238263 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:41:47.238238 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 20:41:47.845789 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:41:47.845756 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:41:54.985119 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:41:54.985086 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:42:02.216473 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:42:02.216439 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:42:02.363020 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:42:02.362988 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:42:02.838886 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:42:02.838853 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:42:02.988783 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:42:02.988755 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:42:03.915369 ip-10-0-133-192 kubenswrapper[4611]: E1030 20:42:03.915339 4611 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hwzc6" containerName="extractor" Oct 30 20:42:07.625368 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:42:07.625334 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:42:07.784072 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:42:07.784043 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:42:20.904368 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:42:20.904332 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:42:21.063893 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:42:21.063860 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:42:33.915584 ip-10-0-133-192 kubenswrapper[4611]: E1030 20:42:33.915548 4611 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hwzc6" containerName="extractor" Oct 30 20:42:47.590351 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:42:47.590316 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:42:47.908934 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:42:47.908901 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:42:48.650973 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:42:48.650939 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:43:03.915309 ip-10-0-133-192 kubenswrapper[4611]: E1030 20:43:03.915272 4611 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hwzc6" containerName="extractor" Oct 30 20:43:08.346154 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:43:08.346118 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:43:22.877065 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:43:22.877037 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:43:23.899101 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:43:23.899063 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:43:33.916033 ip-10-0-133-192 kubenswrapper[4611]: E1030 20:43:33.915994 4611 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hwzc6" containerName="extractor" Oct 30 20:43:41.852321 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:43:41.852289 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:43:42.019198 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:43:42.019166 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:43:43.529793 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:43:43.529758 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:43:43.675877 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:43:43.675843 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:43:44.824712 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:43:44.824680 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:43:47.977166 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:43:47.977132 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:43:48.456083 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:43:48.456048 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:43:48.600097 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:43:48.600061 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:43:49.746795 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:43:49.746766 4611 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-69694c4c9-r6d8g"] Oct 30 20:43:49.754777 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:43:49.754747 4611 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-69694c4c9-r6d8g" Oct 30 20:43:49.762059 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:43:49.762017 4611 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-69694c4c9-r6d8g"] Oct 30 20:43:49.762237 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:43:49.762217 4611 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"openshift-ai-inference-openshift-ai-inference-dockercfg-7dt9v\"" Oct 30 20:43:49.762358 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:43:49.762325 4611 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Oct 30 20:43:49.762422 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:43:49.762397 4611 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Oct 30 20:43:49.762479 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:43:49.762471 4611 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Oct 30 20:43:49.763367 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:43:49.763337 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:43:49.856015 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:43:49.855970 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23e3feb4-2305-4e6e-8d6b-ab240a27cd91-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-69694c4c9-r6d8g\" (UID: \"23e3feb4-2305-4e6e-8d6b-ab240a27cd91\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-69694c4c9-r6d8g" Oct 30 20:43:49.856163 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:43:49.856020 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23e3feb4-2305-4e6e-8d6b-ab240a27cd91-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-69694c4c9-r6d8g\" (UID: \"23e3feb4-2305-4e6e-8d6b-ab240a27cd91\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-69694c4c9-r6d8g" Oct 30 20:43:49.856163 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:43:49.856101 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23e3feb4-2305-4e6e-8d6b-ab240a27cd91-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-69694c4c9-r6d8g\" (UID: \"23e3feb4-2305-4e6e-8d6b-ab240a27cd91\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-69694c4c9-r6d8g" Oct 30 20:43:49.856163 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:43:49.856151 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23e3feb4-2305-4e6e-8d6b-ab240a27cd91-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-69694c4c9-r6d8g\" (UID: \"23e3feb4-2305-4e6e-8d6b-ab240a27cd91\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-69694c4c9-r6d8g" Oct 30 20:43:49.856298 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:43:49.856231 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7gp5f\" (UniqueName: \"kubernetes.io/projected/23e3feb4-2305-4e6e-8d6b-ab240a27cd91-kube-api-access-7gp5f\") pod \"openshift-ai-inference-openshift-ai-inference-69694c4c9-r6d8g\" (UID: \"23e3feb4-2305-4e6e-8d6b-ab240a27cd91\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-69694c4c9-r6d8g" Oct 30 20:43:49.856298 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:43:49.856258 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23e3feb4-2305-4e6e-8d6b-ab240a27cd91-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-69694c4c9-r6d8g\" (UID: \"23e3feb4-2305-4e6e-8d6b-ab240a27cd91\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-69694c4c9-r6d8g" Oct 30 20:43:49.856298 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:43:49.856287 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23e3feb4-2305-4e6e-8d6b-ab240a27cd91-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-69694c4c9-r6d8g\" (UID: \"23e3feb4-2305-4e6e-8d6b-ab240a27cd91\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-69694c4c9-r6d8g" Oct 30 20:43:49.856409 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:43:49.856329 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/23e3feb4-2305-4e6e-8d6b-ab240a27cd91-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-69694c4c9-r6d8g\" (UID: \"23e3feb4-2305-4e6e-8d6b-ab240a27cd91\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-69694c4c9-r6d8g" Oct 30 20:43:49.856409 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:43:49.856367 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23e3feb4-2305-4e6e-8d6b-ab240a27cd91-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-69694c4c9-r6d8g\" (UID: \"23e3feb4-2305-4e6e-8d6b-ab240a27cd91\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-69694c4c9-r6d8g" Oct 30 20:43:49.905984 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:43:49.905955 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:43:49.957425 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:43:49.957403 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23e3feb4-2305-4e6e-8d6b-ab240a27cd91-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-69694c4c9-r6d8g\" (UID: \"23e3feb4-2305-4e6e-8d6b-ab240a27cd91\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-69694c4c9-r6d8g" Oct 30 20:43:49.957500 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:43:49.957436 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23e3feb4-2305-4e6e-8d6b-ab240a27cd91-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-69694c4c9-r6d8g\" (UID: \"23e3feb4-2305-4e6e-8d6b-ab240a27cd91\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-69694c4c9-r6d8g" Oct 30 20:43:49.957500 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:43:49.957454 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23e3feb4-2305-4e6e-8d6b-ab240a27cd91-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-69694c4c9-r6d8g\" (UID: \"23e3feb4-2305-4e6e-8d6b-ab240a27cd91\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-69694c4c9-r6d8g" Oct 30 20:43:49.957500 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:43:49.957469 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23e3feb4-2305-4e6e-8d6b-ab240a27cd91-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-69694c4c9-r6d8g\" (UID: \"23e3feb4-2305-4e6e-8d6b-ab240a27cd91\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-69694c4c9-r6d8g" Oct 30 20:43:49.957633 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:43:49.957541 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23e3feb4-2305-4e6e-8d6b-ab240a27cd91-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-69694c4c9-r6d8g\" (UID: \"23e3feb4-2305-4e6e-8d6b-ab240a27cd91\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-69694c4c9-r6d8g" Oct 30 20:43:49.957633 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:43:49.957598 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7gp5f\" (UniqueName: \"kubernetes.io/projected/23e3feb4-2305-4e6e-8d6b-ab240a27cd91-kube-api-access-7gp5f\") pod \"openshift-ai-inference-openshift-ai-inference-69694c4c9-r6d8g\" (UID: \"23e3feb4-2305-4e6e-8d6b-ab240a27cd91\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-69694c4c9-r6d8g" Oct 30 20:43:49.957714 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:43:49.957630 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23e3feb4-2305-4e6e-8d6b-ab240a27cd91-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-69694c4c9-r6d8g\" (UID: \"23e3feb4-2305-4e6e-8d6b-ab240a27cd91\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-69694c4c9-r6d8g" Oct 30 20:43:49.957714 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:43:49.957661 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23e3feb4-2305-4e6e-8d6b-ab240a27cd91-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-69694c4c9-r6d8g\" (UID: \"23e3feb4-2305-4e6e-8d6b-ab240a27cd91\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-69694c4c9-r6d8g" Oct 30 20:43:49.957714 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:43:49.957696 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/23e3feb4-2305-4e6e-8d6b-ab240a27cd91-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-69694c4c9-r6d8g\" (UID: \"23e3feb4-2305-4e6e-8d6b-ab240a27cd91\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-69694c4c9-r6d8g" Oct 30 20:43:49.957878 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:43:49.957855 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23e3feb4-2305-4e6e-8d6b-ab240a27cd91-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-69694c4c9-r6d8g\" (UID: \"23e3feb4-2305-4e6e-8d6b-ab240a27cd91\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-69694c4c9-r6d8g" Oct 30 20:43:49.957934 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:43:49.957897 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23e3feb4-2305-4e6e-8d6b-ab240a27cd91-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-69694c4c9-r6d8g\" (UID: \"23e3feb4-2305-4e6e-8d6b-ab240a27cd91\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-69694c4c9-r6d8g" Oct 30 20:43:49.958026 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:43:49.958001 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23e3feb4-2305-4e6e-8d6b-ab240a27cd91-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-69694c4c9-r6d8g\" (UID: \"23e3feb4-2305-4e6e-8d6b-ab240a27cd91\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-69694c4c9-r6d8g" Oct 30 20:43:49.958213 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:43:49.958185 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23e3feb4-2305-4e6e-8d6b-ab240a27cd91-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-69694c4c9-r6d8g\" (UID: \"23e3feb4-2305-4e6e-8d6b-ab240a27cd91\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-69694c4c9-r6d8g" Oct 30 20:43:49.958324 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:43:49.958251 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23e3feb4-2305-4e6e-8d6b-ab240a27cd91-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-69694c4c9-r6d8g\" (UID: \"23e3feb4-2305-4e6e-8d6b-ab240a27cd91\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-69694c4c9-r6d8g" Oct 30 20:43:49.959749 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:43:49.959728 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23e3feb4-2305-4e6e-8d6b-ab240a27cd91-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-69694c4c9-r6d8g\" (UID: \"23e3feb4-2305-4e6e-8d6b-ab240a27cd91\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-69694c4c9-r6d8g" Oct 30 20:43:49.960044 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:43:49.960017 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23e3feb4-2305-4e6e-8d6b-ab240a27cd91-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-69694c4c9-r6d8g\" (UID: \"23e3feb4-2305-4e6e-8d6b-ab240a27cd91\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-69694c4c9-r6d8g" Oct 30 20:43:49.968890 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:43:49.968860 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/23e3feb4-2305-4e6e-8d6b-ab240a27cd91-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-69694c4c9-r6d8g\" (UID: \"23e3feb4-2305-4e6e-8d6b-ab240a27cd91\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-69694c4c9-r6d8g" Oct 30 20:43:49.969141 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:43:49.969126 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7gp5f\" (UniqueName: \"kubernetes.io/projected/23e3feb4-2305-4e6e-8d6b-ab240a27cd91-kube-api-access-7gp5f\") pod \"openshift-ai-inference-openshift-ai-inference-69694c4c9-r6d8g\" (UID: \"23e3feb4-2305-4e6e-8d6b-ab240a27cd91\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-69694c4c9-r6d8g" Oct 30 20:43:50.071928 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:43:50.071864 4611 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-69694c4c9-r6d8g" Oct 30 20:43:50.348744 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:43:50.348682 4611 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-69694c4c9-r6d8g"] Oct 30 20:43:50.351561 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:43:50.351501 4611 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23e3feb4_2305_4e6e_8d6b_ab240a27cd91.slice/crio-0f65eb038f9df7063a6c520ac7961e1160855ff974115e4213c7299c5a2c8323 WatchSource:0}: Error finding container 0f65eb038f9df7063a6c520ac7961e1160855ff974115e4213c7299c5a2c8323: Status 404 returned error can't find the container with id 0f65eb038f9df7063a6c520ac7961e1160855ff974115e4213c7299c5a2c8323 Oct 30 20:43:50.634970 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:43:50.634943 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-69694c4c9-r6d8g" event={"ID":"23e3feb4-2305-4e6e-8d6b-ab240a27cd91","Type":"ContainerStarted","Data":"0f65eb038f9df7063a6c520ac7961e1160855ff974115e4213c7299c5a2c8323"} Oct 30 20:43:53.122176 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:43:53.122129 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:43:53.319425 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:43:53.319399 4611 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:43:53.319495 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:43:53.319462 4611 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:43:53.319495 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:43:53.319490 4611 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:43:53.646710 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:43:53.646676 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-69694c4c9-r6d8g" event={"ID":"23e3feb4-2305-4e6e-8d6b-ab240a27cd91","Type":"ContainerStarted","Data":"098b65c6c2084bc3a911dc0db96fe8404e2926b4e9f36ce37fd6a82f0b956be9"} Oct 30 20:43:53.670386 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:43:53.670341 4611 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-69694c4c9-r6d8g" podStartSLOduration=1.705182389 podStartE2EDuration="4.670328121s" totalImagesPullingTime="2.965145732s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:43:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:43:53.669974107 +0000 UTC m=+288.204000722" watchObservedRunningTime="2026-10-30 20:43:53.670328121 +0000 UTC m=+288.204354737" Oct 30 20:43:54.072474 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:43:54.072402 4611 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-69694c4c9-r6d8g" Oct 30 20:43:54.076951 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:43:54.076924 4611 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-69694c4c9-r6d8g" Oct 30 20:43:54.209059 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:43:54.209027 4611 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xq5pw"] Oct 30 20:43:54.212553 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:43:54.212537 4611 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xq5pw" Oct 30 20:43:54.215777 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:43:54.215752 4611 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Oct 30 20:43:54.215890 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:43:54.215792 4611 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Oct 30 20:43:54.215946 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:43:54.215893 4611 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e644421527c169d876fdb59235c927986d6bd2-9d2815fc-dockercfg-xcxhd\"" Oct 30 20:43:54.221688 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:43:54.221669 4611 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xq5pw"] Oct 30 20:43:54.293187 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:43:54.293167 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/533193e0-489f-455c-92aa-bb7fcd2f62e3-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xq5pw\" (UID: \"533193e0-489f-455c-92aa-bb7fcd2f62e3\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xq5pw" Oct 30 20:43:54.293300 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:43:54.293194 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7nmjp\" (UniqueName: \"kubernetes.io/projected/533193e0-489f-455c-92aa-bb7fcd2f62e3-kube-api-access-7nmjp\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xq5pw\" (UID: \"533193e0-489f-455c-92aa-bb7fcd2f62e3\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xq5pw" Oct 30 20:43:54.293300 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:43:54.293224 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/533193e0-489f-455c-92aa-bb7fcd2f62e3-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xq5pw\" (UID: \"533193e0-489f-455c-92aa-bb7fcd2f62e3\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xq5pw" Oct 30 20:43:54.394305 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:43:54.394280 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/533193e0-489f-455c-92aa-bb7fcd2f62e3-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xq5pw\" (UID: \"533193e0-489f-455c-92aa-bb7fcd2f62e3\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xq5pw" Oct 30 20:43:54.394404 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:43:54.394310 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7nmjp\" (UniqueName: \"kubernetes.io/projected/533193e0-489f-455c-92aa-bb7fcd2f62e3-kube-api-access-7nmjp\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xq5pw\" (UID: \"533193e0-489f-455c-92aa-bb7fcd2f62e3\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xq5pw" Oct 30 20:43:54.394404 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:43:54.394340 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/533193e0-489f-455c-92aa-bb7fcd2f62e3-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xq5pw\" (UID: \"533193e0-489f-455c-92aa-bb7fcd2f62e3\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xq5pw" Oct 30 20:43:54.394670 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:43:54.394654 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/533193e0-489f-455c-92aa-bb7fcd2f62e3-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xq5pw\" (UID: \"533193e0-489f-455c-92aa-bb7fcd2f62e3\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xq5pw" Oct 30 20:43:54.394726 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:43:54.394708 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/533193e0-489f-455c-92aa-bb7fcd2f62e3-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xq5pw\" (UID: \"533193e0-489f-455c-92aa-bb7fcd2f62e3\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xq5pw" Oct 30 20:43:54.406386 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:43:54.406360 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7nmjp\" (UniqueName: \"kubernetes.io/projected/533193e0-489f-455c-92aa-bb7fcd2f62e3-kube-api-access-7nmjp\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xq5pw\" (UID: \"533193e0-489f-455c-92aa-bb7fcd2f62e3\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xq5pw" Oct 30 20:43:54.529136 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:43:54.529114 4611 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xq5pw" Oct 30 20:43:54.650806 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:43:54.650740 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-69694c4c9-r6d8g" Oct 30 20:43:54.651736 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:43:54.651715 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-69694c4c9-r6d8g" Oct 30 20:43:54.799928 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:43:54.799893 4611 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xq5pw"] Oct 30 20:43:54.803170 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:43:54.803141 4611 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod533193e0_489f_455c_92aa_bb7fcd2f62e3.slice/crio-4386abb7bd244d435444500a41515f2b2eb0d1b702aa3a3abd536dd5e726565d WatchSource:0}: Error finding container 4386abb7bd244d435444500a41515f2b2eb0d1b702aa3a3abd536dd5e726565d: Status 404 returned error can't find the container with id 4386abb7bd244d435444500a41515f2b2eb0d1b702aa3a3abd536dd5e726565d Oct 30 20:43:55.656069 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:43:55.656034 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xq5pw" event={"ID":"533193e0-489f-455c-92aa-bb7fcd2f62e3","Type":"ContainerStarted","Data":"4386abb7bd244d435444500a41515f2b2eb0d1b702aa3a3abd536dd5e726565d"} Oct 30 20:44:00.681576 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:00.681540 4611 generic.go:356] "Generic (PLEG): container finished" podID="533193e0-489f-455c-92aa-bb7fcd2f62e3" containerID="4ff1611c272959bb85c292f1f8b14ab0d5d6d19e1c3c539abc24fee22bb1b1ef" exitCode=0 Oct 30 20:44:00.681967 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:00.681612 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xq5pw" event={"ID":"533193e0-489f-455c-92aa-bb7fcd2f62e3","Type":"ContainerDied","Data":"4ff1611c272959bb85c292f1f8b14ab0d5d6d19e1c3c539abc24fee22bb1b1ef"} Oct 30 20:44:01.026504 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:01.026434 4611 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6cpgnb"] Oct 30 20:44:01.029679 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:01.029659 4611 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6cpgnb" Oct 30 20:44:01.034924 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:01.034901 4611 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3-b83c8cba-dockercfg-c46n9\"" Oct 30 20:44:01.041331 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:01.041307 4611 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6cpgnb"] Oct 30 20:44:01.060851 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:01.060828 4611 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jjv77"] Oct 30 20:44:01.064064 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:01.064049 4611 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jjv77" Oct 30 20:44:01.074449 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:01.074424 4611 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jjv77"] Oct 30 20:44:01.096267 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:01.096248 4611 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847akmx5c"] Oct 30 20:44:01.099090 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:01.099076 4611 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847akmx5c" Oct 30 20:44:01.109300 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:01.109282 4611 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847akmx5c"] Oct 30 20:44:01.137478 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:01.137459 4611 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"6541d93aaee335d6227a8778ab4e840bbe7f52-f98fbcf8-dockercfg-jtdrc\"" Oct 30 20:44:01.137567 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:01.137544 4611 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"da112bd609bff87f7bacd2ba941225d7cff15c-bd73b2fc-dockercfg-qkdgk\"" Oct 30 20:44:01.154322 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:01.154304 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/dbecf2f9-1c2d-41ae-9add-c603d15532fa-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6cpgnb\" (UID: \"dbecf2f9-1c2d-41ae-9add-c603d15532fa\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6cpgnb" Oct 30 20:44:01.154404 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:01.154348 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pt2bx\" (UniqueName: \"kubernetes.io/projected/4817aa10-669b-407c-8fe7-972876c73a8e-kube-api-access-pt2bx\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jjv77\" (UID: \"4817aa10-669b-407c-8fe7-972876c73a8e\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jjv77" Oct 30 20:44:01.154404 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:01.154371 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-87b2m\" (UniqueName: \"kubernetes.io/projected/c493a337-bffe-41ab-b895-3a75a850aa94-kube-api-access-87b2m\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847akmx5c\" (UID: \"c493a337-bffe-41ab-b895-3a75a850aa94\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847akmx5c" Oct 30 20:44:01.154479 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:01.154461 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z49vx\" (UniqueName: \"kubernetes.io/projected/dbecf2f9-1c2d-41ae-9add-c603d15532fa-kube-api-access-z49vx\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6cpgnb\" (UID: \"dbecf2f9-1c2d-41ae-9add-c603d15532fa\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6cpgnb" Oct 30 20:44:01.154546 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:01.154528 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4817aa10-669b-407c-8fe7-972876c73a8e-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jjv77\" (UID: \"4817aa10-669b-407c-8fe7-972876c73a8e\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jjv77" Oct 30 20:44:01.154589 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:01.154556 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4817aa10-669b-407c-8fe7-972876c73a8e-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jjv77\" (UID: \"4817aa10-669b-407c-8fe7-972876c73a8e\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jjv77" Oct 30 20:44:01.154589 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:01.154576 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/dbecf2f9-1c2d-41ae-9add-c603d15532fa-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6cpgnb\" (UID: \"dbecf2f9-1c2d-41ae-9add-c603d15532fa\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6cpgnb" Oct 30 20:44:01.154651 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:01.154590 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c493a337-bffe-41ab-b895-3a75a850aa94-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847akmx5c\" (UID: \"c493a337-bffe-41ab-b895-3a75a850aa94\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847akmx5c" Oct 30 20:44:01.154689 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:01.154655 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c493a337-bffe-41ab-b895-3a75a850aa94-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847akmx5c\" (UID: \"c493a337-bffe-41ab-b895-3a75a850aa94\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847akmx5c" Oct 30 20:44:01.255661 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:01.255634 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/dbecf2f9-1c2d-41ae-9add-c603d15532fa-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6cpgnb\" (UID: \"dbecf2f9-1c2d-41ae-9add-c603d15532fa\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6cpgnb" Oct 30 20:44:01.255769 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:01.255679 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pt2bx\" (UniqueName: \"kubernetes.io/projected/4817aa10-669b-407c-8fe7-972876c73a8e-kube-api-access-pt2bx\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jjv77\" (UID: \"4817aa10-669b-407c-8fe7-972876c73a8e\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jjv77" Oct 30 20:44:01.255813 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:01.255792 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-87b2m\" (UniqueName: \"kubernetes.io/projected/c493a337-bffe-41ab-b895-3a75a850aa94-kube-api-access-87b2m\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847akmx5c\" (UID: \"c493a337-bffe-41ab-b895-3a75a850aa94\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847akmx5c" Oct 30 20:44:01.255848 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:01.255832 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z49vx\" (UniqueName: \"kubernetes.io/projected/dbecf2f9-1c2d-41ae-9add-c603d15532fa-kube-api-access-z49vx\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6cpgnb\" (UID: \"dbecf2f9-1c2d-41ae-9add-c603d15532fa\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6cpgnb" Oct 30 20:44:01.255885 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:01.255872 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4817aa10-669b-407c-8fe7-972876c73a8e-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jjv77\" (UID: \"4817aa10-669b-407c-8fe7-972876c73a8e\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jjv77" Oct 30 20:44:01.255940 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:01.255890 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4817aa10-669b-407c-8fe7-972876c73a8e-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jjv77\" (UID: \"4817aa10-669b-407c-8fe7-972876c73a8e\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jjv77" Oct 30 20:44:01.255991 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:01.255965 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/dbecf2f9-1c2d-41ae-9add-c603d15532fa-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6cpgnb\" (UID: \"dbecf2f9-1c2d-41ae-9add-c603d15532fa\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6cpgnb" Oct 30 20:44:01.256051 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:01.256003 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c493a337-bffe-41ab-b895-3a75a850aa94-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847akmx5c\" (UID: \"c493a337-bffe-41ab-b895-3a75a850aa94\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847akmx5c" Oct 30 20:44:01.256051 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:01.255977 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/dbecf2f9-1c2d-41ae-9add-c603d15532fa-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6cpgnb\" (UID: \"dbecf2f9-1c2d-41ae-9add-c603d15532fa\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6cpgnb" Oct 30 20:44:01.256051 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:01.256035 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c493a337-bffe-41ab-b895-3a75a850aa94-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847akmx5c\" (UID: \"c493a337-bffe-41ab-b895-3a75a850aa94\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847akmx5c" Oct 30 20:44:01.256212 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:01.256193 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/dbecf2f9-1c2d-41ae-9add-c603d15532fa-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6cpgnb\" (UID: \"dbecf2f9-1c2d-41ae-9add-c603d15532fa\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6cpgnb" Oct 30 20:44:01.256261 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:01.256244 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4817aa10-669b-407c-8fe7-972876c73a8e-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jjv77\" (UID: \"4817aa10-669b-407c-8fe7-972876c73a8e\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jjv77" Oct 30 20:44:01.256295 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:01.256264 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4817aa10-669b-407c-8fe7-972876c73a8e-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jjv77\" (UID: \"4817aa10-669b-407c-8fe7-972876c73a8e\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jjv77" Oct 30 20:44:01.256392 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:01.256374 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c493a337-bffe-41ab-b895-3a75a850aa94-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847akmx5c\" (UID: \"c493a337-bffe-41ab-b895-3a75a850aa94\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847akmx5c" Oct 30 20:44:01.256425 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:01.256395 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c493a337-bffe-41ab-b895-3a75a850aa94-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847akmx5c\" (UID: \"c493a337-bffe-41ab-b895-3a75a850aa94\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847akmx5c" Oct 30 20:44:01.267206 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:01.267181 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pt2bx\" (UniqueName: \"kubernetes.io/projected/4817aa10-669b-407c-8fe7-972876c73a8e-kube-api-access-pt2bx\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jjv77\" (UID: \"4817aa10-669b-407c-8fe7-972876c73a8e\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jjv77" Oct 30 20:44:01.267316 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:01.267271 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-87b2m\" (UniqueName: \"kubernetes.io/projected/c493a337-bffe-41ab-b895-3a75a850aa94-kube-api-access-87b2m\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847akmx5c\" (UID: \"c493a337-bffe-41ab-b895-3a75a850aa94\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847akmx5c" Oct 30 20:44:01.267316 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:01.267278 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z49vx\" (UniqueName: \"kubernetes.io/projected/dbecf2f9-1c2d-41ae-9add-c603d15532fa-kube-api-access-z49vx\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6cpgnb\" (UID: \"dbecf2f9-1c2d-41ae-9add-c603d15532fa\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6cpgnb" Oct 30 20:44:01.346825 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:01.346774 4611 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6cpgnb" Oct 30 20:44:01.380905 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:01.380882 4611 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jjv77" Oct 30 20:44:01.416840 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:01.416800 4611 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847akmx5c" Oct 30 20:44:01.620773 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:01.620733 4611 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6cpgnb"] Oct 30 20:44:01.623835 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:44:01.623812 4611 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddbecf2f9_1c2d_41ae_9add_c603d15532fa.slice/crio-1730a2856895e5e9a239f777e69d5daebb6e23f20b93e592fca0e8a36dcf7994 WatchSource:0}: Error finding container 1730a2856895e5e9a239f777e69d5daebb6e23f20b93e592fca0e8a36dcf7994: Status 404 returned error can't find the container with id 1730a2856895e5e9a239f777e69d5daebb6e23f20b93e592fca0e8a36dcf7994 Oct 30 20:44:01.671774 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:01.671754 4611 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jjv77"] Oct 30 20:44:01.674604 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:44:01.674579 4611 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4817aa10_669b_407c_8fe7_972876c73a8e.slice/crio-222584b37335b64a2934eebcdf7f59b38d4825d2745dd6b58dffbbd35e36673a WatchSource:0}: Error finding container 222584b37335b64a2934eebcdf7f59b38d4825d2745dd6b58dffbbd35e36673a: Status 404 returned error can't find the container with id 222584b37335b64a2934eebcdf7f59b38d4825d2745dd6b58dffbbd35e36673a Oct 30 20:44:01.687489 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:01.687465 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jjv77" event={"ID":"4817aa10-669b-407c-8fe7-972876c73a8e","Type":"ContainerStarted","Data":"222584b37335b64a2934eebcdf7f59b38d4825d2745dd6b58dffbbd35e36673a"} Oct 30 20:44:01.689163 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:01.689137 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6cpgnb" event={"ID":"dbecf2f9-1c2d-41ae-9add-c603d15532fa","Type":"ContainerStarted","Data":"45fc762f6b2ef7546ab2b1e80305cbb470648782aac26b5788706d067dffe127"} Oct 30 20:44:01.689248 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:01.689188 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6cpgnb" event={"ID":"dbecf2f9-1c2d-41ae-9add-c603d15532fa","Type":"ContainerStarted","Data":"1730a2856895e5e9a239f777e69d5daebb6e23f20b93e592fca0e8a36dcf7994"} Oct 30 20:44:01.691187 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:01.691161 4611 generic.go:356] "Generic (PLEG): container finished" podID="533193e0-489f-455c-92aa-bb7fcd2f62e3" containerID="8b3de7f06b8ab6c5eba99c6f03a8d0449f3b7c10fcde97f47db875f11c463bbd" exitCode=0 Oct 30 20:44:01.691271 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:01.691230 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xq5pw" event={"ID":"533193e0-489f-455c-92aa-bb7fcd2f62e3","Type":"ContainerDied","Data":"8b3de7f06b8ab6c5eba99c6f03a8d0449f3b7c10fcde97f47db875f11c463bbd"} Oct 30 20:44:01.699940 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:01.699914 4611 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847akmx5c"] Oct 30 20:44:01.709596 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:44:01.709572 4611 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc493a337_bffe_41ab_b895_3a75a850aa94.slice/crio-9aef15b41fa27a2be9d23234a4e63c288a1e5d24cde07aff79d9933bc7d06a86 WatchSource:0}: Error finding container 9aef15b41fa27a2be9d23234a4e63c288a1e5d24cde07aff79d9933bc7d06a86: Status 404 returned error can't find the container with id 9aef15b41fa27a2be9d23234a4e63c288a1e5d24cde07aff79d9933bc7d06a86 Oct 30 20:44:02.696319 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:02.696281 4611 generic.go:356] "Generic (PLEG): container finished" podID="4817aa10-669b-407c-8fe7-972876c73a8e" containerID="460728fdb7e729fba2495ed47324c9bc096b9c1bc29575b68601d45fe4491ee3" exitCode=0 Oct 30 20:44:02.696909 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:02.696370 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jjv77" event={"ID":"4817aa10-669b-407c-8fe7-972876c73a8e","Type":"ContainerDied","Data":"460728fdb7e729fba2495ed47324c9bc096b9c1bc29575b68601d45fe4491ee3"} Oct 30 20:44:02.698100 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:02.698071 4611 generic.go:356] "Generic (PLEG): container finished" podID="dbecf2f9-1c2d-41ae-9add-c603d15532fa" containerID="45fc762f6b2ef7546ab2b1e80305cbb470648782aac26b5788706d067dffe127" exitCode=0 Oct 30 20:44:02.698208 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:02.698098 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6cpgnb" event={"ID":"dbecf2f9-1c2d-41ae-9add-c603d15532fa","Type":"ContainerDied","Data":"45fc762f6b2ef7546ab2b1e80305cbb470648782aac26b5788706d067dffe127"} Oct 30 20:44:02.700437 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:02.700353 4611 generic.go:356] "Generic (PLEG): container finished" podID="c493a337-bffe-41ab-b895-3a75a850aa94" containerID="9a9185d39e64cfa471c5cc665fe8d700f05a7ad3c022a2615e1876ec23f9d8ef" exitCode=0 Oct 30 20:44:02.700437 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:02.700412 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847akmx5c" event={"ID":"c493a337-bffe-41ab-b895-3a75a850aa94","Type":"ContainerDied","Data":"9a9185d39e64cfa471c5cc665fe8d700f05a7ad3c022a2615e1876ec23f9d8ef"} Oct 30 20:44:02.700559 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:02.700443 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847akmx5c" event={"ID":"c493a337-bffe-41ab-b895-3a75a850aa94","Type":"ContainerStarted","Data":"9aef15b41fa27a2be9d23234a4e63c288a1e5d24cde07aff79d9933bc7d06a86"} Oct 30 20:44:03.707554 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:03.707504 4611 generic.go:356] "Generic (PLEG): container finished" podID="4817aa10-669b-407c-8fe7-972876c73a8e" containerID="87390d89096d54000c6e6b109e4d66f8b6ce3377762abd3d6267ec20bba98804" exitCode=0 Oct 30 20:44:03.707896 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:03.707570 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jjv77" event={"ID":"4817aa10-669b-407c-8fe7-972876c73a8e","Type":"ContainerDied","Data":"87390d89096d54000c6e6b109e4d66f8b6ce3377762abd3d6267ec20bba98804"} Oct 30 20:44:03.919684 ip-10-0-133-192 kubenswrapper[4611]: E1030 20:44:03.919609 4611 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hwzc6" containerName="extractor" Oct 30 20:44:04.713958 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:04.713873 4611 generic.go:356] "Generic (PLEG): container finished" podID="dbecf2f9-1c2d-41ae-9add-c603d15532fa" containerID="55dcb99f2eff869fcabf3f1c0f4e535cc3fbb1a1cb1ee65c6616cacdd368c1fc" exitCode=0 Oct 30 20:44:04.714373 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:04.713959 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6cpgnb" event={"ID":"dbecf2f9-1c2d-41ae-9add-c603d15532fa","Type":"ContainerDied","Data":"55dcb99f2eff869fcabf3f1c0f4e535cc3fbb1a1cb1ee65c6616cacdd368c1fc"} Oct 30 20:44:04.717327 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:04.717301 4611 generic.go:356] "Generic (PLEG): container finished" podID="c493a337-bffe-41ab-b895-3a75a850aa94" containerID="9266dfccb7ac22390d6e256734391c611c5e17a3237eb1d1d2f3557314fef3a5" exitCode=0 Oct 30 20:44:04.717447 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:04.717429 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847akmx5c" event={"ID":"c493a337-bffe-41ab-b895-3a75a850aa94","Type":"ContainerDied","Data":"9266dfccb7ac22390d6e256734391c611c5e17a3237eb1d1d2f3557314fef3a5"} Oct 30 20:44:05.634127 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:05.634100 4611 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7w5wz_7e2574c1-b59c-453c-9501-25ec13a1ccad/ovn-acl-logging/0.log" Oct 30 20:44:05.635782 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:05.635741 4611 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7w5wz_7e2574c1-b59c-453c-9501-25ec13a1ccad/ovn-acl-logging/0.log" Oct 30 20:44:09.746459 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:09.746369 4611 generic.go:356] "Generic (PLEG): container finished" podID="533193e0-489f-455c-92aa-bb7fcd2f62e3" containerID="89c8fae04c933f1c7d1fdf27bb36da958c275e128da04951dd11aa004b9e844f" exitCode=0 Oct 30 20:44:09.746897 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:09.746488 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xq5pw" event={"ID":"533193e0-489f-455c-92aa-bb7fcd2f62e3","Type":"ContainerDied","Data":"89c8fae04c933f1c7d1fdf27bb36da958c275e128da04951dd11aa004b9e844f"} Oct 30 20:44:10.753794 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:10.753714 4611 generic.go:356] "Generic (PLEG): container finished" podID="dbecf2f9-1c2d-41ae-9add-c603d15532fa" containerID="97eb1ab76a7bdb55b5cf77c2f07566c6fd7218c26ac9392daed0a969e8ee508f" exitCode=0 Oct 30 20:44:10.754234 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:10.753797 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6cpgnb" event={"ID":"dbecf2f9-1c2d-41ae-9add-c603d15532fa","Type":"ContainerDied","Data":"97eb1ab76a7bdb55b5cf77c2f07566c6fd7218c26ac9392daed0a969e8ee508f"} Oct 30 20:44:10.756151 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:10.756123 4611 generic.go:356] "Generic (PLEG): container finished" podID="c493a337-bffe-41ab-b895-3a75a850aa94" containerID="11f81bc8a9ab9a925116a24ca4c0c543646cc5566e48f0cc5cf2dc53e706f84e" exitCode=0 Oct 30 20:44:10.756284 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:10.756203 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847akmx5c" event={"ID":"c493a337-bffe-41ab-b895-3a75a850aa94","Type":"ContainerDied","Data":"11f81bc8a9ab9a925116a24ca4c0c543646cc5566e48f0cc5cf2dc53e706f84e"} Oct 30 20:44:10.898933 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:10.898905 4611 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xq5pw" Oct 30 20:44:10.952253 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:10.952223 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/533193e0-489f-455c-92aa-bb7fcd2f62e3-util\" (UniqueName: \"kubernetes.io/empty-dir/533193e0-489f-455c-92aa-bb7fcd2f62e3-util\") pod \"533193e0-489f-455c-92aa-bb7fcd2f62e3\" (UID: \"533193e0-489f-455c-92aa-bb7fcd2f62e3\") " Oct 30 20:44:10.952397 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:10.952268 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/533193e0-489f-455c-92aa-bb7fcd2f62e3-kube-api-access-7nmjp\" (UniqueName: \"kubernetes.io/projected/533193e0-489f-455c-92aa-bb7fcd2f62e3-kube-api-access-7nmjp\") pod \"533193e0-489f-455c-92aa-bb7fcd2f62e3\" (UID: \"533193e0-489f-455c-92aa-bb7fcd2f62e3\") " Oct 30 20:44:10.952397 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:10.952347 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/533193e0-489f-455c-92aa-bb7fcd2f62e3-bundle\" (UniqueName: \"kubernetes.io/empty-dir/533193e0-489f-455c-92aa-bb7fcd2f62e3-bundle\") pod \"533193e0-489f-455c-92aa-bb7fcd2f62e3\" (UID: \"533193e0-489f-455c-92aa-bb7fcd2f62e3\") " Oct 30 20:44:10.953011 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:10.952981 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/533193e0-489f-455c-92aa-bb7fcd2f62e3-bundle" pod "533193e0-489f-455c-92aa-bb7fcd2f62e3" (UID: "533193e0-489f-455c-92aa-bb7fcd2f62e3"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:44:10.954882 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:10.954853 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/533193e0-489f-455c-92aa-bb7fcd2f62e3-kube-api-access-7nmjp" pod "533193e0-489f-455c-92aa-bb7fcd2f62e3" (UID: "533193e0-489f-455c-92aa-bb7fcd2f62e3"). InnerVolumeSpecName "kube-api-access-7nmjp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:44:10.957423 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:10.957401 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/533193e0-489f-455c-92aa-bb7fcd2f62e3-util" pod "533193e0-489f-455c-92aa-bb7fcd2f62e3" (UID: "533193e0-489f-455c-92aa-bb7fcd2f62e3"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:44:11.053487 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:11.053413 4611 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/533193e0-489f-455c-92aa-bb7fcd2f62e3-util\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:44:11.053487 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:11.053445 4611 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7nmjp\" (UniqueName: \"kubernetes.io/projected/533193e0-489f-455c-92aa-bb7fcd2f62e3-kube-api-access-7nmjp\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:44:11.053487 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:11.053457 4611 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/533193e0-489f-455c-92aa-bb7fcd2f62e3-bundle\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:44:11.761712 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:11.761682 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jjv77" event={"ID":"4817aa10-669b-407c-8fe7-972876c73a8e","Type":"ContainerStarted","Data":"49e243047da28478b16f0d64b6fbe16686ae5e84bb36c1cd56f091fb9991846e"} Oct 30 20:44:11.763576 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:11.763537 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xq5pw" event={"ID":"533193e0-489f-455c-92aa-bb7fcd2f62e3","Type":"ContainerDied","Data":"4386abb7bd244d435444500a41515f2b2eb0d1b702aa3a3abd536dd5e726565d"} Oct 30 20:44:11.763697 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:11.763588 4611 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4386abb7bd244d435444500a41515f2b2eb0d1b702aa3a3abd536dd5e726565d" Oct 30 20:44:11.763697 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:11.763553 4611 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xq5pw" Oct 30 20:44:11.783155 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:11.782959 4611 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jjv77" podStartSLOduration=2.151668332 podStartE2EDuration="10.782943981s" totalImagesPullingTime="8.631275649s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:44:01 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:44:11.781381035 +0000 UTC m=+306.315407651" watchObservedRunningTime="2026-10-30 20:44:11.782943981 +0000 UTC m=+306.316970594" Oct 30 20:44:11.890411 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:11.890389 4611 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6cpgnb" Oct 30 20:44:11.904876 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:11.904858 4611 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847akmx5c" Oct 30 20:44:11.960542 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:11.960503 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c493a337-bffe-41ab-b895-3a75a850aa94-kube-api-access-87b2m\" (UniqueName: \"kubernetes.io/projected/c493a337-bffe-41ab-b895-3a75a850aa94-kube-api-access-87b2m\") pod \"c493a337-bffe-41ab-b895-3a75a850aa94\" (UID: \"c493a337-bffe-41ab-b895-3a75a850aa94\") " Oct 30 20:44:11.960655 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:11.960553 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dbecf2f9-1c2d-41ae-9add-c603d15532fa-kube-api-access-z49vx\" (UniqueName: \"kubernetes.io/projected/dbecf2f9-1c2d-41ae-9add-c603d15532fa-kube-api-access-z49vx\") pod \"dbecf2f9-1c2d-41ae-9add-c603d15532fa\" (UID: \"dbecf2f9-1c2d-41ae-9add-c603d15532fa\") " Oct 30 20:44:11.960655 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:11.960575 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c493a337-bffe-41ab-b895-3a75a850aa94-util\" (UniqueName: \"kubernetes.io/empty-dir/c493a337-bffe-41ab-b895-3a75a850aa94-util\") pod \"c493a337-bffe-41ab-b895-3a75a850aa94\" (UID: \"c493a337-bffe-41ab-b895-3a75a850aa94\") " Oct 30 20:44:11.960655 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:11.960607 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dbecf2f9-1c2d-41ae-9add-c603d15532fa-bundle\" (UniqueName: \"kubernetes.io/empty-dir/dbecf2f9-1c2d-41ae-9add-c603d15532fa-bundle\") pod \"dbecf2f9-1c2d-41ae-9add-c603d15532fa\" (UID: \"dbecf2f9-1c2d-41ae-9add-c603d15532fa\") " Oct 30 20:44:11.960655 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:11.960649 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c493a337-bffe-41ab-b895-3a75a850aa94-bundle\" (UniqueName: \"kubernetes.io/empty-dir/c493a337-bffe-41ab-b895-3a75a850aa94-bundle\") pod \"c493a337-bffe-41ab-b895-3a75a850aa94\" (UID: \"c493a337-bffe-41ab-b895-3a75a850aa94\") " Oct 30 20:44:11.960795 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:11.960700 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dbecf2f9-1c2d-41ae-9add-c603d15532fa-util\" (UniqueName: \"kubernetes.io/empty-dir/dbecf2f9-1c2d-41ae-9add-c603d15532fa-util\") pod \"dbecf2f9-1c2d-41ae-9add-c603d15532fa\" (UID: \"dbecf2f9-1c2d-41ae-9add-c603d15532fa\") " Oct 30 20:44:11.961106 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:11.961071 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbecf2f9-1c2d-41ae-9add-c603d15532fa-bundle" pod "dbecf2f9-1c2d-41ae-9add-c603d15532fa" (UID: "dbecf2f9-1c2d-41ae-9add-c603d15532fa"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:44:11.961375 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:11.961352 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c493a337-bffe-41ab-b895-3a75a850aa94-bundle" pod "c493a337-bffe-41ab-b895-3a75a850aa94" (UID: "c493a337-bffe-41ab-b895-3a75a850aa94"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:44:11.963099 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:11.963081 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c493a337-bffe-41ab-b895-3a75a850aa94-kube-api-access-87b2m" pod "c493a337-bffe-41ab-b895-3a75a850aa94" (UID: "c493a337-bffe-41ab-b895-3a75a850aa94"). InnerVolumeSpecName "kube-api-access-87b2m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:44:11.963174 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:11.963121 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dbecf2f9-1c2d-41ae-9add-c603d15532fa-kube-api-access-z49vx" pod "dbecf2f9-1c2d-41ae-9add-c603d15532fa" (UID: "dbecf2f9-1c2d-41ae-9add-c603d15532fa"). InnerVolumeSpecName "kube-api-access-z49vx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:44:11.965275 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:11.965255 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c493a337-bffe-41ab-b895-3a75a850aa94-util" pod "c493a337-bffe-41ab-b895-3a75a850aa94" (UID: "c493a337-bffe-41ab-b895-3a75a850aa94"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:44:11.965444 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:11.965419 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbecf2f9-1c2d-41ae-9add-c603d15532fa-util" pod "dbecf2f9-1c2d-41ae-9add-c603d15532fa" (UID: "dbecf2f9-1c2d-41ae-9add-c603d15532fa"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:44:12.061823 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:12.061799 4611 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c493a337-bffe-41ab-b895-3a75a850aa94-bundle\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:44:12.061823 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:12.061820 4611 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/dbecf2f9-1c2d-41ae-9add-c603d15532fa-util\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:44:12.061952 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:12.061830 4611 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-87b2m\" (UniqueName: \"kubernetes.io/projected/c493a337-bffe-41ab-b895-3a75a850aa94-kube-api-access-87b2m\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:44:12.061952 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:12.061840 4611 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z49vx\" (UniqueName: \"kubernetes.io/projected/dbecf2f9-1c2d-41ae-9add-c603d15532fa-kube-api-access-z49vx\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:44:12.061952 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:12.061850 4611 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c493a337-bffe-41ab-b895-3a75a850aa94-util\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:44:12.061952 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:12.061859 4611 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/dbecf2f9-1c2d-41ae-9add-c603d15532fa-bundle\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:44:12.768817 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:12.768788 4611 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847akmx5c" Oct 30 20:44:12.769227 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:12.768819 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847akmx5c" event={"ID":"c493a337-bffe-41ab-b895-3a75a850aa94","Type":"ContainerDied","Data":"9aef15b41fa27a2be9d23234a4e63c288a1e5d24cde07aff79d9933bc7d06a86"} Oct 30 20:44:12.769227 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:12.768867 4611 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9aef15b41fa27a2be9d23234a4e63c288a1e5d24cde07aff79d9933bc7d06a86" Oct 30 20:44:12.771002 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:12.770980 4611 generic.go:356] "Generic (PLEG): container finished" podID="4817aa10-669b-407c-8fe7-972876c73a8e" containerID="49e243047da28478b16f0d64b6fbe16686ae5e84bb36c1cd56f091fb9991846e" exitCode=0 Oct 30 20:44:12.771109 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:12.771043 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jjv77" event={"ID":"4817aa10-669b-407c-8fe7-972876c73a8e","Type":"ContainerDied","Data":"49e243047da28478b16f0d64b6fbe16686ae5e84bb36c1cd56f091fb9991846e"} Oct 30 20:44:12.772843 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:12.772823 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6cpgnb" event={"ID":"dbecf2f9-1c2d-41ae-9add-c603d15532fa","Type":"ContainerDied","Data":"1730a2856895e5e9a239f777e69d5daebb6e23f20b93e592fca0e8a36dcf7994"} Oct 30 20:44:12.772904 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:12.772860 4611 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1730a2856895e5e9a239f777e69d5daebb6e23f20b93e592fca0e8a36dcf7994" Oct 30 20:44:12.772904 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:12.772868 4611 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6cpgnb" Oct 30 20:44:13.899006 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:13.898983 4611 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jjv77" Oct 30 20:44:13.976121 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:13.976091 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4817aa10-669b-407c-8fe7-972876c73a8e-bundle\" (UniqueName: \"kubernetes.io/empty-dir/4817aa10-669b-407c-8fe7-972876c73a8e-bundle\") pod \"4817aa10-669b-407c-8fe7-972876c73a8e\" (UID: \"4817aa10-669b-407c-8fe7-972876c73a8e\") " Oct 30 20:44:13.976270 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:13.976150 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4817aa10-669b-407c-8fe7-972876c73a8e-kube-api-access-pt2bx\" (UniqueName: \"kubernetes.io/projected/4817aa10-669b-407c-8fe7-972876c73a8e-kube-api-access-pt2bx\") pod \"4817aa10-669b-407c-8fe7-972876c73a8e\" (UID: \"4817aa10-669b-407c-8fe7-972876c73a8e\") " Oct 30 20:44:13.976270 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:13.976222 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4817aa10-669b-407c-8fe7-972876c73a8e-util\" (UniqueName: \"kubernetes.io/empty-dir/4817aa10-669b-407c-8fe7-972876c73a8e-util\") pod \"4817aa10-669b-407c-8fe7-972876c73a8e\" (UID: \"4817aa10-669b-407c-8fe7-972876c73a8e\") " Oct 30 20:44:13.976581 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:13.976555 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4817aa10-669b-407c-8fe7-972876c73a8e-bundle" pod "4817aa10-669b-407c-8fe7-972876c73a8e" (UID: "4817aa10-669b-407c-8fe7-972876c73a8e"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:44:13.978023 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:13.978001 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4817aa10-669b-407c-8fe7-972876c73a8e-kube-api-access-pt2bx" pod "4817aa10-669b-407c-8fe7-972876c73a8e" (UID: "4817aa10-669b-407c-8fe7-972876c73a8e"). InnerVolumeSpecName "kube-api-access-pt2bx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:44:13.979916 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:13.979892 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4817aa10-669b-407c-8fe7-972876c73a8e-util" pod "4817aa10-669b-407c-8fe7-972876c73a8e" (UID: "4817aa10-669b-407c-8fe7-972876c73a8e"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:44:14.077804 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:14.077750 4611 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4817aa10-669b-407c-8fe7-972876c73a8e-util\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:44:14.077804 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:14.077771 4611 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4817aa10-669b-407c-8fe7-972876c73a8e-bundle\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:44:14.077804 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:14.077781 4611 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pt2bx\" (UniqueName: \"kubernetes.io/projected/4817aa10-669b-407c-8fe7-972876c73a8e-kube-api-access-pt2bx\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:44:14.782647 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:14.782612 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jjv77" event={"ID":"4817aa10-669b-407c-8fe7-972876c73a8e","Type":"ContainerDied","Data":"222584b37335b64a2934eebcdf7f59b38d4825d2745dd6b58dffbbd35e36673a"} Oct 30 20:44:14.782647 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:14.782632 4611 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jjv77" Oct 30 20:44:14.782841 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:14.782662 4611 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="222584b37335b64a2934eebcdf7f59b38d4825d2745dd6b58dffbbd35e36673a" Oct 30 20:44:16.039359 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:16.039321 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:44:16.219535 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:16.219488 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:44:16.423206 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:16.423156 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:44:16.703026 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:16.702950 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:44:16.891659 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:16.891624 4611 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-9ffc9ff9f-g8bb7"] Oct 30 20:44:16.892008 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:16.891990 4611 cpu_manager.go:414] "containerMap: removing container" podUID="c493a337-bffe-41ab-b895-3a75a850aa94" containerName="extract" Oct 30 20:44:16.892095 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:16.892010 4611 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c493a337-bffe-41ab-b895-3a75a850aa94" containerName="extract" Oct 30 20:44:16.892095 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:16.892030 4611 cpu_manager.go:414] "containerMap: removing container" podUID="dbecf2f9-1c2d-41ae-9add-c603d15532fa" containerName="extract" Oct 30 20:44:16.892095 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:16.892037 4611 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dbecf2f9-1c2d-41ae-9add-c603d15532fa" containerName="extract" Oct 30 20:44:16.892095 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:16.892063 4611 cpu_manager.go:414] "containerMap: removing container" podUID="4817aa10-669b-407c-8fe7-972876c73a8e" containerName="util" Oct 30 20:44:16.892095 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:16.892070 4611 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4817aa10-669b-407c-8fe7-972876c73a8e" containerName="util" Oct 30 20:44:16.892095 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:16.892089 4611 cpu_manager.go:414] "containerMap: removing container" podUID="4817aa10-669b-407c-8fe7-972876c73a8e" containerName="extract" Oct 30 20:44:16.892095 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:16.892096 4611 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4817aa10-669b-407c-8fe7-972876c73a8e" containerName="extract" Oct 30 20:44:16.892431 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:16.892108 4611 cpu_manager.go:414] "containerMap: removing container" podUID="dbecf2f9-1c2d-41ae-9add-c603d15532fa" containerName="pull" Oct 30 20:44:16.892431 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:16.892115 4611 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dbecf2f9-1c2d-41ae-9add-c603d15532fa" containerName="pull" Oct 30 20:44:16.892431 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:16.892125 4611 cpu_manager.go:414] "containerMap: removing container" podUID="c493a337-bffe-41ab-b895-3a75a850aa94" containerName="pull" Oct 30 20:44:16.892431 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:16.892132 4611 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c493a337-bffe-41ab-b895-3a75a850aa94" containerName="pull" Oct 30 20:44:16.892431 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:16.892147 4611 cpu_manager.go:414] "containerMap: removing container" podUID="533193e0-489f-455c-92aa-bb7fcd2f62e3" containerName="util" Oct 30 20:44:16.892431 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:16.892155 4611 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="533193e0-489f-455c-92aa-bb7fcd2f62e3" containerName="util" Oct 30 20:44:16.892431 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:16.892165 4611 cpu_manager.go:414] "containerMap: removing container" podUID="c493a337-bffe-41ab-b895-3a75a850aa94" containerName="util" Oct 30 20:44:16.892431 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:16.892172 4611 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c493a337-bffe-41ab-b895-3a75a850aa94" containerName="util" Oct 30 20:44:16.892431 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:16.892183 4611 cpu_manager.go:414] "containerMap: removing container" podUID="533193e0-489f-455c-92aa-bb7fcd2f62e3" containerName="extract" Oct 30 20:44:16.892431 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:16.892189 4611 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="533193e0-489f-455c-92aa-bb7fcd2f62e3" containerName="extract" Oct 30 20:44:16.892431 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:16.892203 4611 cpu_manager.go:414] "containerMap: removing container" podUID="533193e0-489f-455c-92aa-bb7fcd2f62e3" containerName="pull" Oct 30 20:44:16.892431 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:16.892210 4611 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="533193e0-489f-455c-92aa-bb7fcd2f62e3" containerName="pull" Oct 30 20:44:16.892431 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:16.892220 4611 cpu_manager.go:414] "containerMap: removing container" podUID="dbecf2f9-1c2d-41ae-9add-c603d15532fa" containerName="util" Oct 30 20:44:16.892431 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:16.892226 4611 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dbecf2f9-1c2d-41ae-9add-c603d15532fa" containerName="util" Oct 30 20:44:16.892431 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:16.892239 4611 cpu_manager.go:414] "containerMap: removing container" podUID="4817aa10-669b-407c-8fe7-972876c73a8e" containerName="pull" Oct 30 20:44:16.892431 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:16.892246 4611 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4817aa10-669b-407c-8fe7-972876c73a8e" containerName="pull" Oct 30 20:44:16.892431 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:16.892320 4611 memory_manager.go:365] "RemoveStaleState removing state" podUID="4817aa10-669b-407c-8fe7-972876c73a8e" containerName="extract" Oct 30 20:44:16.892431 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:16.892332 4611 memory_manager.go:365] "RemoveStaleState removing state" podUID="533193e0-489f-455c-92aa-bb7fcd2f62e3" containerName="extract" Oct 30 20:44:16.892431 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:16.892344 4611 memory_manager.go:365] "RemoveStaleState removing state" podUID="c493a337-bffe-41ab-b895-3a75a850aa94" containerName="extract" Oct 30 20:44:16.892431 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:16.892354 4611 memory_manager.go:365] "RemoveStaleState removing state" podUID="dbecf2f9-1c2d-41ae-9add-c603d15532fa" containerName="extract" Oct 30 20:44:16.924954 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:16.924925 4611 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-9ffc9ff9f-g8bb7" Oct 30 20:44:16.942628 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:16.942601 4611 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-9ffc9ff9f-g8bb7"] Oct 30 20:44:16.960650 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:16.960576 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:44:16.982797 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:16.982774 4611 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\"" Oct 30 20:44:16.982797 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:16.982783 4611 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\"" Oct 30 20:44:16.982939 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:16.982812 4611 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\"" Oct 30 20:44:16.982978 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:16.982939 4611 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-hv2f2\"" Oct 30 20:44:16.983574 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:16.983557 4611 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\"" Oct 30 20:44:16.983623 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:16.983563 4611 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\"" Oct 30 20:44:16.998914 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:16.998889 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c9d2df48-962a-42e0-bc2a-189ed60f9a3d-tmp\") pod \"opendatahub-operator-controller-manager-9ffc9ff9f-g8bb7\" (UID: \"c9d2df48-962a-42e0-bc2a-189ed60f9a3d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-9ffc9ff9f-g8bb7" Oct 30 20:44:16.998987 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:16.998928 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/c9d2df48-962a-42e0-bc2a-189ed60f9a3d-manifests\") pod \"opendatahub-operator-controller-manager-9ffc9ff9f-g8bb7\" (UID: \"c9d2df48-962a-42e0-bc2a-189ed60f9a3d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-9ffc9ff9f-g8bb7" Oct 30 20:44:16.998987 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:16.998956 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/c9d2df48-962a-42e0-bc2a-189ed60f9a3d-metrics-cert\") pod \"opendatahub-operator-controller-manager-9ffc9ff9f-g8bb7\" (UID: \"c9d2df48-962a-42e0-bc2a-189ed60f9a3d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-9ffc9ff9f-g8bb7" Oct 30 20:44:16.999065 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:16.999056 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2vwbn\" (UniqueName: \"kubernetes.io/projected/c9d2df48-962a-42e0-bc2a-189ed60f9a3d-kube-api-access-2vwbn\") pod \"opendatahub-operator-controller-manager-9ffc9ff9f-g8bb7\" (UID: \"c9d2df48-962a-42e0-bc2a-189ed60f9a3d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-9ffc9ff9f-g8bb7" Oct 30 20:44:16.999109 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:16.999082 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c9d2df48-962a-42e0-bc2a-189ed60f9a3d-webhook-cert\") pod \"opendatahub-operator-controller-manager-9ffc9ff9f-g8bb7\" (UID: \"c9d2df48-962a-42e0-bc2a-189ed60f9a3d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-9ffc9ff9f-g8bb7" Oct 30 20:44:16.999109 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:16.999102 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/c9d2df48-962a-42e0-bc2a-189ed60f9a3d-apiservice-cert\") pod \"opendatahub-operator-controller-manager-9ffc9ff9f-g8bb7\" (UID: \"c9d2df48-962a-42e0-bc2a-189ed60f9a3d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-9ffc9ff9f-g8bb7" Oct 30 20:44:17.099832 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:17.099803 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/c9d2df48-962a-42e0-bc2a-189ed60f9a3d-metrics-cert\") pod \"opendatahub-operator-controller-manager-9ffc9ff9f-g8bb7\" (UID: \"c9d2df48-962a-42e0-bc2a-189ed60f9a3d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-9ffc9ff9f-g8bb7" Oct 30 20:44:17.100215 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:17.099860 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2vwbn\" (UniqueName: \"kubernetes.io/projected/c9d2df48-962a-42e0-bc2a-189ed60f9a3d-kube-api-access-2vwbn\") pod \"opendatahub-operator-controller-manager-9ffc9ff9f-g8bb7\" (UID: \"c9d2df48-962a-42e0-bc2a-189ed60f9a3d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-9ffc9ff9f-g8bb7" Oct 30 20:44:17.100215 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:17.099896 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c9d2df48-962a-42e0-bc2a-189ed60f9a3d-webhook-cert\") pod \"opendatahub-operator-controller-manager-9ffc9ff9f-g8bb7\" (UID: \"c9d2df48-962a-42e0-bc2a-189ed60f9a3d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-9ffc9ff9f-g8bb7" Oct 30 20:44:17.100215 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:17.099928 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/c9d2df48-962a-42e0-bc2a-189ed60f9a3d-apiservice-cert\") pod \"opendatahub-operator-controller-manager-9ffc9ff9f-g8bb7\" (UID: \"c9d2df48-962a-42e0-bc2a-189ed60f9a3d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-9ffc9ff9f-g8bb7" Oct 30 20:44:17.100215 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:17.099959 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c9d2df48-962a-42e0-bc2a-189ed60f9a3d-tmp\") pod \"opendatahub-operator-controller-manager-9ffc9ff9f-g8bb7\" (UID: \"c9d2df48-962a-42e0-bc2a-189ed60f9a3d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-9ffc9ff9f-g8bb7" Oct 30 20:44:17.100215 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:17.100006 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/c9d2df48-962a-42e0-bc2a-189ed60f9a3d-manifests\") pod \"opendatahub-operator-controller-manager-9ffc9ff9f-g8bb7\" (UID: \"c9d2df48-962a-42e0-bc2a-189ed60f9a3d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-9ffc9ff9f-g8bb7" Oct 30 20:44:17.100497 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:17.100392 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/c9d2df48-962a-42e0-bc2a-189ed60f9a3d-manifests\") pod \"opendatahub-operator-controller-manager-9ffc9ff9f-g8bb7\" (UID: \"c9d2df48-962a-42e0-bc2a-189ed60f9a3d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-9ffc9ff9f-g8bb7" Oct 30 20:44:17.102117 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:17.102097 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c9d2df48-962a-42e0-bc2a-189ed60f9a3d-tmp\") pod \"opendatahub-operator-controller-manager-9ffc9ff9f-g8bb7\" (UID: \"c9d2df48-962a-42e0-bc2a-189ed60f9a3d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-9ffc9ff9f-g8bb7" Oct 30 20:44:17.102386 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:17.102366 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c9d2df48-962a-42e0-bc2a-189ed60f9a3d-webhook-cert\") pod \"opendatahub-operator-controller-manager-9ffc9ff9f-g8bb7\" (UID: \"c9d2df48-962a-42e0-bc2a-189ed60f9a3d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-9ffc9ff9f-g8bb7" Oct 30 20:44:17.102621 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:17.102603 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/c9d2df48-962a-42e0-bc2a-189ed60f9a3d-metrics-cert\") pod \"opendatahub-operator-controller-manager-9ffc9ff9f-g8bb7\" (UID: \"c9d2df48-962a-42e0-bc2a-189ed60f9a3d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-9ffc9ff9f-g8bb7" Oct 30 20:44:17.102766 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:17.102745 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/c9d2df48-962a-42e0-bc2a-189ed60f9a3d-apiservice-cert\") pod \"opendatahub-operator-controller-manager-9ffc9ff9f-g8bb7\" (UID: \"c9d2df48-962a-42e0-bc2a-189ed60f9a3d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-9ffc9ff9f-g8bb7" Oct 30 20:44:17.111803 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:17.111775 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2vwbn\" (UniqueName: \"kubernetes.io/projected/c9d2df48-962a-42e0-bc2a-189ed60f9a3d-kube-api-access-2vwbn\") pod \"opendatahub-operator-controller-manager-9ffc9ff9f-g8bb7\" (UID: \"c9d2df48-962a-42e0-bc2a-189ed60f9a3d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-9ffc9ff9f-g8bb7" Oct 30 20:44:17.130228 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:17.130202 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:44:17.267431 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:17.267374 4611 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-9ffc9ff9f-g8bb7" Oct 30 20:44:17.341138 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:17.341056 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:44:17.513072 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:17.513041 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:44:17.588948 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:17.588881 4611 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-9ffc9ff9f-g8bb7"] Oct 30 20:44:17.593706 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:44:17.593678 4611 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9d2df48_962a_42e0_bc2a_189ed60f9a3d.slice/crio-a74408b3dee81a83cda2094964d54883cadff704eeb2254beb65f1fe05de69f2 WatchSource:0}: Error finding container a74408b3dee81a83cda2094964d54883cadff704eeb2254beb65f1fe05de69f2: Status 404 returned error can't find the container with id a74408b3dee81a83cda2094964d54883cadff704eeb2254beb65f1fe05de69f2 Oct 30 20:44:17.595351 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:17.595330 4611 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 20:44:17.795839 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:17.795803 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-9ffc9ff9f-g8bb7" event={"ID":"c9d2df48-962a-42e0-bc2a-189ed60f9a3d","Type":"ContainerStarted","Data":"a74408b3dee81a83cda2094964d54883cadff704eeb2254beb65f1fe05de69f2"} Oct 30 20:44:19.613130 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:19.613039 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:44:19.764499 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:19.764464 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:44:21.817107 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:21.817028 4611 generic.go:356] "Generic (PLEG): container finished" podID="c9d2df48-962a-42e0-bc2a-189ed60f9a3d" containerID="3d55bdcdf3ee90352fd6244454b23ca5d8610dc653dc813073206054067bc917" exitCode=0 Oct 30 20:44:21.817525 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:21.817117 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-9ffc9ff9f-g8bb7" event={"ID":"c9d2df48-962a-42e0-bc2a-189ed60f9a3d","Type":"ContainerDied","Data":"3d55bdcdf3ee90352fd6244454b23ca5d8610dc653dc813073206054067bc917"} Oct 30 20:44:22.822822 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:22.822746 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-9ffc9ff9f-g8bb7" event={"ID":"c9d2df48-962a-42e0-bc2a-189ed60f9a3d","Type":"ContainerStarted","Data":"e072c312e977d3c515a34baade5fd2607d22fab6bb8e3d65d21ad5543a1243e3"} Oct 30 20:44:22.823175 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:22.822872 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-9ffc9ff9f-g8bb7" Oct 30 20:44:22.864462 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:22.864420 4611 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-9ffc9ff9f-g8bb7" podStartSLOduration=2.458917423 podStartE2EDuration="6.864405709s" totalImagesPullingTime="3.941353753s" totalInitContainerRuntime="464.134533ms" isStatefulPod=false podCreationTimestamp="2026-10-30 20:44:16 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:44:22.862444828 +0000 UTC m=+317.396471444" watchObservedRunningTime="2026-10-30 20:44:22.864405709 +0000 UTC m=+317.398432325" Oct 30 20:44:23.945829 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:23.945794 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:44:24.086442 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:24.086402 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:44:27.409636 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:27.409601 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:44:29.454693 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:29.454659 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:44:31.605198 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:31.605167 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:44:31.747105 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:31.747077 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:44:33.180976 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:33.180933 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:44:33.383213 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:33.383181 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:44:33.829894 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:33.829862 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-9ffc9ff9f-g8bb7" Oct 30 20:44:33.918556 ip-10-0-133-192 kubenswrapper[4611]: E1030 20:44:33.918523 4611 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hwzc6" containerName="extractor" Oct 30 20:44:34.784719 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:34.784685 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:44:35.135878 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:35.135847 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:44:35.223205 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:35.223176 4611 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-smdl6"] Oct 30 20:44:35.227027 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:35.227008 4611 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-smdl6" Oct 30 20:44:35.239315 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:35.239290 4611 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Oct 30 20:44:35.239435 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:35.239419 4611 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-2gwh7\"" Oct 30 20:44:35.239486 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:35.239434 4611 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Oct 30 20:44:35.243376 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:35.243355 4611 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-smdl6"] Oct 30 20:44:35.342919 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:35.342884 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fpjt8\" (UniqueName: \"kubernetes.io/projected/31d263b9-b3b3-4db4-9026-da84d454dc34-kube-api-access-fpjt8\") pod \"cert-manager-webhook-5dbbbf75cc-smdl6\" (UID: \"31d263b9-b3b3-4db4-9026-da84d454dc34\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-smdl6" Oct 30 20:44:35.343076 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:35.342936 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/31d263b9-b3b3-4db4-9026-da84d454dc34-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-smdl6\" (UID: \"31d263b9-b3b3-4db4-9026-da84d454dc34\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-smdl6" Oct 30 20:44:35.444270 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:35.444190 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fpjt8\" (UniqueName: \"kubernetes.io/projected/31d263b9-b3b3-4db4-9026-da84d454dc34-kube-api-access-fpjt8\") pod \"cert-manager-webhook-5dbbbf75cc-smdl6\" (UID: \"31d263b9-b3b3-4db4-9026-da84d454dc34\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-smdl6" Oct 30 20:44:35.444270 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:35.444232 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/31d263b9-b3b3-4db4-9026-da84d454dc34-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-smdl6\" (UID: \"31d263b9-b3b3-4db4-9026-da84d454dc34\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-smdl6" Oct 30 20:44:35.462430 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:35.462398 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/31d263b9-b3b3-4db4-9026-da84d454dc34-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-smdl6\" (UID: \"31d263b9-b3b3-4db4-9026-da84d454dc34\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-smdl6" Oct 30 20:44:35.467541 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:35.467497 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fpjt8\" (UniqueName: \"kubernetes.io/projected/31d263b9-b3b3-4db4-9026-da84d454dc34-kube-api-access-fpjt8\") pod \"cert-manager-webhook-5dbbbf75cc-smdl6\" (UID: \"31d263b9-b3b3-4db4-9026-da84d454dc34\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-smdl6" Oct 30 20:44:35.549282 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:35.549253 4611 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-smdl6" Oct 30 20:44:35.833077 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:35.832999 4611 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-smdl6"] Oct 30 20:44:35.837719 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:44:35.837692 4611 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod31d263b9_b3b3_4db4_9026_da84d454dc34.slice/crio-0f31ad8247a3bd9ec700b253626d3efe7b7ffad05bc273d7f005825d965a5c12 WatchSource:0}: Error finding container 0f31ad8247a3bd9ec700b253626d3efe7b7ffad05bc273d7f005825d965a5c12: Status 404 returned error can't find the container with id 0f31ad8247a3bd9ec700b253626d3efe7b7ffad05bc273d7f005825d965a5c12 Oct 30 20:44:35.874922 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:35.874893 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-smdl6" event={"ID":"31d263b9-b3b3-4db4-9026-da84d454dc34","Type":"ContainerStarted","Data":"0f31ad8247a3bd9ec700b253626d3efe7b7ffad05bc273d7f005825d965a5c12"} Oct 30 20:44:37.230882 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:37.230808 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:44:37.405392 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:37.405357 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:44:37.489264 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:37.489182 4611 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-w497w"] Oct 30 20:44:37.495459 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:37.495432 4611 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-w497w" Oct 30 20:44:37.516684 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:37.516652 4611 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-4jx76\"" Oct 30 20:44:37.517649 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:37.517626 4611 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-w497w"] Oct 30 20:44:37.559929 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:37.559895 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b2a92b5c-d02a-4ef0-981d-517e0a6cb7c3-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-w497w\" (UID: \"b2a92b5c-d02a-4ef0-981d-517e0a6cb7c3\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-w497w" Oct 30 20:44:37.560068 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:37.559935 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7nc95\" (UniqueName: \"kubernetes.io/projected/b2a92b5c-d02a-4ef0-981d-517e0a6cb7c3-kube-api-access-7nc95\") pod \"cert-manager-cainjector-5c57bbd4fc-w497w\" (UID: \"b2a92b5c-d02a-4ef0-981d-517e0a6cb7c3\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-w497w" Oct 30 20:44:37.660868 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:37.660837 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b2a92b5c-d02a-4ef0-981d-517e0a6cb7c3-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-w497w\" (UID: \"b2a92b5c-d02a-4ef0-981d-517e0a6cb7c3\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-w497w" Oct 30 20:44:37.660868 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:37.660866 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7nc95\" (UniqueName: \"kubernetes.io/projected/b2a92b5c-d02a-4ef0-981d-517e0a6cb7c3-kube-api-access-7nc95\") pod \"cert-manager-cainjector-5c57bbd4fc-w497w\" (UID: \"b2a92b5c-d02a-4ef0-981d-517e0a6cb7c3\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-w497w" Oct 30 20:44:37.678689 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:37.678662 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7nc95\" (UniqueName: \"kubernetes.io/projected/b2a92b5c-d02a-4ef0-981d-517e0a6cb7c3-kube-api-access-7nc95\") pod \"cert-manager-cainjector-5c57bbd4fc-w497w\" (UID: \"b2a92b5c-d02a-4ef0-981d-517e0a6cb7c3\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-w497w" Oct 30 20:44:37.678889 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:37.678866 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b2a92b5c-d02a-4ef0-981d-517e0a6cb7c3-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-w497w\" (UID: \"b2a92b5c-d02a-4ef0-981d-517e0a6cb7c3\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-w497w" Oct 30 20:44:37.809751 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:37.809682 4611 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-w497w" Oct 30 20:44:38.098346 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:38.098247 4611 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-w497w"] Oct 30 20:44:38.101874 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:44:38.101843 4611 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb2a92b5c_d02a_4ef0_981d_517e0a6cb7c3.slice/crio-7417085ae5a770df7202acb26acf78b8fe36932daeb868820f0781bb117725c9 WatchSource:0}: Error finding container 7417085ae5a770df7202acb26acf78b8fe36932daeb868820f0781bb117725c9: Status 404 returned error can't find the container with id 7417085ae5a770df7202acb26acf78b8fe36932daeb868820f0781bb117725c9 Oct 30 20:44:38.893454 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:38.893411 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-w497w" event={"ID":"b2a92b5c-d02a-4ef0-981d-517e0a6cb7c3","Type":"ContainerStarted","Data":"7417085ae5a770df7202acb26acf78b8fe36932daeb868820f0781bb117725c9"} Oct 30 20:44:39.898971 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:39.898928 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-smdl6" event={"ID":"31d263b9-b3b3-4db4-9026-da84d454dc34","Type":"ContainerStarted","Data":"e68bc1ba665cb9a74e753f1f2e25046e06d0f7f52600b5018596a8f4cb1392ea"} Oct 30 20:44:39.899334 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:39.899001 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-smdl6" Oct 30 20:44:39.923569 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:39.923525 4611 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-smdl6" podStartSLOduration=1.605152859 podStartE2EDuration="4.923497723s" totalImagesPullingTime="3.318344864s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:44:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:44:39.923260842 +0000 UTC m=+334.457287459" watchObservedRunningTime="2026-10-30 20:44:39.923497723 +0000 UTC m=+334.457524339" Oct 30 20:44:40.904540 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:40.904489 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-w497w" event={"ID":"b2a92b5c-d02a-4ef0-981d-517e0a6cb7c3","Type":"ContainerStarted","Data":"47e4f776575fa1599fff8583f5584ae5502bb5e6c933339404b04e33438e6860"} Oct 30 20:44:40.921247 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:40.921198 4611 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-w497w" podStartSLOduration=2.106050959 podStartE2EDuration="3.921187016s" totalImagesPullingTime="1.815136057s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:44:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:44:40.920805916 +0000 UTC m=+335.454832531" watchObservedRunningTime="2026-10-30 20:44:40.921187016 +0000 UTC m=+335.455213632" Oct 30 20:44:40.976188 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:40.976148 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:44:42.035904 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:42.035860 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:44:42.230401 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:42.230371 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:44:42.390005 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:42.389964 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:44:43.217156 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:43.217120 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:44:44.740308 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:44.740279 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:44:45.906714 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:45.906684 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-smdl6" Oct 30 20:44:47.750700 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:47.750670 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:44:47.899704 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:47.899667 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:44:48.046821 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:48.046733 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:44:48.983337 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:48.983298 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:44:49.776807 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:49.776723 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:44:52.027140 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:52.027103 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:44:53.637666 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:53.637622 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:44:54.037269 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:54.037180 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:44:54.463544 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:54.463503 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:44:54.751375 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:54.751295 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:44:54.751747 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:54.751385 4611 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-6c6fcc99c4-flnqs"] Oct 30 20:44:54.758901 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:54.758879 4611 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-6c6fcc99c4-flnqs" Oct 30 20:44:54.775943 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:54.775924 4611 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Oct 30 20:44:54.776072 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:54.775960 4611 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-7wl7n\"" Oct 30 20:44:54.776072 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:54.776026 4611 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Oct 30 20:44:54.788057 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:54.788036 4611 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-6c6fcc99c4-flnqs"] Oct 30 20:44:54.801175 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:54.801154 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2hxbx\" (UniqueName: \"kubernetes.io/projected/022d8287-926f-4445-8931-98ab50429582-kube-api-access-2hxbx\") pod \"kube-auth-proxy-6c6fcc99c4-flnqs\" (UID: \"022d8287-926f-4445-8931-98ab50429582\") " pod="openshift-ingress/kube-auth-proxy-6c6fcc99c4-flnqs" Oct 30 20:44:54.801295 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:54.801195 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/022d8287-926f-4445-8931-98ab50429582-tls-certs\") pod \"kube-auth-proxy-6c6fcc99c4-flnqs\" (UID: \"022d8287-926f-4445-8931-98ab50429582\") " pod="openshift-ingress/kube-auth-proxy-6c6fcc99c4-flnqs" Oct 30 20:44:54.801295 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:54.801231 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/022d8287-926f-4445-8931-98ab50429582-tmp\") pod \"kube-auth-proxy-6c6fcc99c4-flnqs\" (UID: \"022d8287-926f-4445-8931-98ab50429582\") " pod="openshift-ingress/kube-auth-proxy-6c6fcc99c4-flnqs" Oct 30 20:44:54.902702 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:54.902676 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/022d8287-926f-4445-8931-98ab50429582-tls-certs\") pod \"kube-auth-proxy-6c6fcc99c4-flnqs\" (UID: \"022d8287-926f-4445-8931-98ab50429582\") " pod="openshift-ingress/kube-auth-proxy-6c6fcc99c4-flnqs" Oct 30 20:44:54.902813 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:54.902728 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/022d8287-926f-4445-8931-98ab50429582-tmp\") pod \"kube-auth-proxy-6c6fcc99c4-flnqs\" (UID: \"022d8287-926f-4445-8931-98ab50429582\") " pod="openshift-ingress/kube-auth-proxy-6c6fcc99c4-flnqs" Oct 30 20:44:54.902813 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:54.902787 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2hxbx\" (UniqueName: \"kubernetes.io/projected/022d8287-926f-4445-8931-98ab50429582-kube-api-access-2hxbx\") pod \"kube-auth-proxy-6c6fcc99c4-flnqs\" (UID: \"022d8287-926f-4445-8931-98ab50429582\") " pod="openshift-ingress/kube-auth-proxy-6c6fcc99c4-flnqs" Oct 30 20:44:54.905043 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:54.905006 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/022d8287-926f-4445-8931-98ab50429582-tmp\") pod \"kube-auth-proxy-6c6fcc99c4-flnqs\" (UID: \"022d8287-926f-4445-8931-98ab50429582\") " pod="openshift-ingress/kube-auth-proxy-6c6fcc99c4-flnqs" Oct 30 20:44:54.905256 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:54.905238 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/022d8287-926f-4445-8931-98ab50429582-tls-certs\") pod \"kube-auth-proxy-6c6fcc99c4-flnqs\" (UID: \"022d8287-926f-4445-8931-98ab50429582\") " pod="openshift-ingress/kube-auth-proxy-6c6fcc99c4-flnqs" Oct 30 20:44:54.927386 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:54.927363 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2hxbx\" (UniqueName: \"kubernetes.io/projected/022d8287-926f-4445-8931-98ab50429582-kube-api-access-2hxbx\") pod \"kube-auth-proxy-6c6fcc99c4-flnqs\" (UID: \"022d8287-926f-4445-8931-98ab50429582\") " pod="openshift-ingress/kube-auth-proxy-6c6fcc99c4-flnqs" Oct 30 20:44:54.995041 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:54.995015 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:44:55.070850 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:55.070790 4611 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-6c6fcc99c4-flnqs" Oct 30 20:44:55.212998 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:55.212970 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:44:55.370127 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:55.370091 4611 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-6c6fcc99c4-flnqs"] Oct 30 20:44:55.372898 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:44:55.372873 4611 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod022d8287_926f_4445_8931_98ab50429582.slice/crio-179878767d46dffbad4737dace3f8d96e6c9fe91a63dae193f2c59d5a272eced WatchSource:0}: Error finding container 179878767d46dffbad4737dace3f8d96e6c9fe91a63dae193f2c59d5a272eced: Status 404 returned error can't find the container with id 179878767d46dffbad4737dace3f8d96e6c9fe91a63dae193f2c59d5a272eced Oct 30 20:44:55.411924 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:55.411894 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:44:55.967631 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:55.967597 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-6c6fcc99c4-flnqs" event={"ID":"022d8287-926f-4445-8931-98ab50429582","Type":"ContainerStarted","Data":"179878767d46dffbad4737dace3f8d96e6c9fe91a63dae193f2c59d5a272eced"} Oct 30 20:44:56.313210 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:56.313134 4611 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-zmvxk"] Oct 30 20:44:56.316849 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:56.316820 4611 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-zmvxk" Oct 30 20:44:56.328978 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:56.328952 4611 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-zmvxk"] Oct 30 20:44:56.395734 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:56.395693 4611 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Oct 30 20:44:56.395879 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:56.395828 4611 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Oct 30 20:44:56.395954 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:56.395880 4611 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Oct 30 20:44:56.414865 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:56.414842 4611 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-fc57p\"" Oct 30 20:44:56.417169 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:56.417146 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mqxg5\" (UniqueName: \"kubernetes.io/projected/2755a243-2222-407a-bf25-a13b54bcc5f4-kube-api-access-mqxg5\") pod \"dns-operator-controller-manager-8557cbdb84-zmvxk\" (UID: \"2755a243-2222-407a-bf25-a13b54bcc5f4\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-zmvxk" Oct 30 20:44:56.517766 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:56.517737 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mqxg5\" (UniqueName: \"kubernetes.io/projected/2755a243-2222-407a-bf25-a13b54bcc5f4-kube-api-access-mqxg5\") pod \"dns-operator-controller-manager-8557cbdb84-zmvxk\" (UID: \"2755a243-2222-407a-bf25-a13b54bcc5f4\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-zmvxk" Oct 30 20:44:56.561466 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:56.561435 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mqxg5\" (UniqueName: \"kubernetes.io/projected/2755a243-2222-407a-bf25-a13b54bcc5f4-kube-api-access-mqxg5\") pod \"dns-operator-controller-manager-8557cbdb84-zmvxk\" (UID: \"2755a243-2222-407a-bf25-a13b54bcc5f4\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-zmvxk" Oct 30 20:44:56.633105 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:56.633073 4611 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-zmvxk" Oct 30 20:44:56.700548 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:56.699979 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:44:56.928845 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:56.928775 4611 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-zmvxk"] Oct 30 20:44:56.932880 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:44:56.932846 4611 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2755a243_2222_407a_bf25_a13b54bcc5f4.slice/crio-7f33f91d03af718052c8bc65e9c533473b9b97c6694d356058716ce8ec21d07b WatchSource:0}: Error finding container 7f33f91d03af718052c8bc65e9c533473b9b97c6694d356058716ce8ec21d07b: Status 404 returned error can't find the container with id 7f33f91d03af718052c8bc65e9c533473b9b97c6694d356058716ce8ec21d07b Oct 30 20:44:56.956649 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:56.956622 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:44:56.974110 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:56.974084 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-zmvxk" event={"ID":"2755a243-2222-407a-bf25-a13b54bcc5f4","Type":"ContainerStarted","Data":"7f33f91d03af718052c8bc65e9c533473b9b97c6694d356058716ce8ec21d07b"} Oct 30 20:44:57.007189 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:57.007166 4611 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-vp92r"] Oct 30 20:44:57.014269 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:57.014251 4611 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-vp92r" Oct 30 20:44:57.023176 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:57.023152 4611 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-vp92r"] Oct 30 20:44:57.023371 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:57.023353 4611 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Oct 30 20:44:57.023500 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:57.023486 4611 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Oct 30 20:44:57.023611 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:57.023580 4611 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-thfd4\"" Oct 30 20:44:57.122445 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:57.122410 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:44:57.128289 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:57.128259 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/860bdddb-14bc-4ba4-bf61-45ef8ffe5ed7-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-vp92r\" (UID: \"860bdddb-14bc-4ba4-bf61-45ef8ffe5ed7\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-vp92r" Oct 30 20:44:57.128425 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:57.128296 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r7ncw\" (UniqueName: \"kubernetes.io/projected/860bdddb-14bc-4ba4-bf61-45ef8ffe5ed7-kube-api-access-r7ncw\") pod \"kuadrant-console-plugin-66c4fb575d-vp92r\" (UID: \"860bdddb-14bc-4ba4-bf61-45ef8ffe5ed7\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-vp92r" Oct 30 20:44:57.128425 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:57.128403 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/860bdddb-14bc-4ba4-bf61-45ef8ffe5ed7-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-vp92r\" (UID: \"860bdddb-14bc-4ba4-bf61-45ef8ffe5ed7\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-vp92r" Oct 30 20:44:57.229480 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:57.229407 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/860bdddb-14bc-4ba4-bf61-45ef8ffe5ed7-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-vp92r\" (UID: \"860bdddb-14bc-4ba4-bf61-45ef8ffe5ed7\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-vp92r" Oct 30 20:44:57.229480 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:57.229447 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r7ncw\" (UniqueName: \"kubernetes.io/projected/860bdddb-14bc-4ba4-bf61-45ef8ffe5ed7-kube-api-access-r7ncw\") pod \"kuadrant-console-plugin-66c4fb575d-vp92r\" (UID: \"860bdddb-14bc-4ba4-bf61-45ef8ffe5ed7\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-vp92r" Oct 30 20:44:57.229680 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:57.229491 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/860bdddb-14bc-4ba4-bf61-45ef8ffe5ed7-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-vp92r\" (UID: \"860bdddb-14bc-4ba4-bf61-45ef8ffe5ed7\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-vp92r" Oct 30 20:44:57.230182 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:57.230156 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/860bdddb-14bc-4ba4-bf61-45ef8ffe5ed7-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-vp92r\" (UID: \"860bdddb-14bc-4ba4-bf61-45ef8ffe5ed7\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-vp92r" Oct 30 20:44:57.231935 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:57.231902 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/860bdddb-14bc-4ba4-bf61-45ef8ffe5ed7-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-vp92r\" (UID: \"860bdddb-14bc-4ba4-bf61-45ef8ffe5ed7\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-vp92r" Oct 30 20:44:57.245616 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:57.245593 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r7ncw\" (UniqueName: \"kubernetes.io/projected/860bdddb-14bc-4ba4-bf61-45ef8ffe5ed7-kube-api-access-r7ncw\") pod \"kuadrant-console-plugin-66c4fb575d-vp92r\" (UID: \"860bdddb-14bc-4ba4-bf61-45ef8ffe5ed7\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-vp92r" Oct 30 20:44:57.281881 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:57.281857 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:44:57.330358 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:57.330333 4611 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-vp92r" Oct 30 20:44:57.611603 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:57.611563 4611 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-vp92r"] Oct 30 20:44:57.614347 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:44:57.614322 4611 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod860bdddb_14bc_4ba4_bf61_45ef8ffe5ed7.slice/crio-d8006b8de63728a78c4b0dc805f37ab1e5f9760ccded16f56264ab5faa9c4d27 WatchSource:0}: Error finding container d8006b8de63728a78c4b0dc805f37ab1e5f9760ccded16f56264ab5faa9c4d27: Status 404 returned error can't find the container with id d8006b8de63728a78c4b0dc805f37ab1e5f9760ccded16f56264ab5faa9c4d27 Oct 30 20:44:57.980177 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:57.980082 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-vp92r" event={"ID":"860bdddb-14bc-4ba4-bf61-45ef8ffe5ed7","Type":"ContainerStarted","Data":"d8006b8de63728a78c4b0dc805f37ab1e5f9760ccded16f56264ab5faa9c4d27"} Oct 30 20:44:57.982234 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:57.982202 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-6c6fcc99c4-flnqs" event={"ID":"022d8287-926f-4445-8931-98ab50429582","Type":"ContainerStarted","Data":"a0ac42752d30c6a698ee137e1679bb78c9f74093915a83f7e3ed3ffa8dc7f1b3"} Oct 30 20:44:58.005577 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:58.005501 4611 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-6c6fcc99c4-flnqs" podStartSLOduration=1.755175919 podStartE2EDuration="4.005487295s" totalImagesPullingTime="2.250311376s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:44:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:44:58.004701219 +0000 UTC m=+352.538727834" watchObservedRunningTime="2026-10-30 20:44:58.005487295 +0000 UTC m=+352.539513913" Oct 30 20:44:58.298973 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:58.298794 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:44:58.530297 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:58.530243 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:44:59.088971 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:44:59.088935 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:45:00.999698 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:45:00.999648 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-zmvxk" event={"ID":"2755a243-2222-407a-bf25-a13b54bcc5f4","Type":"ContainerStarted","Data":"d9773722d32479cb0c6aae75e025aa7ee957eafeee472027b259aa12e80143d5"} Oct 30 20:45:01.000138 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:45:00.999731 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-zmvxk" Oct 30 20:45:01.019014 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:45:01.018968 4611 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-zmvxk" podStartSLOduration=1.699195475 podStartE2EDuration="5.01895394s" totalImagesPullingTime="3.319758465s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:44:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:45:01.017669698 +0000 UTC m=+355.551696314" watchObservedRunningTime="2026-10-30 20:45:01.01895394 +0000 UTC m=+355.552980555" Oct 30 20:45:02.191117 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:45:02.191081 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:45:02.335303 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:45:02.335269 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:45:02.576581 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:45:02.576544 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:45:02.726732 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:45:02.726654 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:45:02.981415 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:45:02.981337 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:45:03.010359 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:45:03.010326 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-vp92r" event={"ID":"860bdddb-14bc-4ba4-bf61-45ef8ffe5ed7","Type":"ContainerStarted","Data":"4f73e68a0933e6797ff7d7afca11eef033203061961ac255fec9601d67a9a82a"} Oct 30 20:45:03.125156 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:45:03.125118 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:45:03.915142 ip-10-0-133-192 kubenswrapper[4611]: E1030 20:45:03.915105 4611 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hwzc6" containerName="extractor" Oct 30 20:45:07.569654 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:45:07.569617 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:45:11.782005 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:45:11.781964 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:45:11.982908 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:45:11.982879 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:45:12.007631 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:45:12.007611 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-zmvxk" Oct 30 20:45:12.031194 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:45:12.031153 4611 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-vp92r" podStartSLOduration=11.253100645 podStartE2EDuration="16.03114059s" totalImagesPullingTime="4.778039945s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:44:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:45:03.027406401 +0000 UTC m=+357.561433018" watchObservedRunningTime="2026-10-30 20:45:12.03114059 +0000 UTC m=+366.565167207" Oct 30 20:45:12.427403 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:45:12.427361 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:45:12.650927 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:45:12.650890 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:45:16.241867 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:45:16.241833 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:45:16.400773 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:45:16.400739 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:45:16.947468 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:45:16.947427 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:45:17.192820 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:45:17.192787 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:45:17.632923 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:45:17.632889 4611 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-57f7f8659f-5rnr8"] Oct 30 20:45:17.638091 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:45:17.638069 4611 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-5rnr8" Oct 30 20:45:17.644178 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:45:17.644153 4611 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-6n5tv\"" Oct 30 20:45:17.645460 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:45:17.645434 4611 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-5rnr8"] Oct 30 20:45:17.738804 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:45:17.738776 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:45:17.792822 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:45:17.792798 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g2dsc\" (UniqueName: \"kubernetes.io/projected/ed4f2d62-25d9-44b0-9227-6003f3bb34b3-kube-api-access-g2dsc\") pod \"authorino-57f7f8659f-5rnr8\" (UID: \"ed4f2d62-25d9-44b0-9227-6003f3bb34b3\") " pod="kuadrant-system/authorino-57f7f8659f-5rnr8" Oct 30 20:45:17.892846 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:45:17.892771 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:45:17.893130 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:45:17.893114 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g2dsc\" (UniqueName: \"kubernetes.io/projected/ed4f2d62-25d9-44b0-9227-6003f3bb34b3-kube-api-access-g2dsc\") pod \"authorino-57f7f8659f-5rnr8\" (UID: \"ed4f2d62-25d9-44b0-9227-6003f3bb34b3\") " pod="kuadrant-system/authorino-57f7f8659f-5rnr8" Oct 30 20:45:17.906946 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:45:17.906915 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g2dsc\" (UniqueName: \"kubernetes.io/projected/ed4f2d62-25d9-44b0-9227-6003f3bb34b3-kube-api-access-g2dsc\") pod \"authorino-57f7f8659f-5rnr8\" (UID: \"ed4f2d62-25d9-44b0-9227-6003f3bb34b3\") " pod="kuadrant-system/authorino-57f7f8659f-5rnr8" Oct 30 20:45:17.948789 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:45:17.948740 4611 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-5rnr8" Oct 30 20:45:18.055608 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:45:18.055577 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:45:18.225440 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:45:18.225373 4611 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-5rnr8"] Oct 30 20:45:18.228050 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:45:18.228024 4611 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poded4f2d62_25d9_44b0_9227_6003f3bb34b3.slice/crio-fa50be0f407bc7698447b886dc67c17b4a6b858dba18dd78c21e82807891deae WatchSource:0}: Error finding container fa50be0f407bc7698447b886dc67c17b4a6b858dba18dd78c21e82807891deae: Status 404 returned error can't find the container with id fa50be0f407bc7698447b886dc67c17b4a6b858dba18dd78c21e82807891deae Oct 30 20:45:18.648785 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:45:18.648755 4611 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-6594f6d9bf-rkj4c"] Oct 30 20:45:18.652584 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:45:18.652552 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:45:18.652745 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:45:18.652730 4611 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-6594f6d9bf-rkj4c" Oct 30 20:45:18.660205 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:45:18.660187 4611 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"openshift-service-ca.crt\"" Oct 30 20:45:18.660383 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:45:18.660367 4611 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-controller-manager-dockercfg-m84c7\"" Oct 30 20:45:18.660438 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:45:18.660378 4611 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"kube-root-ca.crt\"" Oct 30 20:45:18.660438 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:45:18.660370 4611 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"webhook-server-cert\"" Oct 30 20:45:18.660636 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:45:18.660619 4611 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"jobset-manager-config\"" Oct 30 20:45:18.660740 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:45:18.660728 4611 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"metrics-server-cert\"" Oct 30 20:45:18.668892 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:45:18.668872 4611 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-6594f6d9bf-rkj4c"] Oct 30 20:45:18.801183 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:45:18.801157 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9dcf2bde-35e6-4e2d-b3d1-dc1e07def6f0-metrics-certs\") pod \"jobset-controller-manager-6594f6d9bf-rkj4c\" (UID: \"9dcf2bde-35e6-4e2d-b3d1-dc1e07def6f0\") " pod="openshift-jobset-operator/jobset-controller-manager-6594f6d9bf-rkj4c" Oct 30 20:45:18.801300 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:45:18.801216 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9dcf2bde-35e6-4e2d-b3d1-dc1e07def6f0-cert\") pod \"jobset-controller-manager-6594f6d9bf-rkj4c\" (UID: \"9dcf2bde-35e6-4e2d-b3d1-dc1e07def6f0\") " pod="openshift-jobset-operator/jobset-controller-manager-6594f6d9bf-rkj4c" Oct 30 20:45:18.801346 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:45:18.801295 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ldrhh\" (UniqueName: \"kubernetes.io/projected/9dcf2bde-35e6-4e2d-b3d1-dc1e07def6f0-kube-api-access-ldrhh\") pod \"jobset-controller-manager-6594f6d9bf-rkj4c\" (UID: \"9dcf2bde-35e6-4e2d-b3d1-dc1e07def6f0\") " pod="openshift-jobset-operator/jobset-controller-manager-6594f6d9bf-rkj4c" Oct 30 20:45:18.801420 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:45:18.801402 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/9dcf2bde-35e6-4e2d-b3d1-dc1e07def6f0-manager-config\") pod \"jobset-controller-manager-6594f6d9bf-rkj4c\" (UID: \"9dcf2bde-35e6-4e2d-b3d1-dc1e07def6f0\") " pod="openshift-jobset-operator/jobset-controller-manager-6594f6d9bf-rkj4c" Oct 30 20:45:18.902257 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:45:18.902199 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9dcf2bde-35e6-4e2d-b3d1-dc1e07def6f0-cert\") pod \"jobset-controller-manager-6594f6d9bf-rkj4c\" (UID: \"9dcf2bde-35e6-4e2d-b3d1-dc1e07def6f0\") " pod="openshift-jobset-operator/jobset-controller-manager-6594f6d9bf-rkj4c" Oct 30 20:45:18.902257 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:45:18.902227 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ldrhh\" (UniqueName: \"kubernetes.io/projected/9dcf2bde-35e6-4e2d-b3d1-dc1e07def6f0-kube-api-access-ldrhh\") pod \"jobset-controller-manager-6594f6d9bf-rkj4c\" (UID: \"9dcf2bde-35e6-4e2d-b3d1-dc1e07def6f0\") " pod="openshift-jobset-operator/jobset-controller-manager-6594f6d9bf-rkj4c" Oct 30 20:45:18.902397 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:45:18.902273 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/9dcf2bde-35e6-4e2d-b3d1-dc1e07def6f0-manager-config\") pod \"jobset-controller-manager-6594f6d9bf-rkj4c\" (UID: \"9dcf2bde-35e6-4e2d-b3d1-dc1e07def6f0\") " pod="openshift-jobset-operator/jobset-controller-manager-6594f6d9bf-rkj4c" Oct 30 20:45:18.902397 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:45:18.902303 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9dcf2bde-35e6-4e2d-b3d1-dc1e07def6f0-metrics-certs\") pod \"jobset-controller-manager-6594f6d9bf-rkj4c\" (UID: \"9dcf2bde-35e6-4e2d-b3d1-dc1e07def6f0\") " pod="openshift-jobset-operator/jobset-controller-manager-6594f6d9bf-rkj4c" Oct 30 20:45:18.903050 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:45:18.903022 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/9dcf2bde-35e6-4e2d-b3d1-dc1e07def6f0-manager-config\") pod \"jobset-controller-manager-6594f6d9bf-rkj4c\" (UID: \"9dcf2bde-35e6-4e2d-b3d1-dc1e07def6f0\") " pod="openshift-jobset-operator/jobset-controller-manager-6594f6d9bf-rkj4c" Oct 30 20:45:18.904797 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:45:18.904777 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9dcf2bde-35e6-4e2d-b3d1-dc1e07def6f0-cert\") pod \"jobset-controller-manager-6594f6d9bf-rkj4c\" (UID: \"9dcf2bde-35e6-4e2d-b3d1-dc1e07def6f0\") " pod="openshift-jobset-operator/jobset-controller-manager-6594f6d9bf-rkj4c" Oct 30 20:45:18.904990 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:45:18.904969 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9dcf2bde-35e6-4e2d-b3d1-dc1e07def6f0-metrics-certs\") pod \"jobset-controller-manager-6594f6d9bf-rkj4c\" (UID: \"9dcf2bde-35e6-4e2d-b3d1-dc1e07def6f0\") " pod="openshift-jobset-operator/jobset-controller-manager-6594f6d9bf-rkj4c" Oct 30 20:45:18.913959 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:45:18.913935 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ldrhh\" (UniqueName: \"kubernetes.io/projected/9dcf2bde-35e6-4e2d-b3d1-dc1e07def6f0-kube-api-access-ldrhh\") pod \"jobset-controller-manager-6594f6d9bf-rkj4c\" (UID: \"9dcf2bde-35e6-4e2d-b3d1-dc1e07def6f0\") " pod="openshift-jobset-operator/jobset-controller-manager-6594f6d9bf-rkj4c" Oct 30 20:45:18.965915 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:45:18.965891 4611 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-6594f6d9bf-rkj4c" Oct 30 20:45:19.082502 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:45:19.082464 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-5rnr8" event={"ID":"ed4f2d62-25d9-44b0-9227-6003f3bb34b3","Type":"ContainerStarted","Data":"fa50be0f407bc7698447b886dc67c17b4a6b858dba18dd78c21e82807891deae"} Oct 30 20:45:19.238631 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:45:19.238425 4611 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-6594f6d9bf-rkj4c"] Oct 30 20:45:19.241454 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:45:19.241419 4611 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9dcf2bde_35e6_4e2d_b3d1_dc1e07def6f0.slice/crio-9fd11f00c1e46dafd23ed07f1ae0c01919354295179e3977f8d433de3575ba41 WatchSource:0}: Error finding container 9fd11f00c1e46dafd23ed07f1ae0c01919354295179e3977f8d433de3575ba41: Status 404 returned error can't find the container with id 9fd11f00c1e46dafd23ed07f1ae0c01919354295179e3977f8d433de3575ba41 Oct 30 20:45:20.088768 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:45:20.088743 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-6594f6d9bf-rkj4c" event={"ID":"9dcf2bde-35e6-4e2d-b3d1-dc1e07def6f0","Type":"ContainerStarted","Data":"9fd11f00c1e46dafd23ed07f1ae0c01919354295179e3977f8d433de3575ba41"} Oct 30 20:45:20.202742 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:45:20.202698 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:45:20.354550 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:45:20.353387 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:45:21.096715 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:45:21.096676 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-5rnr8" event={"ID":"ed4f2d62-25d9-44b0-9227-6003f3bb34b3","Type":"ContainerStarted","Data":"41fbbcb07223bc95c88792b0cebca7508e237b48a426196432a1f15398db9926"} Oct 30 20:45:21.111198 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:45:21.111154 4611 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-57f7f8659f-5rnr8" podStartSLOduration=2.301710377 podStartE2EDuration="4.11113919s" totalImagesPullingTime="1.809428813s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:45:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:45:21.110796413 +0000 UTC m=+375.644823028" watchObservedRunningTime="2026-10-30 20:45:21.11113919 +0000 UTC m=+375.645165807" Oct 30 20:45:23.108071 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:45:23.108038 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-6594f6d9bf-rkj4c" event={"ID":"9dcf2bde-35e6-4e2d-b3d1-dc1e07def6f0","Type":"ContainerStarted","Data":"c60117a82bd77690bfb08f9ae2cc25cbdae8b326a925de728abbce46b9e43649"} Oct 30 20:45:23.108442 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:45:23.108117 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-jobset-operator/jobset-controller-manager-6594f6d9bf-rkj4c" Oct 30 20:45:23.130111 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:45:23.130063 4611 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-controller-manager-6594f6d9bf-rkj4c" podStartSLOduration=2.103734303 podStartE2EDuration="5.130049644s" totalImagesPullingTime="3.026315341s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:45:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:45:23.129050549 +0000 UTC m=+377.663077163" watchObservedRunningTime="2026-10-30 20:45:23.130049644 +0000 UTC m=+377.664076260" Oct 30 20:45:26.187129 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:45:26.187068 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:45:26.336320 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:45:26.336284 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:45:26.737516 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:45:26.737481 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:45:26.888798 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:45:26.888760 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:45:27.377707 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:45:27.377673 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:45:27.585216 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:45:27.585189 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:45:33.915570 ip-10-0-133-192 kubenswrapper[4611]: E1030 20:45:33.915529 4611 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hwzc6" containerName="extractor" Oct 30 20:45:34.118233 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:45:34.118208 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-jobset-operator/jobset-controller-manager-6594f6d9bf-rkj4c" Oct 30 20:45:35.241433 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:45:35.241394 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:45:35.461215 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:45:35.461179 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:45:35.913298 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:45:35.913267 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:45:36.052573 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:45:36.052543 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:45:37.640099 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:45:37.640062 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:45:37.780566 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:45:37.780532 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:45:39.257197 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:45:39.257157 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:45:42.458882 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:45:42.458827 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:45:42.588377 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:45:42.588346 4611 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-5rnr8"] Oct 30 20:45:42.588579 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:45:42.588542 4611 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-57f7f8659f-5rnr8" podUID="ed4f2d62-25d9-44b0-9227-6003f3bb34b3" containerName="authorino" containerID="cri-o://41fbbcb07223bc95c88792b0cebca7508e237b48a426196432a1f15398db9926" gracePeriod=30 Oct 30 20:45:42.646304 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:45:42.646277 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:45:42.828042 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:45:42.828013 4611 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-5rnr8" Oct 30 20:45:42.892524 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:45:42.892487 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed4f2d62-25d9-44b0-9227-6003f3bb34b3-kube-api-access-g2dsc\" (UniqueName: \"kubernetes.io/projected/ed4f2d62-25d9-44b0-9227-6003f3bb34b3-kube-api-access-g2dsc\") pod \"ed4f2d62-25d9-44b0-9227-6003f3bb34b3\" (UID: \"ed4f2d62-25d9-44b0-9227-6003f3bb34b3\") " Oct 30 20:45:42.894610 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:45:42.894583 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed4f2d62-25d9-44b0-9227-6003f3bb34b3-kube-api-access-g2dsc" pod "ed4f2d62-25d9-44b0-9227-6003f3bb34b3" (UID: "ed4f2d62-25d9-44b0-9227-6003f3bb34b3"). InnerVolumeSpecName "kube-api-access-g2dsc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:45:42.993585 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:45:42.993559 4611 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g2dsc\" (UniqueName: \"kubernetes.io/projected/ed4f2d62-25d9-44b0-9227-6003f3bb34b3-kube-api-access-g2dsc\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:45:43.196048 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:45:43.196021 4611 generic.go:356] "Generic (PLEG): container finished" podID="ed4f2d62-25d9-44b0-9227-6003f3bb34b3" containerID="41fbbcb07223bc95c88792b0cebca7508e237b48a426196432a1f15398db9926" exitCode=0 Oct 30 20:45:43.196193 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:45:43.196074 4611 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-5rnr8" Oct 30 20:45:43.196193 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:45:43.196081 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-5rnr8" event={"ID":"ed4f2d62-25d9-44b0-9227-6003f3bb34b3","Type":"ContainerDied","Data":"41fbbcb07223bc95c88792b0cebca7508e237b48a426196432a1f15398db9926"} Oct 30 20:45:43.196193 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:45:43.196111 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-5rnr8" event={"ID":"ed4f2d62-25d9-44b0-9227-6003f3bb34b3","Type":"ContainerDied","Data":"fa50be0f407bc7698447b886dc67c17b4a6b858dba18dd78c21e82807891deae"} Oct 30 20:45:43.196193 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:45:43.196126 4611 scope.go:122] "RemoveContainer" containerID="41fbbcb07223bc95c88792b0cebca7508e237b48a426196432a1f15398db9926" Oct 30 20:45:43.204020 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:45:43.204000 4611 scope.go:122] "RemoveContainer" containerID="41fbbcb07223bc95c88792b0cebca7508e237b48a426196432a1f15398db9926" Oct 30 20:45:43.204290 ip-10-0-133-192 kubenswrapper[4611]: E1030 20:45:43.204272 4611 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"41fbbcb07223bc95c88792b0cebca7508e237b48a426196432a1f15398db9926\": container with ID starting with 41fbbcb07223bc95c88792b0cebca7508e237b48a426196432a1f15398db9926 not found: ID does not exist" containerID="41fbbcb07223bc95c88792b0cebca7508e237b48a426196432a1f15398db9926" Oct 30 20:45:43.204382 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:45:43.204298 4611 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41fbbcb07223bc95c88792b0cebca7508e237b48a426196432a1f15398db9926"} err="failed to get container status \"41fbbcb07223bc95c88792b0cebca7508e237b48a426196432a1f15398db9926\": rpc error: code = NotFound desc = could not find container \"41fbbcb07223bc95c88792b0cebca7508e237b48a426196432a1f15398db9926\": container with ID starting with 41fbbcb07223bc95c88792b0cebca7508e237b48a426196432a1f15398db9926 not found: ID does not exist" Oct 30 20:45:43.219608 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:45:43.219589 4611 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-5rnr8"] Oct 30 20:45:43.223781 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:45:43.223757 4611 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-5rnr8"] Oct 30 20:45:43.603465 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:45:43.603435 4611 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ed4f2d62-25d9-44b0-9227-6003f3bb34b3" path="/var/lib/kubelet/pods/ed4f2d62-25d9-44b0-9227-6003f3bb34b3/volumes" Oct 30 20:45:47.719754 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:45:47.719709 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:45:47.953288 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:45:47.953214 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:45:48.152666 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:45:48.152627 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:45:48.524337 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:45:48.524256 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:45:51.386451 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:45:51.386417 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:45:51.628498 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:45:51.628463 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:45:51.902243 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:45:51.902212 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:45:52.092246 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:45:52.092213 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:45:53.015167 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:45:53.015136 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:45:53.225192 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:45:53.225159 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:45:53.413120 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:45:53.413091 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:45:56.725815 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:45:56.725783 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:45:56.880833 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:45:56.880801 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:45:57.083386 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:45:57.083314 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:45:57.964394 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:45:57.964365 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:45:58.117530 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:45:58.117481 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:45:59.064377 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:45:59.064322 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:45:59.227926 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:45:59.227876 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:46:03.819669 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:46:03.819638 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:46:03.915032 ip-10-0-133-192 kubenswrapper[4611]: E1030 20:46:03.914996 4611 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hwzc6" containerName="extractor" Oct 30 20:46:03.999307 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:46:03.999276 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:46:08.769181 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:46:08.769141 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:46:28.955592 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:46:28.955554 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:46:29.140297 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:46:29.140253 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:46:29.328559 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:46:29.328471 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:46:32.455331 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:46:32.455297 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:46:32.605385 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:46:32.605352 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:46:33.915531 ip-10-0-133-192 kubenswrapper[4611]: E1030 20:46:33.915479 4611 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hwzc6" containerName="extractor" Oct 30 20:46:33.979722 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:46:33.979686 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:46:34.129874 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:46:34.129844 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:46:48.113633 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:46:48.113594 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:46:52.020149 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:46:52.020112 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:47:03.592611 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:47:03.592564 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:47:03.746751 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:47:03.746725 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:47:03.916039 ip-10-0-133-192 kubenswrapper[4611]: E1030 20:47:03.916009 4611 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hwzc6" containerName="extractor" Oct 30 20:47:04.896262 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:47:04.896227 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:47:04.975373 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:47:04.975350 4611 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-wxl9l"] Oct 30 20:47:04.975679 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:47:04.975666 4611 cpu_manager.go:414] "containerMap: removing container" podUID="ed4f2d62-25d9-44b0-9227-6003f3bb34b3" containerName="authorino" Oct 30 20:47:04.975740 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:47:04.975680 4611 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed4f2d62-25d9-44b0-9227-6003f3bb34b3" containerName="authorino" Oct 30 20:47:04.975779 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:47:04.975765 4611 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed4f2d62-25d9-44b0-9227-6003f3bb34b3" containerName="authorino" Oct 30 20:47:04.978697 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:47:04.978683 4611 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-wxl9l" Oct 30 20:47:04.985621 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:47:04.985561 4611 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 20:47:04.985697 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:47:04.985626 4611 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 20:47:04.986157 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:47:04.986142 4611 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Oct 30 20:47:04.986210 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:47:04.986176 4611 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-k976f\"" Oct 30 20:47:04.987634 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:47:04.987619 4611 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Oct 30 20:47:05.006949 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:47:05.006924 4611 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-wxl9l"] Oct 30 20:47:05.029075 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:47:05.029048 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ff6213b-7f9e-427a-8af0-8f5168f2dd3d-openshift-service-ca\") pod \"maas-api-6d669c7fcd-wxl9l\" (UID: \"8ff6213b-7f9e-427a-8af0-8f5168f2dd3d\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-wxl9l" Oct 30 20:47:05.029199 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:47:05.029160 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8ff6213b-7f9e-427a-8af0-8f5168f2dd3d-maas-api-tls\") pod \"maas-api-6d669c7fcd-wxl9l\" (UID: \"8ff6213b-7f9e-427a-8af0-8f5168f2dd3d\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-wxl9l" Oct 30 20:47:05.029265 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:47:05.029216 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8ff6213b-7f9e-427a-8af0-8f5168f2dd3d-metrics-certs\") pod \"maas-api-6d669c7fcd-wxl9l\" (UID: \"8ff6213b-7f9e-427a-8af0-8f5168f2dd3d\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-wxl9l" Oct 30 20:47:05.029322 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:47:05.029293 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lpxj7\" (UniqueName: \"kubernetes.io/projected/8ff6213b-7f9e-427a-8af0-8f5168f2dd3d-kube-api-access-lpxj7\") pod \"maas-api-6d669c7fcd-wxl9l\" (UID: \"8ff6213b-7f9e-427a-8af0-8f5168f2dd3d\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-wxl9l" Oct 30 20:47:05.054454 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:47:05.054430 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:47:05.129891 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:47:05.129870 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ff6213b-7f9e-427a-8af0-8f5168f2dd3d-openshift-service-ca\") pod \"maas-api-6d669c7fcd-wxl9l\" (UID: \"8ff6213b-7f9e-427a-8af0-8f5168f2dd3d\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-wxl9l" Oct 30 20:47:05.129998 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:47:05.129917 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8ff6213b-7f9e-427a-8af0-8f5168f2dd3d-maas-api-tls\") pod \"maas-api-6d669c7fcd-wxl9l\" (UID: \"8ff6213b-7f9e-427a-8af0-8f5168f2dd3d\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-wxl9l" Oct 30 20:47:05.129998 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:47:05.129933 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8ff6213b-7f9e-427a-8af0-8f5168f2dd3d-metrics-certs\") pod \"maas-api-6d669c7fcd-wxl9l\" (UID: \"8ff6213b-7f9e-427a-8af0-8f5168f2dd3d\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-wxl9l" Oct 30 20:47:05.129998 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:47:05.129953 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lpxj7\" (UniqueName: \"kubernetes.io/projected/8ff6213b-7f9e-427a-8af0-8f5168f2dd3d-kube-api-access-lpxj7\") pod \"maas-api-6d669c7fcd-wxl9l\" (UID: \"8ff6213b-7f9e-427a-8af0-8f5168f2dd3d\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-wxl9l" Oct 30 20:47:05.130629 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:47:05.130608 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ff6213b-7f9e-427a-8af0-8f5168f2dd3d-openshift-service-ca\") pod \"maas-api-6d669c7fcd-wxl9l\" (UID: \"8ff6213b-7f9e-427a-8af0-8f5168f2dd3d\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-wxl9l" Oct 30 20:47:05.132309 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:47:05.132285 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8ff6213b-7f9e-427a-8af0-8f5168f2dd3d-maas-api-tls\") pod \"maas-api-6d669c7fcd-wxl9l\" (UID: \"8ff6213b-7f9e-427a-8af0-8f5168f2dd3d\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-wxl9l" Oct 30 20:47:05.132388 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:47:05.132329 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8ff6213b-7f9e-427a-8af0-8f5168f2dd3d-metrics-certs\") pod \"maas-api-6d669c7fcd-wxl9l\" (UID: \"8ff6213b-7f9e-427a-8af0-8f5168f2dd3d\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-wxl9l" Oct 30 20:47:05.141079 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:47:05.141056 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lpxj7\" (UniqueName: \"kubernetes.io/projected/8ff6213b-7f9e-427a-8af0-8f5168f2dd3d-kube-api-access-lpxj7\") pod \"maas-api-6d669c7fcd-wxl9l\" (UID: \"8ff6213b-7f9e-427a-8af0-8f5168f2dd3d\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-wxl9l" Oct 30 20:47:05.291589 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:47:05.291535 4611 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-wxl9l" Oct 30 20:47:05.568599 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:47:05.568523 4611 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-wxl9l"] Oct 30 20:47:05.571353 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:47:05.571328 4611 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ff6213b_7f9e_427a_8af0_8f5168f2dd3d.slice/crio-2eac0087ff6e7a818d70ab4406fc4c161d2036fe5ba308feb8a441db49421238 WatchSource:0}: Error finding container 2eac0087ff6e7a818d70ab4406fc4c161d2036fe5ba308feb8a441db49421238: Status 404 returned error can't find the container with id 2eac0087ff6e7a818d70ab4406fc4c161d2036fe5ba308feb8a441db49421238 Oct 30 20:47:06.556144 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:47:06.556107 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-wxl9l" event={"ID":"8ff6213b-7f9e-427a-8af0-8f5168f2dd3d","Type":"ContainerStarted","Data":"2eac0087ff6e7a818d70ab4406fc4c161d2036fe5ba308feb8a441db49421238"} Oct 30 20:47:08.567053 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:47:08.567014 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-wxl9l" event={"ID":"8ff6213b-7f9e-427a-8af0-8f5168f2dd3d","Type":"ContainerStarted","Data":"475cdb98d78451a2fb081f1d7d01043c97dbadc51348e2a0da9222ceeba4620d"} Oct 30 20:47:08.567491 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:47:08.567086 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-wxl9l" Oct 30 20:47:08.586372 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:47:08.586326 4611 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-wxl9l" podStartSLOduration=2.32188958 podStartE2EDuration="4.58631289s" totalImagesPullingTime="2.26442331s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:47:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:47:08.58511644 +0000 UTC m=+483.119143057" watchObservedRunningTime="2026-10-30 20:47:08.58631289 +0000 UTC m=+483.120339563" Oct 30 20:47:14.576888 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:47:14.576858 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-wxl9l" Oct 30 20:47:15.071161 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:47:15.071119 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:47:15.245238 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:47:15.245208 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:47:15.354985 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:47:15.354962 4611 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-n9fmf"] Oct 30 20:47:15.358436 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:47:15.358421 4611 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-n9fmf" Oct 30 20:47:15.361320 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:47:15.361303 4611 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-6n5tv\"" Oct 30 20:47:15.366423 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:47:15.366381 4611 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-n9fmf"] Oct 30 20:47:15.428449 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:47:15.428426 4611 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-n9fmf"] Oct 30 20:47:15.428614 ip-10-0-133-192 kubenswrapper[4611]: E1030 20:47:15.428596 4611 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-vvwbm], unattached volumes=[], failed to process volumes=[kube-api-access-vvwbm]: context canceled" pod="kuadrant-system/authorino-6f6944bcdd-n9fmf" podUID="f786f6a2-3dcf-4de5-9eab-61abc3ea16e3" Oct 30 20:47:15.506231 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:47:15.506206 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vvwbm\" (UniqueName: \"kubernetes.io/projected/f786f6a2-3dcf-4de5-9eab-61abc3ea16e3-kube-api-access-vvwbm\") pod \"authorino-6f6944bcdd-n9fmf\" (UID: \"f786f6a2-3dcf-4de5-9eab-61abc3ea16e3\") " pod="kuadrant-system/authorino-6f6944bcdd-n9fmf" Oct 30 20:47:15.596600 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:47:15.596576 4611 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-n9fmf" Oct 30 20:47:15.603629 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:47:15.603611 4611 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-n9fmf" Oct 30 20:47:15.607622 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:47:15.607573 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vvwbm\" (UniqueName: \"kubernetes.io/projected/f786f6a2-3dcf-4de5-9eab-61abc3ea16e3-kube-api-access-vvwbm\") pod \"authorino-6f6944bcdd-n9fmf\" (UID: \"f786f6a2-3dcf-4de5-9eab-61abc3ea16e3\") " pod="kuadrant-system/authorino-6f6944bcdd-n9fmf" Oct 30 20:47:15.619476 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:47:15.619451 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vvwbm\" (UniqueName: \"kubernetes.io/projected/f786f6a2-3dcf-4de5-9eab-61abc3ea16e3-kube-api-access-vvwbm\") pod \"authorino-6f6944bcdd-n9fmf\" (UID: \"f786f6a2-3dcf-4de5-9eab-61abc3ea16e3\") " pod="kuadrant-system/authorino-6f6944bcdd-n9fmf" Oct 30 20:47:15.707832 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:47:15.707807 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f786f6a2-3dcf-4de5-9eab-61abc3ea16e3-kube-api-access-vvwbm\" (UniqueName: \"kubernetes.io/projected/f786f6a2-3dcf-4de5-9eab-61abc3ea16e3-kube-api-access-vvwbm\") pod \"f786f6a2-3dcf-4de5-9eab-61abc3ea16e3\" (UID: \"f786f6a2-3dcf-4de5-9eab-61abc3ea16e3\") " Oct 30 20:47:15.709698 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:47:15.709679 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f786f6a2-3dcf-4de5-9eab-61abc3ea16e3-kube-api-access-vvwbm" pod "f786f6a2-3dcf-4de5-9eab-61abc3ea16e3" (UID: "f786f6a2-3dcf-4de5-9eab-61abc3ea16e3"). InnerVolumeSpecName "kube-api-access-vvwbm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:47:15.808779 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:47:15.808755 4611 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vvwbm\" (UniqueName: \"kubernetes.io/projected/f786f6a2-3dcf-4de5-9eab-61abc3ea16e3-kube-api-access-vvwbm\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:47:15.826397 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:47:15.826371 4611 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-8545cfcdb6-t86wv"] Oct 30 20:47:15.829716 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:47:15.829696 4611 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-8545cfcdb6-t86wv" Oct 30 20:47:15.865901 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:47:15.865854 4611 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Oct 30 20:47:15.867968 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:47:15.867946 4611 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-8545cfcdb6-t86wv"] Oct 30 20:47:15.910002 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:47:15.909977 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vj4pd\" (UniqueName: \"kubernetes.io/projected/501e279a-76c1-42f6-84e3-6337fd9c8181-kube-api-access-vj4pd\") pod \"authorino-8545cfcdb6-t86wv\" (UID: \"501e279a-76c1-42f6-84e3-6337fd9c8181\") " pod="kuadrant-system/authorino-8545cfcdb6-t86wv" Oct 30 20:47:15.910105 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:47:15.910044 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/501e279a-76c1-42f6-84e3-6337fd9c8181-tls-cert\") pod \"authorino-8545cfcdb6-t86wv\" (UID: \"501e279a-76c1-42f6-84e3-6337fd9c8181\") " pod="kuadrant-system/authorino-8545cfcdb6-t86wv" Oct 30 20:47:16.010650 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:47:16.010629 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vj4pd\" (UniqueName: \"kubernetes.io/projected/501e279a-76c1-42f6-84e3-6337fd9c8181-kube-api-access-vj4pd\") pod \"authorino-8545cfcdb6-t86wv\" (UID: \"501e279a-76c1-42f6-84e3-6337fd9c8181\") " pod="kuadrant-system/authorino-8545cfcdb6-t86wv" Oct 30 20:47:16.010724 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:47:16.010677 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/501e279a-76c1-42f6-84e3-6337fd9c8181-tls-cert\") pod \"authorino-8545cfcdb6-t86wv\" (UID: \"501e279a-76c1-42f6-84e3-6337fd9c8181\") " pod="kuadrant-system/authorino-8545cfcdb6-t86wv" Oct 30 20:47:16.013013 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:47:16.012993 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/501e279a-76c1-42f6-84e3-6337fd9c8181-tls-cert\") pod \"authorino-8545cfcdb6-t86wv\" (UID: \"501e279a-76c1-42f6-84e3-6337fd9c8181\") " pod="kuadrant-system/authorino-8545cfcdb6-t86wv" Oct 30 20:47:16.022770 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:47:16.022752 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vj4pd\" (UniqueName: \"kubernetes.io/projected/501e279a-76c1-42f6-84e3-6337fd9c8181-kube-api-access-vj4pd\") pod \"authorino-8545cfcdb6-t86wv\" (UID: \"501e279a-76c1-42f6-84e3-6337fd9c8181\") " pod="kuadrant-system/authorino-8545cfcdb6-t86wv" Oct 30 20:47:16.141049 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:47:16.141005 4611 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-8545cfcdb6-t86wv" Oct 30 20:47:16.415080 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:47:16.415051 4611 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-8545cfcdb6-t86wv"] Oct 30 20:47:16.417670 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:47:16.417641 4611 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod501e279a_76c1_42f6_84e3_6337fd9c8181.slice/crio-a5ba85985a416298f52b077079ad0e3d1a3318fcf127b2deeec070cfe364e825 WatchSource:0}: Error finding container a5ba85985a416298f52b077079ad0e3d1a3318fcf127b2deeec070cfe364e825: Status 404 returned error can't find the container with id a5ba85985a416298f52b077079ad0e3d1a3318fcf127b2deeec070cfe364e825 Oct 30 20:47:16.602028 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:47:16.602004 4611 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-n9fmf" Oct 30 20:47:16.602332 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:47:16.601996 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-8545cfcdb6-t86wv" event={"ID":"501e279a-76c1-42f6-84e3-6337fd9c8181","Type":"ContainerStarted","Data":"a5ba85985a416298f52b077079ad0e3d1a3318fcf127b2deeec070cfe364e825"} Oct 30 20:47:16.630744 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:47:16.630716 4611 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-n9fmf"] Oct 30 20:47:16.638212 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:47:16.638192 4611 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-n9fmf"] Oct 30 20:47:17.605722 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:47:17.605686 4611 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f786f6a2-3dcf-4de5-9eab-61abc3ea16e3" path="/var/lib/kubelet/pods/f786f6a2-3dcf-4de5-9eab-61abc3ea16e3/volumes" Oct 30 20:47:17.608062 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:47:17.608028 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-8545cfcdb6-t86wv" event={"ID":"501e279a-76c1-42f6-84e3-6337fd9c8181","Type":"ContainerStarted","Data":"86879222b77d5d8c672f96a8a62d7f75a993ce55915900f6fc5092926556a88b"} Oct 30 20:47:17.623827 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:47:17.623781 4611 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-8545cfcdb6-t86wv" podStartSLOduration=1.909769949 podStartE2EDuration="2.623764326s" totalImagesPullingTime="713.994377ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:47:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:47:17.622565191 +0000 UTC m=+492.156591807" watchObservedRunningTime="2026-10-30 20:47:17.623764326 +0000 UTC m=+492.157790940" Oct 30 20:47:25.234265 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:47:25.234230 4611 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-wxl9l"] Oct 30 20:47:25.234676 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:47:25.234409 4611 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-wxl9l" podUID="8ff6213b-7f9e-427a-8af0-8f5168f2dd3d" containerName="maas-api" containerID="cri-o://475cdb98d78451a2fb081f1d7d01043c97dbadc51348e2a0da9222ceeba4620d" gracePeriod=30 Oct 30 20:47:25.475755 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:47:25.475733 4611 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-wxl9l" Oct 30 20:47:25.590217 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:47:25.590146 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ff6213b-7f9e-427a-8af0-8f5168f2dd3d-kube-api-access-lpxj7\" (UniqueName: \"kubernetes.io/projected/8ff6213b-7f9e-427a-8af0-8f5168f2dd3d-kube-api-access-lpxj7\") pod \"8ff6213b-7f9e-427a-8af0-8f5168f2dd3d\" (UID: \"8ff6213b-7f9e-427a-8af0-8f5168f2dd3d\") " Oct 30 20:47:25.590217 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:47:25.590202 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8ff6213b-7f9e-427a-8af0-8f5168f2dd3d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ff6213b-7f9e-427a-8af0-8f5168f2dd3d-openshift-service-ca\") pod \"8ff6213b-7f9e-427a-8af0-8f5168f2dd3d\" (UID: \"8ff6213b-7f9e-427a-8af0-8f5168f2dd3d\") " Oct 30 20:47:25.590396 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:47:25.590237 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8ff6213b-7f9e-427a-8af0-8f5168f2dd3d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/8ff6213b-7f9e-427a-8af0-8f5168f2dd3d-metrics-certs\") pod \"8ff6213b-7f9e-427a-8af0-8f5168f2dd3d\" (UID: \"8ff6213b-7f9e-427a-8af0-8f5168f2dd3d\") " Oct 30 20:47:25.590396 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:47:25.590274 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8ff6213b-7f9e-427a-8af0-8f5168f2dd3d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8ff6213b-7f9e-427a-8af0-8f5168f2dd3d-maas-api-tls\") pod \"8ff6213b-7f9e-427a-8af0-8f5168f2dd3d\" (UID: \"8ff6213b-7f9e-427a-8af0-8f5168f2dd3d\") " Oct 30 20:47:25.590654 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:47:25.590628 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ff6213b-7f9e-427a-8af0-8f5168f2dd3d-openshift-service-ca" pod "8ff6213b-7f9e-427a-8af0-8f5168f2dd3d" (UID: "8ff6213b-7f9e-427a-8af0-8f5168f2dd3d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:47:25.592389 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:47:25.592362 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8ff6213b-7f9e-427a-8af0-8f5168f2dd3d-maas-api-tls" pod "8ff6213b-7f9e-427a-8af0-8f5168f2dd3d" (UID: "8ff6213b-7f9e-427a-8af0-8f5168f2dd3d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:47:25.592582 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:47:25.592445 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ff6213b-7f9e-427a-8af0-8f5168f2dd3d-kube-api-access-lpxj7" pod "8ff6213b-7f9e-427a-8af0-8f5168f2dd3d" (UID: "8ff6213b-7f9e-427a-8af0-8f5168f2dd3d"). InnerVolumeSpecName "kube-api-access-lpxj7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:47:25.592582 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:47:25.592477 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8ff6213b-7f9e-427a-8af0-8f5168f2dd3d-metrics-certs" pod "8ff6213b-7f9e-427a-8af0-8f5168f2dd3d" (UID: "8ff6213b-7f9e-427a-8af0-8f5168f2dd3d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:47:25.643091 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:47:25.643067 4611 generic.go:356] "Generic (PLEG): container finished" podID="8ff6213b-7f9e-427a-8af0-8f5168f2dd3d" containerID="475cdb98d78451a2fb081f1d7d01043c97dbadc51348e2a0da9222ceeba4620d" exitCode=0 Oct 30 20:47:25.643196 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:47:25.643109 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-wxl9l" event={"ID":"8ff6213b-7f9e-427a-8af0-8f5168f2dd3d","Type":"ContainerDied","Data":"475cdb98d78451a2fb081f1d7d01043c97dbadc51348e2a0da9222ceeba4620d"} Oct 30 20:47:25.643196 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:47:25.643140 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-wxl9l" event={"ID":"8ff6213b-7f9e-427a-8af0-8f5168f2dd3d","Type":"ContainerDied","Data":"2eac0087ff6e7a818d70ab4406fc4c161d2036fe5ba308feb8a441db49421238"} Oct 30 20:47:25.643196 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:47:25.643160 4611 scope.go:122] "RemoveContainer" containerID="475cdb98d78451a2fb081f1d7d01043c97dbadc51348e2a0da9222ceeba4620d" Oct 30 20:47:25.643196 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:47:25.643111 4611 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-wxl9l" Oct 30 20:47:25.651414 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:47:25.651396 4611 scope.go:122] "RemoveContainer" containerID="475cdb98d78451a2fb081f1d7d01043c97dbadc51348e2a0da9222ceeba4620d" Oct 30 20:47:25.651674 ip-10-0-133-192 kubenswrapper[4611]: E1030 20:47:25.651656 4611 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"475cdb98d78451a2fb081f1d7d01043c97dbadc51348e2a0da9222ceeba4620d\": container with ID starting with 475cdb98d78451a2fb081f1d7d01043c97dbadc51348e2a0da9222ceeba4620d not found: ID does not exist" containerID="475cdb98d78451a2fb081f1d7d01043c97dbadc51348e2a0da9222ceeba4620d" Oct 30 20:47:25.651731 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:47:25.651680 4611 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"475cdb98d78451a2fb081f1d7d01043c97dbadc51348e2a0da9222ceeba4620d"} err="failed to get container status \"475cdb98d78451a2fb081f1d7d01043c97dbadc51348e2a0da9222ceeba4620d\": rpc error: code = NotFound desc = could not find container \"475cdb98d78451a2fb081f1d7d01043c97dbadc51348e2a0da9222ceeba4620d\": container with ID starting with 475cdb98d78451a2fb081f1d7d01043c97dbadc51348e2a0da9222ceeba4620d not found: ID does not exist" Oct 30 20:47:25.658346 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:47:25.658324 4611 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-wxl9l"] Oct 30 20:47:25.662415 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:47:25.662398 4611 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-wxl9l"] Oct 30 20:47:25.691615 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:47:25.691599 4611 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8ff6213b-7f9e-427a-8af0-8f5168f2dd3d-maas-api-tls\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:47:25.691697 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:47:25.691618 4611 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lpxj7\" (UniqueName: \"kubernetes.io/projected/8ff6213b-7f9e-427a-8af0-8f5168f2dd3d-kube-api-access-lpxj7\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:47:25.691697 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:47:25.691630 4611 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ff6213b-7f9e-427a-8af0-8f5168f2dd3d-openshift-service-ca\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:47:25.691697 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:47:25.691639 4611 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8ff6213b-7f9e-427a-8af0-8f5168f2dd3d-metrics-certs\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:47:27.608665 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:47:27.608634 4611 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8ff6213b-7f9e-427a-8af0-8f5168f2dd3d" path="/var/lib/kubelet/pods/8ff6213b-7f9e-427a-8af0-8f5168f2dd3d/volumes" Oct 30 20:47:33.916325 ip-10-0-133-192 kubenswrapper[4611]: E1030 20:47:33.916285 4611 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hwzc6" containerName="extractor" Oct 30 20:47:48.181034 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:47:48.180993 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:47:51.714333 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:47:51.714295 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:47:51.859431 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:47:51.859401 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:48:03.914874 ip-10-0-133-192 kubenswrapper[4611]: E1030 20:48:03.914838 4611 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hwzc6" containerName="extractor" Oct 30 20:48:10.254331 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:48:10.254291 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:48:11.501280 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:48:11.501244 4611 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-s9l7p"] Oct 30 20:48:11.501708 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:48:11.501691 4611 cpu_manager.go:414] "containerMap: removing container" podUID="8ff6213b-7f9e-427a-8af0-8f5168f2dd3d" containerName="maas-api" Oct 30 20:48:11.501751 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:48:11.501733 4611 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ff6213b-7f9e-427a-8af0-8f5168f2dd3d" containerName="maas-api" Oct 30 20:48:11.501812 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:48:11.501803 4611 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ff6213b-7f9e-427a-8af0-8f5168f2dd3d" containerName="maas-api" Oct 30 20:48:11.506001 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:48:11.505982 4611 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-s9l7p" Oct 30 20:48:11.514974 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:48:11.514951 4611 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-c25rl\"" Oct 30 20:48:11.515119 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:48:11.515000 4611 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Oct 30 20:48:11.517093 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:48:11.517070 4611 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-s9l7p"] Oct 30 20:48:11.539225 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:48:11.539197 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a86251b2-e065-4aaa-9b7b-b8b7bee34bc2-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-s9l7p\" (UID: \"a86251b2-e065-4aaa-9b7b-b8b7bee34bc2\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-s9l7p" Oct 30 20:48:11.539333 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:48:11.539235 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a86251b2-e065-4aaa-9b7b-b8b7bee34bc2-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-s9l7p\" (UID: \"a86251b2-e065-4aaa-9b7b-b8b7bee34bc2\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-s9l7p" Oct 30 20:48:11.539333 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:48:11.539253 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a86251b2-e065-4aaa-9b7b-b8b7bee34bc2-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-s9l7p\" (UID: \"a86251b2-e065-4aaa-9b7b-b8b7bee34bc2\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-s9l7p" Oct 30 20:48:11.539412 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:48:11.539358 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a86251b2-e065-4aaa-9b7b-b8b7bee34bc2-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-s9l7p\" (UID: \"a86251b2-e065-4aaa-9b7b-b8b7bee34bc2\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-s9l7p" Oct 30 20:48:11.539412 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:48:11.539385 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l9qcj\" (UniqueName: \"kubernetes.io/projected/a86251b2-e065-4aaa-9b7b-b8b7bee34bc2-kube-api-access-l9qcj\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-s9l7p\" (UID: \"a86251b2-e065-4aaa-9b7b-b8b7bee34bc2\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-s9l7p" Oct 30 20:48:11.539499 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:48:11.539422 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a86251b2-e065-4aaa-9b7b-b8b7bee34bc2-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-s9l7p\" (UID: \"a86251b2-e065-4aaa-9b7b-b8b7bee34bc2\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-s9l7p" Oct 30 20:48:11.596855 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:48:11.596831 4611 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Oct 30 20:48:11.596855 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:48:11.596835 4611 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Oct 30 20:48:11.640653 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:48:11.640626 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a86251b2-e065-4aaa-9b7b-b8b7bee34bc2-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-s9l7p\" (UID: \"a86251b2-e065-4aaa-9b7b-b8b7bee34bc2\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-s9l7p" Oct 30 20:48:11.640771 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:48:11.640661 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a86251b2-e065-4aaa-9b7b-b8b7bee34bc2-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-s9l7p\" (UID: \"a86251b2-e065-4aaa-9b7b-b8b7bee34bc2\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-s9l7p" Oct 30 20:48:11.640771 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:48:11.640682 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a86251b2-e065-4aaa-9b7b-b8b7bee34bc2-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-s9l7p\" (UID: \"a86251b2-e065-4aaa-9b7b-b8b7bee34bc2\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-s9l7p" Oct 30 20:48:11.640771 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:48:11.640715 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a86251b2-e065-4aaa-9b7b-b8b7bee34bc2-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-s9l7p\" (UID: \"a86251b2-e065-4aaa-9b7b-b8b7bee34bc2\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-s9l7p" Oct 30 20:48:11.640771 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:48:11.640731 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l9qcj\" (UniqueName: \"kubernetes.io/projected/a86251b2-e065-4aaa-9b7b-b8b7bee34bc2-kube-api-access-l9qcj\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-s9l7p\" (UID: \"a86251b2-e065-4aaa-9b7b-b8b7bee34bc2\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-s9l7p" Oct 30 20:48:11.640979 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:48:11.640840 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a86251b2-e065-4aaa-9b7b-b8b7bee34bc2-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-s9l7p\" (UID: \"a86251b2-e065-4aaa-9b7b-b8b7bee34bc2\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-s9l7p" Oct 30 20:48:11.641085 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:48:11.641061 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a86251b2-e065-4aaa-9b7b-b8b7bee34bc2-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-s9l7p\" (UID: \"a86251b2-e065-4aaa-9b7b-b8b7bee34bc2\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-s9l7p" Oct 30 20:48:11.641188 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:48:11.641097 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a86251b2-e065-4aaa-9b7b-b8b7bee34bc2-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-s9l7p\" (UID: \"a86251b2-e065-4aaa-9b7b-b8b7bee34bc2\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-s9l7p" Oct 30 20:48:11.641188 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:48:11.641172 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a86251b2-e065-4aaa-9b7b-b8b7bee34bc2-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-s9l7p\" (UID: \"a86251b2-e065-4aaa-9b7b-b8b7bee34bc2\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-s9l7p" Oct 30 20:48:11.642974 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:48:11.642957 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a86251b2-e065-4aaa-9b7b-b8b7bee34bc2-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-s9l7p\" (UID: \"a86251b2-e065-4aaa-9b7b-b8b7bee34bc2\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-s9l7p" Oct 30 20:48:11.643229 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:48:11.643211 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a86251b2-e065-4aaa-9b7b-b8b7bee34bc2-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-s9l7p\" (UID: \"a86251b2-e065-4aaa-9b7b-b8b7bee34bc2\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-s9l7p" Oct 30 20:48:11.652199 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:48:11.652178 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l9qcj\" (UniqueName: \"kubernetes.io/projected/a86251b2-e065-4aaa-9b7b-b8b7bee34bc2-kube-api-access-l9qcj\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-s9l7p\" (UID: \"a86251b2-e065-4aaa-9b7b-b8b7bee34bc2\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-s9l7p" Oct 30 20:48:11.819316 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:48:11.819250 4611 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-s9l7p" Oct 30 20:48:12.021904 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:48:12.021870 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:48:12.097586 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:48:12.096908 4611 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-s9l7p"] Oct 30 20:48:12.100854 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:48:12.100820 4611 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda86251b2_e065_4aaa_9b7b_b8b7bee34bc2.slice/crio-88bf8872719ddc438e7b41d7bca46ae9255d39a2e1905a474105606c0ce7445e WatchSource:0}: Error finding container 88bf8872719ddc438e7b41d7bca46ae9255d39a2e1905a474105606c0ce7445e: Status 404 returned error can't find the container with id 88bf8872719ddc438e7b41d7bca46ae9255d39a2e1905a474105606c0ce7445e Oct 30 20:48:12.849634 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:48:12.849596 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-s9l7p" event={"ID":"a86251b2-e065-4aaa-9b7b-b8b7bee34bc2","Type":"ContainerStarted","Data":"88bf8872719ddc438e7b41d7bca46ae9255d39a2e1905a474105606c0ce7445e"} Oct 30 20:48:13.735479 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:48:13.735391 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:48:13.855014 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:48:13.854979 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-s9l7p" event={"ID":"a86251b2-e065-4aaa-9b7b-b8b7bee34bc2","Type":"ContainerStarted","Data":"267b1bf88a947db2b189e886af9a3b235467b02b2a67e6dd3e896ef04266d05c"} Oct 30 20:48:13.874484 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:48:13.874442 4611 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-s9l7p" podStartSLOduration=1.608290065 podStartE2EDuration="2.874430549s" totalImagesPullingTime="1.266140484s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:48:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:48:13.873447755 +0000 UTC m=+548.407474383" watchObservedRunningTime="2026-10-30 20:48:13.874430549 +0000 UTC m=+548.408457180" Oct 30 20:48:13.915417 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:48:13.915388 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:48:14.819752 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:48:14.819713 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-s9l7p" Oct 30 20:48:15.372974 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:48:15.372943 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:48:15.826342 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:48:15.826273 4611 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-dq6vs"] Oct 30 20:48:15.830327 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:48:15.830304 4611 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-dq6vs" Oct 30 20:48:15.842062 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:48:15.842037 4611 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Oct 30 20:48:15.843474 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:48:15.843453 4611 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-dq6vs"] Oct 30 20:48:15.889604 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:48:15.889578 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f23cb637-ada7-4e6d-897f-83978d56ac7d-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-dq6vs\" (UID: \"f23cb637-ada7-4e6d-897f-83978d56ac7d\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-dq6vs" Oct 30 20:48:15.889712 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:48:15.889612 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f23cb637-ada7-4e6d-897f-83978d56ac7d-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-dq6vs\" (UID: \"f23cb637-ada7-4e6d-897f-83978d56ac7d\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-dq6vs" Oct 30 20:48:15.889712 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:48:15.889699 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f23cb637-ada7-4e6d-897f-83978d56ac7d-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-dq6vs\" (UID: \"f23cb637-ada7-4e6d-897f-83978d56ac7d\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-dq6vs" Oct 30 20:48:15.889785 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:48:15.889755 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-72k2h\" (UniqueName: \"kubernetes.io/projected/f23cb637-ada7-4e6d-897f-83978d56ac7d-kube-api-access-72k2h\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-dq6vs\" (UID: \"f23cb637-ada7-4e6d-897f-83978d56ac7d\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-dq6vs" Oct 30 20:48:15.889819 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:48:15.889810 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f23cb637-ada7-4e6d-897f-83978d56ac7d-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-dq6vs\" (UID: \"f23cb637-ada7-4e6d-897f-83978d56ac7d\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-dq6vs" Oct 30 20:48:15.889865 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:48:15.889852 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f23cb637-ada7-4e6d-897f-83978d56ac7d-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-dq6vs\" (UID: \"f23cb637-ada7-4e6d-897f-83978d56ac7d\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-dq6vs" Oct 30 20:48:15.936495 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:48:15.936469 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:48:15.990805 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:48:15.990783 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f23cb637-ada7-4e6d-897f-83978d56ac7d-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-dq6vs\" (UID: \"f23cb637-ada7-4e6d-897f-83978d56ac7d\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-dq6vs" Oct 30 20:48:15.990895 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:48:15.990814 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f23cb637-ada7-4e6d-897f-83978d56ac7d-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-dq6vs\" (UID: \"f23cb637-ada7-4e6d-897f-83978d56ac7d\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-dq6vs" Oct 30 20:48:15.990949 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:48:15.990930 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f23cb637-ada7-4e6d-897f-83978d56ac7d-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-dq6vs\" (UID: \"f23cb637-ada7-4e6d-897f-83978d56ac7d\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-dq6vs" Oct 30 20:48:15.990995 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:48:15.990974 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-72k2h\" (UniqueName: \"kubernetes.io/projected/f23cb637-ada7-4e6d-897f-83978d56ac7d-kube-api-access-72k2h\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-dq6vs\" (UID: \"f23cb637-ada7-4e6d-897f-83978d56ac7d\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-dq6vs" Oct 30 20:48:15.991099 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:48:15.991076 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f23cb637-ada7-4e6d-897f-83978d56ac7d-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-dq6vs\" (UID: \"f23cb637-ada7-4e6d-897f-83978d56ac7d\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-dq6vs" Oct 30 20:48:15.991178 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:48:15.991105 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f23cb637-ada7-4e6d-897f-83978d56ac7d-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-dq6vs\" (UID: \"f23cb637-ada7-4e6d-897f-83978d56ac7d\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-dq6vs" Oct 30 20:48:15.991398 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:48:15.991373 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f23cb637-ada7-4e6d-897f-83978d56ac7d-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-dq6vs\" (UID: \"f23cb637-ada7-4e6d-897f-83978d56ac7d\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-dq6vs" Oct 30 20:48:15.991474 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:48:15.991405 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f23cb637-ada7-4e6d-897f-83978d56ac7d-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-dq6vs\" (UID: \"f23cb637-ada7-4e6d-897f-83978d56ac7d\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-dq6vs" Oct 30 20:48:15.991474 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:48:15.991436 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f23cb637-ada7-4e6d-897f-83978d56ac7d-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-dq6vs\" (UID: \"f23cb637-ada7-4e6d-897f-83978d56ac7d\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-dq6vs" Oct 30 20:48:15.993146 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:48:15.993123 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f23cb637-ada7-4e6d-897f-83978d56ac7d-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-dq6vs\" (UID: \"f23cb637-ada7-4e6d-897f-83978d56ac7d\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-dq6vs" Oct 30 20:48:15.993358 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:48:15.993337 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f23cb637-ada7-4e6d-897f-83978d56ac7d-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-dq6vs\" (UID: \"f23cb637-ada7-4e6d-897f-83978d56ac7d\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-dq6vs" Oct 30 20:48:16.005640 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:48:16.005617 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-72k2h\" (UniqueName: \"kubernetes.io/projected/f23cb637-ada7-4e6d-897f-83978d56ac7d-kube-api-access-72k2h\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-dq6vs\" (UID: \"f23cb637-ada7-4e6d-897f-83978d56ac7d\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-dq6vs" Oct 30 20:48:16.144311 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:48:16.144289 4611 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-dq6vs" Oct 30 20:48:16.422807 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:48:16.422740 4611 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-dq6vs"] Oct 30 20:48:16.426373 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:48:16.426342 4611 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf23cb637_ada7_4e6d_897f_83978d56ac7d.slice/crio-6a319d0ae96ccab9bc8afef1ec8e98a7cf9f7386ee1fa8ce639e2845b0c3e095 WatchSource:0}: Error finding container 6a319d0ae96ccab9bc8afef1ec8e98a7cf9f7386ee1fa8ce639e2845b0c3e095: Status 404 returned error can't find the container with id 6a319d0ae96ccab9bc8afef1ec8e98a7cf9f7386ee1fa8ce639e2845b0c3e095 Oct 30 20:48:16.709536 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:48:16.709426 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:48:16.870740 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:48:16.870701 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-dq6vs" event={"ID":"f23cb637-ada7-4e6d-897f-83978d56ac7d","Type":"ContainerStarted","Data":"38798be96900f305b764e28befcb860636349d7f8b0d5675f95f57f75f8c022c"} Oct 30 20:48:16.870907 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:48:16.870767 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-dq6vs" event={"ID":"f23cb637-ada7-4e6d-897f-83978d56ac7d","Type":"ContainerStarted","Data":"6a319d0ae96ccab9bc8afef1ec8e98a7cf9f7386ee1fa8ce639e2845b0c3e095"} Oct 30 20:48:16.895556 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:48:16.895483 4611 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-dq6vs" podStartSLOduration=1.578988575 podStartE2EDuration="1.895463336s" totalImagesPullingTime="316.474761ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:48:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:48:16.894931188 +0000 UTC m=+551.428957805" watchObservedRunningTime="2026-10-30 20:48:16.895463336 +0000 UTC m=+551.429489952" Oct 30 20:48:17.144657 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:48:17.144627 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-dq6vs" Oct 30 20:48:17.655837 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:48:17.655806 4611 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-qpdpp"] Oct 30 20:48:17.659451 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:48:17.659426 4611 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-qpdpp" Oct 30 20:48:17.674821 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:48:17.674800 4611 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\"" Oct 30 20:48:17.688297 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:48:17.688271 4611 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-qpdpp"] Oct 30 20:48:17.706935 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:48:17.706902 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aa85e194-38f7-4dfe-8be9-6df7df92f0a1-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-qpdpp\" (UID: \"aa85e194-38f7-4dfe-8be9-6df7df92f0a1\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-qpdpp" Oct 30 20:48:17.707031 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:48:17.706971 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aa85e194-38f7-4dfe-8be9-6df7df92f0a1-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-qpdpp\" (UID: \"aa85e194-38f7-4dfe-8be9-6df7df92f0a1\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-qpdpp" Oct 30 20:48:17.707098 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:48:17.707077 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aa85e194-38f7-4dfe-8be9-6df7df92f0a1-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-qpdpp\" (UID: \"aa85e194-38f7-4dfe-8be9-6df7df92f0a1\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-qpdpp" Oct 30 20:48:17.707215 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:48:17.707193 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aa85e194-38f7-4dfe-8be9-6df7df92f0a1-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-qpdpp\" (UID: \"aa85e194-38f7-4dfe-8be9-6df7df92f0a1\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-qpdpp" Oct 30 20:48:17.707262 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:48:17.707246 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-whbc7\" (UniqueName: \"kubernetes.io/projected/aa85e194-38f7-4dfe-8be9-6df7df92f0a1-kube-api-access-whbc7\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-qpdpp\" (UID: \"aa85e194-38f7-4dfe-8be9-6df7df92f0a1\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-qpdpp" Oct 30 20:48:17.707365 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:48:17.707346 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aa85e194-38f7-4dfe-8be9-6df7df92f0a1-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-qpdpp\" (UID: \"aa85e194-38f7-4dfe-8be9-6df7df92f0a1\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-qpdpp" Oct 30 20:48:17.808142 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:48:17.808076 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aa85e194-38f7-4dfe-8be9-6df7df92f0a1-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-qpdpp\" (UID: \"aa85e194-38f7-4dfe-8be9-6df7df92f0a1\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-qpdpp" Oct 30 20:48:17.808142 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:48:17.808136 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-whbc7\" (UniqueName: \"kubernetes.io/projected/aa85e194-38f7-4dfe-8be9-6df7df92f0a1-kube-api-access-whbc7\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-qpdpp\" (UID: \"aa85e194-38f7-4dfe-8be9-6df7df92f0a1\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-qpdpp" Oct 30 20:48:17.808371 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:48:17.808193 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aa85e194-38f7-4dfe-8be9-6df7df92f0a1-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-qpdpp\" (UID: \"aa85e194-38f7-4dfe-8be9-6df7df92f0a1\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-qpdpp" Oct 30 20:48:17.808371 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:48:17.808233 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aa85e194-38f7-4dfe-8be9-6df7df92f0a1-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-qpdpp\" (UID: \"aa85e194-38f7-4dfe-8be9-6df7df92f0a1\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-qpdpp" Oct 30 20:48:17.808371 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:48:17.808274 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aa85e194-38f7-4dfe-8be9-6df7df92f0a1-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-qpdpp\" (UID: \"aa85e194-38f7-4dfe-8be9-6df7df92f0a1\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-qpdpp" Oct 30 20:48:17.808371 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:48:17.808316 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aa85e194-38f7-4dfe-8be9-6df7df92f0a1-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-qpdpp\" (UID: \"aa85e194-38f7-4dfe-8be9-6df7df92f0a1\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-qpdpp" Oct 30 20:48:17.808793 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:48:17.808597 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aa85e194-38f7-4dfe-8be9-6df7df92f0a1-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-qpdpp\" (UID: \"aa85e194-38f7-4dfe-8be9-6df7df92f0a1\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-qpdpp" Oct 30 20:48:17.808793 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:48:17.808733 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aa85e194-38f7-4dfe-8be9-6df7df92f0a1-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-qpdpp\" (UID: \"aa85e194-38f7-4dfe-8be9-6df7df92f0a1\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-qpdpp" Oct 30 20:48:17.809076 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:48:17.809048 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aa85e194-38f7-4dfe-8be9-6df7df92f0a1-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-qpdpp\" (UID: \"aa85e194-38f7-4dfe-8be9-6df7df92f0a1\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-qpdpp" Oct 30 20:48:17.810789 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:48:17.810765 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aa85e194-38f7-4dfe-8be9-6df7df92f0a1-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-qpdpp\" (UID: \"aa85e194-38f7-4dfe-8be9-6df7df92f0a1\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-qpdpp" Oct 30 20:48:17.811078 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:48:17.811058 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aa85e194-38f7-4dfe-8be9-6df7df92f0a1-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-qpdpp\" (UID: \"aa85e194-38f7-4dfe-8be9-6df7df92f0a1\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-qpdpp" Oct 30 20:48:17.832885 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:48:17.832856 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-whbc7\" (UniqueName: \"kubernetes.io/projected/aa85e194-38f7-4dfe-8be9-6df7df92f0a1-kube-api-access-whbc7\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-qpdpp\" (UID: \"aa85e194-38f7-4dfe-8be9-6df7df92f0a1\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-qpdpp" Oct 30 20:48:17.972742 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:48:17.972654 4611 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-qpdpp" Oct 30 20:48:18.017494 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:48:18.017463 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:48:18.215903 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:48:18.215865 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:48:18.256966 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:48:18.256943 4611 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-qpdpp"] Oct 30 20:48:18.259398 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:48:18.259366 4611 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa85e194_38f7_4dfe_8be9_6df7df92f0a1.slice/crio-8a09dc8c739595d29d54d5abd41b418c6ab514b629f8481c63e5cfa1515562df WatchSource:0}: Error finding container 8a09dc8c739595d29d54d5abd41b418c6ab514b629f8481c63e5cfa1515562df: Status 404 returned error can't find the container with id 8a09dc8c739595d29d54d5abd41b418c6ab514b629f8481c63e5cfa1515562df Oct 30 20:48:18.881009 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:48:18.880972 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-qpdpp" event={"ID":"aa85e194-38f7-4dfe-8be9-6df7df92f0a1","Type":"ContainerStarted","Data":"bf4147bfbca170eaa6305fc60b4d3d38dd738b16e4ee961f9646355d073205cb"} Oct 30 20:48:18.881364 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:48:18.881039 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-qpdpp" event={"ID":"aa85e194-38f7-4dfe-8be9-6df7df92f0a1","Type":"ContainerStarted","Data":"8a09dc8c739595d29d54d5abd41b418c6ab514b629f8481c63e5cfa1515562df"} Oct 30 20:48:18.905376 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:48:18.905332 4611 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-qpdpp" podStartSLOduration=1.5701947779999998 podStartE2EDuration="1.905319309s" totalImagesPullingTime="335.124531ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:48:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:48:18.903693492 +0000 UTC m=+553.437720109" watchObservedRunningTime="2026-10-30 20:48:18.905319309 +0000 UTC m=+553.439345925" Oct 30 20:48:18.973019 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:48:18.972994 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-qpdpp" Oct 30 20:48:19.742421 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:48:19.742384 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:48:21.820008 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:48:21.819972 4611 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-s9l7p" Oct 30 20:48:21.833672 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:48:21.833645 4611 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-s9l7p" Oct 30 20:48:21.905976 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:48:21.905946 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-s9l7p" Oct 30 20:48:26.144667 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:48:26.144632 4611 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-dq6vs" Oct 30 20:48:26.157104 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:48:26.157080 4611 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-dq6vs" Oct 30 20:48:26.930731 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:48:26.930703 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-dq6vs" Oct 30 20:48:27.973576 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:48:27.973547 4611 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-qpdpp" Oct 30 20:48:27.985921 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:48:27.985898 4611 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-qpdpp" Oct 30 20:48:28.941920 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:48:28.941878 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-qpdpp" Oct 30 20:48:33.916142 ip-10-0-133-192 kubenswrapper[4611]: E1030 20:48:33.916046 4611 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hwzc6" containerName="extractor" Oct 30 20:48:35.217237 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:48:35.217193 4611 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-w44qs"] Oct 30 20:48:35.219680 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:48:35.219651 4611 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-w44qs" Oct 30 20:48:35.229376 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:48:35.229355 4611 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-vgmtz\"" Oct 30 20:48:35.240641 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:48:35.240614 4611 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-w44qs"] Oct 30 20:48:35.242819 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:48:35.242795 4611 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8tvvl"] Oct 30 20:48:35.245888 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:48:35.245855 4611 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8tvvl" Oct 30 20:48:35.276272 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:48:35.276245 4611 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8tvvl"] Oct 30 20:48:35.370194 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:48:35.370155 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d6fd4b2-ad0a-4ce3-b5b3-966de34bd8b0-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-w44qs\" (UID: \"6d6fd4b2-ad0a-4ce3-b5b3-966de34bd8b0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-w44qs" Oct 30 20:48:35.370375 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:48:35.370209 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d6fd4b2-ad0a-4ce3-b5b3-966de34bd8b0-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-w44qs\" (UID: \"6d6fd4b2-ad0a-4ce3-b5b3-966de34bd8b0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-w44qs" Oct 30 20:48:35.370375 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:48:35.370234 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46ae03da-7a80-4f78-8abd-391ebc697f2f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-8tvvl\" (UID: \"46ae03da-7a80-4f78-8abd-391ebc697f2f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8tvvl" Oct 30 20:48:35.370494 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:48:35.370391 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46ae03da-7a80-4f78-8abd-391ebc697f2f-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-8tvvl\" (UID: \"46ae03da-7a80-4f78-8abd-391ebc697f2f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8tvvl" Oct 30 20:48:35.370494 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:48:35.370469 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d6fd4b2-ad0a-4ce3-b5b3-966de34bd8b0-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-w44qs\" (UID: \"6d6fd4b2-ad0a-4ce3-b5b3-966de34bd8b0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-w44qs" Oct 30 20:48:35.370631 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:48:35.370500 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lvrsc\" (UniqueName: \"kubernetes.io/projected/46ae03da-7a80-4f78-8abd-391ebc697f2f-kube-api-access-lvrsc\") pod \"maas-default-gateway-openshift-default-6db586dd6c-8tvvl\" (UID: \"46ae03da-7a80-4f78-8abd-391ebc697f2f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8tvvl" Oct 30 20:48:35.370631 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:48:35.370574 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d6fd4b2-ad0a-4ce3-b5b3-966de34bd8b0-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-w44qs\" (UID: \"6d6fd4b2-ad0a-4ce3-b5b3-966de34bd8b0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-w44qs" Oct 30 20:48:35.370714 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:48:35.370650 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qdj6x\" (UniqueName: \"kubernetes.io/projected/6d6fd4b2-ad0a-4ce3-b5b3-966de34bd8b0-kube-api-access-qdj6x\") pod \"maas-default-gateway-openshift-default-6db586dd6c-w44qs\" (UID: \"6d6fd4b2-ad0a-4ce3-b5b3-966de34bd8b0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-w44qs" Oct 30 20:48:35.370714 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:48:35.370688 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46ae03da-7a80-4f78-8abd-391ebc697f2f-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-8tvvl\" (UID: \"46ae03da-7a80-4f78-8abd-391ebc697f2f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8tvvl" Oct 30 20:48:35.370858 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:48:35.370805 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d6fd4b2-ad0a-4ce3-b5b3-966de34bd8b0-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-w44qs\" (UID: \"6d6fd4b2-ad0a-4ce3-b5b3-966de34bd8b0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-w44qs" Oct 30 20:48:35.370924 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:48:35.370878 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46ae03da-7a80-4f78-8abd-391ebc697f2f-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-8tvvl\" (UID: \"46ae03da-7a80-4f78-8abd-391ebc697f2f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8tvvl" Oct 30 20:48:35.370999 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:48:35.370982 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46ae03da-7a80-4f78-8abd-391ebc697f2f-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-8tvvl\" (UID: \"46ae03da-7a80-4f78-8abd-391ebc697f2f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8tvvl" Oct 30 20:48:35.371055 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:48:35.371014 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d6fd4b2-ad0a-4ce3-b5b3-966de34bd8b0-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-w44qs\" (UID: \"6d6fd4b2-ad0a-4ce3-b5b3-966de34bd8b0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-w44qs" Oct 30 20:48:35.371110 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:48:35.371065 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46ae03da-7a80-4f78-8abd-391ebc697f2f-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-8tvvl\" (UID: \"46ae03da-7a80-4f78-8abd-391ebc697f2f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8tvvl" Oct 30 20:48:35.371148 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:48:35.371113 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46ae03da-7a80-4f78-8abd-391ebc697f2f-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-8tvvl\" (UID: \"46ae03da-7a80-4f78-8abd-391ebc697f2f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8tvvl" Oct 30 20:48:35.371148 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:48:35.371140 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46ae03da-7a80-4f78-8abd-391ebc697f2f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-8tvvl\" (UID: \"46ae03da-7a80-4f78-8abd-391ebc697f2f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8tvvl" Oct 30 20:48:35.371242 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:48:35.371181 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d6fd4b2-ad0a-4ce3-b5b3-966de34bd8b0-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-w44qs\" (UID: \"6d6fd4b2-ad0a-4ce3-b5b3-966de34bd8b0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-w44qs" Oct 30 20:48:35.371300 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:48:35.371275 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d6fd4b2-ad0a-4ce3-b5b3-966de34bd8b0-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-w44qs\" (UID: \"6d6fd4b2-ad0a-4ce3-b5b3-966de34bd8b0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-w44qs" Oct 30 20:48:35.472394 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:48:35.472316 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d6fd4b2-ad0a-4ce3-b5b3-966de34bd8b0-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-w44qs\" (UID: \"6d6fd4b2-ad0a-4ce3-b5b3-966de34bd8b0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-w44qs" Oct 30 20:48:35.472394 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:48:35.472351 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qdj6x\" (UniqueName: \"kubernetes.io/projected/6d6fd4b2-ad0a-4ce3-b5b3-966de34bd8b0-kube-api-access-qdj6x\") pod \"maas-default-gateway-openshift-default-6db586dd6c-w44qs\" (UID: \"6d6fd4b2-ad0a-4ce3-b5b3-966de34bd8b0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-w44qs" Oct 30 20:48:35.472394 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:48:35.472372 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46ae03da-7a80-4f78-8abd-391ebc697f2f-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-8tvvl\" (UID: \"46ae03da-7a80-4f78-8abd-391ebc697f2f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8tvvl" Oct 30 20:48:35.472394 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:48:35.472397 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d6fd4b2-ad0a-4ce3-b5b3-966de34bd8b0-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-w44qs\" (UID: \"6d6fd4b2-ad0a-4ce3-b5b3-966de34bd8b0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-w44qs" Oct 30 20:48:35.472793 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:48:35.472418 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46ae03da-7a80-4f78-8abd-391ebc697f2f-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-8tvvl\" (UID: \"46ae03da-7a80-4f78-8abd-391ebc697f2f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8tvvl" Oct 30 20:48:35.472793 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:48:35.472444 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46ae03da-7a80-4f78-8abd-391ebc697f2f-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-8tvvl\" (UID: \"46ae03da-7a80-4f78-8abd-391ebc697f2f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8tvvl" Oct 30 20:48:35.472793 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:48:35.472470 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d6fd4b2-ad0a-4ce3-b5b3-966de34bd8b0-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-w44qs\" (UID: \"6d6fd4b2-ad0a-4ce3-b5b3-966de34bd8b0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-w44qs" Oct 30 20:48:35.472793 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:48:35.472493 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46ae03da-7a80-4f78-8abd-391ebc697f2f-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-8tvvl\" (UID: \"46ae03da-7a80-4f78-8abd-391ebc697f2f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8tvvl" Oct 30 20:48:35.472793 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:48:35.472590 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46ae03da-7a80-4f78-8abd-391ebc697f2f-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-8tvvl\" (UID: \"46ae03da-7a80-4f78-8abd-391ebc697f2f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8tvvl" Oct 30 20:48:35.472793 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:48:35.472613 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46ae03da-7a80-4f78-8abd-391ebc697f2f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-8tvvl\" (UID: \"46ae03da-7a80-4f78-8abd-391ebc697f2f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8tvvl" Oct 30 20:48:35.472793 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:48:35.472650 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d6fd4b2-ad0a-4ce3-b5b3-966de34bd8b0-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-w44qs\" (UID: \"6d6fd4b2-ad0a-4ce3-b5b3-966de34bd8b0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-w44qs" Oct 30 20:48:35.472793 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:48:35.472694 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d6fd4b2-ad0a-4ce3-b5b3-966de34bd8b0-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-w44qs\" (UID: \"6d6fd4b2-ad0a-4ce3-b5b3-966de34bd8b0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-w44qs" Oct 30 20:48:35.472793 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:48:35.472733 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d6fd4b2-ad0a-4ce3-b5b3-966de34bd8b0-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-w44qs\" (UID: \"6d6fd4b2-ad0a-4ce3-b5b3-966de34bd8b0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-w44qs" Oct 30 20:48:35.472793 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:48:35.472778 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d6fd4b2-ad0a-4ce3-b5b3-966de34bd8b0-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-w44qs\" (UID: \"6d6fd4b2-ad0a-4ce3-b5b3-966de34bd8b0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-w44qs" Oct 30 20:48:35.473292 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:48:35.472800 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46ae03da-7a80-4f78-8abd-391ebc697f2f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-8tvvl\" (UID: \"46ae03da-7a80-4f78-8abd-391ebc697f2f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8tvvl" Oct 30 20:48:35.473292 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:48:35.472811 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46ae03da-7a80-4f78-8abd-391ebc697f2f-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-8tvvl\" (UID: \"46ae03da-7a80-4f78-8abd-391ebc697f2f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8tvvl" Oct 30 20:48:35.473292 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:48:35.472836 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46ae03da-7a80-4f78-8abd-391ebc697f2f-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-8tvvl\" (UID: \"46ae03da-7a80-4f78-8abd-391ebc697f2f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8tvvl" Oct 30 20:48:35.473292 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:48:35.472844 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46ae03da-7a80-4f78-8abd-391ebc697f2f-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-8tvvl\" (UID: \"46ae03da-7a80-4f78-8abd-391ebc697f2f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8tvvl" Oct 30 20:48:35.473292 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:48:35.472891 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d6fd4b2-ad0a-4ce3-b5b3-966de34bd8b0-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-w44qs\" (UID: \"6d6fd4b2-ad0a-4ce3-b5b3-966de34bd8b0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-w44qs" Oct 30 20:48:35.473292 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:48:35.472898 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d6fd4b2-ad0a-4ce3-b5b3-966de34bd8b0-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-w44qs\" (UID: \"6d6fd4b2-ad0a-4ce3-b5b3-966de34bd8b0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-w44qs" Oct 30 20:48:35.473292 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:48:35.472955 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lvrsc\" (UniqueName: \"kubernetes.io/projected/46ae03da-7a80-4f78-8abd-391ebc697f2f-kube-api-access-lvrsc\") pod \"maas-default-gateway-openshift-default-6db586dd6c-8tvvl\" (UID: \"46ae03da-7a80-4f78-8abd-391ebc697f2f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8tvvl" Oct 30 20:48:35.473292 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:48:35.473117 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d6fd4b2-ad0a-4ce3-b5b3-966de34bd8b0-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-w44qs\" (UID: \"6d6fd4b2-ad0a-4ce3-b5b3-966de34bd8b0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-w44qs" Oct 30 20:48:35.473292 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:48:35.473146 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46ae03da-7a80-4f78-8abd-391ebc697f2f-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-8tvvl\" (UID: \"46ae03da-7a80-4f78-8abd-391ebc697f2f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8tvvl" Oct 30 20:48:35.473292 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:48:35.473157 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d6fd4b2-ad0a-4ce3-b5b3-966de34bd8b0-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-w44qs\" (UID: \"6d6fd4b2-ad0a-4ce3-b5b3-966de34bd8b0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-w44qs" Oct 30 20:48:35.473863 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:48:35.473366 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46ae03da-7a80-4f78-8abd-391ebc697f2f-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-8tvvl\" (UID: \"46ae03da-7a80-4f78-8abd-391ebc697f2f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8tvvl" Oct 30 20:48:35.473863 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:48:35.473481 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d6fd4b2-ad0a-4ce3-b5b3-966de34bd8b0-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-w44qs\" (UID: \"6d6fd4b2-ad0a-4ce3-b5b3-966de34bd8b0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-w44qs" Oct 30 20:48:35.473863 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:48:35.473772 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46ae03da-7a80-4f78-8abd-391ebc697f2f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-8tvvl\" (UID: \"46ae03da-7a80-4f78-8abd-391ebc697f2f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8tvvl" Oct 30 20:48:35.474332 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:48:35.474307 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d6fd4b2-ad0a-4ce3-b5b3-966de34bd8b0-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-w44qs\" (UID: \"6d6fd4b2-ad0a-4ce3-b5b3-966de34bd8b0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-w44qs" Oct 30 20:48:35.475325 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:48:35.475303 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46ae03da-7a80-4f78-8abd-391ebc697f2f-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-8tvvl\" (UID: \"46ae03da-7a80-4f78-8abd-391ebc697f2f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8tvvl" Oct 30 20:48:35.475734 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:48:35.475706 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d6fd4b2-ad0a-4ce3-b5b3-966de34bd8b0-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-w44qs\" (UID: \"6d6fd4b2-ad0a-4ce3-b5b3-966de34bd8b0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-w44qs" Oct 30 20:48:35.475917 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:48:35.475900 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46ae03da-7a80-4f78-8abd-391ebc697f2f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-8tvvl\" (UID: \"46ae03da-7a80-4f78-8abd-391ebc697f2f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8tvvl" Oct 30 20:48:35.475917 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:48:35.475908 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d6fd4b2-ad0a-4ce3-b5b3-966de34bd8b0-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-w44qs\" (UID: \"6d6fd4b2-ad0a-4ce3-b5b3-966de34bd8b0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-w44qs" Oct 30 20:48:35.489342 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:48:35.489319 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qdj6x\" (UniqueName: \"kubernetes.io/projected/6d6fd4b2-ad0a-4ce3-b5b3-966de34bd8b0-kube-api-access-qdj6x\") pod \"maas-default-gateway-openshift-default-6db586dd6c-w44qs\" (UID: \"6d6fd4b2-ad0a-4ce3-b5b3-966de34bd8b0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-w44qs" Oct 30 20:48:35.494664 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:48:35.494638 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d6fd4b2-ad0a-4ce3-b5b3-966de34bd8b0-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-w44qs\" (UID: \"6d6fd4b2-ad0a-4ce3-b5b3-966de34bd8b0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-w44qs" Oct 30 20:48:35.494858 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:48:35.494838 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lvrsc\" (UniqueName: \"kubernetes.io/projected/46ae03da-7a80-4f78-8abd-391ebc697f2f-kube-api-access-lvrsc\") pod \"maas-default-gateway-openshift-default-6db586dd6c-8tvvl\" (UID: \"46ae03da-7a80-4f78-8abd-391ebc697f2f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8tvvl" Oct 30 20:48:35.498091 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:48:35.498067 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46ae03da-7a80-4f78-8abd-391ebc697f2f-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-8tvvl\" (UID: \"46ae03da-7a80-4f78-8abd-391ebc697f2f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8tvvl" Oct 30 20:48:35.531376 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:48:35.531356 4611 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-w44qs" Oct 30 20:48:35.557907 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:48:35.557881 4611 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8tvvl" Oct 30 20:48:35.855764 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:48:35.855729 4611 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-w44qs"] Oct 30 20:48:35.861880 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:48:35.861851 4611 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d6fd4b2_ad0a_4ce3_b5b3_966de34bd8b0.slice/crio-3c4088239c57821501aa98ef69e2f7811ae80690b15b5844c65bdbd8df07c35a WatchSource:0}: Error finding container 3c4088239c57821501aa98ef69e2f7811ae80690b15b5844c65bdbd8df07c35a: Status 404 returned error can't find the container with id 3c4088239c57821501aa98ef69e2f7811ae80690b15b5844c65bdbd8df07c35a Oct 30 20:48:35.863732 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:48:35.863706 4611 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:48:35.863817 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:48:35.863765 4611 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:48:35.863817 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:48:35.863791 4611 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:48:35.889428 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:48:35.889405 4611 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8tvvl"] Oct 30 20:48:35.894868 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:48:35.894844 4611 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46ae03da_7a80_4f78_8abd_391ebc697f2f.slice/crio-61c7a3fe4fc7b7eb860da5962197197912379df9ccb1dc5e4034f16d6995cf25 WatchSource:0}: Error finding container 61c7a3fe4fc7b7eb860da5962197197912379df9ccb1dc5e4034f16d6995cf25: Status 404 returned error can't find the container with id 61c7a3fe4fc7b7eb860da5962197197912379df9ccb1dc5e4034f16d6995cf25 Oct 30 20:48:35.897167 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:48:35.897138 4611 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:48:35.897271 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:48:35.897209 4611 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:48:35.897271 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:48:35.897250 4611 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:48:35.970416 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:48:35.970379 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-w44qs" event={"ID":"6d6fd4b2-ad0a-4ce3-b5b3-966de34bd8b0","Type":"ContainerStarted","Data":"294c6631ef8999b7e72bd3c483e1a97ecc657e94a3d9ccc2c0234f5fbc212c1e"} Oct 30 20:48:35.970617 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:48:35.970440 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-w44qs" event={"ID":"6d6fd4b2-ad0a-4ce3-b5b3-966de34bd8b0","Type":"ContainerStarted","Data":"3c4088239c57821501aa98ef69e2f7811ae80690b15b5844c65bdbd8df07c35a"} Oct 30 20:48:35.972640 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:48:35.972607 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8tvvl" event={"ID":"46ae03da-7a80-4f78-8abd-391ebc697f2f","Type":"ContainerStarted","Data":"052ae2e95eed9bc6ddfcf31320811ce5a003836a7d00de45f65a7973d480d9bc"} Oct 30 20:48:35.972765 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:48:35.972665 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8tvvl" event={"ID":"46ae03da-7a80-4f78-8abd-391ebc697f2f","Type":"ContainerStarted","Data":"61c7a3fe4fc7b7eb860da5962197197912379df9ccb1dc5e4034f16d6995cf25"} Oct 30 20:48:35.979803 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:48:35.979772 4611 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7b8677f665-47n2d"] Oct 30 20:48:35.982704 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:48:35.982666 4611 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7b8677f665-47n2d" Oct 30 20:48:35.999838 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:48:35.994713 4611 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7b8677f665-47n2d"] Oct 30 20:48:36.008411 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:48:36.005729 4611 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-w44qs" podStartSLOduration=1.005712688 podStartE2EDuration="1.005712688s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:48:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:48:36.004031262 +0000 UTC m=+570.538057878" watchObservedRunningTime="2026-10-30 20:48:36.005712688 +0000 UTC m=+570.539739305" Oct 30 20:48:36.047663 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:48:36.047614 4611 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8tvvl" podStartSLOduration=1.04759799 podStartE2EDuration="1.04759799s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:48:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:48:36.045404857 +0000 UTC m=+570.579431475" watchObservedRunningTime="2026-10-30 20:48:36.04759799 +0000 UTC m=+570.581624606" Oct 30 20:48:36.079040 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:48:36.078787 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/6394d876-8759-4272-9351-9abed42ee4c8-tls-cert\") pod \"authorino-7b8677f665-47n2d\" (UID: \"6394d876-8759-4272-9351-9abed42ee4c8\") " pod="kuadrant-system/authorino-7b8677f665-47n2d" Oct 30 20:48:36.079040 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:48:36.078861 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kz82p\" (UniqueName: \"kubernetes.io/projected/6394d876-8759-4272-9351-9abed42ee4c8-kube-api-access-kz82p\") pod \"authorino-7b8677f665-47n2d\" (UID: \"6394d876-8759-4272-9351-9abed42ee4c8\") " pod="kuadrant-system/authorino-7b8677f665-47n2d" Oct 30 20:48:36.180314 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:48:36.180240 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/6394d876-8759-4272-9351-9abed42ee4c8-tls-cert\") pod \"authorino-7b8677f665-47n2d\" (UID: \"6394d876-8759-4272-9351-9abed42ee4c8\") " pod="kuadrant-system/authorino-7b8677f665-47n2d" Oct 30 20:48:36.180314 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:48:36.180274 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kz82p\" (UniqueName: \"kubernetes.io/projected/6394d876-8759-4272-9351-9abed42ee4c8-kube-api-access-kz82p\") pod \"authorino-7b8677f665-47n2d\" (UID: \"6394d876-8759-4272-9351-9abed42ee4c8\") " pod="kuadrant-system/authorino-7b8677f665-47n2d" Oct 30 20:48:36.182727 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:48:36.182703 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/6394d876-8759-4272-9351-9abed42ee4c8-tls-cert\") pod \"authorino-7b8677f665-47n2d\" (UID: \"6394d876-8759-4272-9351-9abed42ee4c8\") " pod="kuadrant-system/authorino-7b8677f665-47n2d" Oct 30 20:48:36.193583 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:48:36.193557 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kz82p\" (UniqueName: \"kubernetes.io/projected/6394d876-8759-4272-9351-9abed42ee4c8-kube-api-access-kz82p\") pod \"authorino-7b8677f665-47n2d\" (UID: \"6394d876-8759-4272-9351-9abed42ee4c8\") " pod="kuadrant-system/authorino-7b8677f665-47n2d" Oct 30 20:48:36.311277 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:48:36.311242 4611 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7b8677f665-47n2d" Oct 30 20:48:36.532087 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:48:36.532006 4611 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-w44qs" Oct 30 20:48:36.533550 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:48:36.533503 4611 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-w44qs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body= Oct 30 20:48:36.533679 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:48:36.533606 4611 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-w44qs" podUID="6d6fd4b2-ad0a-4ce3-b5b3-966de34bd8b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" Oct 30 20:48:36.572533 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:48:36.572152 4611 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8tvvl" Oct 30 20:48:36.572533 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:48:36.572285 4611 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-8tvvl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body= Oct 30 20:48:36.572533 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:48:36.572326 4611 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8tvvl" podUID="46ae03da-7a80-4f78-8abd-391ebc697f2f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" Oct 30 20:48:36.629069 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:48:36.628089 4611 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7b8677f665-47n2d"] Oct 30 20:48:36.630568 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:48:36.630535 4611 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6394d876_8759_4272_9351_9abed42ee4c8.slice/crio-9ba00f493ac189cb7021d01cc7613366f4b8f175fb84b96d0bc3a8337770f07c WatchSource:0}: Error finding container 9ba00f493ac189cb7021d01cc7613366f4b8f175fb84b96d0bc3a8337770f07c: Status 404 returned error can't find the container with id 9ba00f493ac189cb7021d01cc7613366f4b8f175fb84b96d0bc3a8337770f07c Oct 30 20:48:37.008581 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:48:37.008539 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7b8677f665-47n2d" event={"ID":"6394d876-8759-4272-9351-9abed42ee4c8","Type":"ContainerStarted","Data":"9ba00f493ac189cb7021d01cc7613366f4b8f175fb84b96d0bc3a8337770f07c"} Oct 30 20:48:37.532658 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:48:37.532622 4611 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-w44qs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body= Oct 30 20:48:37.533009 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:48:37.532676 4611 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-w44qs" podUID="6d6fd4b2-ad0a-4ce3-b5b3-966de34bd8b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" Oct 30 20:48:37.562530 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:48:37.558902 4611 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-8tvvl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body= Oct 30 20:48:37.562530 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:48:37.558960 4611 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8tvvl" podUID="46ae03da-7a80-4f78-8abd-391ebc697f2f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" Oct 30 20:48:38.016527 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:48:38.016091 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7b8677f665-47n2d" event={"ID":"6394d876-8759-4272-9351-9abed42ee4c8","Type":"ContainerStarted","Data":"7b6b4c1c3e31130d44cbad29d3a3334342197d48c3a55cff15882e5a7e4a8dc7"} Oct 30 20:48:38.035859 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:48:38.035796 4611 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-7b8677f665-47n2d" podStartSLOduration=2.09460291 podStartE2EDuration="3.035780311s" totalImagesPullingTime="941.177401ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:48:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:48:38.034638536 +0000 UTC m=+572.568665151" watchObservedRunningTime="2026-10-30 20:48:38.035780311 +0000 UTC m=+572.569806928" Oct 30 20:48:38.101533 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:48:38.101143 4611 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-8545cfcdb6-t86wv"] Oct 30 20:48:38.101533 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:48:38.101359 4611 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-8545cfcdb6-t86wv" podUID="501e279a-76c1-42f6-84e3-6337fd9c8181" containerName="authorino" containerID="cri-o://86879222b77d5d8c672f96a8a62d7f75a993ce55915900f6fc5092926556a88b" gracePeriod=30 Oct 30 20:48:38.464577 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:48:38.461666 4611 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-8545cfcdb6-t86wv" Oct 30 20:48:38.521377 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:48:38.521278 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/501e279a-76c1-42f6-84e3-6337fd9c8181-tls-cert\" (UniqueName: \"kubernetes.io/secret/501e279a-76c1-42f6-84e3-6337fd9c8181-tls-cert\") pod \"501e279a-76c1-42f6-84e3-6337fd9c8181\" (UID: \"501e279a-76c1-42f6-84e3-6337fd9c8181\") " Oct 30 20:48:38.521377 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:48:38.521374 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/501e279a-76c1-42f6-84e3-6337fd9c8181-kube-api-access-vj4pd\" (UniqueName: \"kubernetes.io/projected/501e279a-76c1-42f6-84e3-6337fd9c8181-kube-api-access-vj4pd\") pod \"501e279a-76c1-42f6-84e3-6337fd9c8181\" (UID: \"501e279a-76c1-42f6-84e3-6337fd9c8181\") " Oct 30 20:48:38.525793 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:48:38.524668 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/501e279a-76c1-42f6-84e3-6337fd9c8181-kube-api-access-vj4pd" pod "501e279a-76c1-42f6-84e3-6337fd9c8181" (UID: "501e279a-76c1-42f6-84e3-6337fd9c8181"). InnerVolumeSpecName "kube-api-access-vj4pd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:48:38.531781 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:48:38.531757 4611 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-w44qs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body= Oct 30 20:48:38.531905 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:48:38.531811 4611 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-w44qs" podUID="6d6fd4b2-ad0a-4ce3-b5b3-966de34bd8b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" Oct 30 20:48:38.532201 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:48:38.532179 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/501e279a-76c1-42f6-84e3-6337fd9c8181-tls-cert" pod "501e279a-76c1-42f6-84e3-6337fd9c8181" (UID: "501e279a-76c1-42f6-84e3-6337fd9c8181"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:48:38.563531 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:48:38.562920 4611 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-8tvvl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body= Oct 30 20:48:38.563531 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:48:38.562982 4611 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8tvvl" podUID="46ae03da-7a80-4f78-8abd-391ebc697f2f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" Oct 30 20:48:38.622828 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:48:38.622788 4611 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vj4pd\" (UniqueName: \"kubernetes.io/projected/501e279a-76c1-42f6-84e3-6337fd9c8181-kube-api-access-vj4pd\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:48:38.622828 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:48:38.622825 4611 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/501e279a-76c1-42f6-84e3-6337fd9c8181-tls-cert\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:48:38.806269 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:48:38.806234 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:48:39.016936 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:48:39.016904 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:48:39.022295 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:48:39.022241 4611 generic.go:356] "Generic (PLEG): container finished" podID="501e279a-76c1-42f6-84e3-6337fd9c8181" containerID="86879222b77d5d8c672f96a8a62d7f75a993ce55915900f6fc5092926556a88b" exitCode=0 Oct 30 20:48:39.022966 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:48:39.022935 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-8545cfcdb6-t86wv" event={"ID":"501e279a-76c1-42f6-84e3-6337fd9c8181","Type":"ContainerDied","Data":"86879222b77d5d8c672f96a8a62d7f75a993ce55915900f6fc5092926556a88b"} Oct 30 20:48:39.023145 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:48:39.023130 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-8545cfcdb6-t86wv" event={"ID":"501e279a-76c1-42f6-84e3-6337fd9c8181","Type":"ContainerDied","Data":"a5ba85985a416298f52b077079ad0e3d1a3318fcf127b2deeec070cfe364e825"} Oct 30 20:48:39.023325 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:48:39.023313 4611 scope.go:122] "RemoveContainer" containerID="86879222b77d5d8c672f96a8a62d7f75a993ce55915900f6fc5092926556a88b" Oct 30 20:48:39.023541 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:48:39.022972 4611 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-8545cfcdb6-t86wv" Oct 30 20:48:39.042759 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:48:39.042453 4611 scope.go:122] "RemoveContainer" containerID="86879222b77d5d8c672f96a8a62d7f75a993ce55915900f6fc5092926556a88b" Oct 30 20:48:39.042850 ip-10-0-133-192 kubenswrapper[4611]: E1030 20:48:39.042792 4611 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"86879222b77d5d8c672f96a8a62d7f75a993ce55915900f6fc5092926556a88b\": container with ID starting with 86879222b77d5d8c672f96a8a62d7f75a993ce55915900f6fc5092926556a88b not found: ID does not exist" containerID="86879222b77d5d8c672f96a8a62d7f75a993ce55915900f6fc5092926556a88b" Oct 30 20:48:39.042850 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:48:39.042823 4611 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"86879222b77d5d8c672f96a8a62d7f75a993ce55915900f6fc5092926556a88b"} err="failed to get container status \"86879222b77d5d8c672f96a8a62d7f75a993ce55915900f6fc5092926556a88b\": rpc error: code = NotFound desc = could not find container \"86879222b77d5d8c672f96a8a62d7f75a993ce55915900f6fc5092926556a88b\": container with ID starting with 86879222b77d5d8c672f96a8a62d7f75a993ce55915900f6fc5092926556a88b not found: ID does not exist" Oct 30 20:48:39.079636 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:48:39.079572 4611 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-8545cfcdb6-t86wv"] Oct 30 20:48:39.081692 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:48:39.081669 4611 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-8545cfcdb6-t86wv"] Oct 30 20:48:39.532732 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:48:39.532699 4611 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-w44qs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body= Oct 30 20:48:39.533216 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:48:39.532932 4611 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-w44qs" podUID="6d6fd4b2-ad0a-4ce3-b5b3-966de34bd8b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" Oct 30 20:48:39.562532 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:48:39.562100 4611 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-8tvvl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body= Oct 30 20:48:39.562532 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:48:39.562159 4611 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8tvvl" podUID="46ae03da-7a80-4f78-8abd-391ebc697f2f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" Oct 30 20:48:39.619201 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:48:39.618725 4611 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="501e279a-76c1-42f6-84e3-6337fd9c8181" path="/var/lib/kubelet/pods/501e279a-76c1-42f6-84e3-6337fd9c8181/volumes" Oct 30 20:48:40.531886 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:48:40.531855 4611 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-w44qs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body= Oct 30 20:48:40.532127 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:48:40.532105 4611 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-w44qs" podUID="6d6fd4b2-ad0a-4ce3-b5b3-966de34bd8b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" Oct 30 20:48:40.560996 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:48:40.559063 4611 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-8tvvl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body= Oct 30 20:48:40.560996 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:48:40.559120 4611 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8tvvl" podUID="46ae03da-7a80-4f78-8abd-391ebc697f2f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" Oct 30 20:48:41.532060 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:48:41.532019 4611 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-w44qs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body= Oct 30 20:48:41.532453 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:48:41.532080 4611 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-w44qs" podUID="6d6fd4b2-ad0a-4ce3-b5b3-966de34bd8b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" Oct 30 20:48:41.562541 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:48:41.561878 4611 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-8tvvl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body= Oct 30 20:48:41.562541 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:48:41.561937 4611 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8tvvl" podUID="46ae03da-7a80-4f78-8abd-391ebc697f2f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" Oct 30 20:48:42.532589 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:48:42.532545 4611 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-w44qs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body= Oct 30 20:48:42.532986 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:48:42.532619 4611 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-w44qs" podUID="6d6fd4b2-ad0a-4ce3-b5b3-966de34bd8b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" Oct 30 20:48:42.559299 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:48:42.559256 4611 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-8tvvl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body= Oct 30 20:48:42.559497 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:48:42.559337 4611 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8tvvl" podUID="46ae03da-7a80-4f78-8abd-391ebc697f2f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" Oct 30 20:48:43.534534 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:48:43.531208 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:48:43.534534 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:48:43.532310 4611 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-w44qs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body= Oct 30 20:48:43.534534 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:48:43.532379 4611 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-w44qs" podUID="6d6fd4b2-ad0a-4ce3-b5b3-966de34bd8b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" Oct 30 20:48:43.692959 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:48:43.692748 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:48:44.559742 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:48:44.559698 4611 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-8tvvl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:48:44.560180 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:48:44.559772 4611 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8tvvl" podUID="46ae03da-7a80-4f78-8abd-391ebc697f2f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:48:45.579686 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:48:45.579613 4611 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-w44qs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:48:45.580211 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:48:45.579685 4611 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-w44qs" podUID="6d6fd4b2-ad0a-4ce3-b5b3-966de34bd8b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:48:45.605453 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:48:45.604585 4611 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-8tvvl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:48:45.605453 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:48:45.604671 4611 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8tvvl" podUID="46ae03da-7a80-4f78-8abd-391ebc697f2f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:48:46.623685 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:48:46.623637 4611 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-w44qs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:48:46.624189 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:48:46.623705 4611 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-w44qs" podUID="6d6fd4b2-ad0a-4ce3-b5b3-966de34bd8b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:48:46.647255 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:48:46.646640 4611 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-8tvvl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:48:46.647255 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:48:46.646701 4611 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8tvvl" podUID="46ae03da-7a80-4f78-8abd-391ebc697f2f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:48:47.103546 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:48:47.103488 4611 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8tvvl" Oct 30 20:48:47.103913 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:48:47.103867 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8tvvl" Oct 30 20:48:47.104851 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:48:47.104816 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8tvvl" Oct 30 20:48:47.145694 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:48:47.145665 4611 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-w44qs" Oct 30 20:48:47.145989 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:48:47.145968 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-w44qs" Oct 30 20:48:47.146591 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:48:47.146570 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-w44qs" Oct 30 20:48:48.210050 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:48:48.209974 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:49:03.916278 ip-10-0-133-192 kubenswrapper[4611]: E1030 20:49:03.916243 4611 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hwzc6" containerName="extractor" Oct 30 20:49:05.678246 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:49:05.678219 4611 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7w5wz_7e2574c1-b59c-453c-9501-25ec13a1ccad/ovn-acl-logging/0.log" Oct 30 20:49:05.679241 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:49:05.679218 4611 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7w5wz_7e2574c1-b59c-453c-9501-25ec13a1ccad/ovn-acl-logging/0.log" Oct 30 20:49:18.627981 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:49:18.627941 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:49:19.179728 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:49:19.179687 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:49:20.859493 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:49:20.859462 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:49:21.023488 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:49:21.023451 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:49:22.664227 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:49:22.664190 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:49:22.857971 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:49:22.857933 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:49:33.916477 ip-10-0-133-192 kubenswrapper[4611]: E1030 20:49:33.916443 4611 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hwzc6" containerName="extractor" Oct 30 20:49:45.680769 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:49:45.680735 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:49:45.830289 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:49:45.830245 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:49:46.057663 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:49:46.057582 4611 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-hx7mv"] Oct 30 20:49:46.058047 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:49:46.058029 4611 cpu_manager.go:414] "containerMap: removing container" podUID="501e279a-76c1-42f6-84e3-6337fd9c8181" containerName="authorino" Oct 30 20:49:46.058157 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:49:46.058050 4611 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="501e279a-76c1-42f6-84e3-6337fd9c8181" containerName="authorino" Oct 30 20:49:46.058157 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:49:46.058131 4611 memory_manager.go:365] "RemoveStaleState removing state" podUID="501e279a-76c1-42f6-84e3-6337fd9c8181" containerName="authorino" Oct 30 20:49:46.061311 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:49:46.061288 4611 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-hx7mv" Oct 30 20:49:46.074468 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:49:46.074446 4611 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-f2ed1e-openshift-default-dockercfg-g7p8g\"" Oct 30 20:49:46.077424 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:49:46.077403 4611 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-hx7mv"] Oct 30 20:49:46.161889 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:49:46.161864 4611 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-6xtqj"] Oct 30 20:49:46.165376 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:49:46.165355 4611 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-6xtqj" Oct 30 20:49:46.171183 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:49:46.171164 4611 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-ad7ff0-openshift-default-dockercfg-pmg7l\"" Oct 30 20:49:46.173419 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:49:46.173398 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:49:46.181723 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:49:46.181702 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52f7cad0-6a73-4573-9789-59ac6df33da4-istio-token\") pod \"e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-hx7mv\" (UID: \"52f7cad0-6a73-4573-9789-59ac6df33da4\") " pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-hx7mv" Oct 30 20:49:46.181815 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:49:46.181755 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52f7cad0-6a73-4573-9789-59ac6df33da4-istio-podinfo\") pod \"e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-hx7mv\" (UID: \"52f7cad0-6a73-4573-9789-59ac6df33da4\") " pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-hx7mv" Oct 30 20:49:46.181858 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:49:46.181825 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7ktlz\" (UniqueName: \"kubernetes.io/projected/52f7cad0-6a73-4573-9789-59ac6df33da4-kube-api-access-7ktlz\") pod \"e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-hx7mv\" (UID: \"52f7cad0-6a73-4573-9789-59ac6df33da4\") " pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-hx7mv" Oct 30 20:49:46.181858 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:49:46.181854 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52f7cad0-6a73-4573-9789-59ac6df33da4-istio-envoy\") pod \"e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-hx7mv\" (UID: \"52f7cad0-6a73-4573-9789-59ac6df33da4\") " pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-hx7mv" Oct 30 20:49:46.181949 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:49:46.181877 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52f7cad0-6a73-4573-9789-59ac6df33da4-credential-socket\") pod \"e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-hx7mv\" (UID: \"52f7cad0-6a73-4573-9789-59ac6df33da4\") " pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-hx7mv" Oct 30 20:49:46.181949 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:49:46.181920 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52f7cad0-6a73-4573-9789-59ac6df33da4-istiod-ca-cert\") pod \"e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-hx7mv\" (UID: \"52f7cad0-6a73-4573-9789-59ac6df33da4\") " pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-hx7mv" Oct 30 20:49:46.182025 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:49:46.182007 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52f7cad0-6a73-4573-9789-59ac6df33da4-workload-certs\") pod \"e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-hx7mv\" (UID: \"52f7cad0-6a73-4573-9789-59ac6df33da4\") " pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-hx7mv" Oct 30 20:49:46.182090 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:49:46.182078 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52f7cad0-6a73-4573-9789-59ac6df33da4-istio-data\") pod \"e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-hx7mv\" (UID: \"52f7cad0-6a73-4573-9789-59ac6df33da4\") " pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-hx7mv" Oct 30 20:49:46.182160 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:49:46.182143 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52f7cad0-6a73-4573-9789-59ac6df33da4-workload-socket\") pod \"e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-hx7mv\" (UID: \"52f7cad0-6a73-4573-9789-59ac6df33da4\") " pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-hx7mv" Oct 30 20:49:46.193155 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:49:46.193136 4611 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-6xtqj"] Oct 30 20:49:46.274681 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:49:46.274651 4611 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-khv95"] Oct 30 20:49:46.278101 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:49:46.278083 4611 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-khv95" Oct 30 20:49:46.283448 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:49:46.283421 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52f7cad0-6a73-4573-9789-59ac6df33da4-istio-token\") pod \"e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-hx7mv\" (UID: \"52f7cad0-6a73-4573-9789-59ac6df33da4\") " pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-hx7mv" Oct 30 20:49:46.283569 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:49:46.283481 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63d294fd-5293-4752-9116-0aafa81d5e4f-istio-envoy\") pod \"e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-6xtqj\" (UID: \"63d294fd-5293-4752-9116-0aafa81d5e4f\") " pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-6xtqj" Oct 30 20:49:46.283569 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:49:46.283539 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52f7cad0-6a73-4573-9789-59ac6df33da4-istio-podinfo\") pod \"e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-hx7mv\" (UID: \"52f7cad0-6a73-4573-9789-59ac6df33da4\") " pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-hx7mv" Oct 30 20:49:46.283665 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:49:46.283579 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7ktlz\" (UniqueName: \"kubernetes.io/projected/52f7cad0-6a73-4573-9789-59ac6df33da4-kube-api-access-7ktlz\") pod \"e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-hx7mv\" (UID: \"52f7cad0-6a73-4573-9789-59ac6df33da4\") " pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-hx7mv" Oct 30 20:49:46.283665 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:49:46.283615 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52f7cad0-6a73-4573-9789-59ac6df33da4-istio-envoy\") pod \"e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-hx7mv\" (UID: \"52f7cad0-6a73-4573-9789-59ac6df33da4\") " pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-hx7mv" Oct 30 20:49:46.283665 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:49:46.283640 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63d294fd-5293-4752-9116-0aafa81d5e4f-workload-socket\") pod \"e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-6xtqj\" (UID: \"63d294fd-5293-4752-9116-0aafa81d5e4f\") " pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-6xtqj" Oct 30 20:49:46.283665 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:49:46.283661 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52f7cad0-6a73-4573-9789-59ac6df33da4-credential-socket\") pod \"e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-hx7mv\" (UID: \"52f7cad0-6a73-4573-9789-59ac6df33da4\") " pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-hx7mv" Oct 30 20:49:46.283869 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:49:46.283848 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52f7cad0-6a73-4573-9789-59ac6df33da4-istiod-ca-cert\") pod \"e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-hx7mv\" (UID: \"52f7cad0-6a73-4573-9789-59ac6df33da4\") " pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-hx7mv" Oct 30 20:49:46.283921 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:49:46.283893 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52f7cad0-6a73-4573-9789-59ac6df33da4-workload-certs\") pod \"e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-hx7mv\" (UID: \"52f7cad0-6a73-4573-9789-59ac6df33da4\") " pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-hx7mv" Oct 30 20:49:46.283978 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:49:46.283933 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63d294fd-5293-4752-9116-0aafa81d5e4f-istio-token\") pod \"e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-6xtqj\" (UID: \"63d294fd-5293-4752-9116-0aafa81d5e4f\") " pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-6xtqj" Oct 30 20:49:46.284034 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:49:46.284003 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wgzm5\" (UniqueName: \"kubernetes.io/projected/63d294fd-5293-4752-9116-0aafa81d5e4f-kube-api-access-wgzm5\") pod \"e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-6xtqj\" (UID: \"63d294fd-5293-4752-9116-0aafa81d5e4f\") " pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-6xtqj" Oct 30 20:49:46.284034 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:49:46.284019 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52f7cad0-6a73-4573-9789-59ac6df33da4-credential-socket\") pod \"e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-hx7mv\" (UID: \"52f7cad0-6a73-4573-9789-59ac6df33da4\") " pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-hx7mv" Oct 30 20:49:46.284148 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:49:46.284032 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63d294fd-5293-4752-9116-0aafa81d5e4f-istio-podinfo\") pod \"e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-6xtqj\" (UID: \"63d294fd-5293-4752-9116-0aafa81d5e4f\") " pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-6xtqj" Oct 30 20:49:46.284148 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:49:46.284123 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63d294fd-5293-4752-9116-0aafa81d5e4f-istio-data\") pod \"e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-6xtqj\" (UID: \"63d294fd-5293-4752-9116-0aafa81d5e4f\") " pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-6xtqj" Oct 30 20:49:46.284256 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:49:46.284186 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63d294fd-5293-4752-9116-0aafa81d5e4f-istiod-ca-cert\") pod \"e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-6xtqj\" (UID: \"63d294fd-5293-4752-9116-0aafa81d5e4f\") " pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-6xtqj" Oct 30 20:49:46.284256 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:49:46.284226 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52f7cad0-6a73-4573-9789-59ac6df33da4-istio-data\") pod \"e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-hx7mv\" (UID: \"52f7cad0-6a73-4573-9789-59ac6df33da4\") " pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-hx7mv" Oct 30 20:49:46.284256 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:49:46.284237 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52f7cad0-6a73-4573-9789-59ac6df33da4-workload-certs\") pod \"e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-hx7mv\" (UID: \"52f7cad0-6a73-4573-9789-59ac6df33da4\") " pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-hx7mv" Oct 30 20:49:46.284407 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:49:46.284255 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63d294fd-5293-4752-9116-0aafa81d5e4f-credential-socket\") pod \"e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-6xtqj\" (UID: \"63d294fd-5293-4752-9116-0aafa81d5e4f\") " pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-6xtqj" Oct 30 20:49:46.284407 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:49:46.284392 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52f7cad0-6a73-4573-9789-59ac6df33da4-workload-socket\") pod \"e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-hx7mv\" (UID: \"52f7cad0-6a73-4573-9789-59ac6df33da4\") " pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-hx7mv" Oct 30 20:49:46.284533 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:49:46.284432 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63d294fd-5293-4752-9116-0aafa81d5e4f-workload-certs\") pod \"e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-6xtqj\" (UID: \"63d294fd-5293-4752-9116-0aafa81d5e4f\") " pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-6xtqj" Oct 30 20:49:46.284603 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:49:46.284527 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52f7cad0-6a73-4573-9789-59ac6df33da4-istio-data\") pod \"e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-hx7mv\" (UID: \"52f7cad0-6a73-4573-9789-59ac6df33da4\") " pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-hx7mv" Oct 30 20:49:46.284791 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:49:46.284768 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52f7cad0-6a73-4573-9789-59ac6df33da4-istiod-ca-cert\") pod \"e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-hx7mv\" (UID: \"52f7cad0-6a73-4573-9789-59ac6df33da4\") " pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-hx7mv" Oct 30 20:49:46.284791 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:49:46.284781 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52f7cad0-6a73-4573-9789-59ac6df33da4-workload-socket\") pod \"e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-hx7mv\" (UID: \"52f7cad0-6a73-4573-9789-59ac6df33da4\") " pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-hx7mv" Oct 30 20:49:46.286259 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:49:46.286235 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52f7cad0-6a73-4573-9789-59ac6df33da4-istio-envoy\") pod \"e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-hx7mv\" (UID: \"52f7cad0-6a73-4573-9789-59ac6df33da4\") " pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-hx7mv" Oct 30 20:49:46.286349 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:49:46.286235 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52f7cad0-6a73-4573-9789-59ac6df33da4-istio-podinfo\") pod \"e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-hx7mv\" (UID: \"52f7cad0-6a73-4573-9789-59ac6df33da4\") " pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-hx7mv" Oct 30 20:49:46.291973 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:49:46.291955 4611 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-b8aee1-openshift-default-dockercfg-trpph\"" Oct 30 20:49:46.295092 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:49:46.295070 4611 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-khv95"] Oct 30 20:49:46.300699 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:49:46.300679 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52f7cad0-6a73-4573-9789-59ac6df33da4-istio-token\") pod \"e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-hx7mv\" (UID: \"52f7cad0-6a73-4573-9789-59ac6df33da4\") " pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-hx7mv" Oct 30 20:49:46.302244 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:49:46.302224 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7ktlz\" (UniqueName: \"kubernetes.io/projected/52f7cad0-6a73-4573-9789-59ac6df33da4-kube-api-access-7ktlz\") pod \"e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-hx7mv\" (UID: \"52f7cad0-6a73-4573-9789-59ac6df33da4\") " pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-hx7mv" Oct 30 20:49:46.373599 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:49:46.373574 4611 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-hx7mv" Oct 30 20:49:46.385145 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:49:46.385120 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/62c91622-5518-4dc2-9f08-102dd0fcfb84-istio-token\") pod \"e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-khv95\" (UID: \"62c91622-5518-4dc2-9f08-102dd0fcfb84\") " pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-khv95" Oct 30 20:49:46.385262 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:49:46.385207 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wgzm5\" (UniqueName: \"kubernetes.io/projected/63d294fd-5293-4752-9116-0aafa81d5e4f-kube-api-access-wgzm5\") pod \"e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-6xtqj\" (UID: \"63d294fd-5293-4752-9116-0aafa81d5e4f\") " pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-6xtqj" Oct 30 20:49:46.385262 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:49:46.385254 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63d294fd-5293-4752-9116-0aafa81d5e4f-istio-podinfo\") pod \"e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-6xtqj\" (UID: \"63d294fd-5293-4752-9116-0aafa81d5e4f\") " pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-6xtqj" Oct 30 20:49:46.385374 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:49:46.385285 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63d294fd-5293-4752-9116-0aafa81d5e4f-istio-data\") pod \"e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-6xtqj\" (UID: \"63d294fd-5293-4752-9116-0aafa81d5e4f\") " pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-6xtqj" Oct 30 20:49:46.385374 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:49:46.385311 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62c91622-5518-4dc2-9f08-102dd0fcfb84-workload-certs\") pod \"e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-khv95\" (UID: \"62c91622-5518-4dc2-9f08-102dd0fcfb84\") " pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-khv95" Oct 30 20:49:46.385374 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:49:46.385333 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62c91622-5518-4dc2-9f08-102dd0fcfb84-istiod-ca-cert\") pod \"e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-khv95\" (UID: \"62c91622-5518-4dc2-9f08-102dd0fcfb84\") " pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-khv95" Oct 30 20:49:46.385374 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:49:46.385369 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63d294fd-5293-4752-9116-0aafa81d5e4f-istiod-ca-cert\") pod \"e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-6xtqj\" (UID: \"63d294fd-5293-4752-9116-0aafa81d5e4f\") " pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-6xtqj" Oct 30 20:49:46.385624 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:49:46.385495 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63d294fd-5293-4752-9116-0aafa81d5e4f-credential-socket\") pod \"e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-6xtqj\" (UID: \"63d294fd-5293-4752-9116-0aafa81d5e4f\") " pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-6xtqj" Oct 30 20:49:46.385624 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:49:46.385548 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63d294fd-5293-4752-9116-0aafa81d5e4f-workload-certs\") pod \"e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-6xtqj\" (UID: \"63d294fd-5293-4752-9116-0aafa81d5e4f\") " pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-6xtqj" Oct 30 20:49:46.385624 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:49:46.385568 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62c91622-5518-4dc2-9f08-102dd0fcfb84-istio-envoy\") pod \"e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-khv95\" (UID: \"62c91622-5518-4dc2-9f08-102dd0fcfb84\") " pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-khv95" Oct 30 20:49:46.385624 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:49:46.385597 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cv2j9\" (UniqueName: \"kubernetes.io/projected/62c91622-5518-4dc2-9f08-102dd0fcfb84-kube-api-access-cv2j9\") pod \"e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-khv95\" (UID: \"62c91622-5518-4dc2-9f08-102dd0fcfb84\") " pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-khv95" Oct 30 20:49:46.385878 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:49:46.385627 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63d294fd-5293-4752-9116-0aafa81d5e4f-istio-envoy\") pod \"e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-6xtqj\" (UID: \"63d294fd-5293-4752-9116-0aafa81d5e4f\") " pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-6xtqj" Oct 30 20:49:46.385878 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:49:46.385644 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62c91622-5518-4dc2-9f08-102dd0fcfb84-istio-podinfo\") pod \"e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-khv95\" (UID: \"62c91622-5518-4dc2-9f08-102dd0fcfb84\") " pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-khv95" Oct 30 20:49:46.385878 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:49:46.385705 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63d294fd-5293-4752-9116-0aafa81d5e4f-workload-socket\") pod \"e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-6xtqj\" (UID: \"63d294fd-5293-4752-9116-0aafa81d5e4f\") " pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-6xtqj" Oct 30 20:49:46.385878 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:49:46.385716 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63d294fd-5293-4752-9116-0aafa81d5e4f-istio-data\") pod \"e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-6xtqj\" (UID: \"63d294fd-5293-4752-9116-0aafa81d5e4f\") " pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-6xtqj" Oct 30 20:49:46.386124 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:49:46.385952 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63d294fd-5293-4752-9116-0aafa81d5e4f-credential-socket\") pod \"e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-6xtqj\" (UID: \"63d294fd-5293-4752-9116-0aafa81d5e4f\") " pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-6xtqj" Oct 30 20:49:46.386124 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:49:46.386060 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63d294fd-5293-4752-9116-0aafa81d5e4f-workload-certs\") pod \"e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-6xtqj\" (UID: \"63d294fd-5293-4752-9116-0aafa81d5e4f\") " pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-6xtqj" Oct 30 20:49:46.386253 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:49:46.386231 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63d294fd-5293-4752-9116-0aafa81d5e4f-istiod-ca-cert\") pod \"e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-6xtqj\" (UID: \"63d294fd-5293-4752-9116-0aafa81d5e4f\") " pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-6xtqj" Oct 30 20:49:46.386311 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:49:46.386247 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63d294fd-5293-4752-9116-0aafa81d5e4f-workload-socket\") pod \"e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-6xtqj\" (UID: \"63d294fd-5293-4752-9116-0aafa81d5e4f\") " pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-6xtqj" Oct 30 20:49:46.386393 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:49:46.386371 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62c91622-5518-4dc2-9f08-102dd0fcfb84-credential-socket\") pod \"e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-khv95\" (UID: \"62c91622-5518-4dc2-9f08-102dd0fcfb84\") " pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-khv95" Oct 30 20:49:46.386455 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:49:46.386409 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62c91622-5518-4dc2-9f08-102dd0fcfb84-workload-socket\") pod \"e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-khv95\" (UID: \"62c91622-5518-4dc2-9f08-102dd0fcfb84\") " pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-khv95" Oct 30 20:49:46.386455 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:49:46.386442 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62c91622-5518-4dc2-9f08-102dd0fcfb84-istio-data\") pod \"e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-khv95\" (UID: \"62c91622-5518-4dc2-9f08-102dd0fcfb84\") " pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-khv95" Oct 30 20:49:46.386605 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:49:46.386470 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63d294fd-5293-4752-9116-0aafa81d5e4f-istio-token\") pod \"e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-6xtqj\" (UID: \"63d294fd-5293-4752-9116-0aafa81d5e4f\") " pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-6xtqj" Oct 30 20:49:46.387069 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:49:46.387036 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:49:46.388230 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:49:46.388209 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63d294fd-5293-4752-9116-0aafa81d5e4f-istio-podinfo\") pod \"e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-6xtqj\" (UID: \"63d294fd-5293-4752-9116-0aafa81d5e4f\") " pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-6xtqj" Oct 30 20:49:46.388363 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:49:46.388343 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63d294fd-5293-4752-9116-0aafa81d5e4f-istio-envoy\") pod \"e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-6xtqj\" (UID: \"63d294fd-5293-4752-9116-0aafa81d5e4f\") " pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-6xtqj" Oct 30 20:49:46.439038 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:49:46.439004 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wgzm5\" (UniqueName: \"kubernetes.io/projected/63d294fd-5293-4752-9116-0aafa81d5e4f-kube-api-access-wgzm5\") pod \"e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-6xtqj\" (UID: \"63d294fd-5293-4752-9116-0aafa81d5e4f\") " pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-6xtqj" Oct 30 20:49:46.443412 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:49:46.443387 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63d294fd-5293-4752-9116-0aafa81d5e4f-istio-token\") pod \"e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-6xtqj\" (UID: \"63d294fd-5293-4752-9116-0aafa81d5e4f\") " pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-6xtqj" Oct 30 20:49:46.478433 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:49:46.478406 4611 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-6xtqj" Oct 30 20:49:46.487697 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:49:46.487658 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62c91622-5518-4dc2-9f08-102dd0fcfb84-istio-envoy\") pod \"e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-khv95\" (UID: \"62c91622-5518-4dc2-9f08-102dd0fcfb84\") " pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-khv95" Oct 30 20:49:46.487837 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:49:46.487725 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cv2j9\" (UniqueName: \"kubernetes.io/projected/62c91622-5518-4dc2-9f08-102dd0fcfb84-kube-api-access-cv2j9\") pod \"e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-khv95\" (UID: \"62c91622-5518-4dc2-9f08-102dd0fcfb84\") " pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-khv95" Oct 30 20:49:46.487837 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:49:46.487771 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62c91622-5518-4dc2-9f08-102dd0fcfb84-istio-podinfo\") pod \"e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-khv95\" (UID: \"62c91622-5518-4dc2-9f08-102dd0fcfb84\") " pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-khv95" Oct 30 20:49:46.487837 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:49:46.487830 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62c91622-5518-4dc2-9f08-102dd0fcfb84-credential-socket\") pod \"e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-khv95\" (UID: \"62c91622-5518-4dc2-9f08-102dd0fcfb84\") " pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-khv95" Oct 30 20:49:46.488008 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:49:46.487855 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62c91622-5518-4dc2-9f08-102dd0fcfb84-workload-socket\") pod \"e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-khv95\" (UID: \"62c91622-5518-4dc2-9f08-102dd0fcfb84\") " pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-khv95" Oct 30 20:49:46.488008 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:49:46.487887 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62c91622-5518-4dc2-9f08-102dd0fcfb84-istio-data\") pod \"e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-khv95\" (UID: \"62c91622-5518-4dc2-9f08-102dd0fcfb84\") " pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-khv95" Oct 30 20:49:46.488008 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:49:46.487921 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/62c91622-5518-4dc2-9f08-102dd0fcfb84-istio-token\") pod \"e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-khv95\" (UID: \"62c91622-5518-4dc2-9f08-102dd0fcfb84\") " pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-khv95" Oct 30 20:49:46.488008 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:49:46.487982 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62c91622-5518-4dc2-9f08-102dd0fcfb84-workload-certs\") pod \"e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-khv95\" (UID: \"62c91622-5518-4dc2-9f08-102dd0fcfb84\") " pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-khv95" Oct 30 20:49:46.488223 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:49:46.488007 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62c91622-5518-4dc2-9f08-102dd0fcfb84-istiod-ca-cert\") pod \"e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-khv95\" (UID: \"62c91622-5518-4dc2-9f08-102dd0fcfb84\") " pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-khv95" Oct 30 20:49:46.488842 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:49:46.488815 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62c91622-5518-4dc2-9f08-102dd0fcfb84-istiod-ca-cert\") pod \"e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-khv95\" (UID: \"62c91622-5518-4dc2-9f08-102dd0fcfb84\") " pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-khv95" Oct 30 20:49:46.489371 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:49:46.489349 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62c91622-5518-4dc2-9f08-102dd0fcfb84-credential-socket\") pod \"e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-khv95\" (UID: \"62c91622-5518-4dc2-9f08-102dd0fcfb84\") " pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-khv95" Oct 30 20:49:46.489532 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:49:46.489489 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62c91622-5518-4dc2-9f08-102dd0fcfb84-istio-data\") pod \"e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-khv95\" (UID: \"62c91622-5518-4dc2-9f08-102dd0fcfb84\") " pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-khv95" Oct 30 20:49:46.489652 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:49:46.489628 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62c91622-5518-4dc2-9f08-102dd0fcfb84-workload-certs\") pod \"e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-khv95\" (UID: \"62c91622-5518-4dc2-9f08-102dd0fcfb84\") " pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-khv95" Oct 30 20:49:46.489811 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:49:46.489792 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62c91622-5518-4dc2-9f08-102dd0fcfb84-workload-socket\") pod \"e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-khv95\" (UID: \"62c91622-5518-4dc2-9f08-102dd0fcfb84\") " pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-khv95" Oct 30 20:49:46.490384 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:49:46.490357 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62c91622-5518-4dc2-9f08-102dd0fcfb84-istio-envoy\") pod \"e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-khv95\" (UID: \"62c91622-5518-4dc2-9f08-102dd0fcfb84\") " pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-khv95" Oct 30 20:49:46.492321 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:49:46.492294 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62c91622-5518-4dc2-9f08-102dd0fcfb84-istio-podinfo\") pod \"e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-khv95\" (UID: \"62c91622-5518-4dc2-9f08-102dd0fcfb84\") " pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-khv95" Oct 30 20:49:46.498344 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:49:46.498315 4611 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-5f78z"] Oct 30 20:49:46.503847 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:49:46.503819 4611 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-5f78z" Oct 30 20:49:46.519489 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:49:46.518804 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/62c91622-5518-4dc2-9f08-102dd0fcfb84-istio-token\") pod \"e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-khv95\" (UID: \"62c91622-5518-4dc2-9f08-102dd0fcfb84\") " pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-khv95" Oct 30 20:49:46.521430 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:49:46.521286 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cv2j9\" (UniqueName: \"kubernetes.io/projected/62c91622-5518-4dc2-9f08-102dd0fcfb84-kube-api-access-cv2j9\") pod \"e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-khv95\" (UID: \"62c91622-5518-4dc2-9f08-102dd0fcfb84\") " pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-khv95" Oct 30 20:49:46.531039 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:49:46.529729 4611 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-db534b-openshift-default-dockercfg-4psrd\"" Oct 30 20:49:46.531039 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:49:46.530967 4611 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-5f78z"] Oct 30 20:49:46.588492 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:49:46.588466 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06064c28-fd6d-4765-8635-8690665a9b83-workload-certs\") pod \"e2e-worker-w2-db534b-openshift-default-85df4d79d9-5f78z\" (UID: \"06064c28-fd6d-4765-8635-8690665a9b83\") " pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-5f78z" Oct 30 20:49:46.588626 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:49:46.588517 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06064c28-fd6d-4765-8635-8690665a9b83-istio-podinfo\") pod \"e2e-worker-w2-db534b-openshift-default-85df4d79d9-5f78z\" (UID: \"06064c28-fd6d-4765-8635-8690665a9b83\") " pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-5f78z" Oct 30 20:49:46.588626 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:49:46.588606 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/06064c28-fd6d-4765-8635-8690665a9b83-istio-token\") pod \"e2e-worker-w2-db534b-openshift-default-85df4d79d9-5f78z\" (UID: \"06064c28-fd6d-4765-8635-8690665a9b83\") " pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-5f78z" Oct 30 20:49:46.588707 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:49:46.588666 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zls8x\" (UniqueName: \"kubernetes.io/projected/06064c28-fd6d-4765-8635-8690665a9b83-kube-api-access-zls8x\") pod \"e2e-worker-w2-db534b-openshift-default-85df4d79d9-5f78z\" (UID: \"06064c28-fd6d-4765-8635-8690665a9b83\") " pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-5f78z" Oct 30 20:49:46.588749 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:49:46.588740 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06064c28-fd6d-4765-8635-8690665a9b83-workload-socket\") pod \"e2e-worker-w2-db534b-openshift-default-85df4d79d9-5f78z\" (UID: \"06064c28-fd6d-4765-8635-8690665a9b83\") " pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-5f78z" Oct 30 20:49:46.588831 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:49:46.588815 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06064c28-fd6d-4765-8635-8690665a9b83-credential-socket\") pod \"e2e-worker-w2-db534b-openshift-default-85df4d79d9-5f78z\" (UID: \"06064c28-fd6d-4765-8635-8690665a9b83\") " pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-5f78z" Oct 30 20:49:46.588868 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:49:46.588839 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06064c28-fd6d-4765-8635-8690665a9b83-istiod-ca-cert\") pod \"e2e-worker-w2-db534b-openshift-default-85df4d79d9-5f78z\" (UID: \"06064c28-fd6d-4765-8635-8690665a9b83\") " pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-5f78z" Oct 30 20:49:46.588868 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:49:46.588862 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06064c28-fd6d-4765-8635-8690665a9b83-istio-envoy\") pod \"e2e-worker-w2-db534b-openshift-default-85df4d79d9-5f78z\" (UID: \"06064c28-fd6d-4765-8635-8690665a9b83\") " pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-5f78z" Oct 30 20:49:46.588948 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:49:46.588882 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06064c28-fd6d-4765-8635-8690665a9b83-istio-data\") pod \"e2e-worker-w2-db534b-openshift-default-85df4d79d9-5f78z\" (UID: \"06064c28-fd6d-4765-8635-8690665a9b83\") " pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-5f78z" Oct 30 20:49:46.592764 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:49:46.592736 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:49:46.620600 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:49:46.620575 4611 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-k5p2c"] Oct 30 20:49:46.623152 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:49:46.623128 4611 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-khv95" Oct 30 20:49:46.624626 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:49:46.624586 4611 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-k5p2c" Oct 30 20:49:46.652634 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:49:46.647389 4611 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-k5p2c"] Oct 30 20:49:46.663331 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:49:46.663308 4611 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-14cca2-openshift-default-dockercfg-559j6\"" Oct 30 20:49:46.663531 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:49:46.663479 4611 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-hx7mv"] Oct 30 20:49:46.665488 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:49:46.665456 4611 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52f7cad0_6a73_4573_9789_59ac6df33da4.slice/crio-b99d8fe48920544fea38344e40bb203fd45545c58d9ba322cd794a53f7760182 WatchSource:0}: Error finding container b99d8fe48920544fea38344e40bb203fd45545c58d9ba322cd794a53f7760182: Status 404 returned error can't find the container with id b99d8fe48920544fea38344e40bb203fd45545c58d9ba322cd794a53f7760182 Oct 30 20:49:46.667212 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:49:46.667182 4611 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:49:46.667312 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:49:46.667253 4611 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:49:46.667312 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:49:46.667293 4611 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:49:46.690726 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:49:46.690688 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zls8x\" (UniqueName: \"kubernetes.io/projected/06064c28-fd6d-4765-8635-8690665a9b83-kube-api-access-zls8x\") pod \"e2e-worker-w2-db534b-openshift-default-85df4d79d9-5f78z\" (UID: \"06064c28-fd6d-4765-8635-8690665a9b83\") " pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-5f78z" Oct 30 20:49:46.691297 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:49:46.691272 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06064c28-fd6d-4765-8635-8690665a9b83-workload-socket\") pod \"e2e-worker-w2-db534b-openshift-default-85df4d79d9-5f78z\" (UID: \"06064c28-fd6d-4765-8635-8690665a9b83\") " pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-5f78z" Oct 30 20:49:46.692155 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:49:46.692133 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06064c28-fd6d-4765-8635-8690665a9b83-credential-socket\") pod \"e2e-worker-w2-db534b-openshift-default-85df4d79d9-5f78z\" (UID: \"06064c28-fd6d-4765-8635-8690665a9b83\") " pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-5f78z" Oct 30 20:49:46.693048 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:49:46.693026 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06064c28-fd6d-4765-8635-8690665a9b83-istiod-ca-cert\") pod \"e2e-worker-w2-db534b-openshift-default-85df4d79d9-5f78z\" (UID: \"06064c28-fd6d-4765-8635-8690665a9b83\") " pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-5f78z" Oct 30 20:49:46.693157 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:49:46.693076 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06064c28-fd6d-4765-8635-8690665a9b83-istio-envoy\") pod \"e2e-worker-w2-db534b-openshift-default-85df4d79d9-5f78z\" (UID: \"06064c28-fd6d-4765-8635-8690665a9b83\") " pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-5f78z" Oct 30 20:49:46.693157 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:49:46.693100 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06064c28-fd6d-4765-8635-8690665a9b83-istio-data\") pod \"e2e-worker-w2-db534b-openshift-default-85df4d79d9-5f78z\" (UID: \"06064c28-fd6d-4765-8635-8690665a9b83\") " pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-5f78z" Oct 30 20:49:46.693157 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:49:46.693137 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06064c28-fd6d-4765-8635-8690665a9b83-workload-certs\") pod \"e2e-worker-w2-db534b-openshift-default-85df4d79d9-5f78z\" (UID: \"06064c28-fd6d-4765-8635-8690665a9b83\") " pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-5f78z" Oct 30 20:49:46.693316 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:49:46.693165 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06064c28-fd6d-4765-8635-8690665a9b83-istio-podinfo\") pod \"e2e-worker-w2-db534b-openshift-default-85df4d79d9-5f78z\" (UID: \"06064c28-fd6d-4765-8635-8690665a9b83\") " pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-5f78z" Oct 30 20:49:46.693316 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:49:46.693243 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/06064c28-fd6d-4765-8635-8690665a9b83-istio-token\") pod \"e2e-worker-w2-db534b-openshift-default-85df4d79d9-5f78z\" (UID: \"06064c28-fd6d-4765-8635-8690665a9b83\") " pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-5f78z" Oct 30 20:49:46.693825 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:49:46.693748 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06064c28-fd6d-4765-8635-8690665a9b83-istiod-ca-cert\") pod \"e2e-worker-w2-db534b-openshift-default-85df4d79d9-5f78z\" (UID: \"06064c28-fd6d-4765-8635-8690665a9b83\") " pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-5f78z" Oct 30 20:49:46.694044 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:49:46.693999 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06064c28-fd6d-4765-8635-8690665a9b83-istio-data\") pod \"e2e-worker-w2-db534b-openshift-default-85df4d79d9-5f78z\" (UID: \"06064c28-fd6d-4765-8635-8690665a9b83\") " pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-5f78z" Oct 30 20:49:46.694408 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:49:46.694364 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06064c28-fd6d-4765-8635-8690665a9b83-workload-certs\") pod \"e2e-worker-w2-db534b-openshift-default-85df4d79d9-5f78z\" (UID: \"06064c28-fd6d-4765-8635-8690665a9b83\") " pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-5f78z" Oct 30 20:49:46.694536 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:49:46.694488 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06064c28-fd6d-4765-8635-8690665a9b83-workload-socket\") pod \"e2e-worker-w2-db534b-openshift-default-85df4d79d9-5f78z\" (UID: \"06064c28-fd6d-4765-8635-8690665a9b83\") " pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-5f78z" Oct 30 20:49:46.694864 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:49:46.692868 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06064c28-fd6d-4765-8635-8690665a9b83-credential-socket\") pod \"e2e-worker-w2-db534b-openshift-default-85df4d79d9-5f78z\" (UID: \"06064c28-fd6d-4765-8635-8690665a9b83\") " pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-5f78z" Oct 30 20:49:46.698639 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:49:46.698614 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06064c28-fd6d-4765-8635-8690665a9b83-istio-podinfo\") pod \"e2e-worker-w2-db534b-openshift-default-85df4d79d9-5f78z\" (UID: \"06064c28-fd6d-4765-8635-8690665a9b83\") " pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-5f78z" Oct 30 20:49:46.699713 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:49:46.699689 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06064c28-fd6d-4765-8635-8690665a9b83-istio-envoy\") pod \"e2e-worker-w2-db534b-openshift-default-85df4d79d9-5f78z\" (UID: \"06064c28-fd6d-4765-8635-8690665a9b83\") " pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-5f78z" Oct 30 20:49:46.716443 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:49:46.716412 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zls8x\" (UniqueName: \"kubernetes.io/projected/06064c28-fd6d-4765-8635-8690665a9b83-kube-api-access-zls8x\") pod \"e2e-worker-w2-db534b-openshift-default-85df4d79d9-5f78z\" (UID: \"06064c28-fd6d-4765-8635-8690665a9b83\") " pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-5f78z" Oct 30 20:49:46.716661 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:49:46.716641 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/06064c28-fd6d-4765-8635-8690665a9b83-istio-token\") pod \"e2e-worker-w2-db534b-openshift-default-85df4d79d9-5f78z\" (UID: \"06064c28-fd6d-4765-8635-8690665a9b83\") " pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-5f78z" Oct 30 20:49:46.721939 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:49:46.721917 4611 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-trngw"] Oct 30 20:49:46.730679 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:49:46.730660 4611 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-trngw" Oct 30 20:49:46.748280 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:49:46.748254 4611 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-trngw"] Oct 30 20:49:46.769869 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:49:46.769838 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:49:46.774204 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:49:46.774177 4611 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-3d909d-openshift-default-dockercfg-727tj\"" Oct 30 20:49:46.786677 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:49:46.786652 4611 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-6xtqj"] Oct 30 20:49:46.789769 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:49:46.789734 4611 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63d294fd_5293_4752_9116_0aafa81d5e4f.slice/crio-6ebdf9bc66c8d06ec5e5175d22164b5a4a0daa22f36a73ca3d6874bf5ef0ebf2 WatchSource:0}: Error finding container 6ebdf9bc66c8d06ec5e5175d22164b5a4a0daa22f36a73ca3d6874bf5ef0ebf2: Status 404 returned error can't find the container with id 6ebdf9bc66c8d06ec5e5175d22164b5a4a0daa22f36a73ca3d6874bf5ef0ebf2 Oct 30 20:49:46.792479 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:49:46.792452 4611 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:49:46.792563 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:49:46.792535 4611 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:49:46.792614 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:49:46.792567 4611 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:49:46.794987 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:49:46.794911 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85895a05-5edd-47b7-bd4e-8112526dd4d8-credential-socket\") pod \"e2e-worker-w5-14cca2-openshift-default-d7f87db59-k5p2c\" (UID: \"85895a05-5edd-47b7-bd4e-8112526dd4d8\") " pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-k5p2c" Oct 30 20:49:46.794987 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:49:46.794947 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85895a05-5edd-47b7-bd4e-8112526dd4d8-istio-envoy\") pod \"e2e-worker-w5-14cca2-openshift-default-d7f87db59-k5p2c\" (UID: \"85895a05-5edd-47b7-bd4e-8112526dd4d8\") " pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-k5p2c" Oct 30 20:49:46.794987 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:49:46.794979 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85895a05-5edd-47b7-bd4e-8112526dd4d8-istio-data\") pod \"e2e-worker-w5-14cca2-openshift-default-d7f87db59-k5p2c\" (UID: \"85895a05-5edd-47b7-bd4e-8112526dd4d8\") " pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-k5p2c" Oct 30 20:49:46.795100 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:49:46.795042 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85895a05-5edd-47b7-bd4e-8112526dd4d8-istio-podinfo\") pod \"e2e-worker-w5-14cca2-openshift-default-d7f87db59-k5p2c\" (UID: \"85895a05-5edd-47b7-bd4e-8112526dd4d8\") " pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-k5p2c" Oct 30 20:49:46.795193 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:49:46.795115 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85895a05-5edd-47b7-bd4e-8112526dd4d8-workload-socket\") pod \"e2e-worker-w5-14cca2-openshift-default-d7f87db59-k5p2c\" (UID: \"85895a05-5edd-47b7-bd4e-8112526dd4d8\") " pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-k5p2c" Oct 30 20:49:46.795193 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:49:46.795161 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cxx86\" (UniqueName: \"kubernetes.io/projected/85895a05-5edd-47b7-bd4e-8112526dd4d8-kube-api-access-cxx86\") pod \"e2e-worker-w5-14cca2-openshift-default-d7f87db59-k5p2c\" (UID: \"85895a05-5edd-47b7-bd4e-8112526dd4d8\") " pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-k5p2c" Oct 30 20:49:46.795254 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:49:46.795189 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85895a05-5edd-47b7-bd4e-8112526dd4d8-workload-certs\") pod \"e2e-worker-w5-14cca2-openshift-default-d7f87db59-k5p2c\" (UID: \"85895a05-5edd-47b7-bd4e-8112526dd4d8\") " pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-k5p2c" Oct 30 20:49:46.795375 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:49:46.795348 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85895a05-5edd-47b7-bd4e-8112526dd4d8-istiod-ca-cert\") pod \"e2e-worker-w5-14cca2-openshift-default-d7f87db59-k5p2c\" (UID: \"85895a05-5edd-47b7-bd4e-8112526dd4d8\") " pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-k5p2c" Oct 30 20:49:46.795419 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:49:46.795396 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/85895a05-5edd-47b7-bd4e-8112526dd4d8-istio-token\") pod \"e2e-worker-w5-14cca2-openshift-default-d7f87db59-k5p2c\" (UID: \"85895a05-5edd-47b7-bd4e-8112526dd4d8\") " pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-k5p2c" Oct 30 20:49:46.833589 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:49:46.833559 4611 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-5f78z" Oct 30 20:49:46.899824 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:49:46.899794 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ba753c3-5979-4263-8b90-2741747c8592-workload-socket\") pod \"e2e-worker-w6-3d909d-openshift-default-77787695b7-trngw\" (UID: \"5ba753c3-5979-4263-8b90-2741747c8592\") " pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-trngw" Oct 30 20:49:46.899950 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:49:46.899847 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ba753c3-5979-4263-8b90-2741747c8592-credential-socket\") pod \"e2e-worker-w6-3d909d-openshift-default-77787695b7-trngw\" (UID: \"5ba753c3-5979-4263-8b90-2741747c8592\") " pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-trngw" Oct 30 20:49:46.899950 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:49:46.899913 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85895a05-5edd-47b7-bd4e-8112526dd4d8-istiod-ca-cert\") pod \"e2e-worker-w5-14cca2-openshift-default-d7f87db59-k5p2c\" (UID: \"85895a05-5edd-47b7-bd4e-8112526dd4d8\") " pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-k5p2c" Oct 30 20:49:46.899950 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:49:46.899939 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ba753c3-5979-4263-8b90-2741747c8592-istio-envoy\") pod \"e2e-worker-w6-3d909d-openshift-default-77787695b7-trngw\" (UID: \"5ba753c3-5979-4263-8b90-2741747c8592\") " pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-trngw" Oct 30 20:49:46.900109 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:49:46.899977 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/85895a05-5edd-47b7-bd4e-8112526dd4d8-istio-token\") pod \"e2e-worker-w5-14cca2-openshift-default-d7f87db59-k5p2c\" (UID: \"85895a05-5edd-47b7-bd4e-8112526dd4d8\") " pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-k5p2c" Oct 30 20:49:46.900109 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:49:46.900046 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ba753c3-5979-4263-8b90-2741747c8592-workload-certs\") pod \"e2e-worker-w6-3d909d-openshift-default-77787695b7-trngw\" (UID: \"5ba753c3-5979-4263-8b90-2741747c8592\") " pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-trngw" Oct 30 20:49:46.900109 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:49:46.900091 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ba753c3-5979-4263-8b90-2741747c8592-istio-podinfo\") pod \"e2e-worker-w6-3d909d-openshift-default-77787695b7-trngw\" (UID: \"5ba753c3-5979-4263-8b90-2741747c8592\") " pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-trngw" Oct 30 20:49:46.900269 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:49:46.900140 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85895a05-5edd-47b7-bd4e-8112526dd4d8-credential-socket\") pod \"e2e-worker-w5-14cca2-openshift-default-d7f87db59-k5p2c\" (UID: \"85895a05-5edd-47b7-bd4e-8112526dd4d8\") " pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-k5p2c" Oct 30 20:49:46.900269 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:49:46.900168 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85895a05-5edd-47b7-bd4e-8112526dd4d8-istio-envoy\") pod \"e2e-worker-w5-14cca2-openshift-default-d7f87db59-k5p2c\" (UID: \"85895a05-5edd-47b7-bd4e-8112526dd4d8\") " pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-k5p2c" Oct 30 20:49:46.900269 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:49:46.900193 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8j5dq\" (UniqueName: \"kubernetes.io/projected/5ba753c3-5979-4263-8b90-2741747c8592-kube-api-access-8j5dq\") pod \"e2e-worker-w6-3d909d-openshift-default-77787695b7-trngw\" (UID: \"5ba753c3-5979-4263-8b90-2741747c8592\") " pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-trngw" Oct 30 20:49:46.900269 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:49:46.900225 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ba753c3-5979-4263-8b90-2741747c8592-istiod-ca-cert\") pod \"e2e-worker-w6-3d909d-openshift-default-77787695b7-trngw\" (UID: \"5ba753c3-5979-4263-8b90-2741747c8592\") " pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-trngw" Oct 30 20:49:46.900269 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:49:46.900255 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85895a05-5edd-47b7-bd4e-8112526dd4d8-istio-data\") pod \"e2e-worker-w5-14cca2-openshift-default-d7f87db59-k5p2c\" (UID: \"85895a05-5edd-47b7-bd4e-8112526dd4d8\") " pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-k5p2c" Oct 30 20:49:46.900555 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:49:46.900280 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85895a05-5edd-47b7-bd4e-8112526dd4d8-istio-podinfo\") pod \"e2e-worker-w5-14cca2-openshift-default-d7f87db59-k5p2c\" (UID: \"85895a05-5edd-47b7-bd4e-8112526dd4d8\") " pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-k5p2c" Oct 30 20:49:46.900555 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:49:46.900314 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85895a05-5edd-47b7-bd4e-8112526dd4d8-workload-socket\") pod \"e2e-worker-w5-14cca2-openshift-default-d7f87db59-k5p2c\" (UID: \"85895a05-5edd-47b7-bd4e-8112526dd4d8\") " pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-k5p2c" Oct 30 20:49:46.900555 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:49:46.900357 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ba753c3-5979-4263-8b90-2741747c8592-istio-data\") pod \"e2e-worker-w6-3d909d-openshift-default-77787695b7-trngw\" (UID: \"5ba753c3-5979-4263-8b90-2741747c8592\") " pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-trngw" Oct 30 20:49:46.900555 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:49:46.900387 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cxx86\" (UniqueName: \"kubernetes.io/projected/85895a05-5edd-47b7-bd4e-8112526dd4d8-kube-api-access-cxx86\") pod \"e2e-worker-w5-14cca2-openshift-default-d7f87db59-k5p2c\" (UID: \"85895a05-5edd-47b7-bd4e-8112526dd4d8\") " pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-k5p2c" Oct 30 20:49:46.900555 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:49:46.900417 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85895a05-5edd-47b7-bd4e-8112526dd4d8-workload-certs\") pod \"e2e-worker-w5-14cca2-openshift-default-d7f87db59-k5p2c\" (UID: \"85895a05-5edd-47b7-bd4e-8112526dd4d8\") " pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-k5p2c" Oct 30 20:49:46.900555 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:49:46.900455 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ba753c3-5979-4263-8b90-2741747c8592-istio-token\") pod \"e2e-worker-w6-3d909d-openshift-default-77787695b7-trngw\" (UID: \"5ba753c3-5979-4263-8b90-2741747c8592\") " pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-trngw" Oct 30 20:49:46.900900 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:49:46.900874 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85895a05-5edd-47b7-bd4e-8112526dd4d8-credential-socket\") pod \"e2e-worker-w5-14cca2-openshift-default-d7f87db59-k5p2c\" (UID: \"85895a05-5edd-47b7-bd4e-8112526dd4d8\") " pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-k5p2c" Oct 30 20:49:46.919423 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:49:46.902033 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85895a05-5edd-47b7-bd4e-8112526dd4d8-istiod-ca-cert\") pod \"e2e-worker-w5-14cca2-openshift-default-d7f87db59-k5p2c\" (UID: \"85895a05-5edd-47b7-bd4e-8112526dd4d8\") " pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-k5p2c" Oct 30 20:49:46.919423 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:49:46.902297 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85895a05-5edd-47b7-bd4e-8112526dd4d8-istio-data\") pod \"e2e-worker-w5-14cca2-openshift-default-d7f87db59-k5p2c\" (UID: \"85895a05-5edd-47b7-bd4e-8112526dd4d8\") " pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-k5p2c" Oct 30 20:49:46.919423 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:49:46.902548 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85895a05-5edd-47b7-bd4e-8112526dd4d8-workload-socket\") pod \"e2e-worker-w5-14cca2-openshift-default-d7f87db59-k5p2c\" (UID: \"85895a05-5edd-47b7-bd4e-8112526dd4d8\") " pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-k5p2c" Oct 30 20:49:46.919423 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:49:46.903022 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85895a05-5edd-47b7-bd4e-8112526dd4d8-workload-certs\") pod \"e2e-worker-w5-14cca2-openshift-default-d7f87db59-k5p2c\" (UID: \"85895a05-5edd-47b7-bd4e-8112526dd4d8\") " pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-k5p2c" Oct 30 20:49:46.919423 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:49:46.913686 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85895a05-5edd-47b7-bd4e-8112526dd4d8-istio-envoy\") pod \"e2e-worker-w5-14cca2-openshift-default-d7f87db59-k5p2c\" (UID: \"85895a05-5edd-47b7-bd4e-8112526dd4d8\") " pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-k5p2c" Oct 30 20:49:46.919423 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:49:46.914266 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/85895a05-5edd-47b7-bd4e-8112526dd4d8-istio-token\") pod \"e2e-worker-w5-14cca2-openshift-default-d7f87db59-k5p2c\" (UID: \"85895a05-5edd-47b7-bd4e-8112526dd4d8\") " pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-k5p2c" Oct 30 20:49:46.919423 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:49:46.918764 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cxx86\" (UniqueName: \"kubernetes.io/projected/85895a05-5edd-47b7-bd4e-8112526dd4d8-kube-api-access-cxx86\") pod \"e2e-worker-w5-14cca2-openshift-default-d7f87db59-k5p2c\" (UID: \"85895a05-5edd-47b7-bd4e-8112526dd4d8\") " pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-k5p2c" Oct 30 20:49:46.920919 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:49:46.920727 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:49:46.921532 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:49:46.921493 4611 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-khv95"] Oct 30 20:49:46.931839 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:49:46.931720 4611 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:49:46.932044 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:49:46.931968 4611 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:49:46.932230 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:49:46.932171 4611 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:49:46.934175 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:49:46.934020 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85895a05-5edd-47b7-bd4e-8112526dd4d8-istio-podinfo\") pod \"e2e-worker-w5-14cca2-openshift-default-d7f87db59-k5p2c\" (UID: \"85895a05-5edd-47b7-bd4e-8112526dd4d8\") " pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-k5p2c" Oct 30 20:49:46.941202 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:49:46.940943 4611 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-k5p2c" Oct 30 20:49:47.006348 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:49:47.002999 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ba753c3-5979-4263-8b90-2741747c8592-istio-podinfo\") pod \"e2e-worker-w6-3d909d-openshift-default-77787695b7-trngw\" (UID: \"5ba753c3-5979-4263-8b90-2741747c8592\") " pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-trngw" Oct 30 20:49:47.006348 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:49:47.003063 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8j5dq\" (UniqueName: \"kubernetes.io/projected/5ba753c3-5979-4263-8b90-2741747c8592-kube-api-access-8j5dq\") pod \"e2e-worker-w6-3d909d-openshift-default-77787695b7-trngw\" (UID: \"5ba753c3-5979-4263-8b90-2741747c8592\") " pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-trngw" Oct 30 20:49:47.006348 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:49:47.003096 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ba753c3-5979-4263-8b90-2741747c8592-istiod-ca-cert\") pod \"e2e-worker-w6-3d909d-openshift-default-77787695b7-trngw\" (UID: \"5ba753c3-5979-4263-8b90-2741747c8592\") " pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-trngw" Oct 30 20:49:47.006348 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:49:47.003143 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ba753c3-5979-4263-8b90-2741747c8592-istio-data\") pod \"e2e-worker-w6-3d909d-openshift-default-77787695b7-trngw\" (UID: \"5ba753c3-5979-4263-8b90-2741747c8592\") " pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-trngw" Oct 30 20:49:47.006348 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:49:47.003183 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ba753c3-5979-4263-8b90-2741747c8592-istio-token\") pod \"e2e-worker-w6-3d909d-openshift-default-77787695b7-trngw\" (UID: \"5ba753c3-5979-4263-8b90-2741747c8592\") " pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-trngw" Oct 30 20:49:47.006348 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:49:47.003229 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ba753c3-5979-4263-8b90-2741747c8592-workload-socket\") pod \"e2e-worker-w6-3d909d-openshift-default-77787695b7-trngw\" (UID: \"5ba753c3-5979-4263-8b90-2741747c8592\") " pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-trngw" Oct 30 20:49:47.006348 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:49:47.003267 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ba753c3-5979-4263-8b90-2741747c8592-credential-socket\") pod \"e2e-worker-w6-3d909d-openshift-default-77787695b7-trngw\" (UID: \"5ba753c3-5979-4263-8b90-2741747c8592\") " pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-trngw" Oct 30 20:49:47.006348 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:49:47.003333 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ba753c3-5979-4263-8b90-2741747c8592-istio-envoy\") pod \"e2e-worker-w6-3d909d-openshift-default-77787695b7-trngw\" (UID: \"5ba753c3-5979-4263-8b90-2741747c8592\") " pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-trngw" Oct 30 20:49:47.006348 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:49:47.003746 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ba753c3-5979-4263-8b90-2741747c8592-istio-data\") pod \"e2e-worker-w6-3d909d-openshift-default-77787695b7-trngw\" (UID: \"5ba753c3-5979-4263-8b90-2741747c8592\") " pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-trngw" Oct 30 20:49:47.006348 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:49:47.004764 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ba753c3-5979-4263-8b90-2741747c8592-istiod-ca-cert\") pod \"e2e-worker-w6-3d909d-openshift-default-77787695b7-trngw\" (UID: \"5ba753c3-5979-4263-8b90-2741747c8592\") " pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-trngw" Oct 30 20:49:47.006348 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:49:47.005021 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ba753c3-5979-4263-8b90-2741747c8592-workload-socket\") pod \"e2e-worker-w6-3d909d-openshift-default-77787695b7-trngw\" (UID: \"5ba753c3-5979-4263-8b90-2741747c8592\") " pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-trngw" Oct 30 20:49:47.006348 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:49:47.005502 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ba753c3-5979-4263-8b90-2741747c8592-credential-socket\") pod \"e2e-worker-w6-3d909d-openshift-default-77787695b7-trngw\" (UID: \"5ba753c3-5979-4263-8b90-2741747c8592\") " pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-trngw" Oct 30 20:49:47.006348 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:49:47.006022 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ba753c3-5979-4263-8b90-2741747c8592-istio-podinfo\") pod \"e2e-worker-w6-3d909d-openshift-default-77787695b7-trngw\" (UID: \"5ba753c3-5979-4263-8b90-2741747c8592\") " pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-trngw" Oct 30 20:49:47.006944 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:49:47.006598 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ba753c3-5979-4263-8b90-2741747c8592-workload-certs\") pod \"e2e-worker-w6-3d909d-openshift-default-77787695b7-trngw\" (UID: \"5ba753c3-5979-4263-8b90-2741747c8592\") " pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-trngw" Oct 30 20:49:47.007048 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:49:47.007028 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ba753c3-5979-4263-8b90-2741747c8592-workload-certs\") pod \"e2e-worker-w6-3d909d-openshift-default-77787695b7-trngw\" (UID: \"5ba753c3-5979-4263-8b90-2741747c8592\") " pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-trngw" Oct 30 20:49:47.009721 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:49:47.009678 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ba753c3-5979-4263-8b90-2741747c8592-istio-envoy\") pod \"e2e-worker-w6-3d909d-openshift-default-77787695b7-trngw\" (UID: \"5ba753c3-5979-4263-8b90-2741747c8592\") " pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-trngw" Oct 30 20:49:47.016413 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:49:47.016385 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ba753c3-5979-4263-8b90-2741747c8592-istio-token\") pod \"e2e-worker-w6-3d909d-openshift-default-77787695b7-trngw\" (UID: \"5ba753c3-5979-4263-8b90-2741747c8592\") " pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-trngw" Oct 30 20:49:47.016529 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:49:47.016478 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8j5dq\" (UniqueName: \"kubernetes.io/projected/5ba753c3-5979-4263-8b90-2741747c8592-kube-api-access-8j5dq\") pod \"e2e-worker-w6-3d909d-openshift-default-77787695b7-trngw\" (UID: \"5ba753c3-5979-4263-8b90-2741747c8592\") " pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-trngw" Oct 30 20:49:47.051356 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:49:47.047392 4611 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-trngw" Oct 30 20:49:47.113556 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:49:47.113206 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:49:47.129756 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:49:47.129222 4611 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-5f78z"] Oct 30 20:49:47.135164 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:49:47.135130 4611 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06064c28_fd6d_4765_8635_8690665a9b83.slice/crio-494cd8b7fb015b421da4e68261471321425b163ce71c0c90b0ce2ab6664e04a1 WatchSource:0}: Error finding container 494cd8b7fb015b421da4e68261471321425b163ce71c0c90b0ce2ab6664e04a1: Status 404 returned error can't find the container with id 494cd8b7fb015b421da4e68261471321425b163ce71c0c90b0ce2ab6664e04a1 Oct 30 20:49:47.141942 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:49:47.139812 4611 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:49:47.141942 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:49:47.139904 4611 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:49:47.141942 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:49:47.139950 4611 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:49:47.279871 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:49:47.279841 4611 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-k5p2c"] Oct 30 20:49:47.284798 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:49:47.284767 4611 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85895a05_5edd_47b7_bd4e_8112526dd4d8.slice/crio-31393a264ed7e656a1a5cbf87637d3ee5d63f4f888c6a3603e5ee1d3e629b931 WatchSource:0}: Error finding container 31393a264ed7e656a1a5cbf87637d3ee5d63f4f888c6a3603e5ee1d3e629b931: Status 404 returned error can't find the container with id 31393a264ed7e656a1a5cbf87637d3ee5d63f4f888c6a3603e5ee1d3e629b931 Oct 30 20:49:47.287403 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:49:47.287373 4611 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:49:47.287494 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:49:47.287445 4611 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:49:47.287494 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:49:47.287483 4611 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:49:47.357640 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:49:47.357610 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-hx7mv" event={"ID":"52f7cad0-6a73-4573-9789-59ac6df33da4","Type":"ContainerStarted","Data":"91d09dc6096a9bc89e3ea13fab3ce3ddb29cc39689b61711253cc60ce5c9db06"} Oct 30 20:49:47.357780 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:49:47.357659 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-hx7mv" event={"ID":"52f7cad0-6a73-4573-9789-59ac6df33da4","Type":"ContainerStarted","Data":"b99d8fe48920544fea38344e40bb203fd45545c58d9ba322cd794a53f7760182"} Oct 30 20:49:47.360292 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:49:47.359948 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-khv95" event={"ID":"62c91622-5518-4dc2-9f08-102dd0fcfb84","Type":"ContainerStarted","Data":"82498596266e320ce3c9b12b22ae7f60164022ccb0b0900e9a013fe7bf58da19"} Oct 30 20:49:47.360292 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:49:47.359997 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-khv95" event={"ID":"62c91622-5518-4dc2-9f08-102dd0fcfb84","Type":"ContainerStarted","Data":"cdbad39e17b4333ea11936107416c37b744e57792aedc37913d55af8ac11907e"} Oct 30 20:49:47.361850 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:49:47.361825 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-k5p2c" event={"ID":"85895a05-5edd-47b7-bd4e-8112526dd4d8","Type":"ContainerStarted","Data":"31393a264ed7e656a1a5cbf87637d3ee5d63f4f888c6a3603e5ee1d3e629b931"} Oct 30 20:49:47.363330 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:49:47.363300 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-6xtqj" event={"ID":"63d294fd-5293-4752-9116-0aafa81d5e4f","Type":"ContainerStarted","Data":"9616177cca0955ef3462c13286375e84733d645de3ecd8ef6b9861e7317f425c"} Oct 30 20:49:47.363427 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:49:47.363341 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-6xtqj" event={"ID":"63d294fd-5293-4752-9116-0aafa81d5e4f","Type":"ContainerStarted","Data":"6ebdf9bc66c8d06ec5e5175d22164b5a4a0daa22f36a73ca3d6874bf5ef0ebf2"} Oct 30 20:49:47.365313 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:49:47.365285 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-5f78z" event={"ID":"06064c28-fd6d-4765-8635-8690665a9b83","Type":"ContainerStarted","Data":"bbf3a49efec4dd88896aa244e5313f8c00f1dad08d0e259457daea781577c113"} Oct 30 20:49:47.365413 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:49:47.365334 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-5f78z" event={"ID":"06064c28-fd6d-4765-8635-8690665a9b83","Type":"ContainerStarted","Data":"494cd8b7fb015b421da4e68261471321425b163ce71c0c90b0ce2ab6664e04a1"} Oct 30 20:49:47.372535 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:49:47.372296 4611 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-trngw"] Oct 30 20:49:47.374137 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:49:47.374117 4611 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-hx7mv" Oct 30 20:49:47.379607 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:49:47.379481 4611 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-hx7mv" Oct 30 20:49:47.381528 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:49:47.381420 4611 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-hx7mv" podStartSLOduration=1.381404404 podStartE2EDuration="1.381404404s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:49:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:49:47.37911486 +0000 UTC m=+641.913141478" watchObservedRunningTime="2026-10-30 20:49:47.381404404 +0000 UTC m=+641.915431021" Oct 30 20:49:47.383268 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:49:47.383243 4611 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ba753c3_5979_4263_8b90_2741747c8592.slice/crio-069cdb2ff8dc5594c013082f64c626b322e05a04a2d51b61ceb128f5b1fdaeb1 WatchSource:0}: Error finding container 069cdb2ff8dc5594c013082f64c626b322e05a04a2d51b61ceb128f5b1fdaeb1: Status 404 returned error can't find the container with id 069cdb2ff8dc5594c013082f64c626b322e05a04a2d51b61ceb128f5b1fdaeb1 Oct 30 20:49:47.388607 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:49:47.388564 4611 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:49:47.388691 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:49:47.388652 4611 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:49:47.388774 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:49:47.388691 4611 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:49:47.431956 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:49:47.431911 4611 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-5f78z" podStartSLOduration=1.431897099 podStartE2EDuration="1.431897099s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:49:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:49:47.407716997 +0000 UTC m=+641.941743633" watchObservedRunningTime="2026-10-30 20:49:47.431897099 +0000 UTC m=+641.965923714" Oct 30 20:49:47.433231 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:49:47.433193 4611 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-khv95" podStartSLOduration=1.433184225 podStartE2EDuration="1.433184225s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:49:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:49:47.430869147 +0000 UTC m=+641.964895764" watchObservedRunningTime="2026-10-30 20:49:47.433184225 +0000 UTC m=+641.967210842" Oct 30 20:49:47.457902 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:49:47.457834 4611 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-6xtqj" podStartSLOduration=1.457815052 podStartE2EDuration="1.457815052s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:49:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:49:47.453004612 +0000 UTC m=+641.987031228" watchObservedRunningTime="2026-10-30 20:49:47.457815052 +0000 UTC m=+641.991841674" Oct 30 20:49:47.479178 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:49:47.479151 4611 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-6xtqj" Oct 30 20:49:47.486999 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:49:47.486965 4611 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-6xtqj" Oct 30 20:49:47.623746 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:49:47.623707 4611 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-khv95" Oct 30 20:49:47.743634 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:49:47.743557 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:49:47.834461 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:49:47.834431 4611 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-5f78z" Oct 30 20:49:48.266252 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:49:48.266219 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:49:48.372775 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:49:48.372739 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-trngw" event={"ID":"5ba753c3-5979-4263-8b90-2741747c8592","Type":"ContainerStarted","Data":"1aa4d02a6e2697b5e2510958ad678aafedd30c89dd21b88538db4d20e053fe11"} Oct 30 20:49:48.372941 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:49:48.372793 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-trngw" event={"ID":"5ba753c3-5979-4263-8b90-2741747c8592","Type":"ContainerStarted","Data":"069cdb2ff8dc5594c013082f64c626b322e05a04a2d51b61ceb128f5b1fdaeb1"} Oct 30 20:49:48.374415 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:49:48.374380 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-k5p2c" event={"ID":"85895a05-5edd-47b7-bd4e-8112526dd4d8","Type":"ContainerStarted","Data":"a177d3a5bbd6f448456bee62c37268c18299145eed1615119d4d3979cd4e5e79"} Oct 30 20:49:48.375059 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:49:48.375031 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-hx7mv" Oct 30 20:49:48.375193 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:49:48.375080 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-6xtqj" Oct 30 20:49:48.375639 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:49:48.375617 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-hx7mv" Oct 30 20:49:48.375807 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:49:48.375792 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-6xtqj" Oct 30 20:49:48.397181 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:49:48.397123 4611 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-trngw" podStartSLOduration=2.397109719 podStartE2EDuration="2.397109719s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:49:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:49:48.394394288 +0000 UTC m=+642.928420903" watchObservedRunningTime="2026-10-30 20:49:48.397109719 +0000 UTC m=+642.931136335" Oct 30 20:49:48.628394 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:49:48.628368 4611 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-khv95" Oct 30 20:49:48.651142 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:49:48.651095 4611 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-k5p2c" podStartSLOduration=2.6510812059999997 podStartE2EDuration="2.651081206s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:49:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:49:48.463107352 +0000 UTC m=+642.997133969" watchObservedRunningTime="2026-10-30 20:49:48.651081206 +0000 UTC m=+643.185107822" Oct 30 20:49:48.764909 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:49:48.764877 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:49:48.839006 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:49:48.838980 4611 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-5f78z" Oct 30 20:49:48.942235 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:49:48.942172 4611 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-k5p2c" Oct 30 20:49:48.947375 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:49:48.947353 4611 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-k5p2c" Oct 30 20:49:49.047972 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:49:49.047943 4611 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-trngw" Oct 30 20:49:49.052957 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:49:49.052937 4611 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-trngw" Oct 30 20:49:49.381048 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:49:49.381018 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-5f78z" Oct 30 20:49:49.381244 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:49:49.381103 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-trngw" Oct 30 20:49:49.381244 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:49:49.381135 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-khv95" Oct 30 20:49:49.381244 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:49:49.381180 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-k5p2c" Oct 30 20:49:49.381429 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:49:49.381365 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-trngw" Oct 30 20:49:49.381429 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:49:49.381417 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-khv95" Oct 30 20:49:49.381585 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:49:49.381456 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-5f78z" Oct 30 20:49:49.381631 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:49:49.381594 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-k5p2c" Oct 30 20:49:49.788048 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:49:49.787973 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:49:49.954133 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:49:49.954092 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:49:50.167724 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:49:50.167664 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:49:51.012596 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:49:51.012559 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:49:51.187655 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:49:51.187625 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:49:51.368849 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:49:51.368818 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:49:51.757887 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:49:51.757807 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:49:51.913352 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:49:51.913322 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:49:52.203172 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:49:52.203137 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:49:52.588749 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:49:52.588667 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:49:52.754501 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:49:52.754456 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:49:52.963593 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:49:52.963554 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:49:53.160046 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:49:53.160012 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:49:54.355145 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:49:54.355108 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:49:54.564103 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:49:54.564068 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:49:54.767217 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:49:54.767139 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:49:55.337481 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:49:55.337448 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:49:55.640628 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:49:55.640596 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:49:56.340158 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:49:56.340123 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:49:56.565602 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:49:56.565548 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:49:57.220770 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:49:57.220739 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:49:57.973097 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:49:57.973017 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:49:58.872571 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:49:58.872537 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:49:59.886528 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:49:59.886309 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:50:00.124808 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:00.124771 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:50:00.536418 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:00.536334 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:50:00.723007 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:00.722913 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:50:01.097750 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:01.097711 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:50:01.129625 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:01.129593 4611 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-f2ed1e-66d84b8cff-g2w49"] Oct 30 20:50:01.133159 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:01.133139 4611 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-f2ed1e-66d84b8cff-g2w49" Oct 30 20:50:01.143656 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:01.143620 4611 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-f2ed1e-dockercfg-hwznm\"" Oct 30 20:50:01.210927 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:01.210882 4611 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-f2ed1e\"" Oct 30 20:50:01.214670 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:01.214640 4611 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-f2ed1e-66d84b8cff-g2w49"] Oct 30 20:50:01.237226 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:01.237196 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wbxlv\" (UniqueName: \"kubernetes.io/projected/d9298078-08c1-40a5-9117-ced750d46bd8-kube-api-access-wbxlv\") pod \"payload-pre-processing-e2e-worker-w4-f2ed1e-66d84b8cff-g2w49\" (UID: \"d9298078-08c1-40a5-9117-ced750d46bd8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-f2ed1e-66d84b8cff-g2w49" Oct 30 20:50:01.237383 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:01.237364 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9298078-08c1-40a5-9117-ced750d46bd8-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-f2ed1e-66d84b8cff-g2w49\" (UID: \"d9298078-08c1-40a5-9117-ced750d46bd8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-f2ed1e-66d84b8cff-g2w49" Oct 30 20:50:01.339202 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:01.339165 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9298078-08c1-40a5-9117-ced750d46bd8-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-f2ed1e-66d84b8cff-g2w49\" (UID: \"d9298078-08c1-40a5-9117-ced750d46bd8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-f2ed1e-66d84b8cff-g2w49" Oct 30 20:50:01.339384 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:01.339262 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wbxlv\" (UniqueName: \"kubernetes.io/projected/d9298078-08c1-40a5-9117-ced750d46bd8-kube-api-access-wbxlv\") pod \"payload-pre-processing-e2e-worker-w4-f2ed1e-66d84b8cff-g2w49\" (UID: \"d9298078-08c1-40a5-9117-ced750d46bd8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-f2ed1e-66d84b8cff-g2w49" Oct 30 20:50:01.340145 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:01.340113 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9298078-08c1-40a5-9117-ced750d46bd8-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-f2ed1e-66d84b8cff-g2w49\" (UID: \"d9298078-08c1-40a5-9117-ced750d46bd8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-f2ed1e-66d84b8cff-g2w49" Oct 30 20:50:01.404503 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:01.404467 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wbxlv\" (UniqueName: \"kubernetes.io/projected/d9298078-08c1-40a5-9117-ced750d46bd8-kube-api-access-wbxlv\") pod \"payload-pre-processing-e2e-worker-w4-f2ed1e-66d84b8cff-g2w49\" (UID: \"d9298078-08c1-40a5-9117-ced750d46bd8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-f2ed1e-66d84b8cff-g2w49" Oct 30 20:50:01.472543 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:01.472343 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:50:01.486739 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:01.486713 4611 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-f2ed1e-66d84b8cff-g2w49" Oct 30 20:50:01.708245 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:01.708160 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:50:01.833011 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:01.832934 4611 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-f2ed1e-66d84b8cff-g2w49"] Oct 30 20:50:01.838036 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:50:01.838002 4611 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9298078_08c1_40a5_9117_ced750d46bd8.slice/crio-515c987bcb9b42118a95a267c0dfd3ad8ebe4c23cc564cb243e8d849a900ef9e WatchSource:0}: Error finding container 515c987bcb9b42118a95a267c0dfd3ad8ebe4c23cc564cb243e8d849a900ef9e: Status 404 returned error can't find the container with id 515c987bcb9b42118a95a267c0dfd3ad8ebe4c23cc564cb243e8d849a900ef9e Oct 30 20:50:01.840622 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:01.840604 4611 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 20:50:01.925823 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:01.925786 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:50:02.338786 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:02.338753 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:50:02.534160 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:02.534118 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-f2ed1e-66d84b8cff-g2w49" event={"ID":"d9298078-08c1-40a5-9117-ced750d46bd8","Type":"ContainerStarted","Data":"515c987bcb9b42118a95a267c0dfd3ad8ebe4c23cc564cb243e8d849a900ef9e"} Oct 30 20:50:02.902566 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:02.898461 4611 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-3d909d-74bfb44f9b-5f2x8"] Oct 30 20:50:02.903822 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:02.903732 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:50:02.907126 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:02.907099 4611 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-3d909d-74bfb44f9b-5f2x8" Oct 30 20:50:02.922485 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:02.920290 4611 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-3d909d-57b9b6dcf9-l9pwk"] Oct 30 20:50:02.922485 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:02.922186 4611 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w6-3d909d\"" Oct 30 20:50:02.926101 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:02.925901 4611 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-3d909d-57b9b6dcf9-l9pwk" Oct 30 20:50:02.930679 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:02.930655 4611 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-3d909d-74bfb44f9b-5f2x8"] Oct 30 20:50:02.939319 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:02.939295 4611 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-3d909d-57b9b6dcf9-l9pwk"] Oct 30 20:50:02.942781 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:02.942744 4611 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w6-3d909d-dockercfg-mlctq\"" Oct 30 20:50:03.031082 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:03.030687 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mvm6v\" (UniqueName: \"kubernetes.io/projected/46885e54-e3da-4b91-aa7b-433f9b55a0da-kube-api-access-mvm6v\") pod \"payload-processing-e2e-worker-w6-3d909d-74bfb44f9b-5f2x8\" (UID: \"46885e54-e3da-4b91-aa7b-433f9b55a0da\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-3d909d-74bfb44f9b-5f2x8" Oct 30 20:50:03.031082 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:03.030764 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d963de5-2836-4225-9120-d885791b4f71-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-3d909d-57b9b6dcf9-l9pwk\" (UID: \"5d963de5-2836-4225-9120-d885791b4f71\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-3d909d-57b9b6dcf9-l9pwk" Oct 30 20:50:03.031082 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:03.030833 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46885e54-e3da-4b91-aa7b-433f9b55a0da-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-3d909d-74bfb44f9b-5f2x8\" (UID: \"46885e54-e3da-4b91-aa7b-433f9b55a0da\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-3d909d-74bfb44f9b-5f2x8" Oct 30 20:50:03.031082 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:03.030898 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xg67c\" (UniqueName: \"kubernetes.io/projected/5d963de5-2836-4225-9120-d885791b4f71-kube-api-access-xg67c\") pod \"payload-pre-processing-e2e-worker-w6-3d909d-57b9b6dcf9-l9pwk\" (UID: \"5d963de5-2836-4225-9120-d885791b4f71\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-3d909d-57b9b6dcf9-l9pwk" Oct 30 20:50:03.206581 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:03.189718 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46885e54-e3da-4b91-aa7b-433f9b55a0da-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-3d909d-74bfb44f9b-5f2x8\" (UID: \"46885e54-e3da-4b91-aa7b-433f9b55a0da\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-3d909d-74bfb44f9b-5f2x8" Oct 30 20:50:03.206581 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:03.189807 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xg67c\" (UniqueName: \"kubernetes.io/projected/5d963de5-2836-4225-9120-d885791b4f71-kube-api-access-xg67c\") pod \"payload-pre-processing-e2e-worker-w6-3d909d-57b9b6dcf9-l9pwk\" (UID: \"5d963de5-2836-4225-9120-d885791b4f71\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-3d909d-57b9b6dcf9-l9pwk" Oct 30 20:50:03.206581 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:03.189920 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mvm6v\" (UniqueName: \"kubernetes.io/projected/46885e54-e3da-4b91-aa7b-433f9b55a0da-kube-api-access-mvm6v\") pod \"payload-processing-e2e-worker-w6-3d909d-74bfb44f9b-5f2x8\" (UID: \"46885e54-e3da-4b91-aa7b-433f9b55a0da\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-3d909d-74bfb44f9b-5f2x8" Oct 30 20:50:03.206581 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:03.189978 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d963de5-2836-4225-9120-d885791b4f71-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-3d909d-57b9b6dcf9-l9pwk\" (UID: \"5d963de5-2836-4225-9120-d885791b4f71\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-3d909d-57b9b6dcf9-l9pwk" Oct 30 20:50:03.206581 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:03.192371 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46885e54-e3da-4b91-aa7b-433f9b55a0da-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-3d909d-74bfb44f9b-5f2x8\" (UID: \"46885e54-e3da-4b91-aa7b-433f9b55a0da\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-3d909d-74bfb44f9b-5f2x8" Oct 30 20:50:03.206581 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:03.201263 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d963de5-2836-4225-9120-d885791b4f71-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-3d909d-57b9b6dcf9-l9pwk\" (UID: \"5d963de5-2836-4225-9120-d885791b4f71\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-3d909d-57b9b6dcf9-l9pwk" Oct 30 20:50:03.223558 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:03.217265 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:50:03.235573 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:03.235523 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mvm6v\" (UniqueName: \"kubernetes.io/projected/46885e54-e3da-4b91-aa7b-433f9b55a0da-kube-api-access-mvm6v\") pod \"payload-processing-e2e-worker-w6-3d909d-74bfb44f9b-5f2x8\" (UID: \"46885e54-e3da-4b91-aa7b-433f9b55a0da\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-3d909d-74bfb44f9b-5f2x8" Oct 30 20:50:03.238563 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:03.238490 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xg67c\" (UniqueName: \"kubernetes.io/projected/5d963de5-2836-4225-9120-d885791b4f71-kube-api-access-xg67c\") pod \"payload-pre-processing-e2e-worker-w6-3d909d-57b9b6dcf9-l9pwk\" (UID: \"5d963de5-2836-4225-9120-d885791b4f71\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-3d909d-57b9b6dcf9-l9pwk" Oct 30 20:50:03.299047 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:03.298741 4611 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-3d909d-57b9b6dcf9-l9pwk" Oct 30 20:50:03.536092 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:03.535717 4611 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-3d909d-74bfb44f9b-5f2x8" Oct 30 20:50:03.545801 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:03.545484 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:50:03.918529 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:03.913924 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:50:04.021225 ip-10-0-133-192 kubenswrapper[4611]: E1030 20:50:04.021160 4611 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hwzc6" containerName="extractor" Oct 30 20:50:04.087222 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:04.086890 4611 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-3d909d-57b9b6dcf9-l9pwk"] Oct 30 20:50:04.088392 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:04.088141 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:50:04.279245 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:04.277535 4611 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-3d909d-74bfb44f9b-5f2x8"] Oct 30 20:50:04.428689 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:04.428646 4611 patch_prober.go:29] interesting pod/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-6xtqj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:50:04.428844 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:04.428715 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-6xtqj" podUID="63d294fd-5293-4752-9116-0aafa81d5e4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:50:04.613427 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:04.611248 4611 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-f2ed1e-c97788dc8-hnqvv"] Oct 30 20:50:04.616522 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:04.615876 4611 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-f2ed1e-c97788dc8-hnqvv" Oct 30 20:50:04.623603 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:04.623112 4611 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 20:50:04.628488 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:04.628299 4611 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-f2ed1e-c97788dc8-hnqvv"] Oct 30 20:50:04.628608 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:04.628576 4611 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 20:50:04.672547 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:04.671807 4611 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w4-f2ed1e-serving-cert\"" Oct 30 20:50:04.676700 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:04.676124 4611 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-k976f\"" Oct 30 20:50:04.676700 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:04.676252 4611 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Oct 30 20:50:04.676988 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:04.676892 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5052ef1d-3134-4545-8f2e-2c6ef046ccf6-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-f2ed1e-c97788dc8-hnqvv\" (UID: \"5052ef1d-3134-4545-8f2e-2c6ef046ccf6\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-f2ed1e-c97788dc8-hnqvv" Oct 30 20:50:04.676988 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:04.676932 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5052ef1d-3134-4545-8f2e-2c6ef046ccf6-metrics-certs\") pod \"maas-api-e2e-worker-w4-f2ed1e-c97788dc8-hnqvv\" (UID: \"5052ef1d-3134-4545-8f2e-2c6ef046ccf6\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-f2ed1e-c97788dc8-hnqvv" Oct 30 20:50:04.676988 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:04.676969 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x74kg\" (UniqueName: \"kubernetes.io/projected/5052ef1d-3134-4545-8f2e-2c6ef046ccf6-kube-api-access-x74kg\") pod \"maas-api-e2e-worker-w4-f2ed1e-c97788dc8-hnqvv\" (UID: \"5052ef1d-3134-4545-8f2e-2c6ef046ccf6\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-f2ed1e-c97788dc8-hnqvv" Oct 30 20:50:04.677159 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:04.677086 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5052ef1d-3134-4545-8f2e-2c6ef046ccf6-maas-api-tls\") pod \"maas-api-e2e-worker-w4-f2ed1e-c97788dc8-hnqvv\" (UID: \"5052ef1d-3134-4545-8f2e-2c6ef046ccf6\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-f2ed1e-c97788dc8-hnqvv" Oct 30 20:50:04.709397 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:04.709317 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:50:04.778186 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:04.778146 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5052ef1d-3134-4545-8f2e-2c6ef046ccf6-maas-api-tls\") pod \"maas-api-e2e-worker-w4-f2ed1e-c97788dc8-hnqvv\" (UID: \"5052ef1d-3134-4545-8f2e-2c6ef046ccf6\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-f2ed1e-c97788dc8-hnqvv" Oct 30 20:50:04.778346 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:04.778284 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5052ef1d-3134-4545-8f2e-2c6ef046ccf6-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-f2ed1e-c97788dc8-hnqvv\" (UID: \"5052ef1d-3134-4545-8f2e-2c6ef046ccf6\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-f2ed1e-c97788dc8-hnqvv" Oct 30 20:50:04.778346 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:04.778315 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5052ef1d-3134-4545-8f2e-2c6ef046ccf6-metrics-certs\") pod \"maas-api-e2e-worker-w4-f2ed1e-c97788dc8-hnqvv\" (UID: \"5052ef1d-3134-4545-8f2e-2c6ef046ccf6\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-f2ed1e-c97788dc8-hnqvv" Oct 30 20:50:04.778461 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:04.778357 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x74kg\" (UniqueName: \"kubernetes.io/projected/5052ef1d-3134-4545-8f2e-2c6ef046ccf6-kube-api-access-x74kg\") pod \"maas-api-e2e-worker-w4-f2ed1e-c97788dc8-hnqvv\" (UID: \"5052ef1d-3134-4545-8f2e-2c6ef046ccf6\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-f2ed1e-c97788dc8-hnqvv" Oct 30 20:50:04.780368 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:04.780342 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5052ef1d-3134-4545-8f2e-2c6ef046ccf6-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-f2ed1e-c97788dc8-hnqvv\" (UID: \"5052ef1d-3134-4545-8f2e-2c6ef046ccf6\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-f2ed1e-c97788dc8-hnqvv" Oct 30 20:50:04.782147 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:04.782119 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5052ef1d-3134-4545-8f2e-2c6ef046ccf6-maas-api-tls\") pod \"maas-api-e2e-worker-w4-f2ed1e-c97788dc8-hnqvv\" (UID: \"5052ef1d-3134-4545-8f2e-2c6ef046ccf6\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-f2ed1e-c97788dc8-hnqvv" Oct 30 20:50:04.785162 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:04.785135 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5052ef1d-3134-4545-8f2e-2c6ef046ccf6-metrics-certs\") pod \"maas-api-e2e-worker-w4-f2ed1e-c97788dc8-hnqvv\" (UID: \"5052ef1d-3134-4545-8f2e-2c6ef046ccf6\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-f2ed1e-c97788dc8-hnqvv" Oct 30 20:50:04.791813 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:04.791786 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x74kg\" (UniqueName: \"kubernetes.io/projected/5052ef1d-3134-4545-8f2e-2c6ef046ccf6-kube-api-access-x74kg\") pod \"maas-api-e2e-worker-w4-f2ed1e-c97788dc8-hnqvv\" (UID: \"5052ef1d-3134-4545-8f2e-2c6ef046ccf6\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-f2ed1e-c97788dc8-hnqvv" Oct 30 20:50:04.873547 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:04.873262 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:50:04.970583 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:04.945540 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-3d909d-74bfb44f9b-5f2x8" event={"ID":"46885e54-e3da-4b91-aa7b-433f9b55a0da","Type":"ContainerStarted","Data":"5e157bb6d4698ab0d8690569d7f21c73301b0286718525f35f8180addedc96bd"} Oct 30 20:50:04.970583 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:04.945764 4611 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-f2ed1e-c97788dc8-hnqvv" Oct 30 20:50:04.974690 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:04.974654 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-3d909d-57b9b6dcf9-l9pwk" event={"ID":"5d963de5-2836-4225-9120-d885791b4f71","Type":"ContainerStarted","Data":"163dae44e2baad37f2b2c70ce1538eef4db970f635fd74ee9ccf864d62e46c86"} Oct 30 20:50:05.420038 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:05.419986 4611 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-f2ed1e-c97788dc8-hnqvv"] Oct 30 20:50:05.425280 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:05.425199 4611 patch_prober.go:29] interesting pod/e2e-worker-w2-db534b-openshift-default-85df4d79d9-5f78z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:50:05.425280 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:05.425255 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-5f78z" podUID="06064c28-fd6d-4765-8635-8690665a9b83" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:50:05.429897 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:05.425603 4611 patch_prober.go:29] interesting pod/e2e-worker-w5-14cca2-openshift-default-d7f87db59-k5p2c container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:50:05.429897 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:05.425644 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-k5p2c" podUID="85895a05-5edd-47b7-bd4e-8112526dd4d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:50:05.431787 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:50:05.431655 4611 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5052ef1d_3134_4545_8f2e_2c6ef046ccf6.slice/crio-abb64141453fe03188fd2e696a0c51c60500325cacb72fa46e46d4a16153d05a WatchSource:0}: Error finding container abb64141453fe03188fd2e696a0c51c60500325cacb72fa46e46d4a16153d05a: Status 404 returned error can't find the container with id abb64141453fe03188fd2e696a0c51c60500325cacb72fa46e46d4a16153d05a Oct 30 20:50:05.987059 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:05.986998 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:50:06.005498 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:06.005419 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-f2ed1e-c97788dc8-hnqvv" event={"ID":"5052ef1d-3134-4545-8f2e-2c6ef046ccf6","Type":"ContainerStarted","Data":"abb64141453fe03188fd2e696a0c51c60500325cacb72fa46e46d4a16153d05a"} Oct 30 20:50:06.967307 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:06.967206 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:50:07.021252 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:07.020238 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-f2ed1e-c97788dc8-hnqvv" event={"ID":"5052ef1d-3134-4545-8f2e-2c6ef046ccf6","Type":"ContainerStarted","Data":"75fb518252c8b2154f6c0306becaa50e1400b2d57efc9eb8f087e731dcfb3755"} Oct 30 20:50:07.021252 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:07.021197 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-f2ed1e-c97788dc8-hnqvv" Oct 30 20:50:07.162195 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:07.150913 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:50:07.916389 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:07.916335 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:50:08.822361 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:08.820944 4611 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-f2ed1e-c97788dc8-hnqvv" podStartSLOduration=4.215593533 podStartE2EDuration="4.820921829s" totalImagesPullingTime="605.328296ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:50:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:50:07.051097122 +0000 UTC m=+661.585123738" watchObservedRunningTime="2026-10-30 20:50:08.820921829 +0000 UTC m=+663.354948450" Oct 30 20:50:08.823701 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:08.823601 4611 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-cb60a2-76455477c8-6w5qv"] Oct 30 20:50:08.860610 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:08.860579 4611 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-cb60a2-76455477c8-6w5qv" Oct 30 20:50:08.861504 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:08.860451 4611 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-cb60a2-76455477c8-6w5qv"] Oct 30 20:50:08.866652 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:08.865970 4611 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w3-cb60a2-serving-cert\"" Oct 30 20:50:08.945736 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:08.945701 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:50:09.002021 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:09.001984 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p5gfb\" (UniqueName: \"kubernetes.io/projected/a74a618b-03cd-4949-9b6d-f91301dc19d1-kube-api-access-p5gfb\") pod \"maas-api-e2e-worker-w3-cb60a2-76455477c8-6w5qv\" (UID: \"a74a618b-03cd-4949-9b6d-f91301dc19d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-cb60a2-76455477c8-6w5qv" Oct 30 20:50:09.002192 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:09.002149 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a74a618b-03cd-4949-9b6d-f91301dc19d1-maas-api-tls\") pod \"maas-api-e2e-worker-w3-cb60a2-76455477c8-6w5qv\" (UID: \"a74a618b-03cd-4949-9b6d-f91301dc19d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-cb60a2-76455477c8-6w5qv" Oct 30 20:50:09.002276 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:09.002231 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a74a618b-03cd-4949-9b6d-f91301dc19d1-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-cb60a2-76455477c8-6w5qv\" (UID: \"a74a618b-03cd-4949-9b6d-f91301dc19d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-cb60a2-76455477c8-6w5qv" Oct 30 20:50:09.002338 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:09.002301 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a74a618b-03cd-4949-9b6d-f91301dc19d1-metrics-certs\") pod \"maas-api-e2e-worker-w3-cb60a2-76455477c8-6w5qv\" (UID: \"a74a618b-03cd-4949-9b6d-f91301dc19d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-cb60a2-76455477c8-6w5qv" Oct 30 20:50:09.103865 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:09.103826 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a74a618b-03cd-4949-9b6d-f91301dc19d1-maas-api-tls\") pod \"maas-api-e2e-worker-w3-cb60a2-76455477c8-6w5qv\" (UID: \"a74a618b-03cd-4949-9b6d-f91301dc19d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-cb60a2-76455477c8-6w5qv" Oct 30 20:50:09.104052 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:09.103931 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a74a618b-03cd-4949-9b6d-f91301dc19d1-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-cb60a2-76455477c8-6w5qv\" (UID: \"a74a618b-03cd-4949-9b6d-f91301dc19d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-cb60a2-76455477c8-6w5qv" Oct 30 20:50:09.104052 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:09.104001 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a74a618b-03cd-4949-9b6d-f91301dc19d1-metrics-certs\") pod \"maas-api-e2e-worker-w3-cb60a2-76455477c8-6w5qv\" (UID: \"a74a618b-03cd-4949-9b6d-f91301dc19d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-cb60a2-76455477c8-6w5qv" Oct 30 20:50:09.104179 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:09.104052 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p5gfb\" (UniqueName: \"kubernetes.io/projected/a74a618b-03cd-4949-9b6d-f91301dc19d1-kube-api-access-p5gfb\") pod \"maas-api-e2e-worker-w3-cb60a2-76455477c8-6w5qv\" (UID: \"a74a618b-03cd-4949-9b6d-f91301dc19d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-cb60a2-76455477c8-6w5qv" Oct 30 20:50:09.105041 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:09.104992 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a74a618b-03cd-4949-9b6d-f91301dc19d1-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-cb60a2-76455477c8-6w5qv\" (UID: \"a74a618b-03cd-4949-9b6d-f91301dc19d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-cb60a2-76455477c8-6w5qv" Oct 30 20:50:09.119468 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:09.119434 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a74a618b-03cd-4949-9b6d-f91301dc19d1-metrics-certs\") pod \"maas-api-e2e-worker-w3-cb60a2-76455477c8-6w5qv\" (UID: \"a74a618b-03cd-4949-9b6d-f91301dc19d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-cb60a2-76455477c8-6w5qv" Oct 30 20:50:09.126239 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:09.126178 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a74a618b-03cd-4949-9b6d-f91301dc19d1-maas-api-tls\") pod \"maas-api-e2e-worker-w3-cb60a2-76455477c8-6w5qv\" (UID: \"a74a618b-03cd-4949-9b6d-f91301dc19d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-cb60a2-76455477c8-6w5qv" Oct 30 20:50:09.132312 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:09.131129 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:50:09.134571 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:09.134546 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p5gfb\" (UniqueName: \"kubernetes.io/projected/a74a618b-03cd-4949-9b6d-f91301dc19d1-kube-api-access-p5gfb\") pod \"maas-api-e2e-worker-w3-cb60a2-76455477c8-6w5qv\" (UID: \"a74a618b-03cd-4949-9b6d-f91301dc19d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-cb60a2-76455477c8-6w5qv" Oct 30 20:50:09.210871 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:09.210836 4611 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-cb60a2-76455477c8-6w5qv" Oct 30 20:50:09.569481 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:09.569408 4611 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-cb60a2-76455477c8-6w5qv"] Oct 30 20:50:09.663560 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:50:09.663525 4611 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda74a618b_03cd_4949_9b6d_f91301dc19d1.slice/crio-60126af043f2d4f7b92812b7b8a0d3178da3fc4ff26933c4a12557de7e3144e6 WatchSource:0}: Error finding container 60126af043f2d4f7b92812b7b8a0d3178da3fc4ff26933c4a12557de7e3144e6: Status 404 returned error can't find the container with id 60126af043f2d4f7b92812b7b8a0d3178da3fc4ff26933c4a12557de7e3144e6 Oct 30 20:50:11.530052 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:09.919537 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:50:11.530052 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:10.042450 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-cb60a2-76455477c8-6w5qv" event={"ID":"a74a618b-03cd-4949-9b6d-f91301dc19d1","Type":"ContainerStarted","Data":"60126af043f2d4f7b92812b7b8a0d3178da3fc4ff26933c4a12557de7e3144e6"} Oct 30 20:50:11.530052 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:10.044467 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-f2ed1e-66d84b8cff-g2w49" event={"ID":"d9298078-08c1-40a5-9117-ced750d46bd8","Type":"ContainerStarted","Data":"5abfa4a71e3d2a9caa482c69607cb70c5e2edb08b54218a4fc94637fffa80b1d"} Oct 30 20:50:11.530052 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:10.063934 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:50:11.530052 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:10.072898 4611 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-f2ed1e-66d84b8cff-g2w49" podStartSLOduration=1.170588907 podStartE2EDuration="9.07288472s" totalImagesPullingTime="7.902295813s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:50:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:50:10.0690599 +0000 UTC m=+664.603086513" watchObservedRunningTime="2026-10-30 20:50:10.07288472 +0000 UTC m=+664.606911336" Oct 30 20:50:11.530052 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:10.824861 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:50:11.530052 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:10.990346 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:50:11.530052 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:11.099860 4611 generic.go:356] "Generic (PLEG): container finished" podID="63d294fd-5293-4752-9116-0aafa81d5e4f" containerID="9616177cca0955ef3462c13286375e84733d645de3ecd8ef6b9861e7317f425c" exitCode=0 Oct 30 20:50:11.530052 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:11.100133 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-6xtqj" event={"ID":"63d294fd-5293-4752-9116-0aafa81d5e4f","Type":"ContainerDied","Data":"9616177cca0955ef3462c13286375e84733d645de3ecd8ef6b9861e7317f425c"} Oct 30 20:50:11.530052 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:11.100669 4611 scope.go:122] "RemoveContainer" containerID="9616177cca0955ef3462c13286375e84733d645de3ecd8ef6b9861e7317f425c" Oct 30 20:50:11.530052 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:11.102462 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-3d909d-74bfb44f9b-5f2x8" event={"ID":"46885e54-e3da-4b91-aa7b-433f9b55a0da","Type":"ContainerStarted","Data":"ca32235c39e2d40fc731f627b2cb0f906b6da84bf1a9bf3ecdc1a75dbe43d3d6"} Oct 30 20:50:11.530052 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:11.102716 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w6-3d909d-74bfb44f9b-5f2x8" Oct 30 20:50:11.530052 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:11.103352 4611 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:50:11.530052 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:11.103433 4611 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:50:11.530052 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:11.103478 4611 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:50:11.530052 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:11.105630 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-3d909d-57b9b6dcf9-l9pwk" event={"ID":"5d963de5-2836-4225-9120-d885791b4f71","Type":"ContainerStarted","Data":"fe58a91cc76f1810321347afc1a4d4adcbdab7c190d7fef96fd2e34ae9f677c1"} Oct 30 20:50:11.530052 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:11.105702 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-3d909d-57b9b6dcf9-l9pwk" Oct 30 20:50:11.530052 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:11.106444 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-f2ed1e-66d84b8cff-g2w49" Oct 30 20:50:11.530052 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:11.207762 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:50:11.530052 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:11.226494 4611 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-3d909d-57b9b6dcf9-l9pwk" podStartSLOduration=2.764360178 podStartE2EDuration="9.226479488s" totalImagesPullingTime="6.46211931s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:50:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:50:11.204940611 +0000 UTC m=+665.738967228" watchObservedRunningTime="2026-10-30 20:50:11.226479488 +0000 UTC m=+665.760506107" Oct 30 20:50:11.530052 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:11.226732 4611 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w6-3d909d-74bfb44f9b-5f2x8" podStartSLOduration=2.702805287 podStartE2EDuration="9.226726415s" totalImagesPullingTime="6.523921128s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:50:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:50:11.224451247 +0000 UTC m=+665.758477866" watchObservedRunningTime="2026-10-30 20:50:11.226726415 +0000 UTC m=+665.760753030" Oct 30 20:50:11.764799 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:11.764661 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:50:11.943805 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:11.943760 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:50:12.131026 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:12.130982 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-cb60a2-76455477c8-6w5qv" event={"ID":"a74a618b-03cd-4949-9b6d-f91301dc19d1","Type":"ContainerStarted","Data":"7a190eb18771a8d343ae56f5ead972d1a702fda1f89c99959b8ae9ba0a723c73"} Oct 30 20:50:12.132015 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:12.131991 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-cb60a2-76455477c8-6w5qv" Oct 30 20:50:12.133563 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:12.133537 4611 generic.go:356] "Generic (PLEG): container finished" podID="85895a05-5edd-47b7-bd4e-8112526dd4d8" containerID="a177d3a5bbd6f448456bee62c37268c18299145eed1615119d4d3979cd4e5e79" exitCode=0 Oct 30 20:50:12.133683 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:12.133623 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-k5p2c" event={"ID":"85895a05-5edd-47b7-bd4e-8112526dd4d8","Type":"ContainerDied","Data":"a177d3a5bbd6f448456bee62c37268c18299145eed1615119d4d3979cd4e5e79"} Oct 30 20:50:12.133917 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:12.133903 4611 scope.go:122] "RemoveContainer" containerID="a177d3a5bbd6f448456bee62c37268c18299145eed1615119d4d3979cd4e5e79" Oct 30 20:50:12.135845 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:12.135816 4611 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:50:12.135965 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:12.135899 4611 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:50:12.135965 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:12.135945 4611 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:50:12.136880 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:12.136856 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-6xtqj" event={"ID":"63d294fd-5293-4752-9116-0aafa81d5e4f","Type":"ContainerStarted","Data":"a215e69034ca57fda6e5dbb3630c604d2a87905bf54b35fa8f5e49e148efb3ad"} Oct 30 20:50:12.191997 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:12.189002 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:50:12.237968 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:12.232779 4611 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-cb60a2-76455477c8-6w5qv" podStartSLOduration=2.910470993 podStartE2EDuration="4.232743739s" totalImagesPullingTime="1.322272746s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:50:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:50:12.170751952 +0000 UTC m=+666.704778569" watchObservedRunningTime="2026-10-30 20:50:12.232743739 +0000 UTC m=+666.766770359" Oct 30 20:50:12.490632 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:12.490581 4611 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-6xtqj" Oct 30 20:50:12.526656 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:12.526620 4611 patch_prober.go:29] interesting pod/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-6xtqj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body= Oct 30 20:50:12.526863 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:12.526681 4611 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-6xtqj" podUID="63d294fd-5293-4752-9116-0aafa81d5e4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" Oct 30 20:50:12.820271 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:12.815403 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:50:13.149534 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:13.145783 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-k5p2c" event={"ID":"85895a05-5edd-47b7-bd4e-8112526dd4d8","Type":"ContainerStarted","Data":"5bd45afa60cd729f1061937bf05738a9c2230e255ecedb8bea35a1b842e0da08"} Oct 30 20:50:13.176532 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:13.171775 4611 generic.go:356] "Generic (PLEG): container finished" podID="06064c28-fd6d-4765-8635-8690665a9b83" containerID="bbf3a49efec4dd88896aa244e5313f8c00f1dad08d0e259457daea781577c113" exitCode=0 Oct 30 20:50:13.176532 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:13.172554 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-5f78z" event={"ID":"06064c28-fd6d-4765-8635-8690665a9b83","Type":"ContainerDied","Data":"bbf3a49efec4dd88896aa244e5313f8c00f1dad08d0e259457daea781577c113"} Oct 30 20:50:13.176532 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:13.172897 4611 scope.go:122] "RemoveContainer" containerID="bbf3a49efec4dd88896aa244e5313f8c00f1dad08d0e259457daea781577c113" Oct 30 20:50:13.180531 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:13.177846 4611 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:50:13.180531 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:13.177944 4611 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:50:13.180531 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:13.178007 4611 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:50:13.481943 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:13.481908 4611 patch_prober.go:29] interesting pod/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-6xtqj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body= Oct 30 20:50:13.482192 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:13.482162 4611 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-6xtqj" podUID="63d294fd-5293-4752-9116-0aafa81d5e4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" Oct 30 20:50:13.625279 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:13.625234 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:50:13.964588 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:13.952445 4611 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-k5p2c" Oct 30 20:50:13.964588 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:13.952587 4611 patch_prober.go:29] interesting pod/e2e-worker-w5-14cca2-openshift-default-d7f87db59-k5p2c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body= Oct 30 20:50:13.964588 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:13.952633 4611 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-k5p2c" podUID="85895a05-5edd-47b7-bd4e-8112526dd4d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" Oct 30 20:50:14.100544 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:14.099711 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-f2ed1e-c97788dc8-hnqvv" Oct 30 20:50:14.243601 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:14.242408 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-5f78z" event={"ID":"06064c28-fd6d-4765-8635-8690665a9b83","Type":"ContainerStarted","Data":"aabb51bfc31c37d13fadd4cd218f1a931f8a4e4a27b04fa3e107400913404c18"} Oct 30 20:50:14.497590 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:14.497076 4611 patch_prober.go:29] interesting pod/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-6xtqj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body= Oct 30 20:50:14.497590 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:14.497144 4611 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-6xtqj" podUID="63d294fd-5293-4752-9116-0aafa81d5e4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" Oct 30 20:50:14.547601 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:14.547435 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:50:14.837964 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:14.837871 4611 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-5f78z" Oct 30 20:50:14.839258 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:14.839226 4611 patch_prober.go:29] interesting pod/e2e-worker-w2-db534b-openshift-default-85df4d79d9-5f78z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body= Oct 30 20:50:14.839566 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:14.839533 4611 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-5f78z" podUID="06064c28-fd6d-4765-8635-8690665a9b83" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" Oct 30 20:50:14.942068 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:14.942035 4611 patch_prober.go:29] interesting pod/e2e-worker-w5-14cca2-openshift-default-d7f87db59-k5p2c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body= Oct 30 20:50:14.942416 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:14.942384 4611 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-k5p2c" podUID="85895a05-5edd-47b7-bd4e-8112526dd4d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" Oct 30 20:50:15.316991 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:15.316957 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:50:15.555533 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:15.540116 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:50:15.607602 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:15.572881 4611 patch_prober.go:29] interesting pod/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-6xtqj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body= Oct 30 20:50:15.607892 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:15.607855 4611 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-6xtqj" podUID="63d294fd-5293-4752-9116-0aafa81d5e4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" Oct 30 20:50:15.850542 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:15.850398 4611 patch_prober.go:29] interesting pod/e2e-worker-w2-db534b-openshift-default-85df4d79d9-5f78z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body= Oct 30 20:50:15.850542 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:15.850472 4611 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-5f78z" podUID="06064c28-fd6d-4765-8635-8690665a9b83" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" Oct 30 20:50:15.954285 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:15.954193 4611 patch_prober.go:29] interesting pod/e2e-worker-w5-14cca2-openshift-default-d7f87db59-k5p2c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body= Oct 30 20:50:15.954588 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:15.954557 4611 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-k5p2c" podUID="85895a05-5edd-47b7-bd4e-8112526dd4d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" Oct 30 20:50:16.070076 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:16.070039 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:50:16.292369 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:16.292261 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:50:16.503003 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:16.502971 4611 patch_prober.go:29] interesting pod/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-6xtqj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body= Oct 30 20:50:16.503613 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:16.503564 4611 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-6xtqj" podUID="63d294fd-5293-4752-9116-0aafa81d5e4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" Oct 30 20:50:16.731533 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:16.730643 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:50:16.836887 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:16.836409 4611 patch_prober.go:29] interesting pod/e2e-worker-w2-db534b-openshift-default-85df4d79d9-5f78z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body= Oct 30 20:50:16.836887 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:16.836469 4611 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-5f78z" podUID="06064c28-fd6d-4765-8635-8690665a9b83" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" Oct 30 20:50:16.969807 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:16.969351 4611 patch_prober.go:29] interesting pod/e2e-worker-w5-14cca2-openshift-default-d7f87db59-k5p2c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body= Oct 30 20:50:16.969807 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:16.969412 4611 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-k5p2c" podUID="85895a05-5edd-47b7-bd4e-8112526dd4d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" Oct 30 20:50:17.524593 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:17.500403 4611 patch_prober.go:29] interesting pod/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-6xtqj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body= Oct 30 20:50:17.524593 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:17.500478 4611 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-6xtqj" podUID="63d294fd-5293-4752-9116-0aafa81d5e4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" Oct 30 20:50:17.842620 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:17.784780 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:50:17.842620 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:17.837952 4611 patch_prober.go:29] interesting pod/e2e-worker-w2-db534b-openshift-default-85df4d79d9-5f78z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body= Oct 30 20:50:17.842620 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:17.838015 4611 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-5f78z" podUID="06064c28-fd6d-4765-8635-8690665a9b83" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" Oct 30 20:50:17.977589 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:17.943302 4611 patch_prober.go:29] interesting pod/e2e-worker-w5-14cca2-openshift-default-d7f87db59-k5p2c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body= Oct 30 20:50:17.977589 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:17.943376 4611 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-k5p2c" podUID="85895a05-5edd-47b7-bd4e-8112526dd4d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" Oct 30 20:50:18.390533 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:18.380787 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-6xtqj" Oct 30 20:50:18.411357 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:18.408009 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:50:18.516825 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:18.516260 4611 patch_prober.go:29] interesting pod/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-6xtqj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body= Oct 30 20:50:18.516825 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:18.516320 4611 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-6xtqj" podUID="63d294fd-5293-4752-9116-0aafa81d5e4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" Oct 30 20:50:18.615565 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:18.615521 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:50:18.834456 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:18.834367 4611 patch_prober.go:29] interesting pod/e2e-worker-w2-db534b-openshift-default-85df4d79d9-5f78z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body= Oct 30 20:50:18.834675 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:18.834454 4611 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-5f78z" podUID="06064c28-fd6d-4765-8635-8690665a9b83" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" Oct 30 20:50:18.973625 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:18.973589 4611 patch_prober.go:29] interesting pod/e2e-worker-w5-14cca2-openshift-default-d7f87db59-k5p2c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body= Oct 30 20:50:18.974068 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:18.974021 4611 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-k5p2c" podUID="85895a05-5edd-47b7-bd4e-8112526dd4d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" Oct 30 20:50:19.300710 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:19.299637 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-cb60a2-76455477c8-6w5qv" Oct 30 20:50:19.389534 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:19.384085 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-k5p2c" Oct 30 20:50:19.389534 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:19.384963 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-5f78z" Oct 30 20:50:19.443096 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:19.442635 4611 patch_prober.go:29] interesting pod/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-hx7mv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:50:19.443096 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:19.442709 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-hx7mv" podUID="52f7cad0-6a73-4573-9789-59ac6df33da4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:50:19.544139 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:19.540779 4611 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-6xtqj" Oct 30 20:50:19.544139 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:19.543882 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:50:19.556876 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:19.556741 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-6xtqj" Oct 30 20:50:19.838519 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:19.834814 4611 patch_prober.go:29] interesting pod/e2e-worker-w2-db534b-openshift-default-85df4d79d9-5f78z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body= Oct 30 20:50:19.838519 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:19.834870 4611 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-5f78z" podUID="06064c28-fd6d-4765-8635-8690665a9b83" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" Oct 30 20:50:19.942138 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:19.942096 4611 patch_prober.go:29] interesting pod/e2e-worker-w5-14cca2-openshift-default-d7f87db59-k5p2c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body= Oct 30 20:50:19.942300 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:19.942159 4611 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-k5p2c" podUID="85895a05-5edd-47b7-bd4e-8112526dd4d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" Oct 30 20:50:20.335087 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:20.334046 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:50:20.425065 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:20.424656 4611 patch_prober.go:29] interesting pod/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-khv95 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:50:20.425065 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:20.424737 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-khv95" podUID="62c91622-5518-4dc2-9f08-102dd0fcfb84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:50:20.426692 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:20.426654 4611 patch_prober.go:29] interesting pod/e2e-worker-w6-3d909d-openshift-default-77787695b7-trngw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:50:20.426837 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:20.426717 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-trngw" podUID="5ba753c3-5979-4263-8b90-2741747c8592" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:50:20.834810 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:20.834772 4611 patch_prober.go:29] interesting pod/e2e-worker-w2-db534b-openshift-default-85df4d79d9-5f78z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body= Oct 30 20:50:20.835047 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:20.834831 4611 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-5f78z" podUID="06064c28-fd6d-4765-8635-8690665a9b83" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" Oct 30 20:50:20.951166 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:20.949963 4611 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-k5p2c" Oct 30 20:50:20.957952 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:20.957920 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-k5p2c" Oct 30 20:50:21.089316 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:21.089174 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:50:21.502712 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:21.502677 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:50:21.592281 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:21.592241 4611 generic.go:356] "Generic (PLEG): container finished" podID="62c91622-5518-4dc2-9f08-102dd0fcfb84" containerID="82498596266e320ce3c9b12b22ae7f60164022ccb0b0900e9a013fe7bf58da19" exitCode=0 Oct 30 20:50:21.592468 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:21.592316 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-khv95" event={"ID":"62c91622-5518-4dc2-9f08-102dd0fcfb84","Type":"ContainerDied","Data":"82498596266e320ce3c9b12b22ae7f60164022ccb0b0900e9a013fe7bf58da19"} Oct 30 20:50:21.592999 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:21.592978 4611 scope.go:122] "RemoveContainer" containerID="82498596266e320ce3c9b12b22ae7f60164022ccb0b0900e9a013fe7bf58da19" Oct 30 20:50:21.594635 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:21.594606 4611 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:50:21.594718 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:21.594694 4611 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:50:21.594766 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:21.594745 4611 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:50:22.015540 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:22.013731 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:50:22.071035 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:22.071005 4611 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-5f78z" Oct 30 20:50:22.072544 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:22.072500 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-5f78z" Oct 30 20:50:22.145680 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:22.145652 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-f2ed1e-66d84b8cff-g2w49" Oct 30 20:50:22.146711 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:22.146691 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-3d909d-57b9b6dcf9-l9pwk" Oct 30 20:50:22.147273 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:22.147255 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w6-3d909d-74bfb44f9b-5f2x8" Oct 30 20:50:22.601053 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:22.601013 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-khv95" event={"ID":"62c91622-5518-4dc2-9f08-102dd0fcfb84","Type":"ContainerStarted","Data":"e664a227e37bec15f2ce3cd3f40d2ce51bcbda595bf1acec7ad0dd7e33ef3f53"} Oct 30 20:50:22.623550 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:22.623522 4611 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-khv95" Oct 30 20:50:22.623817 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:22.623794 4611 patch_prober.go:29] interesting pod/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-khv95 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body= Oct 30 20:50:22.623876 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:22.623849 4611 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-khv95" podUID="62c91622-5518-4dc2-9f08-102dd0fcfb84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" Oct 30 20:50:23.085586 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:23.077051 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:50:23.611887 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:23.611751 4611 generic.go:356] "Generic (PLEG): container finished" podID="5ba753c3-5979-4263-8b90-2741747c8592" containerID="1aa4d02a6e2697b5e2510958ad678aafedd30c89dd21b88538db4d20e053fe11" exitCode=0 Oct 30 20:50:23.611887 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:23.611837 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-trngw" event={"ID":"5ba753c3-5979-4263-8b90-2741747c8592","Type":"ContainerDied","Data":"1aa4d02a6e2697b5e2510958ad678aafedd30c89dd21b88538db4d20e053fe11"} Oct 30 20:50:23.612733 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:23.612712 4611 scope.go:122] "RemoveContainer" containerID="1aa4d02a6e2697b5e2510958ad678aafedd30c89dd21b88538db4d20e053fe11" Oct 30 20:50:23.614132 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:23.614106 4611 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:50:23.614193 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:23.614183 4611 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:50:23.614233 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:23.614214 4611 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:50:23.617208 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:23.617187 4611 generic.go:356] "Generic (PLEG): container finished" podID="52f7cad0-6a73-4573-9789-59ac6df33da4" containerID="91d09dc6096a9bc89e3ea13fab3ce3ddb29cc39689b61711253cc60ce5c9db06" exitCode=0 Oct 30 20:50:23.618023 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:23.617438 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-hx7mv" event={"ID":"52f7cad0-6a73-4573-9789-59ac6df33da4","Type":"ContainerDied","Data":"91d09dc6096a9bc89e3ea13fab3ce3ddb29cc39689b61711253cc60ce5c9db06"} Oct 30 20:50:23.618023 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:23.617738 4611 scope.go:122] "RemoveContainer" containerID="91d09dc6096a9bc89e3ea13fab3ce3ddb29cc39689b61711253cc60ce5c9db06" Oct 30 20:50:23.621949 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:23.621920 4611 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:50:23.622019 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:23.621999 4611 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:50:23.622061 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:23.622046 4611 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:50:23.623749 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:23.623730 4611 patch_prober.go:29] interesting pod/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-khv95 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body= Oct 30 20:50:23.623859 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:23.623763 4611 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-khv95" podUID="62c91622-5518-4dc2-9f08-102dd0fcfb84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" Oct 30 20:50:24.011587 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:24.007590 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:50:24.624141 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:24.623714 4611 patch_prober.go:29] interesting pod/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-khv95 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body= Oct 30 20:50:24.624141 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:24.623776 4611 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-khv95" podUID="62c91622-5518-4dc2-9f08-102dd0fcfb84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" Oct 30 20:50:24.638095 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:24.637805 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-hx7mv" event={"ID":"52f7cad0-6a73-4573-9789-59ac6df33da4","Type":"ContainerStarted","Data":"0028b06d562d6e065e5b330b2875605dc7238ba219b2e2f402fbd9a06b916e9f"} Oct 30 20:50:24.641527 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:24.640566 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-trngw" event={"ID":"5ba753c3-5979-4263-8b90-2741747c8592","Type":"ContainerStarted","Data":"c10f965c9d3fd5d620ad061d397c593e1191596603f3b4b5f8242467c8cab5b5"} Oct 30 20:50:24.779534 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:24.776134 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:50:25.050219 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:25.049668 4611 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-trngw" Oct 30 20:50:25.050556 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:25.050116 4611 patch_prober.go:29] interesting pod/e2e-worker-w6-3d909d-openshift-default-77787695b7-trngw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body= Oct 30 20:50:25.050758 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:25.050725 4611 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-trngw" podUID="5ba753c3-5979-4263-8b90-2741747c8592" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" Oct 30 20:50:25.378531 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:25.377976 4611 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-hx7mv" Oct 30 20:50:25.378531 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:25.378109 4611 patch_prober.go:29] interesting pod/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-hx7mv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body= Oct 30 20:50:25.378531 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:25.378153 4611 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-hx7mv" podUID="52f7cad0-6a73-4573-9789-59ac6df33da4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" Oct 30 20:50:25.500715 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:25.500683 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:50:25.624726 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:25.624691 4611 patch_prober.go:29] interesting pod/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-khv95 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body= Oct 30 20:50:25.625282 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:25.625250 4611 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-khv95" podUID="62c91622-5518-4dc2-9f08-102dd0fcfb84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" Oct 30 20:50:26.049926 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:26.049890 4611 patch_prober.go:29] interesting pod/e2e-worker-w6-3d909d-openshift-default-77787695b7-trngw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body= Oct 30 20:50:26.050201 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:26.050172 4611 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-trngw" podUID="5ba753c3-5979-4263-8b90-2741747c8592" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" Oct 30 20:50:26.094088 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:26.094053 4611 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-ad7ff0/premium-w0-ad7ff0-kserve-5485d4b948-bpmgf"] Oct 30 20:50:26.135936 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:26.135805 4611 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-ad7ff0/premium-w0-ad7ff0-kserve-5485d4b948-bpmgf"] Oct 30 20:50:26.136109 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:26.135956 4611 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-ad7ff0/premium-w0-ad7ff0-kserve-5485d4b948-bpmgf" Oct 30 20:50:26.143439 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:26.142895 4611 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-ad7ff0\"/\"premium-w0-ad7ff0-kserve-self-signed-certs\"" Oct 30 20:50:26.144884 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:26.144726 4611 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-ad7ff0\"/\"default-dockercfg-rt952\"" Oct 30 20:50:26.144884 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:26.144813 4611 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-ad7ff0\"/\"openshift-service-ca.crt\"" Oct 30 20:50:26.145237 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:26.145214 4611 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-ad7ff0\"/\"kube-root-ca.crt\"" Oct 30 20:50:26.187534 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:26.186838 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:50:26.242006 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:26.241974 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/53cbd692-e9f8-46ba-8fde-3e5215a38904-tmp-dir\") pod \"premium-w0-ad7ff0-kserve-5485d4b948-bpmgf\" (UID: \"53cbd692-e9f8-46ba-8fde-3e5215a38904\") " pod="e2e-models-e2e-worker-w0-ad7ff0/premium-w0-ad7ff0-kserve-5485d4b948-bpmgf" Oct 30 20:50:26.242188 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:26.242048 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/53cbd692-e9f8-46ba-8fde-3e5215a38904-dshm\") pod \"premium-w0-ad7ff0-kserve-5485d4b948-bpmgf\" (UID: \"53cbd692-e9f8-46ba-8fde-3e5215a38904\") " pod="e2e-models-e2e-worker-w0-ad7ff0/premium-w0-ad7ff0-kserve-5485d4b948-bpmgf" Oct 30 20:50:26.242188 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:26.242077 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/53cbd692-e9f8-46ba-8fde-3e5215a38904-model-cache\") pod \"premium-w0-ad7ff0-kserve-5485d4b948-bpmgf\" (UID: \"53cbd692-e9f8-46ba-8fde-3e5215a38904\") " pod="e2e-models-e2e-worker-w0-ad7ff0/premium-w0-ad7ff0-kserve-5485d4b948-bpmgf" Oct 30 20:50:26.242188 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:26.242129 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m6mt5\" (UniqueName: \"kubernetes.io/projected/53cbd692-e9f8-46ba-8fde-3e5215a38904-kube-api-access-m6mt5\") pod \"premium-w0-ad7ff0-kserve-5485d4b948-bpmgf\" (UID: \"53cbd692-e9f8-46ba-8fde-3e5215a38904\") " pod="e2e-models-e2e-worker-w0-ad7ff0/premium-w0-ad7ff0-kserve-5485d4b948-bpmgf" Oct 30 20:50:26.242188 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:26.242162 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/53cbd692-e9f8-46ba-8fde-3e5215a38904-home\") pod \"premium-w0-ad7ff0-kserve-5485d4b948-bpmgf\" (UID: \"53cbd692-e9f8-46ba-8fde-3e5215a38904\") " pod="e2e-models-e2e-worker-w0-ad7ff0/premium-w0-ad7ff0-kserve-5485d4b948-bpmgf" Oct 30 20:50:26.242331 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:26.242208 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/53cbd692-e9f8-46ba-8fde-3e5215a38904-tls-certs\") pod \"premium-w0-ad7ff0-kserve-5485d4b948-bpmgf\" (UID: \"53cbd692-e9f8-46ba-8fde-3e5215a38904\") " pod="e2e-models-e2e-worker-w0-ad7ff0/premium-w0-ad7ff0-kserve-5485d4b948-bpmgf" Oct 30 20:50:26.343603 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:26.342885 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/53cbd692-e9f8-46ba-8fde-3e5215a38904-home\") pod \"premium-w0-ad7ff0-kserve-5485d4b948-bpmgf\" (UID: \"53cbd692-e9f8-46ba-8fde-3e5215a38904\") " pod="e2e-models-e2e-worker-w0-ad7ff0/premium-w0-ad7ff0-kserve-5485d4b948-bpmgf" Oct 30 20:50:26.343603 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:26.342987 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/53cbd692-e9f8-46ba-8fde-3e5215a38904-tls-certs\") pod \"premium-w0-ad7ff0-kserve-5485d4b948-bpmgf\" (UID: \"53cbd692-e9f8-46ba-8fde-3e5215a38904\") " pod="e2e-models-e2e-worker-w0-ad7ff0/premium-w0-ad7ff0-kserve-5485d4b948-bpmgf" Oct 30 20:50:26.343603 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:26.343034 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/53cbd692-e9f8-46ba-8fde-3e5215a38904-tmp-dir\") pod \"premium-w0-ad7ff0-kserve-5485d4b948-bpmgf\" (UID: \"53cbd692-e9f8-46ba-8fde-3e5215a38904\") " pod="e2e-models-e2e-worker-w0-ad7ff0/premium-w0-ad7ff0-kserve-5485d4b948-bpmgf" Oct 30 20:50:26.343603 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:26.343103 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/53cbd692-e9f8-46ba-8fde-3e5215a38904-dshm\") pod \"premium-w0-ad7ff0-kserve-5485d4b948-bpmgf\" (UID: \"53cbd692-e9f8-46ba-8fde-3e5215a38904\") " pod="e2e-models-e2e-worker-w0-ad7ff0/premium-w0-ad7ff0-kserve-5485d4b948-bpmgf" Oct 30 20:50:26.343603 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:26.343138 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/53cbd692-e9f8-46ba-8fde-3e5215a38904-model-cache\") pod \"premium-w0-ad7ff0-kserve-5485d4b948-bpmgf\" (UID: \"53cbd692-e9f8-46ba-8fde-3e5215a38904\") " pod="e2e-models-e2e-worker-w0-ad7ff0/premium-w0-ad7ff0-kserve-5485d4b948-bpmgf" Oct 30 20:50:26.343603 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:26.343216 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m6mt5\" (UniqueName: \"kubernetes.io/projected/53cbd692-e9f8-46ba-8fde-3e5215a38904-kube-api-access-m6mt5\") pod \"premium-w0-ad7ff0-kserve-5485d4b948-bpmgf\" (UID: \"53cbd692-e9f8-46ba-8fde-3e5215a38904\") " pod="e2e-models-e2e-worker-w0-ad7ff0/premium-w0-ad7ff0-kserve-5485d4b948-bpmgf" Oct 30 20:50:26.345187 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:26.344581 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/53cbd692-e9f8-46ba-8fde-3e5215a38904-tmp-dir\") pod \"premium-w0-ad7ff0-kserve-5485d4b948-bpmgf\" (UID: \"53cbd692-e9f8-46ba-8fde-3e5215a38904\") " pod="e2e-models-e2e-worker-w0-ad7ff0/premium-w0-ad7ff0-kserve-5485d4b948-bpmgf" Oct 30 20:50:26.345187 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:26.344809 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/53cbd692-e9f8-46ba-8fde-3e5215a38904-model-cache\") pod \"premium-w0-ad7ff0-kserve-5485d4b948-bpmgf\" (UID: \"53cbd692-e9f8-46ba-8fde-3e5215a38904\") " pod="e2e-models-e2e-worker-w0-ad7ff0/premium-w0-ad7ff0-kserve-5485d4b948-bpmgf" Oct 30 20:50:26.345187 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:26.345056 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/53cbd692-e9f8-46ba-8fde-3e5215a38904-home\") pod \"premium-w0-ad7ff0-kserve-5485d4b948-bpmgf\" (UID: \"53cbd692-e9f8-46ba-8fde-3e5215a38904\") " pod="e2e-models-e2e-worker-w0-ad7ff0/premium-w0-ad7ff0-kserve-5485d4b948-bpmgf" Oct 30 20:50:26.359214 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:26.359151 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/53cbd692-e9f8-46ba-8fde-3e5215a38904-tls-certs\") pod \"premium-w0-ad7ff0-kserve-5485d4b948-bpmgf\" (UID: \"53cbd692-e9f8-46ba-8fde-3e5215a38904\") " pod="e2e-models-e2e-worker-w0-ad7ff0/premium-w0-ad7ff0-kserve-5485d4b948-bpmgf" Oct 30 20:50:26.406544 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:26.394210 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m6mt5\" (UniqueName: \"kubernetes.io/projected/53cbd692-e9f8-46ba-8fde-3e5215a38904-kube-api-access-m6mt5\") pod \"premium-w0-ad7ff0-kserve-5485d4b948-bpmgf\" (UID: \"53cbd692-e9f8-46ba-8fde-3e5215a38904\") " pod="e2e-models-e2e-worker-w0-ad7ff0/premium-w0-ad7ff0-kserve-5485d4b948-bpmgf" Oct 30 20:50:26.406544 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:26.394463 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/53cbd692-e9f8-46ba-8fde-3e5215a38904-dshm\") pod \"premium-w0-ad7ff0-kserve-5485d4b948-bpmgf\" (UID: \"53cbd692-e9f8-46ba-8fde-3e5215a38904\") " pod="e2e-models-e2e-worker-w0-ad7ff0/premium-w0-ad7ff0-kserve-5485d4b948-bpmgf" Oct 30 20:50:26.406544 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:26.395012 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:50:26.406544 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:26.395157 4611 patch_prober.go:29] interesting pod/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-hx7mv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body= Oct 30 20:50:26.406544 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:26.395219 4611 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-hx7mv" podUID="52f7cad0-6a73-4573-9789-59ac6df33da4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" Oct 30 20:50:26.504530 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:26.503729 4611 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-ad7ff0/premium-w0-ad7ff0-kserve-5485d4b948-bpmgf" Oct 30 20:50:26.625265 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:26.624913 4611 patch_prober.go:29] interesting pod/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-khv95 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body= Oct 30 20:50:26.625265 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:26.624975 4611 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-khv95" podUID="62c91622-5518-4dc2-9f08-102dd0fcfb84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" Oct 30 20:50:26.935930 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:26.935837 4611 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-ad7ff0/premium-w0-ad7ff0-kserve-5485d4b948-bpmgf"] Oct 30 20:50:26.939450 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:50:26.939415 4611 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod53cbd692_e9f8_46ba_8fde_3e5215a38904.slice/crio-5a05ccbe8e50a91ebabe3bb62c97d9d7c8063ea376f57f9a7d0e7d04b48750a3 WatchSource:0}: Error finding container 5a05ccbe8e50a91ebabe3bb62c97d9d7c8063ea376f57f9a7d0e7d04b48750a3: Status 404 returned error can't find the container with id 5a05ccbe8e50a91ebabe3bb62c97d9d7c8063ea376f57f9a7d0e7d04b48750a3 Oct 30 20:50:27.020531 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:27.020167 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:50:27.058531 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:27.055488 4611 patch_prober.go:29] interesting pod/e2e-worker-w6-3d909d-openshift-default-77787695b7-trngw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body= Oct 30 20:50:27.058531 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:27.055574 4611 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-trngw" podUID="5ba753c3-5979-4263-8b90-2741747c8592" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" Oct 30 20:50:27.298590 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:27.297919 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:50:27.378528 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:27.378014 4611 patch_prober.go:29] interesting pod/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-hx7mv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body= Oct 30 20:50:27.378528 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:27.378091 4611 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-hx7mv" podUID="52f7cad0-6a73-4573-9789-59ac6df33da4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" Oct 30 20:50:27.536587 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:27.536540 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:50:27.624844 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:27.624808 4611 patch_prober.go:29] interesting pod/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-khv95 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body= Oct 30 20:50:27.625058 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:27.624856 4611 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-khv95" podUID="62c91622-5518-4dc2-9f08-102dd0fcfb84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" Oct 30 20:50:27.795941 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:27.795905 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-ad7ff0/premium-w0-ad7ff0-kserve-5485d4b948-bpmgf" event={"ID":"53cbd692-e9f8-46ba-8fde-3e5215a38904","Type":"ContainerStarted","Data":"01be21bd89a61b89518c9b2365f7320e7b6b1dcc5dfe6c2da5d92a14bd255f70"} Oct 30 20:50:27.796537 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:27.796488 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-ad7ff0/premium-w0-ad7ff0-kserve-5485d4b948-bpmgf" event={"ID":"53cbd692-e9f8-46ba-8fde-3e5215a38904","Type":"ContainerStarted","Data":"5a05ccbe8e50a91ebabe3bb62c97d9d7c8063ea376f57f9a7d0e7d04b48750a3"} Oct 30 20:50:27.825115 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:27.825067 4611 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-ad7ff0/premium-w0-ad7ff0-kserve-5485d4b948-bpmgf" podStartSLOduration=1.82505221 podStartE2EDuration="1.82505221s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:50:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:50:27.823929578 +0000 UTC m=+682.357956195" watchObservedRunningTime="2026-10-30 20:50:27.82505221 +0000 UTC m=+682.359078848" Oct 30 20:50:28.056669 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:28.054607 4611 patch_prober.go:29] interesting pod/e2e-worker-w6-3d909d-openshift-default-77787695b7-trngw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body= Oct 30 20:50:28.056669 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:28.054673 4611 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-trngw" podUID="5ba753c3-5979-4263-8b90-2741747c8592" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" Oct 30 20:50:28.375532 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:28.374976 4611 patch_prober.go:29] interesting pod/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-hx7mv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body= Oct 30 20:50:28.375532 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:28.375050 4611 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-hx7mv" podUID="52f7cad0-6a73-4573-9789-59ac6df33da4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" Oct 30 20:50:28.504697 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:28.504666 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ad7ff0/premium-w0-ad7ff0-kserve-5485d4b948-bpmgf" Oct 30 20:50:28.527829 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:28.527789 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:50:28.627537 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:28.627450 4611 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-khv95" Oct 30 20:50:28.628377 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:28.628361 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-khv95" Oct 30 20:50:28.629022 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:28.629002 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-khv95" Oct 30 20:50:29.049924 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:29.049833 4611 patch_prober.go:29] interesting pod/e2e-worker-w6-3d909d-openshift-default-77787695b7-trngw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body= Oct 30 20:50:29.050530 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:29.050474 4611 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-trngw" podUID="5ba753c3-5979-4263-8b90-2741747c8592" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" Oct 30 20:50:29.314038 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:29.313949 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:50:29.375532 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:29.374606 4611 patch_prober.go:29] interesting pod/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-hx7mv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body= Oct 30 20:50:29.375532 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:29.374679 4611 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-hx7mv" podUID="52f7cad0-6a73-4573-9789-59ac6df33da4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" Oct 30 20:50:30.018149 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:30.018122 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:50:30.049895 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:30.049860 4611 patch_prober.go:29] interesting pod/e2e-worker-w6-3d909d-openshift-default-77787695b7-trngw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body= Oct 30 20:50:30.050186 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:30.050155 4611 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-trngw" podUID="5ba753c3-5979-4263-8b90-2741747c8592" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" Oct 30 20:50:30.176721 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:30.176685 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:50:30.374158 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:30.374121 4611 patch_prober.go:29] interesting pod/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-hx7mv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body= Oct 30 20:50:30.374332 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:30.374179 4611 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-hx7mv" podUID="52f7cad0-6a73-4573-9789-59ac6df33da4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" Oct 30 20:50:30.752003 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:30.751906 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:50:30.950015 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:30.949961 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:50:31.066972 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:31.063781 4611 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-trngw" Oct 30 20:50:31.066972 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:31.065098 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-trngw" Oct 30 20:50:31.072621 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:31.072598 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-trngw" Oct 30 20:50:31.294531 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:31.294258 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:50:31.380219 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:31.380188 4611 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-hx7mv" Oct 30 20:50:31.381360 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:31.381331 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-hx7mv" Oct 30 20:50:31.382845 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:31.382816 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-hx7mv" Oct 30 20:50:31.913433 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:31.913397 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:50:32.311626 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:32.311534 4611 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-cb60a2/simulator-w3-cb60a2-kserve-6fd74b9d8b-vqfzf"] Oct 30 20:50:32.315295 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:32.315272 4611 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-cb60a2/simulator-w3-cb60a2-kserve-6fd74b9d8b-vqfzf" Oct 30 20:50:32.324169 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:32.324147 4611 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-cb60a2\"/\"openshift-service-ca.crt\"" Oct 30 20:50:32.324707 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:32.324682 4611 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-cb60a2\"/\"kube-root-ca.crt\"" Oct 30 20:50:32.324811 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:32.324727 4611 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-cb60a2\"/\"simulator-w3-cb60a2-kserve-self-signed-certs\"" Oct 30 20:50:32.325571 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:32.324728 4611 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-cb60a2\"/\"default-dockercfg-kf8b2\"" Oct 30 20:50:32.328532 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:32.328498 4611 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-cb60a2/simulator-w3-cb60a2-kserve-6fd74b9d8b-vqfzf"] Oct 30 20:50:32.329093 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:32.329074 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/711b167d-4e66-40d6-be75-33ede11f42c5-model-cache\") pod \"simulator-w3-cb60a2-kserve-6fd74b9d8b-vqfzf\" (UID: \"711b167d-4e66-40d6-be75-33ede11f42c5\") " pod="e2e-models-e2e-worker-w3-cb60a2/simulator-w3-cb60a2-kserve-6fd74b9d8b-vqfzf" Oct 30 20:50:32.329199 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:32.329099 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/711b167d-4e66-40d6-be75-33ede11f42c5-tmp-dir\") pod \"simulator-w3-cb60a2-kserve-6fd74b9d8b-vqfzf\" (UID: \"711b167d-4e66-40d6-be75-33ede11f42c5\") " pod="e2e-models-e2e-worker-w3-cb60a2/simulator-w3-cb60a2-kserve-6fd74b9d8b-vqfzf" Oct 30 20:50:32.329261 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:32.329219 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/711b167d-4e66-40d6-be75-33ede11f42c5-dshm\") pod \"simulator-w3-cb60a2-kserve-6fd74b9d8b-vqfzf\" (UID: \"711b167d-4e66-40d6-be75-33ede11f42c5\") " pod="e2e-models-e2e-worker-w3-cb60a2/simulator-w3-cb60a2-kserve-6fd74b9d8b-vqfzf" Oct 30 20:50:32.329261 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:32.329245 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/711b167d-4e66-40d6-be75-33ede11f42c5-home\") pod \"simulator-w3-cb60a2-kserve-6fd74b9d8b-vqfzf\" (UID: \"711b167d-4e66-40d6-be75-33ede11f42c5\") " pod="e2e-models-e2e-worker-w3-cb60a2/simulator-w3-cb60a2-kserve-6fd74b9d8b-vqfzf" Oct 30 20:50:32.329363 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:32.329271 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xvpzz\" (UniqueName: \"kubernetes.io/projected/711b167d-4e66-40d6-be75-33ede11f42c5-kube-api-access-xvpzz\") pod \"simulator-w3-cb60a2-kserve-6fd74b9d8b-vqfzf\" (UID: \"711b167d-4e66-40d6-be75-33ede11f42c5\") " pod="e2e-models-e2e-worker-w3-cb60a2/simulator-w3-cb60a2-kserve-6fd74b9d8b-vqfzf" Oct 30 20:50:32.329423 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:32.329381 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/711b167d-4e66-40d6-be75-33ede11f42c5-tls-certs\") pod \"simulator-w3-cb60a2-kserve-6fd74b9d8b-vqfzf\" (UID: \"711b167d-4e66-40d6-be75-33ede11f42c5\") " pod="e2e-models-e2e-worker-w3-cb60a2/simulator-w3-cb60a2-kserve-6fd74b9d8b-vqfzf" Oct 30 20:50:32.406480 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:32.406447 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:50:32.430031 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:32.430002 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/711b167d-4e66-40d6-be75-33ede11f42c5-tls-certs\") pod \"simulator-w3-cb60a2-kserve-6fd74b9d8b-vqfzf\" (UID: \"711b167d-4e66-40d6-be75-33ede11f42c5\") " pod="e2e-models-e2e-worker-w3-cb60a2/simulator-w3-cb60a2-kserve-6fd74b9d8b-vqfzf" Oct 30 20:50:32.430167 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:32.430100 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/711b167d-4e66-40d6-be75-33ede11f42c5-model-cache\") pod \"simulator-w3-cb60a2-kserve-6fd74b9d8b-vqfzf\" (UID: \"711b167d-4e66-40d6-be75-33ede11f42c5\") " pod="e2e-models-e2e-worker-w3-cb60a2/simulator-w3-cb60a2-kserve-6fd74b9d8b-vqfzf" Oct 30 20:50:32.430167 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:32.430129 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/711b167d-4e66-40d6-be75-33ede11f42c5-tmp-dir\") pod \"simulator-w3-cb60a2-kserve-6fd74b9d8b-vqfzf\" (UID: \"711b167d-4e66-40d6-be75-33ede11f42c5\") " pod="e2e-models-e2e-worker-w3-cb60a2/simulator-w3-cb60a2-kserve-6fd74b9d8b-vqfzf" Oct 30 20:50:32.430284 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:32.430215 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/711b167d-4e66-40d6-be75-33ede11f42c5-dshm\") pod \"simulator-w3-cb60a2-kserve-6fd74b9d8b-vqfzf\" (UID: \"711b167d-4e66-40d6-be75-33ede11f42c5\") " pod="e2e-models-e2e-worker-w3-cb60a2/simulator-w3-cb60a2-kserve-6fd74b9d8b-vqfzf" Oct 30 20:50:32.430284 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:32.430254 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/711b167d-4e66-40d6-be75-33ede11f42c5-home\") pod \"simulator-w3-cb60a2-kserve-6fd74b9d8b-vqfzf\" (UID: \"711b167d-4e66-40d6-be75-33ede11f42c5\") " pod="e2e-models-e2e-worker-w3-cb60a2/simulator-w3-cb60a2-kserve-6fd74b9d8b-vqfzf" Oct 30 20:50:32.430382 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:32.430285 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvpzz\" (UniqueName: \"kubernetes.io/projected/711b167d-4e66-40d6-be75-33ede11f42c5-kube-api-access-xvpzz\") pod \"simulator-w3-cb60a2-kserve-6fd74b9d8b-vqfzf\" (UID: \"711b167d-4e66-40d6-be75-33ede11f42c5\") " pod="e2e-models-e2e-worker-w3-cb60a2/simulator-w3-cb60a2-kserve-6fd74b9d8b-vqfzf" Oct 30 20:50:32.430571 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:32.430544 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/711b167d-4e66-40d6-be75-33ede11f42c5-model-cache\") pod \"simulator-w3-cb60a2-kserve-6fd74b9d8b-vqfzf\" (UID: \"711b167d-4e66-40d6-be75-33ede11f42c5\") " pod="e2e-models-e2e-worker-w3-cb60a2/simulator-w3-cb60a2-kserve-6fd74b9d8b-vqfzf" Oct 30 20:50:32.430682 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:32.430578 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/711b167d-4e66-40d6-be75-33ede11f42c5-tmp-dir\") pod \"simulator-w3-cb60a2-kserve-6fd74b9d8b-vqfzf\" (UID: \"711b167d-4e66-40d6-be75-33ede11f42c5\") " pod="e2e-models-e2e-worker-w3-cb60a2/simulator-w3-cb60a2-kserve-6fd74b9d8b-vqfzf" Oct 30 20:50:32.430682 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:32.430628 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/711b167d-4e66-40d6-be75-33ede11f42c5-home\") pod \"simulator-w3-cb60a2-kserve-6fd74b9d8b-vqfzf\" (UID: \"711b167d-4e66-40d6-be75-33ede11f42c5\") " pod="e2e-models-e2e-worker-w3-cb60a2/simulator-w3-cb60a2-kserve-6fd74b9d8b-vqfzf" Oct 30 20:50:32.432326 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:32.432302 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/711b167d-4e66-40d6-be75-33ede11f42c5-dshm\") pod \"simulator-w3-cb60a2-kserve-6fd74b9d8b-vqfzf\" (UID: \"711b167d-4e66-40d6-be75-33ede11f42c5\") " pod="e2e-models-e2e-worker-w3-cb60a2/simulator-w3-cb60a2-kserve-6fd74b9d8b-vqfzf" Oct 30 20:50:32.432594 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:32.432576 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/711b167d-4e66-40d6-be75-33ede11f42c5-tls-certs\") pod \"simulator-w3-cb60a2-kserve-6fd74b9d8b-vqfzf\" (UID: \"711b167d-4e66-40d6-be75-33ede11f42c5\") " pod="e2e-models-e2e-worker-w3-cb60a2/simulator-w3-cb60a2-kserve-6fd74b9d8b-vqfzf" Oct 30 20:50:32.441926 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:32.441905 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xvpzz\" (UniqueName: \"kubernetes.io/projected/711b167d-4e66-40d6-be75-33ede11f42c5-kube-api-access-xvpzz\") pod \"simulator-w3-cb60a2-kserve-6fd74b9d8b-vqfzf\" (UID: \"711b167d-4e66-40d6-be75-33ede11f42c5\") " pod="e2e-models-e2e-worker-w3-cb60a2/simulator-w3-cb60a2-kserve-6fd74b9d8b-vqfzf" Oct 30 20:50:32.605584 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:32.605546 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:50:32.627521 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:32.627479 4611 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-cb60a2/simulator-w3-cb60a2-kserve-6fd74b9d8b-vqfzf" Oct 30 20:50:32.919844 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:32.919818 4611 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-cb60a2/simulator-w3-cb60a2-kserve-6fd74b9d8b-vqfzf"] Oct 30 20:50:32.921184 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:50:32.921158 4611 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod711b167d_4e66_40d6_be75_33ede11f42c5.slice/crio-4c228916f6d3f64e0cdd9605efff747a55b580bace8ffa02febff8f046316694 WatchSource:0}: Error finding container 4c228916f6d3f64e0cdd9605efff747a55b580bace8ffa02febff8f046316694: Status 404 returned error can't find the container with id 4c228916f6d3f64e0cdd9605efff747a55b580bace8ffa02febff8f046316694 Oct 30 20:50:33.309982 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:33.309855 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:50:33.558949 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:33.558914 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:50:33.868092 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:33.868053 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-cb60a2/simulator-w3-cb60a2-kserve-6fd74b9d8b-vqfzf" event={"ID":"711b167d-4e66-40d6-be75-33ede11f42c5","Type":"ContainerStarted","Data":"9f55ee67ad822c60030083e1f0ed09869e45b9eb2d5d2eeef4659752733052d8"} Oct 30 20:50:33.868273 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:33.868102 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-cb60a2/simulator-w3-cb60a2-kserve-6fd74b9d8b-vqfzf" event={"ID":"711b167d-4e66-40d6-be75-33ede11f42c5","Type":"ContainerStarted","Data":"4c228916f6d3f64e0cdd9605efff747a55b580bace8ffa02febff8f046316694"} Oct 30 20:50:33.915943 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:33.915888 4611 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-cb60a2/simulator-w3-cb60a2-kserve-6fd74b9d8b-vqfzf" podStartSLOduration=1.9158718970000002 podStartE2EDuration="1.915871897s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:50:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:50:33.915726949 +0000 UTC m=+688.449753567" watchObservedRunningTime="2026-10-30 20:50:33.915871897 +0000 UTC m=+688.449898514" Oct 30 20:50:33.918774 ip-10-0-133-192 kubenswrapper[4611]: E1030 20:50:33.918750 4611 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hwzc6" containerName="extractor" Oct 30 20:50:34.406900 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:34.406859 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:50:34.628374 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:34.628340 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-cb60a2/simulator-w3-cb60a2-kserve-6fd74b9d8b-vqfzf" Oct 30 20:50:35.122578 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:35.122541 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:50:35.785555 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:35.785467 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:50:36.504482 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:36.504445 4611 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-ad7ff0/premium-w0-ad7ff0-kserve-5485d4b948-bpmgf" Oct 30 20:50:36.516756 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:36.516733 4611 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-ad7ff0/premium-w0-ad7ff0-kserve-5485d4b948-bpmgf" Oct 30 20:50:36.589804 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:36.589764 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:50:36.894489 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:36.894459 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ad7ff0/premium-w0-ad7ff0-kserve-5485d4b948-bpmgf" Oct 30 20:50:37.295084 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:37.294990 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:50:38.101638 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:38.101602 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:50:38.995988 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:38.995956 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:50:39.501982 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:39.501951 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:50:39.791118 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:39.791007 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:50:40.373986 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:40.373950 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:50:40.593921 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:40.593879 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:50:41.198959 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:41.198904 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:50:41.803222 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:41.803188 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:50:42.592159 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:42.592109 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:50:42.628099 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:42.628064 4611 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-cb60a2/simulator-w3-cb60a2-kserve-6fd74b9d8b-vqfzf" Oct 30 20:50:42.640637 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:42.640611 4611 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-cb60a2/simulator-w3-cb60a2-kserve-6fd74b9d8b-vqfzf" Oct 30 20:50:42.969103 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:42.969071 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-cb60a2/simulator-w3-cb60a2-kserve-6fd74b9d8b-vqfzf" Oct 30 20:50:43.265494 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:43.265407 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:50:43.931320 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:43.931274 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:50:44.680621 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:44.680586 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:50:45.555546 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:45.555493 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:50:46.394480 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:46.394443 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:50:47.236022 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:47.235988 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:50:47.888813 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:47.888775 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:50:48.345776 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:48.345696 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:50:48.507995 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:48.507963 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:50:49.401155 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:49.401086 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:50:49.848429 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:49.848348 4611 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-b8aee1/premium-w1-b8aee1-kserve-786d575599-htc8d"] Oct 30 20:50:49.853490 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:49.853467 4611 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-b8aee1/premium-w1-b8aee1-kserve-786d575599-htc8d" Oct 30 20:50:49.862432 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:49.862404 4611 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-b8aee1/premium-w1-b8aee1-kserve-786d575599-htc8d"] Oct 30 20:50:49.862722 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:49.862706 4611 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-b8aee1\"/\"kube-root-ca.crt\"" Oct 30 20:50:49.862782 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:49.862753 4611 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-b8aee1\"/\"openshift-service-ca.crt\"" Oct 30 20:50:49.862823 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:49.862808 4611 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-b8aee1\"/\"premium-w1-b8aee1-kserve-self-signed-certs\"" Oct 30 20:50:49.862860 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:49.862813 4611 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-b8aee1\"/\"default-dockercfg-qmhc7\"" Oct 30 20:50:49.930383 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:49.930352 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/77368afe-edc8-4fa5-8010-d321965fb38e-dshm\") pod \"premium-w1-b8aee1-kserve-786d575599-htc8d\" (UID: \"77368afe-edc8-4fa5-8010-d321965fb38e\") " pod="e2e-models-e2e-worker-w1-b8aee1/premium-w1-b8aee1-kserve-786d575599-htc8d" Oct 30 20:50:49.930570 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:49.930390 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/77368afe-edc8-4fa5-8010-d321965fb38e-tmp-dir\") pod \"premium-w1-b8aee1-kserve-786d575599-htc8d\" (UID: \"77368afe-edc8-4fa5-8010-d321965fb38e\") " pod="e2e-models-e2e-worker-w1-b8aee1/premium-w1-b8aee1-kserve-786d575599-htc8d" Oct 30 20:50:49.930570 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:49.930419 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/77368afe-edc8-4fa5-8010-d321965fb38e-model-cache\") pod \"premium-w1-b8aee1-kserve-786d575599-htc8d\" (UID: \"77368afe-edc8-4fa5-8010-d321965fb38e\") " pod="e2e-models-e2e-worker-w1-b8aee1/premium-w1-b8aee1-kserve-786d575599-htc8d" Oct 30 20:50:49.930658 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:49.930623 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5wgpw\" (UniqueName: \"kubernetes.io/projected/77368afe-edc8-4fa5-8010-d321965fb38e-kube-api-access-5wgpw\") pod \"premium-w1-b8aee1-kserve-786d575599-htc8d\" (UID: \"77368afe-edc8-4fa5-8010-d321965fb38e\") " pod="e2e-models-e2e-worker-w1-b8aee1/premium-w1-b8aee1-kserve-786d575599-htc8d" Oct 30 20:50:49.930711 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:49.930670 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/77368afe-edc8-4fa5-8010-d321965fb38e-home\") pod \"premium-w1-b8aee1-kserve-786d575599-htc8d\" (UID: \"77368afe-edc8-4fa5-8010-d321965fb38e\") " pod="e2e-models-e2e-worker-w1-b8aee1/premium-w1-b8aee1-kserve-786d575599-htc8d" Oct 30 20:50:49.930802 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:49.930781 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/77368afe-edc8-4fa5-8010-d321965fb38e-tls-certs\") pod \"premium-w1-b8aee1-kserve-786d575599-htc8d\" (UID: \"77368afe-edc8-4fa5-8010-d321965fb38e\") " pod="e2e-models-e2e-worker-w1-b8aee1/premium-w1-b8aee1-kserve-786d575599-htc8d" Oct 30 20:50:49.943000 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:49.942965 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:50:50.034359 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:50.034316 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5wgpw\" (UniqueName: \"kubernetes.io/projected/77368afe-edc8-4fa5-8010-d321965fb38e-kube-api-access-5wgpw\") pod \"premium-w1-b8aee1-kserve-786d575599-htc8d\" (UID: \"77368afe-edc8-4fa5-8010-d321965fb38e\") " pod="e2e-models-e2e-worker-w1-b8aee1/premium-w1-b8aee1-kserve-786d575599-htc8d" Oct 30 20:50:50.034982 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:50.034952 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/77368afe-edc8-4fa5-8010-d321965fb38e-home\") pod \"premium-w1-b8aee1-kserve-786d575599-htc8d\" (UID: \"77368afe-edc8-4fa5-8010-d321965fb38e\") " pod="e2e-models-e2e-worker-w1-b8aee1/premium-w1-b8aee1-kserve-786d575599-htc8d" Oct 30 20:50:50.035202 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:50.035184 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/77368afe-edc8-4fa5-8010-d321965fb38e-tls-certs\") pod \"premium-w1-b8aee1-kserve-786d575599-htc8d\" (UID: \"77368afe-edc8-4fa5-8010-d321965fb38e\") " pod="e2e-models-e2e-worker-w1-b8aee1/premium-w1-b8aee1-kserve-786d575599-htc8d" Oct 30 20:50:50.035540 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:50.035501 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/77368afe-edc8-4fa5-8010-d321965fb38e-dshm\") pod \"premium-w1-b8aee1-kserve-786d575599-htc8d\" (UID: \"77368afe-edc8-4fa5-8010-d321965fb38e\") " pod="e2e-models-e2e-worker-w1-b8aee1/premium-w1-b8aee1-kserve-786d575599-htc8d" Oct 30 20:50:50.035705 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:50.035679 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/77368afe-edc8-4fa5-8010-d321965fb38e-tmp-dir\") pod \"premium-w1-b8aee1-kserve-786d575599-htc8d\" (UID: \"77368afe-edc8-4fa5-8010-d321965fb38e\") " pod="e2e-models-e2e-worker-w1-b8aee1/premium-w1-b8aee1-kserve-786d575599-htc8d" Oct 30 20:50:50.035848 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:50.035833 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/77368afe-edc8-4fa5-8010-d321965fb38e-model-cache\") pod \"premium-w1-b8aee1-kserve-786d575599-htc8d\" (UID: \"77368afe-edc8-4fa5-8010-d321965fb38e\") " pod="e2e-models-e2e-worker-w1-b8aee1/premium-w1-b8aee1-kserve-786d575599-htc8d" Oct 30 20:50:50.036391 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:50.036369 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/77368afe-edc8-4fa5-8010-d321965fb38e-model-cache\") pod \"premium-w1-b8aee1-kserve-786d575599-htc8d\" (UID: \"77368afe-edc8-4fa5-8010-d321965fb38e\") " pod="e2e-models-e2e-worker-w1-b8aee1/premium-w1-b8aee1-kserve-786d575599-htc8d" Oct 30 20:50:50.036829 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:50.036811 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/77368afe-edc8-4fa5-8010-d321965fb38e-home\") pod \"premium-w1-b8aee1-kserve-786d575599-htc8d\" (UID: \"77368afe-edc8-4fa5-8010-d321965fb38e\") " pod="e2e-models-e2e-worker-w1-b8aee1/premium-w1-b8aee1-kserve-786d575599-htc8d" Oct 30 20:50:50.038820 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:50.038797 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/77368afe-edc8-4fa5-8010-d321965fb38e-tmp-dir\") pod \"premium-w1-b8aee1-kserve-786d575599-htc8d\" (UID: \"77368afe-edc8-4fa5-8010-d321965fb38e\") " pod="e2e-models-e2e-worker-w1-b8aee1/premium-w1-b8aee1-kserve-786d575599-htc8d" Oct 30 20:50:50.042150 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:50.042095 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/77368afe-edc8-4fa5-8010-d321965fb38e-tls-certs\") pod \"premium-w1-b8aee1-kserve-786d575599-htc8d\" (UID: \"77368afe-edc8-4fa5-8010-d321965fb38e\") " pod="e2e-models-e2e-worker-w1-b8aee1/premium-w1-b8aee1-kserve-786d575599-htc8d" Oct 30 20:50:50.044654 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:50.044605 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/77368afe-edc8-4fa5-8010-d321965fb38e-dshm\") pod \"premium-w1-b8aee1-kserve-786d575599-htc8d\" (UID: \"77368afe-edc8-4fa5-8010-d321965fb38e\") " pod="e2e-models-e2e-worker-w1-b8aee1/premium-w1-b8aee1-kserve-786d575599-htc8d" Oct 30 20:50:50.057048 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:50.057019 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5wgpw\" (UniqueName: \"kubernetes.io/projected/77368afe-edc8-4fa5-8010-d321965fb38e-kube-api-access-5wgpw\") pod \"premium-w1-b8aee1-kserve-786d575599-htc8d\" (UID: \"77368afe-edc8-4fa5-8010-d321965fb38e\") " pod="e2e-models-e2e-worker-w1-b8aee1/premium-w1-b8aee1-kserve-786d575599-htc8d" Oct 30 20:50:50.100604 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:50.100535 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:50:50.168213 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:50.168189 4611 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-b8aee1/premium-w1-b8aee1-kserve-786d575599-htc8d" Oct 30 20:50:50.464122 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:50.464096 4611 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-b8aee1/premium-w1-b8aee1-kserve-786d575599-htc8d"] Oct 30 20:50:50.465847 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:50:50.465819 4611 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod77368afe_edc8_4fa5_8010_d321965fb38e.slice/crio-f4ae98e2cf300b0abdeedf0e8e61248b7eb03e6a8795aace301d395247412194 WatchSource:0}: Error finding container f4ae98e2cf300b0abdeedf0e8e61248b7eb03e6a8795aace301d395247412194: Status 404 returned error can't find the container with id f4ae98e2cf300b0abdeedf0e8e61248b7eb03e6a8795aace301d395247412194 Oct 30 20:50:50.909593 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:50.909558 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:50:51.006239 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:51.006199 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-b8aee1/premium-w1-b8aee1-kserve-786d575599-htc8d" event={"ID":"77368afe-edc8-4fa5-8010-d321965fb38e","Type":"ContainerStarted","Data":"9446b05e516e9d84fb6c0e27f5f22ca129ff843c39d4c225c6ee73c7219a0892"} Oct 30 20:50:51.006438 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:51.006248 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-b8aee1/premium-w1-b8aee1-kserve-786d575599-htc8d" event={"ID":"77368afe-edc8-4fa5-8010-d321965fb38e","Type":"ContainerStarted","Data":"f4ae98e2cf300b0abdeedf0e8e61248b7eb03e6a8795aace301d395247412194"} Oct 30 20:50:51.025931 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:51.025881 4611 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-b8aee1/premium-w1-b8aee1-kserve-786d575599-htc8d" podStartSLOduration=2.025867894 podStartE2EDuration="2.025867894s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:50:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:50:51.025254826 +0000 UTC m=+705.559281442" watchObservedRunningTime="2026-10-30 20:50:51.025867894 +0000 UTC m=+705.559894510" Oct 30 20:50:51.169211 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:51.169137 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b8aee1/premium-w1-b8aee1-kserve-786d575599-htc8d" Oct 30 20:50:51.517362 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:51.517211 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:50:52.103120 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:52.103080 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:50:52.816170 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:52.816140 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:50:52.995648 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:52.995607 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:50:53.508181 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:53.508143 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:50:54.046304 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:54.046270 4611 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-14cca2/premium-w5-14cca2-kserve-6878777857-wpjtj"] Oct 30 20:50:54.052593 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:54.051421 4611 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-14cca2/premium-w5-14cca2-kserve-6878777857-wpjtj" Oct 30 20:50:54.062337 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:54.062305 4611 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-14cca2\"/\"openshift-service-ca.crt\"" Oct 30 20:50:54.062691 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:54.062668 4611 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-14cca2\"/\"default-dockercfg-lspws\"" Oct 30 20:50:54.062899 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:54.062876 4611 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-14cca2\"/\"kube-root-ca.crt\"" Oct 30 20:50:54.063054 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:54.062888 4611 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-14cca2\"/\"premium-w5-14cca2-kserve-self-signed-certs\"" Oct 30 20:50:54.063476 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:54.063450 4611 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-14cca2/premium-w5-14cca2-kserve-6878777857-wpjtj"] Oct 30 20:50:54.147657 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:54.147623 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:50:54.185435 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:54.185414 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b99eb76b-d1c6-49e9-9809-c351c0f5a4de-tmp-dir\") pod \"premium-w5-14cca2-kserve-6878777857-wpjtj\" (UID: \"b99eb76b-d1c6-49e9-9809-c351c0f5a4de\") " pod="e2e-models-e2e-worker-w5-14cca2/premium-w5-14cca2-kserve-6878777857-wpjtj" Oct 30 20:50:54.185586 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:54.185457 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b99eb76b-d1c6-49e9-9809-c351c0f5a4de-home\") pod \"premium-w5-14cca2-kserve-6878777857-wpjtj\" (UID: \"b99eb76b-d1c6-49e9-9809-c351c0f5a4de\") " pod="e2e-models-e2e-worker-w5-14cca2/premium-w5-14cca2-kserve-6878777857-wpjtj" Oct 30 20:50:54.185586 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:54.185480 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b99eb76b-d1c6-49e9-9809-c351c0f5a4de-dshm\") pod \"premium-w5-14cca2-kserve-6878777857-wpjtj\" (UID: \"b99eb76b-d1c6-49e9-9809-c351c0f5a4de\") " pod="e2e-models-e2e-worker-w5-14cca2/premium-w5-14cca2-kserve-6878777857-wpjtj" Oct 30 20:50:54.185586 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:54.185567 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b99eb76b-d1c6-49e9-9809-c351c0f5a4de-tls-certs\") pod \"premium-w5-14cca2-kserve-6878777857-wpjtj\" (UID: \"b99eb76b-d1c6-49e9-9809-c351c0f5a4de\") " pod="e2e-models-e2e-worker-w5-14cca2/premium-w5-14cca2-kserve-6878777857-wpjtj" Oct 30 20:50:54.185719 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:54.185672 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wbl24\" (UniqueName: \"kubernetes.io/projected/b99eb76b-d1c6-49e9-9809-c351c0f5a4de-kube-api-access-wbl24\") pod \"premium-w5-14cca2-kserve-6878777857-wpjtj\" (UID: \"b99eb76b-d1c6-49e9-9809-c351c0f5a4de\") " pod="e2e-models-e2e-worker-w5-14cca2/premium-w5-14cca2-kserve-6878777857-wpjtj" Oct 30 20:50:54.185779 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:54.185715 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b99eb76b-d1c6-49e9-9809-c351c0f5a4de-model-cache\") pod \"premium-w5-14cca2-kserve-6878777857-wpjtj\" (UID: \"b99eb76b-d1c6-49e9-9809-c351c0f5a4de\") " pod="e2e-models-e2e-worker-w5-14cca2/premium-w5-14cca2-kserve-6878777857-wpjtj" Oct 30 20:50:54.287129 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:54.287094 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b99eb76b-d1c6-49e9-9809-c351c0f5a4de-home\") pod \"premium-w5-14cca2-kserve-6878777857-wpjtj\" (UID: \"b99eb76b-d1c6-49e9-9809-c351c0f5a4de\") " pod="e2e-models-e2e-worker-w5-14cca2/premium-w5-14cca2-kserve-6878777857-wpjtj" Oct 30 20:50:54.287326 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:54.287141 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b99eb76b-d1c6-49e9-9809-c351c0f5a4de-dshm\") pod \"premium-w5-14cca2-kserve-6878777857-wpjtj\" (UID: \"b99eb76b-d1c6-49e9-9809-c351c0f5a4de\") " pod="e2e-models-e2e-worker-w5-14cca2/premium-w5-14cca2-kserve-6878777857-wpjtj" Oct 30 20:50:54.287326 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:54.287177 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b99eb76b-d1c6-49e9-9809-c351c0f5a4de-tls-certs\") pod \"premium-w5-14cca2-kserve-6878777857-wpjtj\" (UID: \"b99eb76b-d1c6-49e9-9809-c351c0f5a4de\") " pod="e2e-models-e2e-worker-w5-14cca2/premium-w5-14cca2-kserve-6878777857-wpjtj" Oct 30 20:50:54.287326 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:54.287231 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wbl24\" (UniqueName: \"kubernetes.io/projected/b99eb76b-d1c6-49e9-9809-c351c0f5a4de-kube-api-access-wbl24\") pod \"premium-w5-14cca2-kserve-6878777857-wpjtj\" (UID: \"b99eb76b-d1c6-49e9-9809-c351c0f5a4de\") " pod="e2e-models-e2e-worker-w5-14cca2/premium-w5-14cca2-kserve-6878777857-wpjtj" Oct 30 20:50:54.287326 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:54.287271 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b99eb76b-d1c6-49e9-9809-c351c0f5a4de-model-cache\") pod \"premium-w5-14cca2-kserve-6878777857-wpjtj\" (UID: \"b99eb76b-d1c6-49e9-9809-c351c0f5a4de\") " pod="e2e-models-e2e-worker-w5-14cca2/premium-w5-14cca2-kserve-6878777857-wpjtj" Oct 30 20:50:54.287326 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:54.287310 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b99eb76b-d1c6-49e9-9809-c351c0f5a4de-tmp-dir\") pod \"premium-w5-14cca2-kserve-6878777857-wpjtj\" (UID: \"b99eb76b-d1c6-49e9-9809-c351c0f5a4de\") " pod="e2e-models-e2e-worker-w5-14cca2/premium-w5-14cca2-kserve-6878777857-wpjtj" Oct 30 20:50:54.287675 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:54.287651 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b99eb76b-d1c6-49e9-9809-c351c0f5a4de-home\") pod \"premium-w5-14cca2-kserve-6878777857-wpjtj\" (UID: \"b99eb76b-d1c6-49e9-9809-c351c0f5a4de\") " pod="e2e-models-e2e-worker-w5-14cca2/premium-w5-14cca2-kserve-6878777857-wpjtj" Oct 30 20:50:54.287747 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:54.287670 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b99eb76b-d1c6-49e9-9809-c351c0f5a4de-model-cache\") pod \"premium-w5-14cca2-kserve-6878777857-wpjtj\" (UID: \"b99eb76b-d1c6-49e9-9809-c351c0f5a4de\") " pod="e2e-models-e2e-worker-w5-14cca2/premium-w5-14cca2-kserve-6878777857-wpjtj" Oct 30 20:50:54.287811 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:54.287791 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b99eb76b-d1c6-49e9-9809-c351c0f5a4de-tmp-dir\") pod \"premium-w5-14cca2-kserve-6878777857-wpjtj\" (UID: \"b99eb76b-d1c6-49e9-9809-c351c0f5a4de\") " pod="e2e-models-e2e-worker-w5-14cca2/premium-w5-14cca2-kserve-6878777857-wpjtj" Oct 30 20:50:54.289486 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:54.289459 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b99eb76b-d1c6-49e9-9809-c351c0f5a4de-dshm\") pod \"premium-w5-14cca2-kserve-6878777857-wpjtj\" (UID: \"b99eb76b-d1c6-49e9-9809-c351c0f5a4de\") " pod="e2e-models-e2e-worker-w5-14cca2/premium-w5-14cca2-kserve-6878777857-wpjtj" Oct 30 20:50:54.289767 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:54.289751 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b99eb76b-d1c6-49e9-9809-c351c0f5a4de-tls-certs\") pod \"premium-w5-14cca2-kserve-6878777857-wpjtj\" (UID: \"b99eb76b-d1c6-49e9-9809-c351c0f5a4de\") " pod="e2e-models-e2e-worker-w5-14cca2/premium-w5-14cca2-kserve-6878777857-wpjtj" Oct 30 20:50:54.298351 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:54.298283 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wbl24\" (UniqueName: \"kubernetes.io/projected/b99eb76b-d1c6-49e9-9809-c351c0f5a4de-kube-api-access-wbl24\") pod \"premium-w5-14cca2-kserve-6878777857-wpjtj\" (UID: \"b99eb76b-d1c6-49e9-9809-c351c0f5a4de\") " pod="e2e-models-e2e-worker-w5-14cca2/premium-w5-14cca2-kserve-6878777857-wpjtj" Oct 30 20:50:54.357912 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:54.357882 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:50:54.372422 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:54.372398 4611 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-14cca2/premium-w5-14cca2-kserve-6878777857-wpjtj" Oct 30 20:50:54.650814 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:54.650790 4611 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-14cca2/premium-w5-14cca2-kserve-6878777857-wpjtj"] Oct 30 20:50:54.653455 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:50:54.653428 4611 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb99eb76b_d1c6_49e9_9809_c351c0f5a4de.slice/crio-76a0c637a5b71d45e9c6b3862658c946946c7fb05aa6e54316e8326be92fd82e WatchSource:0}: Error finding container 76a0c637a5b71d45e9c6b3862658c946946c7fb05aa6e54316e8326be92fd82e: Status 404 returned error can't find the container with id 76a0c637a5b71d45e9c6b3862658c946946c7fb05aa6e54316e8326be92fd82e Oct 30 20:50:54.949222 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:54.949131 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:50:55.029734 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:55.029696 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-14cca2/premium-w5-14cca2-kserve-6878777857-wpjtj" event={"ID":"b99eb76b-d1c6-49e9-9809-c351c0f5a4de","Type":"ContainerStarted","Data":"d406ff442d0c9a437cc2ff961f6cb0177e6a528cd2212db4510a648c6f8e2446"} Oct 30 20:50:55.029734 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:55.029739 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-14cca2/premium-w5-14cca2-kserve-6878777857-wpjtj" event={"ID":"b99eb76b-d1c6-49e9-9809-c351c0f5a4de","Type":"ContainerStarted","Data":"76a0c637a5b71d45e9c6b3862658c946946c7fb05aa6e54316e8326be92fd82e"} Oct 30 20:50:55.048856 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:55.048808 4611 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-14cca2/premium-w5-14cca2-kserve-6878777857-wpjtj" podStartSLOduration=1.048794518 podStartE2EDuration="1.048794518s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:50:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:50:55.046888003 +0000 UTC m=+709.580914619" watchObservedRunningTime="2026-10-30 20:50:55.048794518 +0000 UTC m=+709.582821441" Oct 30 20:50:55.373169 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:55.373133 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-14cca2/premium-w5-14cca2-kserve-6878777857-wpjtj" Oct 30 20:50:55.772794 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:55.772711 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:50:56.612964 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:56.612935 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:50:56.764294 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:56.764262 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:50:57.306373 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:57.306338 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:50:57.810976 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:57.810938 4611 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-3d909d/premium-w6-3d909d-kserve-765648946f-2qvd2"] Oct 30 20:50:57.815270 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:57.815249 4611 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-3d909d/premium-w6-3d909d-kserve-765648946f-2qvd2" Oct 30 20:50:57.824486 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:57.824464 4611 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-3d909d\"/\"openshift-service-ca.crt\"" Oct 30 20:50:57.824635 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:57.824617 4611 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-3d909d\"/\"default-dockercfg-2wvvq\"" Oct 30 20:50:57.824714 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:57.824700 4611 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-3d909d\"/\"premium-w6-3d909d-kserve-self-signed-certs\"" Oct 30 20:50:57.826726 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:57.826706 4611 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-3d909d\"/\"kube-root-ca.crt\"" Oct 30 20:50:57.827282 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:57.827263 4611 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-3d909d/premium-w6-3d909d-kserve-765648946f-2qvd2"] Oct 30 20:50:57.905449 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:57.905414 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:50:57.925316 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:57.925285 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fca5d697-109d-4e90-ab41-c40ad31af874-dshm\") pod \"premium-w6-3d909d-kserve-765648946f-2qvd2\" (UID: \"fca5d697-109d-4e90-ab41-c40ad31af874\") " pod="e2e-models-e2e-worker-w6-3d909d/premium-w6-3d909d-kserve-765648946f-2qvd2" Oct 30 20:50:57.925456 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:57.925393 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t4984\" (UniqueName: \"kubernetes.io/projected/fca5d697-109d-4e90-ab41-c40ad31af874-kube-api-access-t4984\") pod \"premium-w6-3d909d-kserve-765648946f-2qvd2\" (UID: \"fca5d697-109d-4e90-ab41-c40ad31af874\") " pod="e2e-models-e2e-worker-w6-3d909d/premium-w6-3d909d-kserve-765648946f-2qvd2" Oct 30 20:50:57.925522 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:57.925457 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fca5d697-109d-4e90-ab41-c40ad31af874-home\") pod \"premium-w6-3d909d-kserve-765648946f-2qvd2\" (UID: \"fca5d697-109d-4e90-ab41-c40ad31af874\") " pod="e2e-models-e2e-worker-w6-3d909d/premium-w6-3d909d-kserve-765648946f-2qvd2" Oct 30 20:50:57.925576 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:57.925535 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fca5d697-109d-4e90-ab41-c40ad31af874-tmp-dir\") pod \"premium-w6-3d909d-kserve-765648946f-2qvd2\" (UID: \"fca5d697-109d-4e90-ab41-c40ad31af874\") " pod="e2e-models-e2e-worker-w6-3d909d/premium-w6-3d909d-kserve-765648946f-2qvd2" Oct 30 20:50:57.925576 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:57.925559 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fca5d697-109d-4e90-ab41-c40ad31af874-model-cache\") pod \"premium-w6-3d909d-kserve-765648946f-2qvd2\" (UID: \"fca5d697-109d-4e90-ab41-c40ad31af874\") " pod="e2e-models-e2e-worker-w6-3d909d/premium-w6-3d909d-kserve-765648946f-2qvd2" Oct 30 20:50:57.925656 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:57.925586 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fca5d697-109d-4e90-ab41-c40ad31af874-tls-certs\") pod \"premium-w6-3d909d-kserve-765648946f-2qvd2\" (UID: \"fca5d697-109d-4e90-ab41-c40ad31af874\") " pod="e2e-models-e2e-worker-w6-3d909d/premium-w6-3d909d-kserve-765648946f-2qvd2" Oct 30 20:50:58.031531 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:58.027254 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fca5d697-109d-4e90-ab41-c40ad31af874-tmp-dir\") pod \"premium-w6-3d909d-kserve-765648946f-2qvd2\" (UID: \"fca5d697-109d-4e90-ab41-c40ad31af874\") " pod="e2e-models-e2e-worker-w6-3d909d/premium-w6-3d909d-kserve-765648946f-2qvd2" Oct 30 20:50:58.031531 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:58.027308 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fca5d697-109d-4e90-ab41-c40ad31af874-model-cache\") pod \"premium-w6-3d909d-kserve-765648946f-2qvd2\" (UID: \"fca5d697-109d-4e90-ab41-c40ad31af874\") " pod="e2e-models-e2e-worker-w6-3d909d/premium-w6-3d909d-kserve-765648946f-2qvd2" Oct 30 20:50:58.031531 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:58.027349 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fca5d697-109d-4e90-ab41-c40ad31af874-tls-certs\") pod \"premium-w6-3d909d-kserve-765648946f-2qvd2\" (UID: \"fca5d697-109d-4e90-ab41-c40ad31af874\") " pod="e2e-models-e2e-worker-w6-3d909d/premium-w6-3d909d-kserve-765648946f-2qvd2" Oct 30 20:50:58.031531 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:58.027473 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fca5d697-109d-4e90-ab41-c40ad31af874-dshm\") pod \"premium-w6-3d909d-kserve-765648946f-2qvd2\" (UID: \"fca5d697-109d-4e90-ab41-c40ad31af874\") " pod="e2e-models-e2e-worker-w6-3d909d/premium-w6-3d909d-kserve-765648946f-2qvd2" Oct 30 20:50:58.031531 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:58.027573 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t4984\" (UniqueName: \"kubernetes.io/projected/fca5d697-109d-4e90-ab41-c40ad31af874-kube-api-access-t4984\") pod \"premium-w6-3d909d-kserve-765648946f-2qvd2\" (UID: \"fca5d697-109d-4e90-ab41-c40ad31af874\") " pod="e2e-models-e2e-worker-w6-3d909d/premium-w6-3d909d-kserve-765648946f-2qvd2" Oct 30 20:50:58.031531 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:58.027611 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fca5d697-109d-4e90-ab41-c40ad31af874-home\") pod \"premium-w6-3d909d-kserve-765648946f-2qvd2\" (UID: \"fca5d697-109d-4e90-ab41-c40ad31af874\") " pod="e2e-models-e2e-worker-w6-3d909d/premium-w6-3d909d-kserve-765648946f-2qvd2" Oct 30 20:50:58.031531 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:58.028068 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fca5d697-109d-4e90-ab41-c40ad31af874-home\") pod \"premium-w6-3d909d-kserve-765648946f-2qvd2\" (UID: \"fca5d697-109d-4e90-ab41-c40ad31af874\") " pod="e2e-models-e2e-worker-w6-3d909d/premium-w6-3d909d-kserve-765648946f-2qvd2" Oct 30 20:50:58.031531 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:58.028333 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fca5d697-109d-4e90-ab41-c40ad31af874-tmp-dir\") pod \"premium-w6-3d909d-kserve-765648946f-2qvd2\" (UID: \"fca5d697-109d-4e90-ab41-c40ad31af874\") " pod="e2e-models-e2e-worker-w6-3d909d/premium-w6-3d909d-kserve-765648946f-2qvd2" Oct 30 20:50:58.031531 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:58.028590 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fca5d697-109d-4e90-ab41-c40ad31af874-model-cache\") pod \"premium-w6-3d909d-kserve-765648946f-2qvd2\" (UID: \"fca5d697-109d-4e90-ab41-c40ad31af874\") " pod="e2e-models-e2e-worker-w6-3d909d/premium-w6-3d909d-kserve-765648946f-2qvd2" Oct 30 20:50:58.032171 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:58.032120 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fca5d697-109d-4e90-ab41-c40ad31af874-tls-certs\") pod \"premium-w6-3d909d-kserve-765648946f-2qvd2\" (UID: \"fca5d697-109d-4e90-ab41-c40ad31af874\") " pod="e2e-models-e2e-worker-w6-3d909d/premium-w6-3d909d-kserve-765648946f-2qvd2" Oct 30 20:50:58.046242 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:58.046208 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fca5d697-109d-4e90-ab41-c40ad31af874-dshm\") pod \"premium-w6-3d909d-kserve-765648946f-2qvd2\" (UID: \"fca5d697-109d-4e90-ab41-c40ad31af874\") " pod="e2e-models-e2e-worker-w6-3d909d/premium-w6-3d909d-kserve-765648946f-2qvd2" Oct 30 20:50:58.052629 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:58.052487 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:50:58.056018 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:58.055949 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t4984\" (UniqueName: \"kubernetes.io/projected/fca5d697-109d-4e90-ab41-c40ad31af874-kube-api-access-t4984\") pod \"premium-w6-3d909d-kserve-765648946f-2qvd2\" (UID: \"fca5d697-109d-4e90-ab41-c40ad31af874\") " pod="e2e-models-e2e-worker-w6-3d909d/premium-w6-3d909d-kserve-765648946f-2qvd2" Oct 30 20:50:58.128775 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:58.128737 4611 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-3d909d/premium-w6-3d909d-kserve-765648946f-2qvd2" Oct 30 20:50:58.408138 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:58.408103 4611 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-3d909d/premium-w6-3d909d-kserve-765648946f-2qvd2"] Oct 30 20:50:58.409302 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:50:58.409279 4611 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfca5d697_109d_4e90_ab41_c40ad31af874.slice/crio-e886a59e35f43dc34aa7fa2211f0bf8bb4818372a50a502b72a2d7be0f68a1df WatchSource:0}: Error finding container e886a59e35f43dc34aa7fa2211f0bf8bb4818372a50a502b72a2d7be0f68a1df: Status 404 returned error can't find the container with id e886a59e35f43dc34aa7fa2211f0bf8bb4818372a50a502b72a2d7be0f68a1df Oct 30 20:50:58.791267 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:58.791120 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:50:59.059065 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:59.058991 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-3d909d/premium-w6-3d909d-kserve-765648946f-2qvd2" event={"ID":"fca5d697-109d-4e90-ab41-c40ad31af874","Type":"ContainerStarted","Data":"65bafa3a9f701bd8a0b3de653b025974cd022196ab21c4096ab2c3222e53b795"} Oct 30 20:50:59.059065 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:59.059035 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-3d909d/premium-w6-3d909d-kserve-765648946f-2qvd2" event={"ID":"fca5d697-109d-4e90-ab41-c40ad31af874","Type":"ContainerStarted","Data":"e886a59e35f43dc34aa7fa2211f0bf8bb4818372a50a502b72a2d7be0f68a1df"} Oct 30 20:50:59.078061 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:59.078014 4611 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-3d909d/premium-w6-3d909d-kserve-765648946f-2qvd2" podStartSLOduration=2.078001412 podStartE2EDuration="2.078001412s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:50:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:50:59.07530224 +0000 UTC m=+713.609328855" watchObservedRunningTime="2026-10-30 20:50:59.078001412 +0000 UTC m=+713.612028028" Oct 30 20:50:59.129995 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:59.129969 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-3d909d/premium-w6-3d909d-kserve-765648946f-2qvd2" Oct 30 20:50:59.397654 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:50:59.397621 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:51:00.169241 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:51:00.169210 4611 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-b8aee1/premium-w1-b8aee1-kserve-786d575599-htc8d" Oct 30 20:51:00.181633 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:51:00.181601 4611 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-b8aee1/premium-w1-b8aee1-kserve-786d575599-htc8d" Oct 30 20:51:00.190040 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:51:00.190008 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:51:00.801403 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:51:00.801369 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:51:01.081815 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:51:01.081741 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b8aee1/premium-w1-b8aee1-kserve-786d575599-htc8d" Oct 30 20:51:01.485008 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:51:01.484953 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:51:02.114802 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:51:02.114766 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:51:02.777698 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:51:02.777664 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:51:03.394399 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:51:03.394364 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:51:03.917007 ip-10-0-133-192 kubenswrapper[4611]: E1030 20:51:03.916974 4611 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hwzc6" containerName="extractor" Oct 30 20:51:04.092973 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:51:04.092938 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:51:04.372834 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:51:04.372799 4611 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-14cca2/premium-w5-14cca2-kserve-6878777857-wpjtj" Oct 30 20:51:04.386012 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:51:04.385980 4611 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-14cca2/premium-w5-14cca2-kserve-6878777857-wpjtj" Oct 30 20:51:04.823217 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:51:04.823135 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:51:05.120348 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:51:05.120318 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-14cca2/premium-w5-14cca2-kserve-6878777857-wpjtj" Oct 30 20:51:05.618723 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:51:05.618576 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:51:06.405109 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:51:06.405003 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:51:07.008353 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:51:07.008314 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:51:07.680818 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:51:07.680778 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:51:08.129076 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:51:08.129048 4611 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-3d909d/premium-w6-3d909d-kserve-765648946f-2qvd2" Oct 30 20:51:08.141252 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:51:08.141227 4611 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-3d909d/premium-w6-3d909d-kserve-765648946f-2qvd2" Oct 30 20:51:08.303187 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:51:08.303145 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:51:08.475966 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:51:08.475892 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:51:09.081614 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:51:09.081580 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:51:09.141211 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:51:09.141182 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-3d909d/premium-w6-3d909d-kserve-765648946f-2qvd2" Oct 30 20:51:09.693461 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:51:09.693424 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:51:10.390095 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:51:10.390065 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:51:11.089161 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:51:11.089130 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:51:11.677844 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:51:11.677644 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:51:12.292831 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:51:12.292678 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:51:12.985758 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:51:12.985724 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:51:13.589709 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:51:13.589669 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:51:14.185812 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:51:14.185778 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:51:14.803630 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:51:14.803590 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:51:15.960687 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:51:15.960644 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:51:16.786539 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:51:16.786404 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:51:17.770453 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:51:17.770419 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:51:18.589060 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:51:18.589025 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:51:19.261435 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:51:19.261382 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:51:19.929443 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:51:19.929338 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:51:20.601671 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:51:20.601639 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:51:21.208065 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:51:21.208029 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:51:21.824130 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:51:21.824091 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:51:22.555123 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:51:22.555084 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:51:23.136714 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:51:23.136679 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:51:23.717008 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:51:23.716973 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:51:24.392801 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:51:24.392769 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:51:25.330554 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:51:25.330492 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:51:26.086530 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:51:26.086473 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:51:26.823538 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:51:26.821715 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:51:27.629435 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:51:27.629400 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:51:28.310141 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:51:28.310092 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:51:29.119001 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:51:29.118964 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:51:30.154101 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:51:30.154064 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:51:31.131888 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:51:31.131849 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:51:31.911591 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:51:31.911559 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:51:32.064011 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:51:32.063948 4611 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-sxmcf"] Oct 30 20:51:32.069231 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:51:32.069213 4611 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-sxmcf" Oct 30 20:51:32.077758 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:51:32.077734 4611 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-cb60a2-openshift-default-dockercfg-zl72q\"" Oct 30 20:51:32.081364 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:51:32.081342 4611 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-sxmcf"] Oct 30 20:51:32.164941 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:51:32.164864 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cfbb7630-0e90-41df-9d4e-ade03ff1f522-istio-data\") pod \"e2e-worker-w3-cb60a2-openshift-default-57db98bd79-sxmcf\" (UID: \"cfbb7630-0e90-41df-9d4e-ade03ff1f522\") " pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-sxmcf" Oct 30 20:51:32.164941 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:51:32.164906 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfbb7630-0e90-41df-9d4e-ade03ff1f522-credential-socket\") pod \"e2e-worker-w3-cb60a2-openshift-default-57db98bd79-sxmcf\" (UID: \"cfbb7630-0e90-41df-9d4e-ade03ff1f522\") " pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-sxmcf" Oct 30 20:51:32.165122 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:51:32.164997 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cfbb7630-0e90-41df-9d4e-ade03ff1f522-istio-token\") pod \"e2e-worker-w3-cb60a2-openshift-default-57db98bd79-sxmcf\" (UID: \"cfbb7630-0e90-41df-9d4e-ade03ff1f522\") " pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-sxmcf" Oct 30 20:51:32.165122 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:51:32.165042 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfbb7630-0e90-41df-9d4e-ade03ff1f522-workload-socket\") pod \"e2e-worker-w3-cb60a2-openshift-default-57db98bd79-sxmcf\" (UID: \"cfbb7630-0e90-41df-9d4e-ade03ff1f522\") " pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-sxmcf" Oct 30 20:51:32.165122 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:51:32.165105 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gg4dc\" (UniqueName: \"kubernetes.io/projected/cfbb7630-0e90-41df-9d4e-ade03ff1f522-kube-api-access-gg4dc\") pod \"e2e-worker-w3-cb60a2-openshift-default-57db98bd79-sxmcf\" (UID: \"cfbb7630-0e90-41df-9d4e-ade03ff1f522\") " pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-sxmcf" Oct 30 20:51:32.165267 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:51:32.165142 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cfbb7630-0e90-41df-9d4e-ade03ff1f522-workload-certs\") pod \"e2e-worker-w3-cb60a2-openshift-default-57db98bd79-sxmcf\" (UID: \"cfbb7630-0e90-41df-9d4e-ade03ff1f522\") " pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-sxmcf" Oct 30 20:51:32.165267 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:51:32.165246 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cfbb7630-0e90-41df-9d4e-ade03ff1f522-istiod-ca-cert\") pod \"e2e-worker-w3-cb60a2-openshift-default-57db98bd79-sxmcf\" (UID: \"cfbb7630-0e90-41df-9d4e-ade03ff1f522\") " pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-sxmcf" Oct 30 20:51:32.165350 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:51:32.165287 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cfbb7630-0e90-41df-9d4e-ade03ff1f522-istio-envoy\") pod \"e2e-worker-w3-cb60a2-openshift-default-57db98bd79-sxmcf\" (UID: \"cfbb7630-0e90-41df-9d4e-ade03ff1f522\") " pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-sxmcf" Oct 30 20:51:32.165350 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:51:32.165328 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cfbb7630-0e90-41df-9d4e-ade03ff1f522-istio-podinfo\") pod \"e2e-worker-w3-cb60a2-openshift-default-57db98bd79-sxmcf\" (UID: \"cfbb7630-0e90-41df-9d4e-ade03ff1f522\") " pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-sxmcf" Oct 30 20:51:32.266405 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:51:32.266372 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cfbb7630-0e90-41df-9d4e-ade03ff1f522-istio-envoy\") pod \"e2e-worker-w3-cb60a2-openshift-default-57db98bd79-sxmcf\" (UID: \"cfbb7630-0e90-41df-9d4e-ade03ff1f522\") " pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-sxmcf" Oct 30 20:51:32.266556 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:51:32.266430 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cfbb7630-0e90-41df-9d4e-ade03ff1f522-istio-podinfo\") pod \"e2e-worker-w3-cb60a2-openshift-default-57db98bd79-sxmcf\" (UID: \"cfbb7630-0e90-41df-9d4e-ade03ff1f522\") " pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-sxmcf" Oct 30 20:51:32.266556 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:51:32.266492 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cfbb7630-0e90-41df-9d4e-ade03ff1f522-istio-data\") pod \"e2e-worker-w3-cb60a2-openshift-default-57db98bd79-sxmcf\" (UID: \"cfbb7630-0e90-41df-9d4e-ade03ff1f522\") " pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-sxmcf" Oct 30 20:51:32.266632 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:51:32.266557 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfbb7630-0e90-41df-9d4e-ade03ff1f522-credential-socket\") pod \"e2e-worker-w3-cb60a2-openshift-default-57db98bd79-sxmcf\" (UID: \"cfbb7630-0e90-41df-9d4e-ade03ff1f522\") " pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-sxmcf" Oct 30 20:51:32.266632 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:51:32.266615 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cfbb7630-0e90-41df-9d4e-ade03ff1f522-istio-token\") pod \"e2e-worker-w3-cb60a2-openshift-default-57db98bd79-sxmcf\" (UID: \"cfbb7630-0e90-41df-9d4e-ade03ff1f522\") " pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-sxmcf" Oct 30 20:51:32.266726 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:51:32.266659 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfbb7630-0e90-41df-9d4e-ade03ff1f522-workload-socket\") pod \"e2e-worker-w3-cb60a2-openshift-default-57db98bd79-sxmcf\" (UID: \"cfbb7630-0e90-41df-9d4e-ade03ff1f522\") " pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-sxmcf" Oct 30 20:51:32.266726 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:51:32.266702 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gg4dc\" (UniqueName: \"kubernetes.io/projected/cfbb7630-0e90-41df-9d4e-ade03ff1f522-kube-api-access-gg4dc\") pod \"e2e-worker-w3-cb60a2-openshift-default-57db98bd79-sxmcf\" (UID: \"cfbb7630-0e90-41df-9d4e-ade03ff1f522\") " pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-sxmcf" Oct 30 20:51:32.266821 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:51:32.266761 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cfbb7630-0e90-41df-9d4e-ade03ff1f522-workload-certs\") pod \"e2e-worker-w3-cb60a2-openshift-default-57db98bd79-sxmcf\" (UID: \"cfbb7630-0e90-41df-9d4e-ade03ff1f522\") " pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-sxmcf" Oct 30 20:51:32.266877 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:51:32.266831 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cfbb7630-0e90-41df-9d4e-ade03ff1f522-istiod-ca-cert\") pod \"e2e-worker-w3-cb60a2-openshift-default-57db98bd79-sxmcf\" (UID: \"cfbb7630-0e90-41df-9d4e-ade03ff1f522\") " pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-sxmcf" Oct 30 20:51:32.267071 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:51:32.266915 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cfbb7630-0e90-41df-9d4e-ade03ff1f522-istio-data\") pod \"e2e-worker-w3-cb60a2-openshift-default-57db98bd79-sxmcf\" (UID: \"cfbb7630-0e90-41df-9d4e-ade03ff1f522\") " pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-sxmcf" Oct 30 20:51:32.267071 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:51:32.266948 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfbb7630-0e90-41df-9d4e-ade03ff1f522-credential-socket\") pod \"e2e-worker-w3-cb60a2-openshift-default-57db98bd79-sxmcf\" (UID: \"cfbb7630-0e90-41df-9d4e-ade03ff1f522\") " pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-sxmcf" Oct 30 20:51:32.267249 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:51:32.267230 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cfbb7630-0e90-41df-9d4e-ade03ff1f522-workload-certs\") pod \"e2e-worker-w3-cb60a2-openshift-default-57db98bd79-sxmcf\" (UID: \"cfbb7630-0e90-41df-9d4e-ade03ff1f522\") " pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-sxmcf" Oct 30 20:51:32.267249 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:51:32.267238 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfbb7630-0e90-41df-9d4e-ade03ff1f522-workload-socket\") pod \"e2e-worker-w3-cb60a2-openshift-default-57db98bd79-sxmcf\" (UID: \"cfbb7630-0e90-41df-9d4e-ade03ff1f522\") " pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-sxmcf" Oct 30 20:51:32.267595 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:51:32.267571 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cfbb7630-0e90-41df-9d4e-ade03ff1f522-istiod-ca-cert\") pod \"e2e-worker-w3-cb60a2-openshift-default-57db98bd79-sxmcf\" (UID: \"cfbb7630-0e90-41df-9d4e-ade03ff1f522\") " pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-sxmcf" Oct 30 20:51:32.268760 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:51:32.268732 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cfbb7630-0e90-41df-9d4e-ade03ff1f522-istio-envoy\") pod \"e2e-worker-w3-cb60a2-openshift-default-57db98bd79-sxmcf\" (UID: \"cfbb7630-0e90-41df-9d4e-ade03ff1f522\") " pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-sxmcf" Oct 30 20:51:32.268868 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:51:32.268828 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cfbb7630-0e90-41df-9d4e-ade03ff1f522-istio-podinfo\") pod \"e2e-worker-w3-cb60a2-openshift-default-57db98bd79-sxmcf\" (UID: \"cfbb7630-0e90-41df-9d4e-ade03ff1f522\") " pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-sxmcf" Oct 30 20:51:32.280074 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:51:32.280050 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cfbb7630-0e90-41df-9d4e-ade03ff1f522-istio-token\") pod \"e2e-worker-w3-cb60a2-openshift-default-57db98bd79-sxmcf\" (UID: \"cfbb7630-0e90-41df-9d4e-ade03ff1f522\") " pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-sxmcf" Oct 30 20:51:32.280260 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:51:32.280240 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gg4dc\" (UniqueName: \"kubernetes.io/projected/cfbb7630-0e90-41df-9d4e-ade03ff1f522-kube-api-access-gg4dc\") pod \"e2e-worker-w3-cb60a2-openshift-default-57db98bd79-sxmcf\" (UID: \"cfbb7630-0e90-41df-9d4e-ade03ff1f522\") " pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-sxmcf" Oct 30 20:51:32.383665 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:51:32.383632 4611 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-sxmcf" Oct 30 20:51:32.667852 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:51:32.667832 4611 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-sxmcf"] Oct 30 20:51:32.669865 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:51:32.669836 4611 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcfbb7630_0e90_41df_9d4e_ade03ff1f522.slice/crio-e223e1927a99adac226fd76e9ba1b52086516f9760b25e1c9ee4698db6a76bd3 WatchSource:0}: Error finding container e223e1927a99adac226fd76e9ba1b52086516f9760b25e1c9ee4698db6a76bd3: Status 404 returned error can't find the container with id e223e1927a99adac226fd76e9ba1b52086516f9760b25e1c9ee4698db6a76bd3 Oct 30 20:51:32.672023 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:51:32.671990 4611 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:51:32.672104 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:51:32.672051 4611 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:51:32.672104 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:51:32.672080 4611 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:51:32.705598 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:51:32.705562 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:51:33.277694 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:51:33.277651 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-sxmcf" event={"ID":"cfbb7630-0e90-41df-9d4e-ade03ff1f522","Type":"ContainerStarted","Data":"27e240f53d2b9886252ca3e879894bca4d4306d4b9d49918f76959e495672fa2"} Oct 30 20:51:33.278089 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:51:33.277707 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-sxmcf" event={"ID":"cfbb7630-0e90-41df-9d4e-ade03ff1f522","Type":"ContainerStarted","Data":"e223e1927a99adac226fd76e9ba1b52086516f9760b25e1c9ee4698db6a76bd3"} Oct 30 20:51:33.300780 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:51:33.300714 4611 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-sxmcf" podStartSLOduration=1.300697348 podStartE2EDuration="1.300697348s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:51:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:51:33.300450002 +0000 UTC m=+747.834476620" watchObservedRunningTime="2026-10-30 20:51:33.300697348 +0000 UTC m=+747.834723965" Oct 30 20:51:33.383900 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:51:33.383870 4611 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-sxmcf" Oct 30 20:51:33.385140 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:51:33.385120 4611 patch_prober.go:29] interesting pod/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-sxmcf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body= Oct 30 20:51:33.385195 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:51:33.385163 4611 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-sxmcf" podUID="cfbb7630-0e90-41df-9d4e-ade03ff1f522" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" Oct 30 20:51:33.499734 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:51:33.499697 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:51:33.916949 ip-10-0-133-192 kubenswrapper[4611]: E1030 20:51:33.916902 4611 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hwzc6" containerName="extractor" Oct 30 20:51:34.295572 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:51:34.295473 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:51:34.384243 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:51:34.384206 4611 patch_prober.go:29] interesting pod/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-sxmcf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body= Oct 30 20:51:34.384397 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:51:34.384253 4611 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-sxmcf" podUID="cfbb7630-0e90-41df-9d4e-ade03ff1f522" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" Oct 30 20:51:34.983406 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:51:34.983371 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:51:35.384289 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:51:35.384246 4611 patch_prober.go:29] interesting pod/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-sxmcf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body= Oct 30 20:51:35.384782 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:51:35.384309 4611 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-sxmcf" podUID="cfbb7630-0e90-41df-9d4e-ade03ff1f522" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" Oct 30 20:51:35.643459 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:51:35.643362 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:51:36.323460 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:51:36.323426 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:51:36.384417 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:51:36.384385 4611 patch_prober.go:29] interesting pod/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-sxmcf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body= Oct 30 20:51:36.384805 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:51:36.384437 4611 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-sxmcf" podUID="cfbb7630-0e90-41df-9d4e-ade03ff1f522" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" Oct 30 20:51:37.028654 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:51:37.028619 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:51:37.384450 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:51:37.384412 4611 patch_prober.go:29] interesting pod/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-sxmcf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body= Oct 30 20:51:37.384874 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:51:37.384479 4611 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-sxmcf" podUID="cfbb7630-0e90-41df-9d4e-ade03ff1f522" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" Oct 30 20:51:37.953554 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:51:37.952966 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:51:38.384882 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:51:38.384847 4611 patch_prober.go:29] interesting pod/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-sxmcf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body= Oct 30 20:51:38.385468 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:51:38.385435 4611 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-sxmcf" podUID="cfbb7630-0e90-41df-9d4e-ade03ff1f522" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" Oct 30 20:51:38.695177 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:51:38.695086 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:51:39.388224 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:51:39.388196 4611 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-sxmcf" Oct 30 20:51:39.388743 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:51:39.388557 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-sxmcf" Oct 30 20:51:39.389254 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:51:39.389235 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-sxmcf" Oct 30 20:51:39.460846 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:51:39.460810 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:51:40.201441 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:51:40.201394 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:51:40.954053 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:51:40.952661 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:51:41.590220 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:51:41.590183 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:51:42.298379 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:51:42.298290 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:51:43.085783 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:51:43.085748 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:51:44.068035 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:51:44.068001 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:51:45.008118 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:51:45.008084 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:51:45.680133 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:51:45.680094 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:51:46.419646 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:51:46.419613 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:51:47.168680 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:51:47.168648 4611 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-de5833d3-v55zb"] Oct 30 20:51:47.172755 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:51:47.172737 4611 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de5833d3-v55zb" Oct 30 20:51:47.178235 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:51:47.178213 4611 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:51:47.178959 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:51:47.178939 4611 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7hf8r\"" Oct 30 20:51:47.183409 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:51:47.183384 4611 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-de5833d3-v55zb"] Oct 30 20:51:47.226337 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:51:47.226303 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:51:47.287173 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:51:47.287138 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-66j9t\" (UniqueName: \"kubernetes.io/projected/93074e7e-2063-4c32-9c4c-36137d25d437-kube-api-access-66j9t\") pod \"maas-api-revoke-sub-de5833d3-v55zb\" (UID: \"93074e7e-2063-4c32-9c4c-36137d25d437\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de5833d3-v55zb" Oct 30 20:51:47.287338 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:51:47.287254 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93074e7e-2063-4c32-9c4c-36137d25d437-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-de5833d3-v55zb\" (UID: \"93074e7e-2063-4c32-9c4c-36137d25d437\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de5833d3-v55zb" Oct 30 20:51:47.287405 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:51:47.287386 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93074e7e-2063-4c32-9c4c-36137d25d437-maas-api-xks-ca\") pod \"maas-api-revoke-sub-de5833d3-v55zb\" (UID: \"93074e7e-2063-4c32-9c4c-36137d25d437\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de5833d3-v55zb" Oct 30 20:51:47.388347 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:51:47.388313 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93074e7e-2063-4c32-9c4c-36137d25d437-maas-api-xks-ca\") pod \"maas-api-revoke-sub-de5833d3-v55zb\" (UID: \"93074e7e-2063-4c32-9c4c-36137d25d437\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de5833d3-v55zb" Oct 30 20:51:47.388548 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:51:47.388461 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-66j9t\" (UniqueName: \"kubernetes.io/projected/93074e7e-2063-4c32-9c4c-36137d25d437-kube-api-access-66j9t\") pod \"maas-api-revoke-sub-de5833d3-v55zb\" (UID: \"93074e7e-2063-4c32-9c4c-36137d25d437\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de5833d3-v55zb" Oct 30 20:51:47.388548 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:51:47.388524 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93074e7e-2063-4c32-9c4c-36137d25d437-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-de5833d3-v55zb\" (UID: \"93074e7e-2063-4c32-9c4c-36137d25d437\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de5833d3-v55zb" Oct 30 20:51:47.389140 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:51:47.389110 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93074e7e-2063-4c32-9c4c-36137d25d437-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-de5833d3-v55zb\" (UID: \"93074e7e-2063-4c32-9c4c-36137d25d437\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de5833d3-v55zb" Oct 30 20:51:47.391469 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:51:47.391440 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93074e7e-2063-4c32-9c4c-36137d25d437-maas-api-xks-ca\") pod \"maas-api-revoke-sub-de5833d3-v55zb\" (UID: \"93074e7e-2063-4c32-9c4c-36137d25d437\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de5833d3-v55zb" Oct 30 20:51:47.402384 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:51:47.402365 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-66j9t\" (UniqueName: \"kubernetes.io/projected/93074e7e-2063-4c32-9c4c-36137d25d437-kube-api-access-66j9t\") pod \"maas-api-revoke-sub-de5833d3-v55zb\" (UID: \"93074e7e-2063-4c32-9c4c-36137d25d437\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de5833d3-v55zb" Oct 30 20:51:47.489326 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:51:47.489257 4611 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de5833d3-v55zb" Oct 30 20:51:47.766911 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:51:47.766884 4611 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-de5833d3-v55zb"] Oct 30 20:51:47.768572 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:51:47.768549 4611 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93074e7e_2063_4c32_9c4c_36137d25d437.slice/crio-26839963e77c6d8dea365bc482b19eca818f391cd9be395d348a21483bdfda5d WatchSource:0}: Error finding container 26839963e77c6d8dea365bc482b19eca818f391cd9be395d348a21483bdfda5d: Status 404 returned error can't find the container with id 26839963e77c6d8dea365bc482b19eca818f391cd9be395d348a21483bdfda5d Oct 30 20:51:48.016239 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:51:48.016105 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:51:48.373653 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:51:48.373616 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:51:48.381055 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:51:48.381018 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de5833d3-v55zb" event={"ID":"93074e7e-2063-4c32-9c4c-36137d25d437","Type":"ContainerStarted","Data":"26839963e77c6d8dea365bc482b19eca818f391cd9be395d348a21483bdfda5d"} Oct 30 20:51:48.724570 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:51:48.724448 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:51:49.393929 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:51:49.393880 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:51:50.092535 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:51:50.092472 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:51:50.710166 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:51:50.708869 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:51:51.368021 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:51:51.367975 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:51:51.400911 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:51:51.400874 4611 generic.go:356] "Generic (PLEG): container finished" podID="93074e7e-2063-4c32-9c4c-36137d25d437" containerID="3ae27b1e50bb852c8549eef0ab5d09d2a49dd6f9440a828e529f497524e3ece1" exitCode=0 Oct 30 20:51:51.401096 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:51:51.400954 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de5833d3-v55zb" event={"ID":"93074e7e-2063-4c32-9c4c-36137d25d437","Type":"ContainerDied","Data":"3ae27b1e50bb852c8549eef0ab5d09d2a49dd6f9440a828e529f497524e3ece1"} Oct 30 20:51:52.037083 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:51:52.037045 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:51:52.601351 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:51:52.601326 4611 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de5833d3-v55zb" Oct 30 20:51:52.763992 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:51:52.763902 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/93074e7e-2063-4c32-9c4c-36137d25d437-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93074e7e-2063-4c32-9c4c-36137d25d437-maas-api-openshift-ca\") pod \"93074e7e-2063-4c32-9c4c-36137d25d437\" (UID: \"93074e7e-2063-4c32-9c4c-36137d25d437\") " Oct 30 20:51:52.763992 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:51:52.763965 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93074e7e-2063-4c32-9c4c-36137d25d437-kube-api-access-66j9t\" (UniqueName: \"kubernetes.io/projected/93074e7e-2063-4c32-9c4c-36137d25d437-kube-api-access-66j9t\") pod \"93074e7e-2063-4c32-9c4c-36137d25d437\" (UID: \"93074e7e-2063-4c32-9c4c-36137d25d437\") " Oct 30 20:51:52.764225 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:51:52.764015 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/93074e7e-2063-4c32-9c4c-36137d25d437-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93074e7e-2063-4c32-9c4c-36137d25d437-maas-api-xks-ca\") pod \"93074e7e-2063-4c32-9c4c-36137d25d437\" (UID: \"93074e7e-2063-4c32-9c4c-36137d25d437\") " Oct 30 20:51:52.764350 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:51:52.764321 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93074e7e-2063-4c32-9c4c-36137d25d437-maas-api-openshift-ca" pod "93074e7e-2063-4c32-9c4c-36137d25d437" (UID: "93074e7e-2063-4c32-9c4c-36137d25d437"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:51:52.764975 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:51:52.764495 4611 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93074e7e-2063-4c32-9c4c-36137d25d437-maas-api-openshift-ca\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:51:52.766438 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:51:52.766410 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/93074e7e-2063-4c32-9c4c-36137d25d437-maas-api-xks-ca" pod "93074e7e-2063-4c32-9c4c-36137d25d437" (UID: "93074e7e-2063-4c32-9c4c-36137d25d437"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:51:52.766773 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:51:52.766744 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93074e7e-2063-4c32-9c4c-36137d25d437-kube-api-access-66j9t" pod "93074e7e-2063-4c32-9c4c-36137d25d437" (UID: "93074e7e-2063-4c32-9c4c-36137d25d437"). InnerVolumeSpecName "kube-api-access-66j9t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:51:52.865316 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:51:52.865278 4611 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-66j9t\" (UniqueName: \"kubernetes.io/projected/93074e7e-2063-4c32-9c4c-36137d25d437-kube-api-access-66j9t\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:51:52.865316 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:51:52.865305 4611 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93074e7e-2063-4c32-9c4c-36137d25d437-maas-api-xks-ca\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:51:52.916927 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:51:52.916893 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:51:53.418148 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:51:53.418114 4611 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de5833d3-v55zb" Oct 30 20:51:53.418562 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:51:53.418148 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de5833d3-v55zb" event={"ID":"93074e7e-2063-4c32-9c4c-36137d25d437","Type":"ContainerDied","Data":"26839963e77c6d8dea365bc482b19eca818f391cd9be395d348a21483bdfda5d"} Oct 30 20:51:53.418562 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:51:53.418201 4611 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="26839963e77c6d8dea365bc482b19eca818f391cd9be395d348a21483bdfda5d" Oct 30 20:51:53.647949 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:51:53.647914 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:51:54.287533 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:51:54.287456 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:51:54.932143 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:51:54.932106 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:51:55.586549 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:51:55.586122 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:51:56.510011 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:51:56.509970 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:51:57.215672 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:51:57.215631 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:51:57.924181 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:51:57.924146 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:51:58.632472 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:51:58.632435 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:51:59.269671 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:51:59.269635 4611 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d728d037-45fkc"] Oct 30 20:51:59.270142 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:51:59.270106 4611 cpu_manager.go:414] "containerMap: removing container" podUID="93074e7e-2063-4c32-9c4c-36137d25d437" containerName="revoke-keys" Oct 30 20:51:59.270142 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:51:59.270116 4611 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93074e7e-2063-4c32-9c4c-36137d25d437" containerName="revoke-keys" Oct 30 20:51:59.270266 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:51:59.270213 4611 memory_manager.go:365] "RemoveStaleState removing state" podUID="93074e7e-2063-4c32-9c4c-36137d25d437" containerName="revoke-keys" Oct 30 20:51:59.273955 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:51:59.273934 4611 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d728d037-45fkc" Oct 30 20:51:59.284034 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:51:59.284006 4611 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7hf8r\"" Oct 30 20:51:59.284034 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:51:59.284017 4611 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:51:59.288594 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:51:59.288569 4611 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d728d037-45fkc"] Oct 30 20:51:59.307407 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:51:59.307378 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:51:59.442005 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:51:59.441966 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b15af64-44d3-4e30-8600-732e59af6877-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d728d037-45fkc\" (UID: \"1b15af64-44d3-4e30-8600-732e59af6877\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d728d037-45fkc" Oct 30 20:51:59.442199 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:51:59.442052 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bdhcq\" (UniqueName: \"kubernetes.io/projected/1b15af64-44d3-4e30-8600-732e59af6877-kube-api-access-bdhcq\") pod \"maas-api-revoke-sub-d728d037-45fkc\" (UID: \"1b15af64-44d3-4e30-8600-732e59af6877\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d728d037-45fkc" Oct 30 20:51:59.442199 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:51:59.442093 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b15af64-44d3-4e30-8600-732e59af6877-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d728d037-45fkc\" (UID: \"1b15af64-44d3-4e30-8600-732e59af6877\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d728d037-45fkc" Oct 30 20:51:59.543449 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:51:59.543377 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b15af64-44d3-4e30-8600-732e59af6877-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d728d037-45fkc\" (UID: \"1b15af64-44d3-4e30-8600-732e59af6877\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d728d037-45fkc" Oct 30 20:51:59.543623 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:51:59.543483 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b15af64-44d3-4e30-8600-732e59af6877-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d728d037-45fkc\" (UID: \"1b15af64-44d3-4e30-8600-732e59af6877\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d728d037-45fkc" Oct 30 20:51:59.543623 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:51:59.543587 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bdhcq\" (UniqueName: \"kubernetes.io/projected/1b15af64-44d3-4e30-8600-732e59af6877-kube-api-access-bdhcq\") pod \"maas-api-revoke-sub-d728d037-45fkc\" (UID: \"1b15af64-44d3-4e30-8600-732e59af6877\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d728d037-45fkc" Oct 30 20:51:59.544060 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:51:59.544030 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b15af64-44d3-4e30-8600-732e59af6877-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d728d037-45fkc\" (UID: \"1b15af64-44d3-4e30-8600-732e59af6877\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d728d037-45fkc" Oct 30 20:51:59.545829 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:51:59.545808 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b15af64-44d3-4e30-8600-732e59af6877-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d728d037-45fkc\" (UID: \"1b15af64-44d3-4e30-8600-732e59af6877\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d728d037-45fkc" Oct 30 20:51:59.556885 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:51:59.556862 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bdhcq\" (UniqueName: \"kubernetes.io/projected/1b15af64-44d3-4e30-8600-732e59af6877-kube-api-access-bdhcq\") pod \"maas-api-revoke-sub-d728d037-45fkc\" (UID: \"1b15af64-44d3-4e30-8600-732e59af6877\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d728d037-45fkc" Oct 30 20:51:59.598634 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:51:59.598601 4611 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d728d037-45fkc" Oct 30 20:51:59.883494 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:51:59.883467 4611 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d728d037-45fkc"] Oct 30 20:51:59.885586 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:51:59.885559 4611 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b15af64_44d3_4e30_8600_732e59af6877.slice/crio-3f6eba92c5e7d5f266a54aed9417342bbc5d1a21125cae907f38a2fa15311e6b WatchSource:0}: Error finding container 3f6eba92c5e7d5f266a54aed9417342bbc5d1a21125cae907f38a2fa15311e6b: Status 404 returned error can't find the container with id 3f6eba92c5e7d5f266a54aed9417342bbc5d1a21125cae907f38a2fa15311e6b Oct 30 20:52:00.042804 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:00.041353 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:52:00.460947 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:00.460871 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d728d037-45fkc" event={"ID":"1b15af64-44d3-4e30-8600-732e59af6877","Type":"ContainerStarted","Data":"a9c16efe2304ca324ef84b191e5dbeaa4bf4950620f7877cae1f835e3e4c29df"} Oct 30 20:52:00.460947 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:00.460921 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d728d037-45fkc" event={"ID":"1b15af64-44d3-4e30-8600-732e59af6877","Type":"ContainerStarted","Data":"3f6eba92c5e7d5f266a54aed9417342bbc5d1a21125cae907f38a2fa15311e6b"} Oct 30 20:52:00.479504 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:00.479456 4611 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d728d037-45fkc" podStartSLOduration=1.479441662 podStartE2EDuration="1.479441662s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:51:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:52:00.477951135 +0000 UTC m=+775.011977750" watchObservedRunningTime="2026-10-30 20:52:00.479441662 +0000 UTC m=+775.013468278" Oct 30 20:52:00.707595 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:00.707547 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:52:01.309666 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:01.309630 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:52:01.468171 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:01.468134 4611 generic.go:356] "Generic (PLEG): container finished" podID="1b15af64-44d3-4e30-8600-732e59af6877" containerID="a9c16efe2304ca324ef84b191e5dbeaa4bf4950620f7877cae1f835e3e4c29df" exitCode=0 Oct 30 20:52:01.468552 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:01.468217 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d728d037-45fkc" event={"ID":"1b15af64-44d3-4e30-8600-732e59af6877","Type":"ContainerDied","Data":"a9c16efe2304ca324ef84b191e5dbeaa4bf4950620f7877cae1f835e3e4c29df"} Oct 30 20:52:01.578404 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:01.578328 4611 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-kpn5r"] Oct 30 20:52:01.583244 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:01.583224 4611 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-kpn5r" Oct 30 20:52:01.596427 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:01.596401 4611 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-kpn5r"] Oct 30 20:52:01.670165 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:01.670116 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25d72035-53b1-4de2-b594-34c64b9b1560-workload-socket\") pod \"e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-kpn5r\" (UID: \"25d72035-53b1-4de2-b594-34c64b9b1560\") " pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-kpn5r" Oct 30 20:52:01.670351 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:01.670251 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/25d72035-53b1-4de2-b594-34c64b9b1560-istio-token\") pod \"e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-kpn5r\" (UID: \"25d72035-53b1-4de2-b594-34c64b9b1560\") " pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-kpn5r" Oct 30 20:52:01.670351 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:01.670291 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25d72035-53b1-4de2-b594-34c64b9b1560-credential-socket\") pod \"e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-kpn5r\" (UID: \"25d72035-53b1-4de2-b594-34c64b9b1560\") " pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-kpn5r" Oct 30 20:52:01.670351 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:01.670320 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25d72035-53b1-4de2-b594-34c64b9b1560-istio-podinfo\") pod \"e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-kpn5r\" (UID: \"25d72035-53b1-4de2-b594-34c64b9b1560\") " pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-kpn5r" Oct 30 20:52:01.670552 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:01.670457 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25d72035-53b1-4de2-b594-34c64b9b1560-istiod-ca-cert\") pod \"e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-kpn5r\" (UID: \"25d72035-53b1-4de2-b594-34c64b9b1560\") " pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-kpn5r" Oct 30 20:52:01.670552 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:01.670491 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v9222\" (UniqueName: \"kubernetes.io/projected/25d72035-53b1-4de2-b594-34c64b9b1560-kube-api-access-v9222\") pod \"e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-kpn5r\" (UID: \"25d72035-53b1-4de2-b594-34c64b9b1560\") " pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-kpn5r" Oct 30 20:52:01.670675 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:01.670582 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25d72035-53b1-4de2-b594-34c64b9b1560-istio-envoy\") pod \"e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-kpn5r\" (UID: \"25d72035-53b1-4de2-b594-34c64b9b1560\") " pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-kpn5r" Oct 30 20:52:01.670675 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:01.670612 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25d72035-53b1-4de2-b594-34c64b9b1560-istio-data\") pod \"e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-kpn5r\" (UID: \"25d72035-53b1-4de2-b594-34c64b9b1560\") " pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-kpn5r" Oct 30 20:52:01.670784 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:01.670759 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25d72035-53b1-4de2-b594-34c64b9b1560-workload-certs\") pod \"e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-kpn5r\" (UID: \"25d72035-53b1-4de2-b594-34c64b9b1560\") " pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-kpn5r" Oct 30 20:52:01.771953 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:01.771914 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25d72035-53b1-4de2-b594-34c64b9b1560-workload-socket\") pod \"e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-kpn5r\" (UID: \"25d72035-53b1-4de2-b594-34c64b9b1560\") " pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-kpn5r" Oct 30 20:52:01.772130 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:01.771986 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/25d72035-53b1-4de2-b594-34c64b9b1560-istio-token\") pod \"e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-kpn5r\" (UID: \"25d72035-53b1-4de2-b594-34c64b9b1560\") " pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-kpn5r" Oct 30 20:52:01.772130 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:01.772012 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25d72035-53b1-4de2-b594-34c64b9b1560-credential-socket\") pod \"e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-kpn5r\" (UID: \"25d72035-53b1-4de2-b594-34c64b9b1560\") " pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-kpn5r" Oct 30 20:52:01.772130 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:01.772032 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25d72035-53b1-4de2-b594-34c64b9b1560-istio-podinfo\") pod \"e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-kpn5r\" (UID: \"25d72035-53b1-4de2-b594-34c64b9b1560\") " pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-kpn5r" Oct 30 20:52:01.772130 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:01.772057 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25d72035-53b1-4de2-b594-34c64b9b1560-istiod-ca-cert\") pod \"e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-kpn5r\" (UID: \"25d72035-53b1-4de2-b594-34c64b9b1560\") " pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-kpn5r" Oct 30 20:52:01.772130 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:01.772081 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9222\" (UniqueName: \"kubernetes.io/projected/25d72035-53b1-4de2-b594-34c64b9b1560-kube-api-access-v9222\") pod \"e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-kpn5r\" (UID: \"25d72035-53b1-4de2-b594-34c64b9b1560\") " pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-kpn5r" Oct 30 20:52:01.772130 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:01.772126 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25d72035-53b1-4de2-b594-34c64b9b1560-istio-envoy\") pod \"e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-kpn5r\" (UID: \"25d72035-53b1-4de2-b594-34c64b9b1560\") " pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-kpn5r" Oct 30 20:52:01.772466 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:01.772156 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25d72035-53b1-4de2-b594-34c64b9b1560-istio-data\") pod \"e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-kpn5r\" (UID: \"25d72035-53b1-4de2-b594-34c64b9b1560\") " pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-kpn5r" Oct 30 20:52:01.772466 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:01.772250 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25d72035-53b1-4de2-b594-34c64b9b1560-workload-certs\") pod \"e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-kpn5r\" (UID: \"25d72035-53b1-4de2-b594-34c64b9b1560\") " pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-kpn5r" Oct 30 20:52:01.772466 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:01.772374 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25d72035-53b1-4de2-b594-34c64b9b1560-workload-socket\") pod \"e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-kpn5r\" (UID: \"25d72035-53b1-4de2-b594-34c64b9b1560\") " pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-kpn5r" Oct 30 20:52:01.772466 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:01.772411 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25d72035-53b1-4de2-b594-34c64b9b1560-credential-socket\") pod \"e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-kpn5r\" (UID: \"25d72035-53b1-4de2-b594-34c64b9b1560\") " pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-kpn5r" Oct 30 20:52:01.772787 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:01.772768 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25d72035-53b1-4de2-b594-34c64b9b1560-workload-certs\") pod \"e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-kpn5r\" (UID: \"25d72035-53b1-4de2-b594-34c64b9b1560\") " pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-kpn5r" Oct 30 20:52:01.772851 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:01.772804 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25d72035-53b1-4de2-b594-34c64b9b1560-istio-data\") pod \"e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-kpn5r\" (UID: \"25d72035-53b1-4de2-b594-34c64b9b1560\") " pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-kpn5r" Oct 30 20:52:01.772944 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:01.772922 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25d72035-53b1-4de2-b594-34c64b9b1560-istiod-ca-cert\") pod \"e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-kpn5r\" (UID: \"25d72035-53b1-4de2-b594-34c64b9b1560\") " pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-kpn5r" Oct 30 20:52:01.774489 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:01.774468 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25d72035-53b1-4de2-b594-34c64b9b1560-istio-podinfo\") pod \"e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-kpn5r\" (UID: \"25d72035-53b1-4de2-b594-34c64b9b1560\") " pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-kpn5r" Oct 30 20:52:01.774591 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:01.774492 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25d72035-53b1-4de2-b594-34c64b9b1560-istio-envoy\") pod \"e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-kpn5r\" (UID: \"25d72035-53b1-4de2-b594-34c64b9b1560\") " pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-kpn5r" Oct 30 20:52:01.783441 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:01.783416 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/25d72035-53b1-4de2-b594-34c64b9b1560-istio-token\") pod \"e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-kpn5r\" (UID: \"25d72035-53b1-4de2-b594-34c64b9b1560\") " pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-kpn5r" Oct 30 20:52:01.783598 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:01.783468 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v9222\" (UniqueName: \"kubernetes.io/projected/25d72035-53b1-4de2-b594-34c64b9b1560-kube-api-access-v9222\") pod \"e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-kpn5r\" (UID: \"25d72035-53b1-4de2-b594-34c64b9b1560\") " pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-kpn5r" Oct 30 20:52:01.899261 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:01.899225 4611 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-kpn5r" Oct 30 20:52:01.992033 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:01.991992 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:52:02.189173 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:02.189136 4611 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-kpn5r"] Oct 30 20:52:02.191071 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:52:02.191043 4611 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25d72035_53b1_4de2_b594_34c64b9b1560.slice/crio-9e26e9839e37016426ef8bb1778a5245222a9ea492ad2b23182087f6a869867b WatchSource:0}: Error finding container 9e26e9839e37016426ef8bb1778a5245222a9ea492ad2b23182087f6a869867b: Status 404 returned error can't find the container with id 9e26e9839e37016426ef8bb1778a5245222a9ea492ad2b23182087f6a869867b Oct 30 20:52:02.199130 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:02.199095 4611 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:52:02.199234 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:02.199171 4611 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:52:02.199234 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:02.199214 4611 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:52:02.475629 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:02.475545 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-kpn5r" event={"ID":"25d72035-53b1-4de2-b594-34c64b9b1560","Type":"ContainerStarted","Data":"702ebef0ccce8010474bb2a51a9b783cde0b5a29f31f3643c02904f15a075355"} Oct 30 20:52:02.475629 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:02.475610 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-kpn5r" event={"ID":"25d72035-53b1-4de2-b594-34c64b9b1560","Type":"ContainerStarted","Data":"9e26e9839e37016426ef8bb1778a5245222a9ea492ad2b23182087f6a869867b"} Oct 30 20:52:02.500921 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:02.500862 4611 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-kpn5r" podStartSLOduration=1.500843642 podStartE2EDuration="1.500843642s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:52:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:52:02.497017662 +0000 UTC m=+777.031044277" watchObservedRunningTime="2026-10-30 20:52:02.500843642 +0000 UTC m=+777.034870260" Oct 30 20:52:02.608503 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:02.608471 4611 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d728d037-45fkc" Oct 30 20:52:02.685661 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:02.685627 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1b15af64-44d3-4e30-8600-732e59af6877-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b15af64-44d3-4e30-8600-732e59af6877-maas-api-xks-ca\") pod \"1b15af64-44d3-4e30-8600-732e59af6877\" (UID: \"1b15af64-44d3-4e30-8600-732e59af6877\") " Oct 30 20:52:02.685824 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:02.685768 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b15af64-44d3-4e30-8600-732e59af6877-kube-api-access-bdhcq\" (UniqueName: \"kubernetes.io/projected/1b15af64-44d3-4e30-8600-732e59af6877-kube-api-access-bdhcq\") pod \"1b15af64-44d3-4e30-8600-732e59af6877\" (UID: \"1b15af64-44d3-4e30-8600-732e59af6877\") " Oct 30 20:52:02.685824 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:02.685819 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1b15af64-44d3-4e30-8600-732e59af6877-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b15af64-44d3-4e30-8600-732e59af6877-maas-api-openshift-ca\") pod \"1b15af64-44d3-4e30-8600-732e59af6877\" (UID: \"1b15af64-44d3-4e30-8600-732e59af6877\") " Oct 30 20:52:02.686194 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:02.686165 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b15af64-44d3-4e30-8600-732e59af6877-maas-api-openshift-ca" pod "1b15af64-44d3-4e30-8600-732e59af6877" (UID: "1b15af64-44d3-4e30-8600-732e59af6877"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:52:02.687728 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:02.687703 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b15af64-44d3-4e30-8600-732e59af6877-maas-api-xks-ca" pod "1b15af64-44d3-4e30-8600-732e59af6877" (UID: "1b15af64-44d3-4e30-8600-732e59af6877"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:52:02.687909 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:02.687884 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b15af64-44d3-4e30-8600-732e59af6877-kube-api-access-bdhcq" pod "1b15af64-44d3-4e30-8600-732e59af6877" (UID: "1b15af64-44d3-4e30-8600-732e59af6877"). InnerVolumeSpecName "kube-api-access-bdhcq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:52:02.787400 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:02.787375 4611 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bdhcq\" (UniqueName: \"kubernetes.io/projected/1b15af64-44d3-4e30-8600-732e59af6877-kube-api-access-bdhcq\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:52:02.787532 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:02.787400 4611 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b15af64-44d3-4e30-8600-732e59af6877-maas-api-openshift-ca\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:52:02.787532 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:02.787411 4611 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b15af64-44d3-4e30-8600-732e59af6877-maas-api-xks-ca\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:52:02.900128 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:02.900094 4611 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-kpn5r" Oct 30 20:52:03.011193 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:03.011157 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:52:03.482777 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:03.482736 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d728d037-45fkc" event={"ID":"1b15af64-44d3-4e30-8600-732e59af6877","Type":"ContainerDied","Data":"3f6eba92c5e7d5f266a54aed9417342bbc5d1a21125cae907f38a2fa15311e6b"} Oct 30 20:52:03.483170 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:03.482797 4611 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3f6eba92c5e7d5f266a54aed9417342bbc5d1a21125cae907f38a2fa15311e6b" Oct 30 20:52:03.483170 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:03.482756 4611 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d728d037-45fkc" Oct 30 20:52:03.901223 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:03.901188 4611 patch_prober.go:29] interesting pod/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-kpn5r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body= Oct 30 20:52:03.901391 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:03.901257 4611 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-kpn5r" podUID="25d72035-53b1-4de2-b594-34c64b9b1560" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" Oct 30 20:52:03.908408 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:03.908374 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:52:03.921363 ip-10-0-133-192 kubenswrapper[4611]: E1030 20:52:03.921326 4611 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hwzc6" containerName="extractor" Oct 30 20:52:04.709281 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:04.709238 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:52:04.900453 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:04.900418 4611 patch_prober.go:29] interesting pod/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-kpn5r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body= Oct 30 20:52:04.900646 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:04.900486 4611 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-kpn5r" podUID="25d72035-53b1-4de2-b594-34c64b9b1560" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" Oct 30 20:52:05.489344 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:05.489309 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:52:05.899957 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:05.899931 4611 patch_prober.go:29] interesting pod/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-kpn5r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body= Oct 30 20:52:05.900319 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:05.899976 4611 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-kpn5r" podUID="25d72035-53b1-4de2-b594-34c64b9b1560" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" Oct 30 20:52:06.538147 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:06.538110 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:52:06.911532 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:06.905897 4611 patch_prober.go:29] interesting pod/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-kpn5r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body= Oct 30 20:52:06.911532 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:06.905960 4611 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-kpn5r" podUID="25d72035-53b1-4de2-b594-34c64b9b1560" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" Oct 30 20:52:07.906533 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:07.905879 4611 patch_prober.go:29] interesting pod/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-kpn5r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body= Oct 30 20:52:07.906533 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:07.905946 4611 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-kpn5r" podUID="25d72035-53b1-4de2-b594-34c64b9b1560" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" Oct 30 20:52:07.928745 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:07.928432 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:52:08.076532 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:08.075619 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b8aee1/premium-w1-b8aee1-kserve-786d575599-htc8d" podUID="77368afe-edc8-4fa5-8010-d321965fb38e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:52:08.109673 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:08.109625 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-14cca2/premium-w5-14cca2-kserve-6878777857-wpjtj" podUID="b99eb76b-d1c6-49e9-9809-c351c0f5a4de" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:52:08.141638 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:08.141593 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-3d909d/premium-w6-3d909d-kserve-765648946f-2qvd2" podUID="fca5d697-109d-4e90-ab41-c40ad31af874" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded" Oct 30 20:52:08.904369 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:08.904331 4611 patch_prober.go:29] interesting pod/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-kpn5r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body= Oct 30 20:52:08.904648 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:08.904617 4611 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-kpn5r" podUID="25d72035-53b1-4de2-b594-34c64b9b1560" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" Oct 30 20:52:08.908994 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:08.908967 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:52:09.881532 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:09.881323 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:52:10.786534 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:10.782395 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:52:10.941649 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:10.941606 4611 patch_prober.go:29] interesting pod/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-kpn5r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:52:10.942103 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:10.941656 4611 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-kpn5r" podUID="25d72035-53b1-4de2-b594-34c64b9b1560" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:52:11.009313 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:11.009278 4611 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-87079fdf-mf2f5"] Oct 30 20:52:11.009906 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:11.009884 4611 cpu_manager.go:414] "containerMap: removing container" podUID="1b15af64-44d3-4e30-8600-732e59af6877" containerName="revoke-keys" Oct 30 20:52:11.009906 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:11.009905 4611 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b15af64-44d3-4e30-8600-732e59af6877" containerName="revoke-keys" Oct 30 20:52:11.010076 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:11.010042 4611 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b15af64-44d3-4e30-8600-732e59af6877" containerName="revoke-keys" Oct 30 20:52:11.015359 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:11.015339 4611 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-87079fdf-mf2f5" Oct 30 20:52:11.030364 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:11.030336 4611 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:52:11.031233 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:11.031211 4611 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7hf8r\"" Oct 30 20:52:11.052981 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:11.052896 4611 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-87079fdf-mf2f5"] Oct 30 20:52:11.055260 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:11.054215 4611 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-kpn5r" Oct 30 20:52:11.056083 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:11.055578 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-kpn5r" Oct 30 20:52:11.056603 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:11.056556 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-kpn5r" Oct 30 20:52:11.123888 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:11.123167 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58075892-0276-4403-aa80-69c374f8283d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-87079fdf-mf2f5\" (UID: \"58075892-0276-4403-aa80-69c374f8283d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87079fdf-mf2f5" Oct 30 20:52:11.123888 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:11.123363 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58075892-0276-4403-aa80-69c374f8283d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-87079fdf-mf2f5\" (UID: \"58075892-0276-4403-aa80-69c374f8283d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87079fdf-mf2f5" Oct 30 20:52:11.123888 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:11.123382 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w9rcs\" (UniqueName: \"kubernetes.io/projected/58075892-0276-4403-aa80-69c374f8283d-kube-api-access-w9rcs\") pod \"maas-api-revoke-sub-87079fdf-mf2f5\" (UID: \"58075892-0276-4403-aa80-69c374f8283d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87079fdf-mf2f5" Oct 30 20:52:11.225268 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:11.225189 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58075892-0276-4403-aa80-69c374f8283d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-87079fdf-mf2f5\" (UID: \"58075892-0276-4403-aa80-69c374f8283d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87079fdf-mf2f5" Oct 30 20:52:11.225459 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:11.225286 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w9rcs\" (UniqueName: \"kubernetes.io/projected/58075892-0276-4403-aa80-69c374f8283d-kube-api-access-w9rcs\") pod \"maas-api-revoke-sub-87079fdf-mf2f5\" (UID: \"58075892-0276-4403-aa80-69c374f8283d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87079fdf-mf2f5" Oct 30 20:52:11.225459 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:11.225433 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58075892-0276-4403-aa80-69c374f8283d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-87079fdf-mf2f5\" (UID: \"58075892-0276-4403-aa80-69c374f8283d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87079fdf-mf2f5" Oct 30 20:52:11.226348 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:11.226321 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58075892-0276-4403-aa80-69c374f8283d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-87079fdf-mf2f5\" (UID: \"58075892-0276-4403-aa80-69c374f8283d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87079fdf-mf2f5" Oct 30 20:52:11.228145 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:11.228122 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58075892-0276-4403-aa80-69c374f8283d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-87079fdf-mf2f5\" (UID: \"58075892-0276-4403-aa80-69c374f8283d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87079fdf-mf2f5" Oct 30 20:52:11.238272 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:11.238251 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w9rcs\" (UniqueName: \"kubernetes.io/projected/58075892-0276-4403-aa80-69c374f8283d-kube-api-access-w9rcs\") pod \"maas-api-revoke-sub-87079fdf-mf2f5\" (UID: \"58075892-0276-4403-aa80-69c374f8283d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87079fdf-mf2f5" Oct 30 20:52:11.351126 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:11.351096 4611 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-87079fdf-mf2f5" Oct 30 20:52:11.514282 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:11.514251 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:52:11.633910 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:11.633887 4611 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-87079fdf-mf2f5"] Oct 30 20:52:11.635958 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:52:11.635935 4611 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58075892_0276_4403_aa80_69c374f8283d.slice/crio-048de947ace68583e162d105c713f0bc3043c55abbf84cd1d32eac6c8fbd8419 WatchSource:0}: Error finding container 048de947ace68583e162d105c713f0bc3043c55abbf84cd1d32eac6c8fbd8419: Status 404 returned error can't find the container with id 048de947ace68583e162d105c713f0bc3043c55abbf84cd1d32eac6c8fbd8419 Oct 30 20:52:12.334955 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:12.334708 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:52:12.610807 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:12.610778 4611 generic.go:356] "Generic (PLEG): container finished" podID="58075892-0276-4403-aa80-69c374f8283d" containerID="1c479d7a9e1eac8587c32d7ceca486804b3fd524eece15219dd5eb4cfed6c0ab" exitCode=0 Oct 30 20:52:12.610958 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:12.610857 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-87079fdf-mf2f5" event={"ID":"58075892-0276-4403-aa80-69c374f8283d","Type":"ContainerDied","Data":"1c479d7a9e1eac8587c32d7ceca486804b3fd524eece15219dd5eb4cfed6c0ab"} Oct 30 20:52:12.610958 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:12.610906 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-87079fdf-mf2f5" event={"ID":"58075892-0276-4403-aa80-69c374f8283d","Type":"ContainerStarted","Data":"048de947ace68583e162d105c713f0bc3043c55abbf84cd1d32eac6c8fbd8419"} Oct 30 20:52:12.747267 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:12.747233 4611 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-14cca2/distinct-w5-14cca2-kserve-587f6996b8-sgf6h"] Oct 30 20:52:12.751876 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:12.751857 4611 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-14cca2/distinct-w5-14cca2-kserve-587f6996b8-sgf6h" Oct 30 20:52:12.759830 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:12.759802 4611 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-14cca2\"/\"distinct-w5-14cca2-kserve-self-signed-certs\"" Oct 30 20:52:12.761703 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:12.761675 4611 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-14cca2/distinct-w5-14cca2-kserve-587f6996b8-sgf6h"] Oct 30 20:52:12.840939 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:12.840906 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:52:12.844188 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:12.844164 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/02b652c3-0724-4a50-9e95-2a9890cc97c4-model-cache\") pod \"distinct-w5-14cca2-kserve-587f6996b8-sgf6h\" (UID: \"02b652c3-0724-4a50-9e95-2a9890cc97c4\") " pod="e2e-models-e2e-worker-w5-14cca2/distinct-w5-14cca2-kserve-587f6996b8-sgf6h" Oct 30 20:52:12.844271 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:12.844199 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/02b652c3-0724-4a50-9e95-2a9890cc97c4-tls-certs\") pod \"distinct-w5-14cca2-kserve-587f6996b8-sgf6h\" (UID: \"02b652c3-0724-4a50-9e95-2a9890cc97c4\") " pod="e2e-models-e2e-worker-w5-14cca2/distinct-w5-14cca2-kserve-587f6996b8-sgf6h" Oct 30 20:52:12.844370 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:12.844349 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/02b652c3-0724-4a50-9e95-2a9890cc97c4-dshm\") pod \"distinct-w5-14cca2-kserve-587f6996b8-sgf6h\" (UID: \"02b652c3-0724-4a50-9e95-2a9890cc97c4\") " pod="e2e-models-e2e-worker-w5-14cca2/distinct-w5-14cca2-kserve-587f6996b8-sgf6h" Oct 30 20:52:12.844430 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:12.844419 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5fx85\" (UniqueName: \"kubernetes.io/projected/02b652c3-0724-4a50-9e95-2a9890cc97c4-kube-api-access-5fx85\") pod \"distinct-w5-14cca2-kserve-587f6996b8-sgf6h\" (UID: \"02b652c3-0724-4a50-9e95-2a9890cc97c4\") " pod="e2e-models-e2e-worker-w5-14cca2/distinct-w5-14cca2-kserve-587f6996b8-sgf6h" Oct 30 20:52:12.844479 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:12.844468 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/02b652c3-0724-4a50-9e95-2a9890cc97c4-home\") pod \"distinct-w5-14cca2-kserve-587f6996b8-sgf6h\" (UID: \"02b652c3-0724-4a50-9e95-2a9890cc97c4\") " pod="e2e-models-e2e-worker-w5-14cca2/distinct-w5-14cca2-kserve-587f6996b8-sgf6h" Oct 30 20:52:12.844614 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:12.844591 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/02b652c3-0724-4a50-9e95-2a9890cc97c4-tmp-dir\") pod \"distinct-w5-14cca2-kserve-587f6996b8-sgf6h\" (UID: \"02b652c3-0724-4a50-9e95-2a9890cc97c4\") " pod="e2e-models-e2e-worker-w5-14cca2/distinct-w5-14cca2-kserve-587f6996b8-sgf6h" Oct 30 20:52:12.951091 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:12.950998 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/02b652c3-0724-4a50-9e95-2a9890cc97c4-home\") pod \"distinct-w5-14cca2-kserve-587f6996b8-sgf6h\" (UID: \"02b652c3-0724-4a50-9e95-2a9890cc97c4\") " pod="e2e-models-e2e-worker-w5-14cca2/distinct-w5-14cca2-kserve-587f6996b8-sgf6h" Oct 30 20:52:12.951091 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:12.951083 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/02b652c3-0724-4a50-9e95-2a9890cc97c4-tmp-dir\") pod \"distinct-w5-14cca2-kserve-587f6996b8-sgf6h\" (UID: \"02b652c3-0724-4a50-9e95-2a9890cc97c4\") " pod="e2e-models-e2e-worker-w5-14cca2/distinct-w5-14cca2-kserve-587f6996b8-sgf6h" Oct 30 20:52:12.951333 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:12.951185 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/02b652c3-0724-4a50-9e95-2a9890cc97c4-model-cache\") pod \"distinct-w5-14cca2-kserve-587f6996b8-sgf6h\" (UID: \"02b652c3-0724-4a50-9e95-2a9890cc97c4\") " pod="e2e-models-e2e-worker-w5-14cca2/distinct-w5-14cca2-kserve-587f6996b8-sgf6h" Oct 30 20:52:12.951333 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:12.951224 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/02b652c3-0724-4a50-9e95-2a9890cc97c4-tls-certs\") pod \"distinct-w5-14cca2-kserve-587f6996b8-sgf6h\" (UID: \"02b652c3-0724-4a50-9e95-2a9890cc97c4\") " pod="e2e-models-e2e-worker-w5-14cca2/distinct-w5-14cca2-kserve-587f6996b8-sgf6h" Oct 30 20:52:12.951333 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:12.951274 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/02b652c3-0724-4a50-9e95-2a9890cc97c4-dshm\") pod \"distinct-w5-14cca2-kserve-587f6996b8-sgf6h\" (UID: \"02b652c3-0724-4a50-9e95-2a9890cc97c4\") " pod="e2e-models-e2e-worker-w5-14cca2/distinct-w5-14cca2-kserve-587f6996b8-sgf6h" Oct 30 20:52:12.951523 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:12.951330 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5fx85\" (UniqueName: \"kubernetes.io/projected/02b652c3-0724-4a50-9e95-2a9890cc97c4-kube-api-access-5fx85\") pod \"distinct-w5-14cca2-kserve-587f6996b8-sgf6h\" (UID: \"02b652c3-0724-4a50-9e95-2a9890cc97c4\") " pod="e2e-models-e2e-worker-w5-14cca2/distinct-w5-14cca2-kserve-587f6996b8-sgf6h" Oct 30 20:52:12.952074 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:12.952042 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/02b652c3-0724-4a50-9e95-2a9890cc97c4-home\") pod \"distinct-w5-14cca2-kserve-587f6996b8-sgf6h\" (UID: \"02b652c3-0724-4a50-9e95-2a9890cc97c4\") " pod="e2e-models-e2e-worker-w5-14cca2/distinct-w5-14cca2-kserve-587f6996b8-sgf6h" Oct 30 20:52:12.952358 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:12.952325 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/02b652c3-0724-4a50-9e95-2a9890cc97c4-model-cache\") pod \"distinct-w5-14cca2-kserve-587f6996b8-sgf6h\" (UID: \"02b652c3-0724-4a50-9e95-2a9890cc97c4\") " pod="e2e-models-e2e-worker-w5-14cca2/distinct-w5-14cca2-kserve-587f6996b8-sgf6h" Oct 30 20:52:12.952657 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:12.952634 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/02b652c3-0724-4a50-9e95-2a9890cc97c4-tmp-dir\") pod \"distinct-w5-14cca2-kserve-587f6996b8-sgf6h\" (UID: \"02b652c3-0724-4a50-9e95-2a9890cc97c4\") " pod="e2e-models-e2e-worker-w5-14cca2/distinct-w5-14cca2-kserve-587f6996b8-sgf6h" Oct 30 20:52:12.955051 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:12.955023 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/02b652c3-0724-4a50-9e95-2a9890cc97c4-dshm\") pod \"distinct-w5-14cca2-kserve-587f6996b8-sgf6h\" (UID: \"02b652c3-0724-4a50-9e95-2a9890cc97c4\") " pod="e2e-models-e2e-worker-w5-14cca2/distinct-w5-14cca2-kserve-587f6996b8-sgf6h" Oct 30 20:52:12.955363 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:12.955342 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/02b652c3-0724-4a50-9e95-2a9890cc97c4-tls-certs\") pod \"distinct-w5-14cca2-kserve-587f6996b8-sgf6h\" (UID: \"02b652c3-0724-4a50-9e95-2a9890cc97c4\") " pod="e2e-models-e2e-worker-w5-14cca2/distinct-w5-14cca2-kserve-587f6996b8-sgf6h" Oct 30 20:52:12.970168 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:12.968376 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5fx85\" (UniqueName: \"kubernetes.io/projected/02b652c3-0724-4a50-9e95-2a9890cc97c4-kube-api-access-5fx85\") pod \"distinct-w5-14cca2-kserve-587f6996b8-sgf6h\" (UID: \"02b652c3-0724-4a50-9e95-2a9890cc97c4\") " pod="e2e-models-e2e-worker-w5-14cca2/distinct-w5-14cca2-kserve-587f6996b8-sgf6h" Oct 30 20:52:13.070691 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:13.070655 4611 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-14cca2/distinct-w5-14cca2-kserve-587f6996b8-sgf6h" Oct 30 20:52:13.119990 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:13.119941 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:52:13.372170 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:13.372146 4611 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-14cca2/distinct-w5-14cca2-kserve-587f6996b8-sgf6h"] Oct 30 20:52:13.374244 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:52:13.374213 4611 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod02b652c3_0724_4a50_9e95_2a9890cc97c4.slice/crio-669182bc8bed8cd3666e25e5f0c0d558a1f950b836b942e983b0c59d6445d118 WatchSource:0}: Error finding container 669182bc8bed8cd3666e25e5f0c0d558a1f950b836b942e983b0c59d6445d118: Status 404 returned error can't find the container with id 669182bc8bed8cd3666e25e5f0c0d558a1f950b836b942e983b0c59d6445d118 Oct 30 20:52:13.617502 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:13.617468 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-14cca2/distinct-w5-14cca2-kserve-587f6996b8-sgf6h" event={"ID":"02b652c3-0724-4a50-9e95-2a9890cc97c4","Type":"ContainerStarted","Data":"d1b4cba96b77d59a8b7e17e5ea7041b95d5bac7a658002706fdc139ef29a806f"} Oct 30 20:52:13.617502 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:13.617524 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-14cca2/distinct-w5-14cca2-kserve-587f6996b8-sgf6h" event={"ID":"02b652c3-0724-4a50-9e95-2a9890cc97c4","Type":"ContainerStarted","Data":"669182bc8bed8cd3666e25e5f0c0d558a1f950b836b942e983b0c59d6445d118"} Oct 30 20:52:13.636963 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:13.636854 4611 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-14cca2/distinct-w5-14cca2-kserve-587f6996b8-sgf6h" podStartSLOduration=1.63683969 podStartE2EDuration="1.63683969s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:52:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:52:13.634880808 +0000 UTC m=+788.168907426" watchObservedRunningTime="2026-10-30 20:52:13.63683969 +0000 UTC m=+788.170866306" Oct 30 20:52:13.764112 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:13.764086 4611 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-87079fdf-mf2f5" Oct 30 20:52:13.798117 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:13.798070 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:52:13.863839 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:13.863804 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/58075892-0276-4403-aa80-69c374f8283d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58075892-0276-4403-aa80-69c374f8283d-maas-api-openshift-ca\") pod \"58075892-0276-4403-aa80-69c374f8283d\" (UID: \"58075892-0276-4403-aa80-69c374f8283d\") " Oct 30 20:52:13.864025 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:13.864011 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/58075892-0276-4403-aa80-69c374f8283d-kube-api-access-w9rcs\" (UniqueName: \"kubernetes.io/projected/58075892-0276-4403-aa80-69c374f8283d-kube-api-access-w9rcs\") pod \"58075892-0276-4403-aa80-69c374f8283d\" (UID: \"58075892-0276-4403-aa80-69c374f8283d\") " Oct 30 20:52:13.864071 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:13.864036 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/58075892-0276-4403-aa80-69c374f8283d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58075892-0276-4403-aa80-69c374f8283d-maas-api-xks-ca\") pod \"58075892-0276-4403-aa80-69c374f8283d\" (UID: \"58075892-0276-4403-aa80-69c374f8283d\") " Oct 30 20:52:13.864276 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:13.864251 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58075892-0276-4403-aa80-69c374f8283d-maas-api-openshift-ca" pod "58075892-0276-4403-aa80-69c374f8283d" (UID: "58075892-0276-4403-aa80-69c374f8283d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:52:13.864575 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:13.864553 4611 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58075892-0276-4403-aa80-69c374f8283d-maas-api-openshift-ca\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:52:13.866165 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:13.866148 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58075892-0276-4403-aa80-69c374f8283d-kube-api-access-w9rcs" pod "58075892-0276-4403-aa80-69c374f8283d" (UID: "58075892-0276-4403-aa80-69c374f8283d"). InnerVolumeSpecName "kube-api-access-w9rcs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:52:13.867029 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:13.867008 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/58075892-0276-4403-aa80-69c374f8283d-maas-api-xks-ca" pod "58075892-0276-4403-aa80-69c374f8283d" (UID: "58075892-0276-4403-aa80-69c374f8283d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:52:13.965290 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:13.965225 4611 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w9rcs\" (UniqueName: \"kubernetes.io/projected/58075892-0276-4403-aa80-69c374f8283d-kube-api-access-w9rcs\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:52:13.965290 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:13.965247 4611 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58075892-0276-4403-aa80-69c374f8283d-maas-api-xks-ca\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:52:14.070942 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:14.070915 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-14cca2/distinct-w5-14cca2-kserve-587f6996b8-sgf6h" Oct 30 20:52:14.583290 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:14.583192 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:52:14.632358 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:14.632329 4611 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-87079fdf-mf2f5" Oct 30 20:52:14.632540 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:14.632372 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-87079fdf-mf2f5" event={"ID":"58075892-0276-4403-aa80-69c374f8283d","Type":"ContainerDied","Data":"048de947ace68583e162d105c713f0bc3043c55abbf84cd1d32eac6c8fbd8419"} Oct 30 20:52:14.632540 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:14.632457 4611 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="048de947ace68583e162d105c713f0bc3043c55abbf84cd1d32eac6c8fbd8419" Oct 30 20:52:15.456980 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:15.456939 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:52:16.304900 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:16.304708 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:52:16.942038 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:16.937875 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:52:17.695414 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:17.695375 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:52:18.309069 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:18.309032 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:52:18.986941 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:18.986901 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:52:19.712033 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:19.711949 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:52:20.569242 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:20.569189 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:52:21.662672 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:21.662631 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:52:22.683009 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:22.682974 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:52:23.071273 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:23.071196 4611 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-14cca2/distinct-w5-14cca2-kserve-587f6996b8-sgf6h" Oct 30 20:52:23.083850 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:23.083828 4611 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-14cca2/distinct-w5-14cca2-kserve-587f6996b8-sgf6h" Oct 30 20:52:23.638672 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:23.638615 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:52:23.713314 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:23.713286 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-14cca2/distinct-w5-14cca2-kserve-587f6996b8-sgf6h" Oct 30 20:52:24.317793 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:24.317754 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:52:24.487232 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:24.487197 4611 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-cb60a2/distinct-w3-cb60a2-kserve-5bb76dc74-86fds"] Oct 30 20:52:24.487833 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:24.487813 4611 cpu_manager.go:414] "containerMap: removing container" podUID="58075892-0276-4403-aa80-69c374f8283d" containerName="revoke-keys" Oct 30 20:52:24.487910 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:24.487834 4611 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="58075892-0276-4403-aa80-69c374f8283d" containerName="revoke-keys" Oct 30 20:52:24.487980 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:24.487949 4611 memory_manager.go:365] "RemoveStaleState removing state" podUID="58075892-0276-4403-aa80-69c374f8283d" containerName="revoke-keys" Oct 30 20:52:24.496281 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:24.496257 4611 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-cb60a2/distinct-w3-cb60a2-kserve-5bb76dc74-86fds" Oct 30 20:52:24.500090 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:24.500064 4611 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-cb60a2/distinct-w3-cb60a2-kserve-5bb76dc74-86fds"] Oct 30 20:52:24.508060 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:24.508040 4611 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-cb60a2\"/\"distinct-w3-cb60a2-kserve-self-signed-certs\"" Oct 30 20:52:24.609060 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:24.609028 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:52:24.613233 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:24.613207 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/52e301d9-f378-4e54-9515-01a6680d9fd7-tls-certs\") pod \"distinct-w3-cb60a2-kserve-5bb76dc74-86fds\" (UID: \"52e301d9-f378-4e54-9515-01a6680d9fd7\") " pod="e2e-models-e2e-worker-w3-cb60a2/distinct-w3-cb60a2-kserve-5bb76dc74-86fds" Oct 30 20:52:24.613356 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:24.613247 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tl6c9\" (UniqueName: \"kubernetes.io/projected/52e301d9-f378-4e54-9515-01a6680d9fd7-kube-api-access-tl6c9\") pod \"distinct-w3-cb60a2-kserve-5bb76dc74-86fds\" (UID: \"52e301d9-f378-4e54-9515-01a6680d9fd7\") " pod="e2e-models-e2e-worker-w3-cb60a2/distinct-w3-cb60a2-kserve-5bb76dc74-86fds" Oct 30 20:52:24.613406 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:24.613374 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/52e301d9-f378-4e54-9515-01a6680d9fd7-tmp-dir\") pod \"distinct-w3-cb60a2-kserve-5bb76dc74-86fds\" (UID: \"52e301d9-f378-4e54-9515-01a6680d9fd7\") " pod="e2e-models-e2e-worker-w3-cb60a2/distinct-w3-cb60a2-kserve-5bb76dc74-86fds" Oct 30 20:52:24.613457 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:24.613426 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/52e301d9-f378-4e54-9515-01a6680d9fd7-model-cache\") pod \"distinct-w3-cb60a2-kserve-5bb76dc74-86fds\" (UID: \"52e301d9-f378-4e54-9515-01a6680d9fd7\") " pod="e2e-models-e2e-worker-w3-cb60a2/distinct-w3-cb60a2-kserve-5bb76dc74-86fds" Oct 30 20:52:24.613572 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:24.613558 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/52e301d9-f378-4e54-9515-01a6680d9fd7-dshm\") pod \"distinct-w3-cb60a2-kserve-5bb76dc74-86fds\" (UID: \"52e301d9-f378-4e54-9515-01a6680d9fd7\") " pod="e2e-models-e2e-worker-w3-cb60a2/distinct-w3-cb60a2-kserve-5bb76dc74-86fds" Oct 30 20:52:24.613613 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:24.613597 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/52e301d9-f378-4e54-9515-01a6680d9fd7-home\") pod \"distinct-w3-cb60a2-kserve-5bb76dc74-86fds\" (UID: \"52e301d9-f378-4e54-9515-01a6680d9fd7\") " pod="e2e-models-e2e-worker-w3-cb60a2/distinct-w3-cb60a2-kserve-5bb76dc74-86fds" Oct 30 20:52:24.714544 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:24.714498 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/52e301d9-f378-4e54-9515-01a6680d9fd7-tls-certs\") pod \"distinct-w3-cb60a2-kserve-5bb76dc74-86fds\" (UID: \"52e301d9-f378-4e54-9515-01a6680d9fd7\") " pod="e2e-models-e2e-worker-w3-cb60a2/distinct-w3-cb60a2-kserve-5bb76dc74-86fds" Oct 30 20:52:24.715027 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:24.714622 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tl6c9\" (UniqueName: \"kubernetes.io/projected/52e301d9-f378-4e54-9515-01a6680d9fd7-kube-api-access-tl6c9\") pod \"distinct-w3-cb60a2-kserve-5bb76dc74-86fds\" (UID: \"52e301d9-f378-4e54-9515-01a6680d9fd7\") " pod="e2e-models-e2e-worker-w3-cb60a2/distinct-w3-cb60a2-kserve-5bb76dc74-86fds" Oct 30 20:52:24.715027 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:24.714681 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/52e301d9-f378-4e54-9515-01a6680d9fd7-tmp-dir\") pod \"distinct-w3-cb60a2-kserve-5bb76dc74-86fds\" (UID: \"52e301d9-f378-4e54-9515-01a6680d9fd7\") " pod="e2e-models-e2e-worker-w3-cb60a2/distinct-w3-cb60a2-kserve-5bb76dc74-86fds" Oct 30 20:52:24.715027 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:24.714723 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/52e301d9-f378-4e54-9515-01a6680d9fd7-model-cache\") pod \"distinct-w3-cb60a2-kserve-5bb76dc74-86fds\" (UID: \"52e301d9-f378-4e54-9515-01a6680d9fd7\") " pod="e2e-models-e2e-worker-w3-cb60a2/distinct-w3-cb60a2-kserve-5bb76dc74-86fds" Oct 30 20:52:24.715027 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:24.714799 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/52e301d9-f378-4e54-9515-01a6680d9fd7-dshm\") pod \"distinct-w3-cb60a2-kserve-5bb76dc74-86fds\" (UID: \"52e301d9-f378-4e54-9515-01a6680d9fd7\") " pod="e2e-models-e2e-worker-w3-cb60a2/distinct-w3-cb60a2-kserve-5bb76dc74-86fds" Oct 30 20:52:24.715027 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:24.714842 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/52e301d9-f378-4e54-9515-01a6680d9fd7-home\") pod \"distinct-w3-cb60a2-kserve-5bb76dc74-86fds\" (UID: \"52e301d9-f378-4e54-9515-01a6680d9fd7\") " pod="e2e-models-e2e-worker-w3-cb60a2/distinct-w3-cb60a2-kserve-5bb76dc74-86fds" Oct 30 20:52:24.715266 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:24.715156 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/52e301d9-f378-4e54-9515-01a6680d9fd7-tmp-dir\") pod \"distinct-w3-cb60a2-kserve-5bb76dc74-86fds\" (UID: \"52e301d9-f378-4e54-9515-01a6680d9fd7\") " pod="e2e-models-e2e-worker-w3-cb60a2/distinct-w3-cb60a2-kserve-5bb76dc74-86fds" Oct 30 20:52:24.715266 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:24.715192 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/52e301d9-f378-4e54-9515-01a6680d9fd7-home\") pod \"distinct-w3-cb60a2-kserve-5bb76dc74-86fds\" (UID: \"52e301d9-f378-4e54-9515-01a6680d9fd7\") " pod="e2e-models-e2e-worker-w3-cb60a2/distinct-w3-cb60a2-kserve-5bb76dc74-86fds" Oct 30 20:52:24.715342 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:24.715312 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/52e301d9-f378-4e54-9515-01a6680d9fd7-model-cache\") pod \"distinct-w3-cb60a2-kserve-5bb76dc74-86fds\" (UID: \"52e301d9-f378-4e54-9515-01a6680d9fd7\") " pod="e2e-models-e2e-worker-w3-cb60a2/distinct-w3-cb60a2-kserve-5bb76dc74-86fds" Oct 30 20:52:24.716885 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:24.716861 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/52e301d9-f378-4e54-9515-01a6680d9fd7-dshm\") pod \"distinct-w3-cb60a2-kserve-5bb76dc74-86fds\" (UID: \"52e301d9-f378-4e54-9515-01a6680d9fd7\") " pod="e2e-models-e2e-worker-w3-cb60a2/distinct-w3-cb60a2-kserve-5bb76dc74-86fds" Oct 30 20:52:24.717156 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:24.717137 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/52e301d9-f378-4e54-9515-01a6680d9fd7-tls-certs\") pod \"distinct-w3-cb60a2-kserve-5bb76dc74-86fds\" (UID: \"52e301d9-f378-4e54-9515-01a6680d9fd7\") " pod="e2e-models-e2e-worker-w3-cb60a2/distinct-w3-cb60a2-kserve-5bb76dc74-86fds" Oct 30 20:52:24.725997 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:24.725976 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tl6c9\" (UniqueName: \"kubernetes.io/projected/52e301d9-f378-4e54-9515-01a6680d9fd7-kube-api-access-tl6c9\") pod \"distinct-w3-cb60a2-kserve-5bb76dc74-86fds\" (UID: \"52e301d9-f378-4e54-9515-01a6680d9fd7\") " pod="e2e-models-e2e-worker-w3-cb60a2/distinct-w3-cb60a2-kserve-5bb76dc74-86fds" Oct 30 20:52:24.812632 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:24.812596 4611 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-cb60a2/distinct-w3-cb60a2-kserve-5bb76dc74-86fds" Oct 30 20:52:25.094439 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:25.094396 4611 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-cb60a2/distinct-w3-cb60a2-kserve-5bb76dc74-86fds"] Oct 30 20:52:25.097275 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:52:25.097249 4611 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod52e301d9_f378_4e54_9515_01a6680d9fd7.slice/crio-5573e26a002b83469b1768d80b0d819d38c1c8aea0f4663b2c212c6f0bc1c2b7 WatchSource:0}: Error finding container 5573e26a002b83469b1768d80b0d819d38c1c8aea0f4663b2c212c6f0bc1c2b7: Status 404 returned error can't find the container with id 5573e26a002b83469b1768d80b0d819d38c1c8aea0f4663b2c212c6f0bc1c2b7 Oct 30 20:52:25.112656 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:25.112585 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:52:25.716957 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:25.716918 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-cb60a2/distinct-w3-cb60a2-kserve-5bb76dc74-86fds" event={"ID":"52e301d9-f378-4e54-9515-01a6680d9fd7","Type":"ContainerStarted","Data":"36cf0baf044dd3dc38d449c968df0fbd734cf76c3a1acfd9d63f3083987ced58"} Oct 30 20:52:25.717367 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:25.716971 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-cb60a2/distinct-w3-cb60a2-kserve-5bb76dc74-86fds" event={"ID":"52e301d9-f378-4e54-9515-01a6680d9fd7","Type":"ContainerStarted","Data":"5573e26a002b83469b1768d80b0d819d38c1c8aea0f4663b2c212c6f0bc1c2b7"} Oct 30 20:52:25.741688 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:25.741640 4611 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-cb60a2/distinct-w3-cb60a2-kserve-5bb76dc74-86fds" podStartSLOduration=1.741625763 podStartE2EDuration="1.741625763s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:52:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:52:25.73639526 +0000 UTC m=+800.270421876" watchObservedRunningTime="2026-10-30 20:52:25.741625763 +0000 UTC m=+800.275652380" Oct 30 20:52:25.776086 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:25.776059 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:52:25.812823 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:25.812792 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-cb60a2/distinct-w3-cb60a2-kserve-5bb76dc74-86fds" Oct 30 20:52:26.409617 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:26.409571 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:52:27.104301 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:27.104254 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:52:27.721692 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:27.721653 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:52:28.407741 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:28.407677 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:52:29.117101 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:29.117071 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:52:29.772833 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:29.772795 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:52:30.389288 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:30.389246 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:52:31.100273 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:31.100236 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:52:31.866174 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:31.866138 4611 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-gl9z9"] Oct 30 20:52:31.870467 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:31.870442 4611 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-gl9z9" Oct 30 20:52:31.918539 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:31.916993 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29cef334-f72f-471a-9793-f7907b62e622-istio-envoy\") pod \"e2e-worker-w2-db534b-openshift-default-85df4d79d9-gl9z9\" (UID: \"29cef334-f72f-471a-9793-f7907b62e622\") " pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-gl9z9" Oct 30 20:52:31.918539 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:31.917052 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sdl6p\" (UniqueName: \"kubernetes.io/projected/29cef334-f72f-471a-9793-f7907b62e622-kube-api-access-sdl6p\") pod \"e2e-worker-w2-db534b-openshift-default-85df4d79d9-gl9z9\" (UID: \"29cef334-f72f-471a-9793-f7907b62e622\") " pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-gl9z9" Oct 30 20:52:31.918539 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:31.917105 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29cef334-f72f-471a-9793-f7907b62e622-istiod-ca-cert\") pod \"e2e-worker-w2-db534b-openshift-default-85df4d79d9-gl9z9\" (UID: \"29cef334-f72f-471a-9793-f7907b62e622\") " pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-gl9z9" Oct 30 20:52:31.918539 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:31.917318 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29cef334-f72f-471a-9793-f7907b62e622-istio-data\") pod \"e2e-worker-w2-db534b-openshift-default-85df4d79d9-gl9z9\" (UID: \"29cef334-f72f-471a-9793-f7907b62e622\") " pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-gl9z9" Oct 30 20:52:31.918539 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:31.917400 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29cef334-f72f-471a-9793-f7907b62e622-credential-socket\") pod \"e2e-worker-w2-db534b-openshift-default-85df4d79d9-gl9z9\" (UID: \"29cef334-f72f-471a-9793-f7907b62e622\") " pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-gl9z9" Oct 30 20:52:31.918539 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:31.917478 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/29cef334-f72f-471a-9793-f7907b62e622-istio-token\") pod \"e2e-worker-w2-db534b-openshift-default-85df4d79d9-gl9z9\" (UID: \"29cef334-f72f-471a-9793-f7907b62e622\") " pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-gl9z9" Oct 30 20:52:31.918539 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:31.917594 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29cef334-f72f-471a-9793-f7907b62e622-istio-podinfo\") pod \"e2e-worker-w2-db534b-openshift-default-85df4d79d9-gl9z9\" (UID: \"29cef334-f72f-471a-9793-f7907b62e622\") " pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-gl9z9" Oct 30 20:52:31.918539 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:31.917626 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29cef334-f72f-471a-9793-f7907b62e622-workload-socket\") pod \"e2e-worker-w2-db534b-openshift-default-85df4d79d9-gl9z9\" (UID: \"29cef334-f72f-471a-9793-f7907b62e622\") " pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-gl9z9" Oct 30 20:52:31.918539 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:31.917664 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29cef334-f72f-471a-9793-f7907b62e622-workload-certs\") pod \"e2e-worker-w2-db534b-openshift-default-85df4d79d9-gl9z9\" (UID: \"29cef334-f72f-471a-9793-f7907b62e622\") " pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-gl9z9" Oct 30 20:52:31.922167 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:31.922137 4611 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-gl9z9"] Oct 30 20:52:31.934890 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:31.934838 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:52:31.968622 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:31.968581 4611 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-6pfl5"] Oct 30 20:52:31.972658 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:31.972635 4611 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-6pfl5" Oct 30 20:52:31.997040 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:31.997012 4611 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-6pfl5"] Oct 30 20:52:32.018527 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:32.018478 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29cef334-f72f-471a-9793-f7907b62e622-istio-podinfo\") pod \"e2e-worker-w2-db534b-openshift-default-85df4d79d9-gl9z9\" (UID: \"29cef334-f72f-471a-9793-f7907b62e622\") " pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-gl9z9" Oct 30 20:52:32.018685 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:32.018540 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29cef334-f72f-471a-9793-f7907b62e622-workload-socket\") pod \"e2e-worker-w2-db534b-openshift-default-85df4d79d9-gl9z9\" (UID: \"29cef334-f72f-471a-9793-f7907b62e622\") " pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-gl9z9" Oct 30 20:52:32.018685 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:32.018575 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29cef334-f72f-471a-9793-f7907b62e622-workload-certs\") pod \"e2e-worker-w2-db534b-openshift-default-85df4d79d9-gl9z9\" (UID: \"29cef334-f72f-471a-9793-f7907b62e622\") " pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-gl9z9" Oct 30 20:52:32.018685 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:32.018637 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29cef334-f72f-471a-9793-f7907b62e622-istio-envoy\") pod \"e2e-worker-w2-db534b-openshift-default-85df4d79d9-gl9z9\" (UID: \"29cef334-f72f-471a-9793-f7907b62e622\") " pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-gl9z9" Oct 30 20:52:32.018685 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:32.018668 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sdl6p\" (UniqueName: \"kubernetes.io/projected/29cef334-f72f-471a-9793-f7907b62e622-kube-api-access-sdl6p\") pod \"e2e-worker-w2-db534b-openshift-default-85df4d79d9-gl9z9\" (UID: \"29cef334-f72f-471a-9793-f7907b62e622\") " pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-gl9z9" Oct 30 20:52:32.018909 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:32.018713 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29cef334-f72f-471a-9793-f7907b62e622-istiod-ca-cert\") pod \"e2e-worker-w2-db534b-openshift-default-85df4d79d9-gl9z9\" (UID: \"29cef334-f72f-471a-9793-f7907b62e622\") " pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-gl9z9" Oct 30 20:52:32.018909 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:32.018760 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/06695006-f33e-474a-bfa5-40eb62bd42f8-istio-token\") pod \"e2e-worker-w5-14cca2-openshift-default-d7f87db59-6pfl5\" (UID: \"06695006-f33e-474a-bfa5-40eb62bd42f8\") " pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-6pfl5" Oct 30 20:52:32.018909 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:32.018788 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06695006-f33e-474a-bfa5-40eb62bd42f8-istio-envoy\") pod \"e2e-worker-w5-14cca2-openshift-default-d7f87db59-6pfl5\" (UID: \"06695006-f33e-474a-bfa5-40eb62bd42f8\") " pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-6pfl5" Oct 30 20:52:32.018909 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:32.018810 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06695006-f33e-474a-bfa5-40eb62bd42f8-istio-podinfo\") pod \"e2e-worker-w5-14cca2-openshift-default-d7f87db59-6pfl5\" (UID: \"06695006-f33e-474a-bfa5-40eb62bd42f8\") " pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-6pfl5" Oct 30 20:52:32.018909 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:32.018844 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29cef334-f72f-471a-9793-f7907b62e622-istio-data\") pod \"e2e-worker-w2-db534b-openshift-default-85df4d79d9-gl9z9\" (UID: \"29cef334-f72f-471a-9793-f7907b62e622\") " pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-gl9z9" Oct 30 20:52:32.018909 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:32.018879 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06695006-f33e-474a-bfa5-40eb62bd42f8-workload-socket\") pod \"e2e-worker-w5-14cca2-openshift-default-d7f87db59-6pfl5\" (UID: \"06695006-f33e-474a-bfa5-40eb62bd42f8\") " pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-6pfl5" Oct 30 20:52:32.019203 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:32.018918 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06695006-f33e-474a-bfa5-40eb62bd42f8-istio-data\") pod \"e2e-worker-w5-14cca2-openshift-default-d7f87db59-6pfl5\" (UID: \"06695006-f33e-474a-bfa5-40eb62bd42f8\") " pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-6pfl5" Oct 30 20:52:32.019203 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:32.018956 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29cef334-f72f-471a-9793-f7907b62e622-credential-socket\") pod \"e2e-worker-w2-db534b-openshift-default-85df4d79d9-gl9z9\" (UID: \"29cef334-f72f-471a-9793-f7907b62e622\") " pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-gl9z9" Oct 30 20:52:32.019203 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:32.018980 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06695006-f33e-474a-bfa5-40eb62bd42f8-istiod-ca-cert\") pod \"e2e-worker-w5-14cca2-openshift-default-d7f87db59-6pfl5\" (UID: \"06695006-f33e-474a-bfa5-40eb62bd42f8\") " pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-6pfl5" Oct 30 20:52:32.019203 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:32.019028 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/29cef334-f72f-471a-9793-f7907b62e622-istio-token\") pod \"e2e-worker-w2-db534b-openshift-default-85df4d79d9-gl9z9\" (UID: \"29cef334-f72f-471a-9793-f7907b62e622\") " pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-gl9z9" Oct 30 20:52:32.019203 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:32.019043 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06695006-f33e-474a-bfa5-40eb62bd42f8-workload-certs\") pod \"e2e-worker-w5-14cca2-openshift-default-d7f87db59-6pfl5\" (UID: \"06695006-f33e-474a-bfa5-40eb62bd42f8\") " pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-6pfl5" Oct 30 20:52:32.019203 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:32.019062 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06695006-f33e-474a-bfa5-40eb62bd42f8-credential-socket\") pod \"e2e-worker-w5-14cca2-openshift-default-d7f87db59-6pfl5\" (UID: \"06695006-f33e-474a-bfa5-40eb62bd42f8\") " pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-6pfl5" Oct 30 20:52:32.019203 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:32.019080 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-76brf\" (UniqueName: \"kubernetes.io/projected/06695006-f33e-474a-bfa5-40eb62bd42f8-kube-api-access-76brf\") pod \"e2e-worker-w5-14cca2-openshift-default-d7f87db59-6pfl5\" (UID: \"06695006-f33e-474a-bfa5-40eb62bd42f8\") " pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-6pfl5" Oct 30 20:52:32.021020 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:32.020989 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29cef334-f72f-471a-9793-f7907b62e622-credential-socket\") pod \"e2e-worker-w2-db534b-openshift-default-85df4d79d9-gl9z9\" (UID: \"29cef334-f72f-471a-9793-f7907b62e622\") " pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-gl9z9" Oct 30 20:52:32.021135 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:32.021025 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29cef334-f72f-471a-9793-f7907b62e622-istio-data\") pod \"e2e-worker-w2-db534b-openshift-default-85df4d79d9-gl9z9\" (UID: \"29cef334-f72f-471a-9793-f7907b62e622\") " pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-gl9z9" Oct 30 20:52:32.021232 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:32.021207 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29cef334-f72f-471a-9793-f7907b62e622-workload-socket\") pod \"e2e-worker-w2-db534b-openshift-default-85df4d79d9-gl9z9\" (UID: \"29cef334-f72f-471a-9793-f7907b62e622\") " pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-gl9z9" Oct 30 20:52:32.021393 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:32.021369 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29cef334-f72f-471a-9793-f7907b62e622-workload-certs\") pod \"e2e-worker-w2-db534b-openshift-default-85df4d79d9-gl9z9\" (UID: \"29cef334-f72f-471a-9793-f7907b62e622\") " pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-gl9z9" Oct 30 20:52:32.021740 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:32.021708 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29cef334-f72f-471a-9793-f7907b62e622-istiod-ca-cert\") pod \"e2e-worker-w2-db534b-openshift-default-85df4d79d9-gl9z9\" (UID: \"29cef334-f72f-471a-9793-f7907b62e622\") " pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-gl9z9" Oct 30 20:52:32.022185 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:32.022163 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29cef334-f72f-471a-9793-f7907b62e622-istio-podinfo\") pod \"e2e-worker-w2-db534b-openshift-default-85df4d79d9-gl9z9\" (UID: \"29cef334-f72f-471a-9793-f7907b62e622\") " pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-gl9z9" Oct 30 20:52:32.024147 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:32.024127 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29cef334-f72f-471a-9793-f7907b62e622-istio-envoy\") pod \"e2e-worker-w2-db534b-openshift-default-85df4d79d9-gl9z9\" (UID: \"29cef334-f72f-471a-9793-f7907b62e622\") " pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-gl9z9" Oct 30 20:52:32.034226 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:32.034176 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sdl6p\" (UniqueName: \"kubernetes.io/projected/29cef334-f72f-471a-9793-f7907b62e622-kube-api-access-sdl6p\") pod \"e2e-worker-w2-db534b-openshift-default-85df4d79d9-gl9z9\" (UID: \"29cef334-f72f-471a-9793-f7907b62e622\") " pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-gl9z9" Oct 30 20:52:32.036239 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:32.036218 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/29cef334-f72f-471a-9793-f7907b62e622-istio-token\") pod \"e2e-worker-w2-db534b-openshift-default-85df4d79d9-gl9z9\" (UID: \"29cef334-f72f-471a-9793-f7907b62e622\") " pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-gl9z9" Oct 30 20:52:32.120612 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:32.120532 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/06695006-f33e-474a-bfa5-40eb62bd42f8-istio-token\") pod \"e2e-worker-w5-14cca2-openshift-default-d7f87db59-6pfl5\" (UID: \"06695006-f33e-474a-bfa5-40eb62bd42f8\") " pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-6pfl5" Oct 30 20:52:32.120612 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:32.120580 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06695006-f33e-474a-bfa5-40eb62bd42f8-istio-envoy\") pod \"e2e-worker-w5-14cca2-openshift-default-d7f87db59-6pfl5\" (UID: \"06695006-f33e-474a-bfa5-40eb62bd42f8\") " pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-6pfl5" Oct 30 20:52:32.121072 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:32.120750 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06695006-f33e-474a-bfa5-40eb62bd42f8-istio-podinfo\") pod \"e2e-worker-w5-14cca2-openshift-default-d7f87db59-6pfl5\" (UID: \"06695006-f33e-474a-bfa5-40eb62bd42f8\") " pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-6pfl5" Oct 30 20:52:32.121072 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:32.120796 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06695006-f33e-474a-bfa5-40eb62bd42f8-workload-socket\") pod \"e2e-worker-w5-14cca2-openshift-default-d7f87db59-6pfl5\" (UID: \"06695006-f33e-474a-bfa5-40eb62bd42f8\") " pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-6pfl5" Oct 30 20:52:32.121072 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:32.120827 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06695006-f33e-474a-bfa5-40eb62bd42f8-istio-data\") pod \"e2e-worker-w5-14cca2-openshift-default-d7f87db59-6pfl5\" (UID: \"06695006-f33e-474a-bfa5-40eb62bd42f8\") " pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-6pfl5" Oct 30 20:52:32.121072 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:32.120880 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06695006-f33e-474a-bfa5-40eb62bd42f8-istiod-ca-cert\") pod \"e2e-worker-w5-14cca2-openshift-default-d7f87db59-6pfl5\" (UID: \"06695006-f33e-474a-bfa5-40eb62bd42f8\") " pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-6pfl5" Oct 30 20:52:32.121072 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:32.120951 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06695006-f33e-474a-bfa5-40eb62bd42f8-workload-certs\") pod \"e2e-worker-w5-14cca2-openshift-default-d7f87db59-6pfl5\" (UID: \"06695006-f33e-474a-bfa5-40eb62bd42f8\") " pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-6pfl5" Oct 30 20:52:32.121072 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:32.120981 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06695006-f33e-474a-bfa5-40eb62bd42f8-credential-socket\") pod \"e2e-worker-w5-14cca2-openshift-default-d7f87db59-6pfl5\" (UID: \"06695006-f33e-474a-bfa5-40eb62bd42f8\") " pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-6pfl5" Oct 30 20:52:32.121072 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:32.121010 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-76brf\" (UniqueName: \"kubernetes.io/projected/06695006-f33e-474a-bfa5-40eb62bd42f8-kube-api-access-76brf\") pod \"e2e-worker-w5-14cca2-openshift-default-d7f87db59-6pfl5\" (UID: \"06695006-f33e-474a-bfa5-40eb62bd42f8\") " pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-6pfl5" Oct 30 20:52:32.121424 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:32.121207 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06695006-f33e-474a-bfa5-40eb62bd42f8-workload-socket\") pod \"e2e-worker-w5-14cca2-openshift-default-d7f87db59-6pfl5\" (UID: \"06695006-f33e-474a-bfa5-40eb62bd42f8\") " pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-6pfl5" Oct 30 20:52:32.121424 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:32.121251 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06695006-f33e-474a-bfa5-40eb62bd42f8-istio-data\") pod \"e2e-worker-w5-14cca2-openshift-default-d7f87db59-6pfl5\" (UID: \"06695006-f33e-474a-bfa5-40eb62bd42f8\") " pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-6pfl5" Oct 30 20:52:32.121581 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:32.121559 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06695006-f33e-474a-bfa5-40eb62bd42f8-credential-socket\") pod \"e2e-worker-w5-14cca2-openshift-default-d7f87db59-6pfl5\" (UID: \"06695006-f33e-474a-bfa5-40eb62bd42f8\") " pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-6pfl5" Oct 30 20:52:32.121726 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:32.121683 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06695006-f33e-474a-bfa5-40eb62bd42f8-workload-certs\") pod \"e2e-worker-w5-14cca2-openshift-default-d7f87db59-6pfl5\" (UID: \"06695006-f33e-474a-bfa5-40eb62bd42f8\") " pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-6pfl5" Oct 30 20:52:32.121916 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:32.121893 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06695006-f33e-474a-bfa5-40eb62bd42f8-istiod-ca-cert\") pod \"e2e-worker-w5-14cca2-openshift-default-d7f87db59-6pfl5\" (UID: \"06695006-f33e-474a-bfa5-40eb62bd42f8\") " pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-6pfl5" Oct 30 20:52:32.123475 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:32.123446 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06695006-f33e-474a-bfa5-40eb62bd42f8-istio-envoy\") pod \"e2e-worker-w5-14cca2-openshift-default-d7f87db59-6pfl5\" (UID: \"06695006-f33e-474a-bfa5-40eb62bd42f8\") " pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-6pfl5" Oct 30 20:52:32.123692 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:32.123675 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06695006-f33e-474a-bfa5-40eb62bd42f8-istio-podinfo\") pod \"e2e-worker-w5-14cca2-openshift-default-d7f87db59-6pfl5\" (UID: \"06695006-f33e-474a-bfa5-40eb62bd42f8\") " pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-6pfl5" Oct 30 20:52:32.136719 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:32.136692 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/06695006-f33e-474a-bfa5-40eb62bd42f8-istio-token\") pod \"e2e-worker-w5-14cca2-openshift-default-d7f87db59-6pfl5\" (UID: \"06695006-f33e-474a-bfa5-40eb62bd42f8\") " pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-6pfl5" Oct 30 20:52:32.136835 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:32.136745 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-76brf\" (UniqueName: \"kubernetes.io/projected/06695006-f33e-474a-bfa5-40eb62bd42f8-kube-api-access-76brf\") pod \"e2e-worker-w5-14cca2-openshift-default-d7f87db59-6pfl5\" (UID: \"06695006-f33e-474a-bfa5-40eb62bd42f8\") " pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-6pfl5" Oct 30 20:52:32.209858 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:32.209828 4611 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-gl9z9" Oct 30 20:52:32.352991 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:32.349753 4611 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-6pfl5" Oct 30 20:52:32.519904 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:32.519811 4611 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-gl9z9"] Oct 30 20:52:32.542144 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:52:32.542092 4611 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29cef334_f72f_471a_9793_f7907b62e622.slice/crio-0f1409b2989e74d8f1ca2b08e2e5b6f93b49918b63e50c79675bdfc85ffeb599 WatchSource:0}: Error finding container 0f1409b2989e74d8f1ca2b08e2e5b6f93b49918b63e50c79675bdfc85ffeb599: Status 404 returned error can't find the container with id 0f1409b2989e74d8f1ca2b08e2e5b6f93b49918b63e50c79675bdfc85ffeb599 Oct 30 20:52:32.547264 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:32.545843 4611 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:52:32.547264 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:32.545930 4611 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:52:32.547264 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:32.545977 4611 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:52:32.727543 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:32.723778 4611 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-6pfl5"] Oct 30 20:52:32.731082 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:32.731042 4611 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:52:32.731195 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:32.731134 4611 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:52:32.731195 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:32.731181 4611 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:52:32.772060 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:32.771992 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-6pfl5" event={"ID":"06695006-f33e-474a-bfa5-40eb62bd42f8","Type":"ContainerStarted","Data":"f1d317ed8399fd402fd0848fc383a41e38b436aa21d27912592be2aa68f9bdac"} Oct 30 20:52:32.774052 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:32.773640 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-gl9z9" event={"ID":"29cef334-f72f-471a-9793-f7907b62e622","Type":"ContainerStarted","Data":"e577f4543d3da626fa3715088ef07c55db24bb6b443dd39a8c4530fd7118b3cd"} Oct 30 20:52:32.774052 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:32.773676 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-gl9z9" event={"ID":"29cef334-f72f-471a-9793-f7907b62e622","Type":"ContainerStarted","Data":"0f1409b2989e74d8f1ca2b08e2e5b6f93b49918b63e50c79675bdfc85ffeb599"} Oct 30 20:52:32.790737 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:32.790713 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:52:32.807539 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:32.805808 4611 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-gl9z9" podStartSLOduration=1.8057911359999999 podStartE2EDuration="1.805791136s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:52:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:52:32.804004468 +0000 UTC m=+807.338031084" watchObservedRunningTime="2026-10-30 20:52:32.805791136 +0000 UTC m=+807.339817753" Oct 30 20:52:33.218526 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:33.217684 4611 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-gl9z9" Oct 30 20:52:33.229669 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:33.229632 4611 patch_prober.go:29] interesting pod/e2e-worker-w2-db534b-openshift-default-85df4d79d9-gl9z9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body= Oct 30 20:52:33.229905 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:33.229869 4611 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-gl9z9" podUID="29cef334-f72f-471a-9793-f7907b62e622" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" Oct 30 20:52:33.589250 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:33.589017 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:52:33.840634 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:33.835672 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-6pfl5" event={"ID":"06695006-f33e-474a-bfa5-40eb62bd42f8","Type":"ContainerStarted","Data":"087ddc58167f65b47f6e684ebfdfb7d6fc341e0c4e55750c35fe52dce7ad0473"} Oct 30 20:52:34.042536 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:33.996984 4611 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-6pfl5" podStartSLOduration=2.996959525 podStartE2EDuration="2.996959525s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:52:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:52:33.88900683 +0000 UTC m=+808.423033445" watchObservedRunningTime="2026-10-30 20:52:33.996959525 +0000 UTC m=+808.530986142" Oct 30 20:52:34.046128 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:34.043882 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:52:34.228534 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:34.213169 4611 patch_prober.go:29] interesting pod/e2e-worker-w2-db534b-openshift-default-85df4d79d9-gl9z9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body= Oct 30 20:52:34.240730 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:34.240663 4611 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-gl9z9" podUID="29cef334-f72f-471a-9793-f7907b62e622" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" Oct 30 20:52:34.371531 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:34.369901 4611 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-6pfl5" Oct 30 20:52:34.389902 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:34.389865 4611 patch_prober.go:29] interesting pod/e2e-worker-w5-14cca2-openshift-default-d7f87db59-6pfl5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Oct 30 20:52:34.390174 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:34.390134 4611 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-6pfl5" podUID="06695006-f33e-474a-bfa5-40eb62bd42f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Oct 30 20:52:34.440560 ip-10-0-133-192 kubenswrapper[4611]: E1030 20:52:34.433018 4611 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hwzc6" containerName="extractor" Oct 30 20:52:34.530575 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:34.524955 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:52:34.813914 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:34.813837 4611 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-cb60a2/distinct-w3-cb60a2-kserve-5bb76dc74-86fds" Oct 30 20:52:34.831702 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:34.831671 4611 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-cb60a2/distinct-w3-cb60a2-kserve-5bb76dc74-86fds" Oct 30 20:52:34.902137 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:34.902104 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-cb60a2/distinct-w3-cb60a2-kserve-5bb76dc74-86fds" Oct 30 20:52:35.028905 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:35.027890 4611 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-020e7427-mrljk"] Oct 30 20:52:35.035108 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:35.035077 4611 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-020e7427-mrljk" Oct 30 20:52:35.038613 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:35.038589 4611 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:52:35.038753 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:35.038655 4611 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7hf8r\"" Oct 30 20:52:35.040775 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:35.040548 4611 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-020e7427-mrljk"] Oct 30 20:52:35.044453 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:35.044211 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a41f107-a2bb-45e9-a071-265238a96fc7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-020e7427-mrljk\" (UID: \"7a41f107-a2bb-45e9-a071-265238a96fc7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-020e7427-mrljk" Oct 30 20:52:35.044453 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:35.044256 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a41f107-a2bb-45e9-a071-265238a96fc7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-020e7427-mrljk\" (UID: \"7a41f107-a2bb-45e9-a071-265238a96fc7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-020e7427-mrljk" Oct 30 20:52:35.044453 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:35.044351 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k75ll\" (UniqueName: \"kubernetes.io/projected/7a41f107-a2bb-45e9-a071-265238a96fc7-kube-api-access-k75ll\") pod \"maas-api-revoke-sub-020e7427-mrljk\" (UID: \"7a41f107-a2bb-45e9-a071-265238a96fc7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-020e7427-mrljk" Oct 30 20:52:35.196533 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:35.173685 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a41f107-a2bb-45e9-a071-265238a96fc7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-020e7427-mrljk\" (UID: \"7a41f107-a2bb-45e9-a071-265238a96fc7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-020e7427-mrljk" Oct 30 20:52:35.197649 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:35.197615 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k75ll\" (UniqueName: \"kubernetes.io/projected/7a41f107-a2bb-45e9-a071-265238a96fc7-kube-api-access-k75ll\") pod \"maas-api-revoke-sub-020e7427-mrljk\" (UID: \"7a41f107-a2bb-45e9-a071-265238a96fc7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-020e7427-mrljk" Oct 30 20:52:35.197964 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:35.197936 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a41f107-a2bb-45e9-a071-265238a96fc7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-020e7427-mrljk\" (UID: \"7a41f107-a2bb-45e9-a071-265238a96fc7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-020e7427-mrljk" Oct 30 20:52:35.200129 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:35.175702 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a41f107-a2bb-45e9-a071-265238a96fc7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-020e7427-mrljk\" (UID: \"7a41f107-a2bb-45e9-a071-265238a96fc7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-020e7427-mrljk" Oct 30 20:52:35.206535 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:35.206492 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a41f107-a2bb-45e9-a071-265238a96fc7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-020e7427-mrljk\" (UID: \"7a41f107-a2bb-45e9-a071-265238a96fc7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-020e7427-mrljk" Oct 30 20:52:35.210648 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:35.210623 4611 patch_prober.go:29] interesting pod/e2e-worker-w2-db534b-openshift-default-85df4d79d9-gl9z9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body= Oct 30 20:52:35.210742 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:35.210664 4611 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-gl9z9" podUID="29cef334-f72f-471a-9793-f7907b62e622" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" Oct 30 20:52:35.212004 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:35.211980 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k75ll\" (UniqueName: \"kubernetes.io/projected/7a41f107-a2bb-45e9-a071-265238a96fc7-kube-api-access-k75ll\") pod \"maas-api-revoke-sub-020e7427-mrljk\" (UID: \"7a41f107-a2bb-45e9-a071-265238a96fc7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-020e7427-mrljk" Oct 30 20:52:35.302304 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:35.302260 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:52:35.367531 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:35.351199 4611 patch_prober.go:29] interesting pod/e2e-worker-w5-14cca2-openshift-default-d7f87db59-6pfl5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Oct 30 20:52:35.367531 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:35.351262 4611 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-6pfl5" podUID="06695006-f33e-474a-bfa5-40eb62bd42f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Oct 30 20:52:35.409598 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:35.408608 4611 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-020e7427-mrljk" Oct 30 20:52:35.708591 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:35.705435 4611 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-cwrqt"] Oct 30 20:52:35.717545 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:35.717495 4611 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-tg9jv"] Oct 30 20:52:35.718172 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:35.718151 4611 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-cwrqt" Oct 30 20:52:35.728209 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:35.727807 4611 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-cwrqt"] Oct 30 20:52:35.728209 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:35.727838 4611 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-tg9jv"] Oct 30 20:52:35.728209 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:35.727905 4611 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-tg9jv" Oct 30 20:52:35.814053 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:35.814025 4611 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-020e7427-mrljk"] Oct 30 20:52:35.816295 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:35.816270 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee56d291-6c05-4674-895a-88f339e18e44-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-tg9jv\" (UID: \"ee56d291-6c05-4674-895a-88f339e18e44\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-tg9jv" Oct 30 20:52:35.816387 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:35.816315 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee56d291-6c05-4674-895a-88f339e18e44-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-tg9jv\" (UID: \"ee56d291-6c05-4674-895a-88f339e18e44\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-tg9jv" Oct 30 20:52:35.816430 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:35.816374 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5da54d6d-30b1-4c0c-9824-e06f7a4ae0ee-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-cwrqt\" (UID: \"5da54d6d-30b1-4c0c-9824-e06f7a4ae0ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-cwrqt" Oct 30 20:52:35.816430 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:35.816423 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rxn7c\" (UniqueName: \"kubernetes.io/projected/5da54d6d-30b1-4c0c-9824-e06f7a4ae0ee-kube-api-access-rxn7c\") pod \"maas-default-gateway-openshift-default-6db586dd6c-cwrqt\" (UID: \"5da54d6d-30b1-4c0c-9824-e06f7a4ae0ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-cwrqt" Oct 30 20:52:35.816498 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:35.816458 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5da54d6d-30b1-4c0c-9824-e06f7a4ae0ee-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-cwrqt\" (UID: \"5da54d6d-30b1-4c0c-9824-e06f7a4ae0ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-cwrqt" Oct 30 20:52:35.816498 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:35.816488 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee56d291-6c05-4674-895a-88f339e18e44-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-tg9jv\" (UID: \"ee56d291-6c05-4674-895a-88f339e18e44\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-tg9jv" Oct 30 20:52:35.816590 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:35.816538 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5da54d6d-30b1-4c0c-9824-e06f7a4ae0ee-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-cwrqt\" (UID: \"5da54d6d-30b1-4c0c-9824-e06f7a4ae0ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-cwrqt" Oct 30 20:52:35.816590 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:35.816579 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5da54d6d-30b1-4c0c-9824-e06f7a4ae0ee-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-cwrqt\" (UID: \"5da54d6d-30b1-4c0c-9824-e06f7a4ae0ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-cwrqt" Oct 30 20:52:35.816653 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:35.816610 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee56d291-6c05-4674-895a-88f339e18e44-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-tg9jv\" (UID: \"ee56d291-6c05-4674-895a-88f339e18e44\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-tg9jv" Oct 30 20:52:35.816653 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:35.816645 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5da54d6d-30b1-4c0c-9824-e06f7a4ae0ee-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-cwrqt\" (UID: \"5da54d6d-30b1-4c0c-9824-e06f7a4ae0ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-cwrqt" Oct 30 20:52:35.816715 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:35.816668 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee56d291-6c05-4674-895a-88f339e18e44-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-tg9jv\" (UID: \"ee56d291-6c05-4674-895a-88f339e18e44\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-tg9jv" Oct 30 20:52:35.816715 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:35.816689 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee56d291-6c05-4674-895a-88f339e18e44-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-tg9jv\" (UID: \"ee56d291-6c05-4674-895a-88f339e18e44\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-tg9jv" Oct 30 20:52:35.816715 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:35.816704 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5da54d6d-30b1-4c0c-9824-e06f7a4ae0ee-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-cwrqt\" (UID: \"5da54d6d-30b1-4c0c-9824-e06f7a4ae0ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-cwrqt" Oct 30 20:52:35.816805 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:35.816735 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee56d291-6c05-4674-895a-88f339e18e44-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-tg9jv\" (UID: \"ee56d291-6c05-4674-895a-88f339e18e44\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-tg9jv" Oct 30 20:52:35.816805 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:35.816762 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6xxsv\" (UniqueName: \"kubernetes.io/projected/ee56d291-6c05-4674-895a-88f339e18e44-kube-api-access-6xxsv\") pod \"maas-default-gateway-openshift-default-6db586dd6c-tg9jv\" (UID: \"ee56d291-6c05-4674-895a-88f339e18e44\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-tg9jv" Oct 30 20:52:35.816805 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:35.816778 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5da54d6d-30b1-4c0c-9824-e06f7a4ae0ee-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-cwrqt\" (UID: \"5da54d6d-30b1-4c0c-9824-e06f7a4ae0ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-cwrqt" Oct 30 20:52:35.816889 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:35.816805 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5da54d6d-30b1-4c0c-9824-e06f7a4ae0ee-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-cwrqt\" (UID: \"5da54d6d-30b1-4c0c-9824-e06f7a4ae0ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-cwrqt" Oct 30 20:52:35.816889 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:35.816826 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee56d291-6c05-4674-895a-88f339e18e44-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-tg9jv\" (UID: \"ee56d291-6c05-4674-895a-88f339e18e44\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-tg9jv" Oct 30 20:52:35.901866 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:35.901550 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-020e7427-mrljk" event={"ID":"7a41f107-a2bb-45e9-a071-265238a96fc7","Type":"ContainerStarted","Data":"552ca8eaf25d175217953c4db98e42a8024139e6df2b2c5664c498176e07535d"} Oct 30 20:52:35.917453 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:35.917418 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5da54d6d-30b1-4c0c-9824-e06f7a4ae0ee-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-cwrqt\" (UID: \"5da54d6d-30b1-4c0c-9824-e06f7a4ae0ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-cwrqt" Oct 30 20:52:35.917600 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:35.917458 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rxn7c\" (UniqueName: \"kubernetes.io/projected/5da54d6d-30b1-4c0c-9824-e06f7a4ae0ee-kube-api-access-rxn7c\") pod \"maas-default-gateway-openshift-default-6db586dd6c-cwrqt\" (UID: \"5da54d6d-30b1-4c0c-9824-e06f7a4ae0ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-cwrqt" Oct 30 20:52:35.917600 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:35.917523 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5da54d6d-30b1-4c0c-9824-e06f7a4ae0ee-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-cwrqt\" (UID: \"5da54d6d-30b1-4c0c-9824-e06f7a4ae0ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-cwrqt" Oct 30 20:52:35.917737 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:35.917612 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee56d291-6c05-4674-895a-88f339e18e44-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-tg9jv\" (UID: \"ee56d291-6c05-4674-895a-88f339e18e44\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-tg9jv" Oct 30 20:52:35.917737 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:35.917642 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5da54d6d-30b1-4c0c-9824-e06f7a4ae0ee-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-cwrqt\" (UID: \"5da54d6d-30b1-4c0c-9824-e06f7a4ae0ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-cwrqt" Oct 30 20:52:35.917737 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:35.917717 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5da54d6d-30b1-4c0c-9824-e06f7a4ae0ee-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-cwrqt\" (UID: \"5da54d6d-30b1-4c0c-9824-e06f7a4ae0ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-cwrqt" Oct 30 20:52:35.917897 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:35.917785 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee56d291-6c05-4674-895a-88f339e18e44-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-tg9jv\" (UID: \"ee56d291-6c05-4674-895a-88f339e18e44\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-tg9jv" Oct 30 20:52:35.917897 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:35.917848 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5da54d6d-30b1-4c0c-9824-e06f7a4ae0ee-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-cwrqt\" (UID: \"5da54d6d-30b1-4c0c-9824-e06f7a4ae0ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-cwrqt" Oct 30 20:52:35.917997 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:35.917904 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee56d291-6c05-4674-895a-88f339e18e44-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-tg9jv\" (UID: \"ee56d291-6c05-4674-895a-88f339e18e44\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-tg9jv" Oct 30 20:52:35.917997 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:35.917940 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee56d291-6c05-4674-895a-88f339e18e44-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-tg9jv\" (UID: \"ee56d291-6c05-4674-895a-88f339e18e44\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-tg9jv" Oct 30 20:52:35.917997 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:35.917957 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5da54d6d-30b1-4c0c-9824-e06f7a4ae0ee-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-cwrqt\" (UID: \"5da54d6d-30b1-4c0c-9824-e06f7a4ae0ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-cwrqt" Oct 30 20:52:35.917997 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:35.917963 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5da54d6d-30b1-4c0c-9824-e06f7a4ae0ee-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-cwrqt\" (UID: \"5da54d6d-30b1-4c0c-9824-e06f7a4ae0ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-cwrqt" Oct 30 20:52:35.918197 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:35.918057 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee56d291-6c05-4674-895a-88f339e18e44-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-tg9jv\" (UID: \"ee56d291-6c05-4674-895a-88f339e18e44\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-tg9jv" Oct 30 20:52:35.918197 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:35.918113 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xxsv\" (UniqueName: \"kubernetes.io/projected/ee56d291-6c05-4674-895a-88f339e18e44-kube-api-access-6xxsv\") pod \"maas-default-gateway-openshift-default-6db586dd6c-tg9jv\" (UID: \"ee56d291-6c05-4674-895a-88f339e18e44\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-tg9jv" Oct 30 20:52:35.918197 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:35.918132 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5da54d6d-30b1-4c0c-9824-e06f7a4ae0ee-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-cwrqt\" (UID: \"5da54d6d-30b1-4c0c-9824-e06f7a4ae0ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-cwrqt" Oct 30 20:52:35.918197 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:35.918170 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5da54d6d-30b1-4c0c-9824-e06f7a4ae0ee-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-cwrqt\" (UID: \"5da54d6d-30b1-4c0c-9824-e06f7a4ae0ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-cwrqt" Oct 30 20:52:35.918197 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:35.918192 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee56d291-6c05-4674-895a-88f339e18e44-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-tg9jv\" (UID: \"ee56d291-6c05-4674-895a-88f339e18e44\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-tg9jv" Oct 30 20:52:35.918454 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:35.918212 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee56d291-6c05-4674-895a-88f339e18e44-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-tg9jv\" (UID: \"ee56d291-6c05-4674-895a-88f339e18e44\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-tg9jv" Oct 30 20:52:35.918454 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:35.918242 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee56d291-6c05-4674-895a-88f339e18e44-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-tg9jv\" (UID: \"ee56d291-6c05-4674-895a-88f339e18e44\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-tg9jv" Oct 30 20:52:35.918454 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:35.918266 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5da54d6d-30b1-4c0c-9824-e06f7a4ae0ee-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-cwrqt\" (UID: \"5da54d6d-30b1-4c0c-9824-e06f7a4ae0ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-cwrqt" Oct 30 20:52:35.918454 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:35.918354 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee56d291-6c05-4674-895a-88f339e18e44-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-tg9jv\" (UID: \"ee56d291-6c05-4674-895a-88f339e18e44\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-tg9jv" Oct 30 20:52:35.918687 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:35.918554 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee56d291-6c05-4674-895a-88f339e18e44-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-tg9jv\" (UID: \"ee56d291-6c05-4674-895a-88f339e18e44\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-tg9jv" Oct 30 20:52:35.918863 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:35.918843 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee56d291-6c05-4674-895a-88f339e18e44-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-tg9jv\" (UID: \"ee56d291-6c05-4674-895a-88f339e18e44\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-tg9jv" Oct 30 20:52:35.919148 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:35.919109 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee56d291-6c05-4674-895a-88f339e18e44-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-tg9jv\" (UID: \"ee56d291-6c05-4674-895a-88f339e18e44\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-tg9jv" Oct 30 20:52:35.919444 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:35.919424 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5da54d6d-30b1-4c0c-9824-e06f7a4ae0ee-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-cwrqt\" (UID: \"5da54d6d-30b1-4c0c-9824-e06f7a4ae0ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-cwrqt" Oct 30 20:52:35.919753 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:35.919729 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee56d291-6c05-4674-895a-88f339e18e44-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-tg9jv\" (UID: \"ee56d291-6c05-4674-895a-88f339e18e44\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-tg9jv" Oct 30 20:52:35.919930 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:35.919910 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5da54d6d-30b1-4c0c-9824-e06f7a4ae0ee-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-cwrqt\" (UID: \"5da54d6d-30b1-4c0c-9824-e06f7a4ae0ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-cwrqt" Oct 30 20:52:35.920589 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:35.920345 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5da54d6d-30b1-4c0c-9824-e06f7a4ae0ee-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-cwrqt\" (UID: \"5da54d6d-30b1-4c0c-9824-e06f7a4ae0ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-cwrqt" Oct 30 20:52:35.920589 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:35.920409 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5da54d6d-30b1-4c0c-9824-e06f7a4ae0ee-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-cwrqt\" (UID: \"5da54d6d-30b1-4c0c-9824-e06f7a4ae0ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-cwrqt" Oct 30 20:52:35.921066 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:35.921047 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee56d291-6c05-4674-895a-88f339e18e44-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-tg9jv\" (UID: \"ee56d291-6c05-4674-895a-88f339e18e44\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-tg9jv" Oct 30 20:52:35.922116 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:35.922095 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5da54d6d-30b1-4c0c-9824-e06f7a4ae0ee-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-cwrqt\" (UID: \"5da54d6d-30b1-4c0c-9824-e06f7a4ae0ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-cwrqt" Oct 30 20:52:35.922989 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:35.922969 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee56d291-6c05-4674-895a-88f339e18e44-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-tg9jv\" (UID: \"ee56d291-6c05-4674-895a-88f339e18e44\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-tg9jv" Oct 30 20:52:35.928878 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:35.928856 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rxn7c\" (UniqueName: \"kubernetes.io/projected/5da54d6d-30b1-4c0c-9824-e06f7a4ae0ee-kube-api-access-rxn7c\") pod \"maas-default-gateway-openshift-default-6db586dd6c-cwrqt\" (UID: \"5da54d6d-30b1-4c0c-9824-e06f7a4ae0ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-cwrqt" Oct 30 20:52:35.930056 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:35.929933 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6xxsv\" (UniqueName: \"kubernetes.io/projected/ee56d291-6c05-4674-895a-88f339e18e44-kube-api-access-6xxsv\") pod \"maas-default-gateway-openshift-default-6db586dd6c-tg9jv\" (UID: \"ee56d291-6c05-4674-895a-88f339e18e44\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-tg9jv" Oct 30 20:52:35.930056 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:35.930018 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5da54d6d-30b1-4c0c-9824-e06f7a4ae0ee-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-cwrqt\" (UID: \"5da54d6d-30b1-4c0c-9824-e06f7a4ae0ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-cwrqt" Oct 30 20:52:35.930719 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:35.930696 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee56d291-6c05-4674-895a-88f339e18e44-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-tg9jv\" (UID: \"ee56d291-6c05-4674-895a-88f339e18e44\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-tg9jv" Oct 30 20:52:36.044483 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:36.044403 4611 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-cwrqt" Oct 30 20:52:36.104275 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:36.104245 4611 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-tg9jv" Oct 30 20:52:36.104492 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:36.104465 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:52:36.215081 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:36.215044 4611 patch_prober.go:29] interesting pod/e2e-worker-w2-db534b-openshift-default-85df4d79d9-gl9z9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body= Oct 30 20:52:36.215218 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:36.215104 4611 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-gl9z9" podUID="29cef334-f72f-471a-9793-f7907b62e622" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" Oct 30 20:52:36.371346 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:36.370203 4611 patch_prober.go:29] interesting pod/e2e-worker-w5-14cca2-openshift-default-d7f87db59-6pfl5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Oct 30 20:52:36.371346 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:36.370269 4611 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-6pfl5" podUID="06695006-f33e-474a-bfa5-40eb62bd42f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Oct 30 20:52:36.506101 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:36.504169 4611 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:52:36.506101 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:36.504268 4611 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:52:36.506101 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:36.504310 4611 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:52:36.512006 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:36.511767 4611 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-cwrqt"] Oct 30 20:52:36.534310 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:36.530425 4611 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-tg9jv"] Oct 30 20:52:36.537539 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:36.537469 4611 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:52:36.537619 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:36.537575 4611 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:52:36.537619 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:36.537607 4611 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:52:36.818574 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:36.809997 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:52:36.942178 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:36.936775 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-020e7427-mrljk" event={"ID":"7a41f107-a2bb-45e9-a071-265238a96fc7","Type":"ContainerStarted","Data":"5a112870a7bd0d7ec22f2e6ac21b8fec865daabbcbd37215a94c46c0242807a7"} Oct 30 20:52:36.942178 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:36.938429 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-cwrqt" event={"ID":"5da54d6d-30b1-4c0c-9824-e06f7a4ae0ee","Type":"ContainerStarted","Data":"9c714c781d32fade35666567542cb168264cfee21b36c83a7f19bf26970f441b"} Oct 30 20:52:36.942178 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:36.938475 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-cwrqt" event={"ID":"5da54d6d-30b1-4c0c-9824-e06f7a4ae0ee","Type":"ContainerStarted","Data":"fdbf7de60cf0ba663da5d4339c441692ff5dba593481f701014d1efd8f9776ca"} Oct 30 20:52:36.942178 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:36.941551 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-tg9jv" event={"ID":"ee56d291-6c05-4674-895a-88f339e18e44","Type":"ContainerStarted","Data":"289f6b0c2f22dcd6e6d93f4870c65c6516738680641d94232273824ef00a4158"} Oct 30 20:52:36.942178 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:36.941593 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-tg9jv" event={"ID":"ee56d291-6c05-4674-895a-88f339e18e44","Type":"ContainerStarted","Data":"3119a90975846ee6602ff878b35a462eedb94cb1507167a3f57921f1741c9b77"} Oct 30 20:52:37.014063 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:37.013981 4611 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-cwrqt" podStartSLOduration=2.013962716 podStartE2EDuration="2.013962716s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:52:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:52:37.000194433 +0000 UTC m=+811.534221086" watchObservedRunningTime="2026-10-30 20:52:37.013962716 +0000 UTC m=+811.547989336" Oct 30 20:52:37.037756 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:37.037706 4611 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-tg9jv" podStartSLOduration=2.037688406 podStartE2EDuration="2.037688406s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:52:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:52:37.033962131 +0000 UTC m=+811.567988747" watchObservedRunningTime="2026-10-30 20:52:37.037688406 +0000 UTC m=+811.571715023" Oct 30 20:52:37.054530 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:37.047307 4611 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-cwrqt" Oct 30 20:52:37.054530 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:37.047455 4611 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-cwrqt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Oct 30 20:52:37.054530 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:37.047500 4611 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-cwrqt" podUID="5da54d6d-30b1-4c0c-9824-e06f7a4ae0ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Oct 30 20:52:37.113688 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:37.113608 4611 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-tg9jv" Oct 30 20:52:37.114005 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:37.113980 4611 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-tg9jv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Oct 30 20:52:37.114179 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:37.114143 4611 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-tg9jv" podUID="ee56d291-6c05-4674-895a-88f339e18e44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Oct 30 20:52:37.218581 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:37.218096 4611 patch_prober.go:29] interesting pod/e2e-worker-w2-db534b-openshift-default-85df4d79d9-gl9z9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body= Oct 30 20:52:37.218581 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:37.218154 4611 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-gl9z9" podUID="29cef334-f72f-471a-9793-f7907b62e622" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" Oct 30 20:52:37.356536 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:37.354846 4611 patch_prober.go:29] interesting pod/e2e-worker-w5-14cca2-openshift-default-d7f87db59-6pfl5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Oct 30 20:52:37.356536 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:37.354908 4611 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-6pfl5" podUID="06695006-f33e-474a-bfa5-40eb62bd42f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Oct 30 20:52:37.702563 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:37.691541 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:52:38.020629 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:38.016316 4611 generic.go:356] "Generic (PLEG): container finished" podID="7a41f107-a2bb-45e9-a071-265238a96fc7" containerID="5a112870a7bd0d7ec22f2e6ac21b8fec865daabbcbd37215a94c46c0242807a7" exitCode=0 Oct 30 20:52:38.020629 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:38.017467 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-020e7427-mrljk" event={"ID":"7a41f107-a2bb-45e9-a071-265238a96fc7","Type":"ContainerDied","Data":"5a112870a7bd0d7ec22f2e6ac21b8fec865daabbcbd37215a94c46c0242807a7"} Oct 30 20:52:38.054010 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:38.053973 4611 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-cwrqt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Oct 30 20:52:38.054271 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:38.054240 4611 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-cwrqt" podUID="5da54d6d-30b1-4c0c-9824-e06f7a4ae0ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Oct 30 20:52:38.115535 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:38.104922 4611 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-tg9jv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Oct 30 20:52:38.115535 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:38.104988 4611 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-tg9jv" podUID="ee56d291-6c05-4674-895a-88f339e18e44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Oct 30 20:52:38.237684 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:38.237641 4611 patch_prober.go:29] interesting pod/e2e-worker-w2-db534b-openshift-default-85df4d79d9-gl9z9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body= Oct 30 20:52:38.237988 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:38.237955 4611 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-gl9z9" podUID="29cef334-f72f-471a-9793-f7907b62e622" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" Oct 30 20:52:38.352005 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:38.351966 4611 patch_prober.go:29] interesting pod/e2e-worker-w5-14cca2-openshift-default-d7f87db59-6pfl5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Oct 30 20:52:38.352179 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:38.352044 4611 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-6pfl5" podUID="06695006-f33e-474a-bfa5-40eb62bd42f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Oct 30 20:52:38.706726 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:38.706623 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-14cca2/distinct-w5-14cca2-kserve-587f6996b8-sgf6h" podUID="02b652c3-0724-4a50-9e95-2a9890cc97c4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:52:38.890866 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:38.890814 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cb60a2/distinct-w3-cb60a2-kserve-5bb76dc74-86fds" podUID="52e301d9-f378-4e54-9515-01a6680d9fd7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded" Oct 30 20:52:38.933599 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:38.908845 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad7ff0/premium-w0-ad7ff0-kserve-5485d4b948-bpmgf" podUID="53cbd692-e9f8-46ba-8fde-3e5215a38904" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:52:39.004533 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:38.980776 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cb60a2/simulator-w3-cb60a2-kserve-6fd74b9d8b-vqfzf" podUID="711b167d-4e66-40d6-be75-33ede11f42c5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 20:52:39.210545 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:39.175580 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-3d909d/premium-w6-3d909d-kserve-765648946f-2qvd2" podUID="fca5d697-109d-4e90-ab41-c40ad31af874" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:52:39.210545 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:39.176199 4611 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-3d909d/premium-w6-3d909d-kserve-765648946f-2qvd2" podUID="fca5d697-109d-4e90-ab41-c40ad31af874" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:52:39.210545 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:39.176314 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b8aee1/premium-w1-b8aee1-kserve-786d575599-htc8d" podUID="77368afe-edc8-4fa5-8010-d321965fb38e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded" Oct 30 20:52:39.210545 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:39.177060 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-14cca2/premium-w5-14cca2-kserve-6878777857-wpjtj" podUID="b99eb76b-d1c6-49e9-9809-c351c0f5a4de" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:52:39.210545 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:39.209238 4611 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-cwrqt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Oct 30 20:52:39.210545 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:39.209307 4611 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-cwrqt" podUID="5da54d6d-30b1-4c0c-9824-e06f7a4ae0ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Oct 30 20:52:39.214577 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:39.212749 4611 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-tg9jv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Oct 30 20:52:39.214577 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:39.212806 4611 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-tg9jv" podUID="ee56d291-6c05-4674-895a-88f339e18e44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Oct 30 20:52:39.240546 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:39.239082 4611 patch_prober.go:29] interesting pod/e2e-worker-w2-db534b-openshift-default-85df4d79d9-gl9z9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body= Oct 30 20:52:39.240546 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:39.239131 4611 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-gl9z9" podUID="29cef334-f72f-471a-9793-f7907b62e622" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" Oct 30 20:52:39.309532 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:39.298373 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:52:39.378534 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:39.376107 4611 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-020e7427-mrljk" Oct 30 20:52:39.378534 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:39.376986 4611 patch_prober.go:29] interesting pod/e2e-worker-w5-14cca2-openshift-default-d7f87db59-6pfl5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Oct 30 20:52:39.378534 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:39.377036 4611 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-6pfl5" podUID="06695006-f33e-474a-bfa5-40eb62bd42f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Oct 30 20:52:39.390533 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:39.388477 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a41f107-a2bb-45e9-a071-265238a96fc7-kube-api-access-k75ll\" (UniqueName: \"kubernetes.io/projected/7a41f107-a2bb-45e9-a071-265238a96fc7-kube-api-access-k75ll\") pod \"7a41f107-a2bb-45e9-a071-265238a96fc7\" (UID: \"7a41f107-a2bb-45e9-a071-265238a96fc7\") " Oct 30 20:52:39.390533 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:39.388579 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7a41f107-a2bb-45e9-a071-265238a96fc7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a41f107-a2bb-45e9-a071-265238a96fc7-maas-api-openshift-ca\") pod \"7a41f107-a2bb-45e9-a071-265238a96fc7\" (UID: \"7a41f107-a2bb-45e9-a071-265238a96fc7\") " Oct 30 20:52:39.390533 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:39.388782 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7a41f107-a2bb-45e9-a071-265238a96fc7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a41f107-a2bb-45e9-a071-265238a96fc7-maas-api-xks-ca\") pod \"7a41f107-a2bb-45e9-a071-265238a96fc7\" (UID: \"7a41f107-a2bb-45e9-a071-265238a96fc7\") " Oct 30 20:52:39.394415 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:39.393722 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a41f107-a2bb-45e9-a071-265238a96fc7-maas-api-openshift-ca" pod "7a41f107-a2bb-45e9-a071-265238a96fc7" (UID: "7a41f107-a2bb-45e9-a071-265238a96fc7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:52:39.404585 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:39.404553 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-020e7427-mrljk" event={"ID":"7a41f107-a2bb-45e9-a071-265238a96fc7","Type":"ContainerDied","Data":"552ca8eaf25d175217953c4db98e42a8024139e6df2b2c5664c498176e07535d"} Oct 30 20:52:39.404832 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:39.404812 4611 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="552ca8eaf25d175217953c4db98e42a8024139e6df2b2c5664c498176e07535d" Oct 30 20:52:39.487783 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:39.487690 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a41f107-a2bb-45e9-a071-265238a96fc7-kube-api-access-k75ll" pod "7a41f107-a2bb-45e9-a071-265238a96fc7" (UID: "7a41f107-a2bb-45e9-a071-265238a96fc7"). InnerVolumeSpecName "kube-api-access-k75ll". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:52:39.574191 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:39.488034 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7a41f107-a2bb-45e9-a071-265238a96fc7-maas-api-xks-ca" pod "7a41f107-a2bb-45e9-a071-265238a96fc7" (UID: "7a41f107-a2bb-45e9-a071-265238a96fc7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:52:39.678537 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:39.659068 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:52:39.743590 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:39.683321 4611 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a41f107-a2bb-45e9-a071-265238a96fc7-maas-api-xks-ca\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:52:39.745929 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:39.745903 4611 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k75ll\" (UniqueName: \"kubernetes.io/projected/7a41f107-a2bb-45e9-a071-265238a96fc7-kube-api-access-k75ll\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:52:39.746131 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:39.746116 4611 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a41f107-a2bb-45e9-a071-265238a96fc7-maas-api-openshift-ca\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:52:39.776570 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:39.745268 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-14cca2/distinct-w5-14cca2-kserve-587f6996b8-sgf6h" podUID="02b652c3-0724-4a50-9e95-2a9890cc97c4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:52:39.887943 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:39.869786 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-14cca2/distinct-w5-14cca2-kserve-587f6996b8-sgf6h" Oct 30 20:52:39.968642 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:39.968590 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad7ff0/premium-w0-ad7ff0-kserve-5485d4b948-bpmgf" podUID="53cbd692-e9f8-46ba-8fde-3e5215a38904" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:52:39.969041 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:39.969016 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ad7ff0/premium-w0-ad7ff0-kserve-5485d4b948-bpmgf" Oct 30 20:52:40.000647 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:39.979475 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cb60a2/distinct-w3-cb60a2-kserve-5bb76dc74-86fds" podUID="52e301d9-f378-4e54-9515-01a6680d9fd7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:52:40.005453 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:40.000973 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-cb60a2/distinct-w3-cb60a2-kserve-5bb76dc74-86fds" Oct 30 20:52:40.123039 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:40.093217 4611 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-cwrqt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Oct 30 20:52:40.123039 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:40.121469 4611 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-cwrqt" podUID="5da54d6d-30b1-4c0c-9824-e06f7a4ae0ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Oct 30 20:52:40.132542 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:40.132269 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ad7ff0/premium-w0-ad7ff0-kserve-5485d4b948-bpmgf" Oct 30 20:52:40.164531 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:40.133291 4611 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-tg9jv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Oct 30 20:52:40.164531 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:40.133335 4611 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-tg9jv" podUID="ee56d291-6c05-4674-895a-88f339e18e44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Oct 30 20:52:40.237571 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:40.237531 4611 patch_prober.go:29] interesting pod/e2e-worker-w2-db534b-openshift-default-85df4d79d9-gl9z9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body= Oct 30 20:52:40.241111 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:40.241067 4611 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-gl9z9" podUID="29cef334-f72f-471a-9793-f7907b62e622" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" Oct 30 20:52:40.249662 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:40.249640 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-cb60a2/distinct-w3-cb60a2-kserve-5bb76dc74-86fds" Oct 30 20:52:40.278825 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:40.278740 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-14cca2/distinct-w5-14cca2-kserve-587f6996b8-sgf6h" Oct 30 20:52:40.405536 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:40.403227 4611 patch_prober.go:29] interesting pod/e2e-worker-w5-14cca2-openshift-default-d7f87db59-6pfl5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Oct 30 20:52:40.405536 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:40.403286 4611 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-6pfl5" podUID="06695006-f33e-474a-bfa5-40eb62bd42f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Oct 30 20:52:40.457563 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:40.446703 4611 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-020e7427-mrljk" Oct 30 20:52:40.457563 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:40.447322 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:52:41.059533 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:41.059084 4611 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-cwrqt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Oct 30 20:52:41.059533 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:41.059161 4611 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-cwrqt" podUID="5da54d6d-30b1-4c0c-9824-e06f7a4ae0ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Oct 30 20:52:41.178546 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:41.149915 4611 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-tg9jv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Oct 30 20:52:41.179103 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:41.179063 4611 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-tg9jv" podUID="ee56d291-6c05-4674-895a-88f339e18e44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Oct 30 20:52:41.332583 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:41.311952 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:52:41.333668 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:41.333359 4611 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-gl9z9" Oct 30 20:52:41.359534 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:41.334896 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-gl9z9" Oct 30 20:52:41.376542 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:41.372765 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-gl9z9" Oct 30 20:52:41.675529 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:41.667837 4611 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-6pfl5" Oct 30 20:52:41.675529 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:41.669283 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-6pfl5" Oct 30 20:52:41.700865 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:41.700836 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-6pfl5" Oct 30 20:52:42.045033 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:42.044952 4611 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-cwrqt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Oct 30 20:52:42.045033 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:42.045003 4611 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-cwrqt" podUID="5da54d6d-30b1-4c0c-9824-e06f7a4ae0ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Oct 30 20:52:42.105175 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:42.105140 4611 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-tg9jv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Oct 30 20:52:42.105547 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:42.105197 4611 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-tg9jv" podUID="ee56d291-6c05-4674-895a-88f339e18e44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Oct 30 20:52:42.126021 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:42.125991 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:52:42.909902 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:42.909871 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:52:43.045411 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:43.045376 4611 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-cwrqt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Oct 30 20:52:43.045635 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:43.045436 4611 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-cwrqt" podUID="5da54d6d-30b1-4c0c-9824-e06f7a4ae0ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Oct 30 20:52:43.105144 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:43.105109 4611 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-tg9jv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Oct 30 20:52:43.105297 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:43.105158 4611 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-tg9jv" podUID="ee56d291-6c05-4674-895a-88f339e18e44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Oct 30 20:52:43.694752 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:43.694710 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:52:44.045853 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:44.045774 4611 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-cwrqt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Oct 30 20:52:44.045853 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:44.045839 4611 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-cwrqt" podUID="5da54d6d-30b1-4c0c-9824-e06f7a4ae0ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Oct 30 20:52:44.105056 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:44.105023 4611 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-tg9jv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Oct 30 20:52:44.105234 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:44.105083 4611 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-tg9jv" podUID="ee56d291-6c05-4674-895a-88f339e18e44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Oct 30 20:52:44.387416 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:44.387369 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:52:45.042295 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:45.042258 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:52:45.045126 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:45.045096 4611 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-cwrqt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Oct 30 20:52:45.045271 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:45.045148 4611 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-cwrqt" podUID="5da54d6d-30b1-4c0c-9824-e06f7a4ae0ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Oct 30 20:52:45.105085 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:45.105049 4611 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-tg9jv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Oct 30 20:52:45.105370 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:45.105327 4611 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-tg9jv" podUID="ee56d291-6c05-4674-895a-88f339e18e44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Oct 30 20:52:45.261153 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:45.261119 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:52:46.100550 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:46.100497 4611 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-cwrqt" Oct 30 20:52:46.101760 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:46.101738 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-cwrqt" Oct 30 20:52:46.104195 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:46.104172 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-cwrqt" Oct 30 20:52:46.138237 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:46.138204 4611 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-tg9jv" Oct 30 20:52:46.138818 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:46.138740 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-tg9jv" Oct 30 20:52:46.139980 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:46.139959 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-tg9jv" Oct 30 20:52:46.151587 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:46.151545 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:52:46.386085 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:46.386051 4611 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-qvfk5"] Oct 30 20:52:46.386827 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:46.386808 4611 cpu_manager.go:414] "containerMap: removing container" podUID="7a41f107-a2bb-45e9-a071-265238a96fc7" containerName="revoke-keys" Oct 30 20:52:46.386893 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:46.386830 4611 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a41f107-a2bb-45e9-a071-265238a96fc7" containerName="revoke-keys" Oct 30 20:52:46.387006 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:46.386994 4611 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a41f107-a2bb-45e9-a071-265238a96fc7" containerName="revoke-keys" Oct 30 20:52:46.392030 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:46.392010 4611 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-qvfk5" Oct 30 20:52:46.399728 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:46.399699 4611 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-qvfk5"] Oct 30 20:52:46.492827 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:46.492797 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/527a4fc6-4bfe-49c5-9352-d5b82a74e7ff-workload-certs\") pod \"e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-qvfk5\" (UID: \"527a4fc6-4bfe-49c5-9352-d5b82a74e7ff\") " pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-qvfk5" Oct 30 20:52:46.492997 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:46.492946 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/527a4fc6-4bfe-49c5-9352-d5b82a74e7ff-istio-data\") pod \"e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-qvfk5\" (UID: \"527a4fc6-4bfe-49c5-9352-d5b82a74e7ff\") " pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-qvfk5" Oct 30 20:52:46.492997 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:46.492983 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/527a4fc6-4bfe-49c5-9352-d5b82a74e7ff-istio-token\") pod \"e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-qvfk5\" (UID: \"527a4fc6-4bfe-49c5-9352-d5b82a74e7ff\") " pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-qvfk5" Oct 30 20:52:46.493088 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:46.493021 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/527a4fc6-4bfe-49c5-9352-d5b82a74e7ff-istio-envoy\") pod \"e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-qvfk5\" (UID: \"527a4fc6-4bfe-49c5-9352-d5b82a74e7ff\") " pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-qvfk5" Oct 30 20:52:46.493172 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:46.493147 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/527a4fc6-4bfe-49c5-9352-d5b82a74e7ff-workload-socket\") pod \"e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-qvfk5\" (UID: \"527a4fc6-4bfe-49c5-9352-d5b82a74e7ff\") " pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-qvfk5" Oct 30 20:52:46.493214 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:46.493199 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5smv6\" (UniqueName: \"kubernetes.io/projected/527a4fc6-4bfe-49c5-9352-d5b82a74e7ff-kube-api-access-5smv6\") pod \"e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-qvfk5\" (UID: \"527a4fc6-4bfe-49c5-9352-d5b82a74e7ff\") " pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-qvfk5" Oct 30 20:52:46.493269 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:46.493253 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/527a4fc6-4bfe-49c5-9352-d5b82a74e7ff-istio-podinfo\") pod \"e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-qvfk5\" (UID: \"527a4fc6-4bfe-49c5-9352-d5b82a74e7ff\") " pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-qvfk5" Oct 30 20:52:46.493312 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:46.493300 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/527a4fc6-4bfe-49c5-9352-d5b82a74e7ff-istiod-ca-cert\") pod \"e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-qvfk5\" (UID: \"527a4fc6-4bfe-49c5-9352-d5b82a74e7ff\") " pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-qvfk5" Oct 30 20:52:46.493350 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:46.493339 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/527a4fc6-4bfe-49c5-9352-d5b82a74e7ff-credential-socket\") pod \"e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-qvfk5\" (UID: \"527a4fc6-4bfe-49c5-9352-d5b82a74e7ff\") " pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-qvfk5" Oct 30 20:52:46.594818 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:46.594791 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/527a4fc6-4bfe-49c5-9352-d5b82a74e7ff-istio-data\") pod \"e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-qvfk5\" (UID: \"527a4fc6-4bfe-49c5-9352-d5b82a74e7ff\") " pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-qvfk5" Oct 30 20:52:46.594972 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:46.594828 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/527a4fc6-4bfe-49c5-9352-d5b82a74e7ff-istio-token\") pod \"e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-qvfk5\" (UID: \"527a4fc6-4bfe-49c5-9352-d5b82a74e7ff\") " pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-qvfk5" Oct 30 20:52:46.594972 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:46.594859 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/527a4fc6-4bfe-49c5-9352-d5b82a74e7ff-istio-envoy\") pod \"e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-qvfk5\" (UID: \"527a4fc6-4bfe-49c5-9352-d5b82a74e7ff\") " pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-qvfk5" Oct 30 20:52:46.594972 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:46.594894 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/527a4fc6-4bfe-49c5-9352-d5b82a74e7ff-workload-socket\") pod \"e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-qvfk5\" (UID: \"527a4fc6-4bfe-49c5-9352-d5b82a74e7ff\") " pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-qvfk5" Oct 30 20:52:46.594972 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:46.594921 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5smv6\" (UniqueName: \"kubernetes.io/projected/527a4fc6-4bfe-49c5-9352-d5b82a74e7ff-kube-api-access-5smv6\") pod \"e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-qvfk5\" (UID: \"527a4fc6-4bfe-49c5-9352-d5b82a74e7ff\") " pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-qvfk5" Oct 30 20:52:46.594972 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:46.594944 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/527a4fc6-4bfe-49c5-9352-d5b82a74e7ff-istio-podinfo\") pod \"e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-qvfk5\" (UID: \"527a4fc6-4bfe-49c5-9352-d5b82a74e7ff\") " pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-qvfk5" Oct 30 20:52:46.594972 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:46.594968 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/527a4fc6-4bfe-49c5-9352-d5b82a74e7ff-istiod-ca-cert\") pod \"e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-qvfk5\" (UID: \"527a4fc6-4bfe-49c5-9352-d5b82a74e7ff\") " pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-qvfk5" Oct 30 20:52:46.595273 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:46.595022 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/527a4fc6-4bfe-49c5-9352-d5b82a74e7ff-credential-socket\") pod \"e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-qvfk5\" (UID: \"527a4fc6-4bfe-49c5-9352-d5b82a74e7ff\") " pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-qvfk5" Oct 30 20:52:46.595273 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:46.595054 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/527a4fc6-4bfe-49c5-9352-d5b82a74e7ff-workload-certs\") pod \"e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-qvfk5\" (UID: \"527a4fc6-4bfe-49c5-9352-d5b82a74e7ff\") " pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-qvfk5" Oct 30 20:52:46.595384 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:46.595286 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/527a4fc6-4bfe-49c5-9352-d5b82a74e7ff-istio-data\") pod \"e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-qvfk5\" (UID: \"527a4fc6-4bfe-49c5-9352-d5b82a74e7ff\") " pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-qvfk5" Oct 30 20:52:46.595384 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:46.595364 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/527a4fc6-4bfe-49c5-9352-d5b82a74e7ff-workload-socket\") pod \"e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-qvfk5\" (UID: \"527a4fc6-4bfe-49c5-9352-d5b82a74e7ff\") " pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-qvfk5" Oct 30 20:52:46.595499 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:46.595433 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/527a4fc6-4bfe-49c5-9352-d5b82a74e7ff-credential-socket\") pod \"e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-qvfk5\" (UID: \"527a4fc6-4bfe-49c5-9352-d5b82a74e7ff\") " pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-qvfk5" Oct 30 20:52:46.595590 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:46.595573 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/527a4fc6-4bfe-49c5-9352-d5b82a74e7ff-workload-certs\") pod \"e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-qvfk5\" (UID: \"527a4fc6-4bfe-49c5-9352-d5b82a74e7ff\") " pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-qvfk5" Oct 30 20:52:46.595889 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:46.595864 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/527a4fc6-4bfe-49c5-9352-d5b82a74e7ff-istiod-ca-cert\") pod \"e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-qvfk5\" (UID: \"527a4fc6-4bfe-49c5-9352-d5b82a74e7ff\") " pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-qvfk5" Oct 30 20:52:46.597185 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:46.597167 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/527a4fc6-4bfe-49c5-9352-d5b82a74e7ff-istio-envoy\") pod \"e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-qvfk5\" (UID: \"527a4fc6-4bfe-49c5-9352-d5b82a74e7ff\") " pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-qvfk5" Oct 30 20:52:46.597393 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:46.597373 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/527a4fc6-4bfe-49c5-9352-d5b82a74e7ff-istio-podinfo\") pod \"e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-qvfk5\" (UID: \"527a4fc6-4bfe-49c5-9352-d5b82a74e7ff\") " pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-qvfk5" Oct 30 20:52:46.605829 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:46.605802 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/527a4fc6-4bfe-49c5-9352-d5b82a74e7ff-istio-token\") pod \"e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-qvfk5\" (UID: \"527a4fc6-4bfe-49c5-9352-d5b82a74e7ff\") " pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-qvfk5" Oct 30 20:52:46.606134 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:46.606117 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5smv6\" (UniqueName: \"kubernetes.io/projected/527a4fc6-4bfe-49c5-9352-d5b82a74e7ff-kube-api-access-5smv6\") pod \"e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-qvfk5\" (UID: \"527a4fc6-4bfe-49c5-9352-d5b82a74e7ff\") " pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-qvfk5" Oct 30 20:52:46.713033 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:46.712973 4611 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-qvfk5" Oct 30 20:52:46.987744 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:46.987650 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:52:47.042623 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:47.042590 4611 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-qvfk5"] Oct 30 20:52:47.049588 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:52:47.049495 4611 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod527a4fc6_4bfe_49c5_9352_d5b82a74e7ff.slice/crio-d1397b503ba4140455a1da1c7696506684d9fc3d6fcdfb2f8b09e7fa9c3b6c80 WatchSource:0}: Error finding container d1397b503ba4140455a1da1c7696506684d9fc3d6fcdfb2f8b09e7fa9c3b6c80: Status 404 returned error can't find the container with id d1397b503ba4140455a1da1c7696506684d9fc3d6fcdfb2f8b09e7fa9c3b6c80 Oct 30 20:52:47.053565 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:47.053527 4611 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:52:47.053747 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:47.053725 4611 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:52:47.053883 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:47.053864 4611 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:52:47.577671 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:47.577635 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-qvfk5" event={"ID":"527a4fc6-4bfe-49c5-9352-d5b82a74e7ff","Type":"ContainerStarted","Data":"07f6f4705c682af2e74dcdad8363c9ce1eee6a59a57f010b76fd28ad40d54386"} Oct 30 20:52:47.578077 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:47.577689 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-qvfk5" event={"ID":"527a4fc6-4bfe-49c5-9352-d5b82a74e7ff","Type":"ContainerStarted","Data":"d1397b503ba4140455a1da1c7696506684d9fc3d6fcdfb2f8b09e7fa9c3b6c80"} Oct 30 20:52:47.603256 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:47.603209 4611 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-qvfk5" podStartSLOduration=1.603195395 podStartE2EDuration="1.603195395s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:52:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:52:47.598638198 +0000 UTC m=+822.132664818" watchObservedRunningTime="2026-10-30 20:52:47.603195395 +0000 UTC m=+822.137222010" Oct 30 20:52:47.714032 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:47.714005 4611 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-qvfk5" Oct 30 20:52:47.876652 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:47.876609 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:52:48.419725 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:48.419681 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:52:48.699038 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:48.698944 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:52:48.716422 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:48.716385 4611 patch_prober.go:29] interesting pod/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-qvfk5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Oct 30 20:52:48.716618 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:48.716452 4611 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-qvfk5" podUID="527a4fc6-4bfe-49c5-9352-d5b82a74e7ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Oct 30 20:52:49.472541 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:49.472274 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:52:49.714106 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:49.714069 4611 patch_prober.go:29] interesting pod/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-qvfk5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Oct 30 20:52:49.714484 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:49.714142 4611 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-qvfk5" podUID="527a4fc6-4bfe-49c5-9352-d5b82a74e7ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Oct 30 20:52:50.287048 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:50.287003 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:52:50.713923 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:50.713890 4611 patch_prober.go:29] interesting pod/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-qvfk5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Oct 30 20:52:50.714166 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:50.713941 4611 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-qvfk5" podUID="527a4fc6-4bfe-49c5-9352-d5b82a74e7ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Oct 30 20:52:51.094020 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:51.093913 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:52:51.713658 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:51.713613 4611 patch_prober.go:29] interesting pod/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-qvfk5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Oct 30 20:52:51.713846 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:51.713681 4611 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-qvfk5" podUID="527a4fc6-4bfe-49c5-9352-d5b82a74e7ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Oct 30 20:52:51.813815 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:51.813775 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:52:52.497140 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:52.497103 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:52:52.714177 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:52.714143 4611 patch_prober.go:29] interesting pod/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-qvfk5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Oct 30 20:52:52.714340 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:52.714196 4611 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-qvfk5" podUID="527a4fc6-4bfe-49c5-9352-d5b82a74e7ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Oct 30 20:52:53.238054 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:53.237765 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:52:53.805474 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:53.805437 4611 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-qvfk5" Oct 30 20:52:53.806289 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:53.806256 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-qvfk5" Oct 30 20:52:53.807345 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:53.807319 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-qvfk5" Oct 30 20:52:54.114076 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:54.114042 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:52:54.916545 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:54.916068 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:52:55.607536 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:55.606194 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:52:56.217087 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:56.217052 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:52:56.901202 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:56.901168 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:52:57.697857 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:57.697823 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:52:57.841924 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:57.841888 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:52:58.387961 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:58.387921 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:52:59.023996 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:59.023950 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:52:59.692092 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:52:59.691903 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:53:00.166040 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:00.166009 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:53:00.510739 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:00.510653 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:53:01.147145 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:01.147095 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:53:01.355452 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:01.355403 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:53:01.687598 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:01.687563 4611 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-mwkjq"] Oct 30 20:53:01.692317 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:01.692292 4611 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-mwkjq" Oct 30 20:53:01.697564 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:01.697539 4611 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-mwkjq"] Oct 30 20:53:01.806648 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:01.806616 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59bd8e32-17bd-4753-ab30-1e32a1a2085f-workload-certs\") pod \"e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-mwkjq\" (UID: \"59bd8e32-17bd-4753-ab30-1e32a1a2085f\") " pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-mwkjq" Oct 30 20:53:01.806823 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:01.806678 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9vr9n\" (UniqueName: \"kubernetes.io/projected/59bd8e32-17bd-4753-ab30-1e32a1a2085f-kube-api-access-9vr9n\") pod \"e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-mwkjq\" (UID: \"59bd8e32-17bd-4753-ab30-1e32a1a2085f\") " pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-mwkjq" Oct 30 20:53:01.806823 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:01.806778 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59bd8e32-17bd-4753-ab30-1e32a1a2085f-istiod-ca-cert\") pod \"e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-mwkjq\" (UID: \"59bd8e32-17bd-4753-ab30-1e32a1a2085f\") " pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-mwkjq" Oct 30 20:53:01.806823 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:01.806805 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59bd8e32-17bd-4753-ab30-1e32a1a2085f-istio-data\") pod \"e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-mwkjq\" (UID: \"59bd8e32-17bd-4753-ab30-1e32a1a2085f\") " pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-mwkjq" Oct 30 20:53:01.806959 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:01.806917 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59bd8e32-17bd-4753-ab30-1e32a1a2085f-istio-token\") pod \"e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-mwkjq\" (UID: \"59bd8e32-17bd-4753-ab30-1e32a1a2085f\") " pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-mwkjq" Oct 30 20:53:01.806996 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:01.806982 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59bd8e32-17bd-4753-ab30-1e32a1a2085f-istio-podinfo\") pod \"e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-mwkjq\" (UID: \"59bd8e32-17bd-4753-ab30-1e32a1a2085f\") " pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-mwkjq" Oct 30 20:53:01.807087 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:01.807067 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59bd8e32-17bd-4753-ab30-1e32a1a2085f-credential-socket\") pod \"e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-mwkjq\" (UID: \"59bd8e32-17bd-4753-ab30-1e32a1a2085f\") " pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-mwkjq" Oct 30 20:53:01.807128 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:01.807114 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59bd8e32-17bd-4753-ab30-1e32a1a2085f-workload-socket\") pod \"e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-mwkjq\" (UID: \"59bd8e32-17bd-4753-ab30-1e32a1a2085f\") " pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-mwkjq" Oct 30 20:53:01.807176 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:01.807153 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59bd8e32-17bd-4753-ab30-1e32a1a2085f-istio-envoy\") pod \"e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-mwkjq\" (UID: \"59bd8e32-17bd-4753-ab30-1e32a1a2085f\") " pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-mwkjq" Oct 30 20:53:01.908559 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:01.908519 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59bd8e32-17bd-4753-ab30-1e32a1a2085f-workload-certs\") pod \"e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-mwkjq\" (UID: \"59bd8e32-17bd-4753-ab30-1e32a1a2085f\") " pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-mwkjq" Oct 30 20:53:01.908739 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:01.908583 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9vr9n\" (UniqueName: \"kubernetes.io/projected/59bd8e32-17bd-4753-ab30-1e32a1a2085f-kube-api-access-9vr9n\") pod \"e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-mwkjq\" (UID: \"59bd8e32-17bd-4753-ab30-1e32a1a2085f\") " pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-mwkjq" Oct 30 20:53:01.908739 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:01.908617 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59bd8e32-17bd-4753-ab30-1e32a1a2085f-istiod-ca-cert\") pod \"e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-mwkjq\" (UID: \"59bd8e32-17bd-4753-ab30-1e32a1a2085f\") " pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-mwkjq" Oct 30 20:53:01.908739 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:01.908634 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59bd8e32-17bd-4753-ab30-1e32a1a2085f-istio-data\") pod \"e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-mwkjq\" (UID: \"59bd8e32-17bd-4753-ab30-1e32a1a2085f\") " pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-mwkjq" Oct 30 20:53:01.908739 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:01.908680 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59bd8e32-17bd-4753-ab30-1e32a1a2085f-istio-token\") pod \"e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-mwkjq\" (UID: \"59bd8e32-17bd-4753-ab30-1e32a1a2085f\") " pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-mwkjq" Oct 30 20:53:01.908739 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:01.908727 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59bd8e32-17bd-4753-ab30-1e32a1a2085f-istio-podinfo\") pod \"e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-mwkjq\" (UID: \"59bd8e32-17bd-4753-ab30-1e32a1a2085f\") " pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-mwkjq" Oct 30 20:53:01.909289 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:01.908799 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59bd8e32-17bd-4753-ab30-1e32a1a2085f-credential-socket\") pod \"e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-mwkjq\" (UID: \"59bd8e32-17bd-4753-ab30-1e32a1a2085f\") " pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-mwkjq" Oct 30 20:53:01.909289 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:01.908857 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59bd8e32-17bd-4753-ab30-1e32a1a2085f-workload-socket\") pod \"e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-mwkjq\" (UID: \"59bd8e32-17bd-4753-ab30-1e32a1a2085f\") " pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-mwkjq" Oct 30 20:53:01.909289 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:01.908912 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59bd8e32-17bd-4753-ab30-1e32a1a2085f-workload-certs\") pod \"e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-mwkjq\" (UID: \"59bd8e32-17bd-4753-ab30-1e32a1a2085f\") " pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-mwkjq" Oct 30 20:53:01.909289 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:01.908917 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59bd8e32-17bd-4753-ab30-1e32a1a2085f-istio-envoy\") pod \"e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-mwkjq\" (UID: \"59bd8e32-17bd-4753-ab30-1e32a1a2085f\") " pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-mwkjq" Oct 30 20:53:01.909289 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:01.908989 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59bd8e32-17bd-4753-ab30-1e32a1a2085f-istio-data\") pod \"e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-mwkjq\" (UID: \"59bd8e32-17bd-4753-ab30-1e32a1a2085f\") " pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-mwkjq" Oct 30 20:53:01.909289 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:01.909214 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59bd8e32-17bd-4753-ab30-1e32a1a2085f-workload-socket\") pod \"e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-mwkjq\" (UID: \"59bd8e32-17bd-4753-ab30-1e32a1a2085f\") " pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-mwkjq" Oct 30 20:53:01.909289 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:01.909291 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59bd8e32-17bd-4753-ab30-1e32a1a2085f-credential-socket\") pod \"e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-mwkjq\" (UID: \"59bd8e32-17bd-4753-ab30-1e32a1a2085f\") " pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-mwkjq" Oct 30 20:53:01.909689 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:01.909531 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59bd8e32-17bd-4753-ab30-1e32a1a2085f-istiod-ca-cert\") pod \"e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-mwkjq\" (UID: \"59bd8e32-17bd-4753-ab30-1e32a1a2085f\") " pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-mwkjq" Oct 30 20:53:01.911236 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:01.911212 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59bd8e32-17bd-4753-ab30-1e32a1a2085f-istio-envoy\") pod \"e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-mwkjq\" (UID: \"59bd8e32-17bd-4753-ab30-1e32a1a2085f\") " pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-mwkjq" Oct 30 20:53:01.911442 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:01.911426 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59bd8e32-17bd-4753-ab30-1e32a1a2085f-istio-podinfo\") pod \"e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-mwkjq\" (UID: \"59bd8e32-17bd-4753-ab30-1e32a1a2085f\") " pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-mwkjq" Oct 30 20:53:01.919648 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:01.919625 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59bd8e32-17bd-4753-ab30-1e32a1a2085f-istio-token\") pod \"e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-mwkjq\" (UID: \"59bd8e32-17bd-4753-ab30-1e32a1a2085f\") " pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-mwkjq" Oct 30 20:53:01.919775 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:01.919760 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9vr9n\" (UniqueName: \"kubernetes.io/projected/59bd8e32-17bd-4753-ab30-1e32a1a2085f-kube-api-access-9vr9n\") pod \"e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-mwkjq\" (UID: \"59bd8e32-17bd-4753-ab30-1e32a1a2085f\") " pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-mwkjq" Oct 30 20:53:02.016764 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:02.016711 4611 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-mwkjq" Oct 30 20:53:02.148318 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:02.148285 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:53:02.272799 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:02.272728 4611 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-2pw94"] Oct 30 20:53:02.278529 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:02.278080 4611 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-2pw94" Oct 30 20:53:02.296068 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:02.296022 4611 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-2pw94"] Oct 30 20:53:02.310842 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:02.310821 4611 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-mwkjq"] Oct 30 20:53:02.316108 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:53:02.316080 4611 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59bd8e32_17bd_4753_ab30_1e32a1a2085f.slice/crio-e089fb9ae62e8db71d7725f4dddb373e78880da2c4552048c2f57da16e09e185 WatchSource:0}: Error finding container e089fb9ae62e8db71d7725f4dddb373e78880da2c4552048c2f57da16e09e185: Status 404 returned error can't find the container with id e089fb9ae62e8db71d7725f4dddb373e78880da2c4552048c2f57da16e09e185 Oct 30 20:53:02.319338 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:02.319305 4611 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:53:02.319413 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:02.319387 4611 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:53:02.319451 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:02.319431 4611 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:53:02.418839 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:02.418797 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8603ecfb-6faf-4f18-bb0b-2aee0341b0e3-workload-socket\") pod \"e2e-worker-w3-cb60a2-openshift-default-57db98bd79-2pw94\" (UID: \"8603ecfb-6faf-4f18-bb0b-2aee0341b0e3\") " pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-2pw94" Oct 30 20:53:02.419258 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:02.418845 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8603ecfb-6faf-4f18-bb0b-2aee0341b0e3-istio-podinfo\") pod \"e2e-worker-w3-cb60a2-openshift-default-57db98bd79-2pw94\" (UID: \"8603ecfb-6faf-4f18-bb0b-2aee0341b0e3\") " pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-2pw94" Oct 30 20:53:02.419258 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:02.418945 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8603ecfb-6faf-4f18-bb0b-2aee0341b0e3-istio-token\") pod \"e2e-worker-w3-cb60a2-openshift-default-57db98bd79-2pw94\" (UID: \"8603ecfb-6faf-4f18-bb0b-2aee0341b0e3\") " pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-2pw94" Oct 30 20:53:02.419258 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:02.419009 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8603ecfb-6faf-4f18-bb0b-2aee0341b0e3-istiod-ca-cert\") pod \"e2e-worker-w3-cb60a2-openshift-default-57db98bd79-2pw94\" (UID: \"8603ecfb-6faf-4f18-bb0b-2aee0341b0e3\") " pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-2pw94" Oct 30 20:53:02.419258 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:02.419106 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gsjg7\" (UniqueName: \"kubernetes.io/projected/8603ecfb-6faf-4f18-bb0b-2aee0341b0e3-kube-api-access-gsjg7\") pod \"e2e-worker-w3-cb60a2-openshift-default-57db98bd79-2pw94\" (UID: \"8603ecfb-6faf-4f18-bb0b-2aee0341b0e3\") " pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-2pw94" Oct 30 20:53:02.419258 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:02.419143 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8603ecfb-6faf-4f18-bb0b-2aee0341b0e3-istio-data\") pod \"e2e-worker-w3-cb60a2-openshift-default-57db98bd79-2pw94\" (UID: \"8603ecfb-6faf-4f18-bb0b-2aee0341b0e3\") " pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-2pw94" Oct 30 20:53:02.419258 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:02.419245 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8603ecfb-6faf-4f18-bb0b-2aee0341b0e3-credential-socket\") pod \"e2e-worker-w3-cb60a2-openshift-default-57db98bd79-2pw94\" (UID: \"8603ecfb-6faf-4f18-bb0b-2aee0341b0e3\") " pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-2pw94" Oct 30 20:53:02.419590 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:02.419269 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8603ecfb-6faf-4f18-bb0b-2aee0341b0e3-workload-certs\") pod \"e2e-worker-w3-cb60a2-openshift-default-57db98bd79-2pw94\" (UID: \"8603ecfb-6faf-4f18-bb0b-2aee0341b0e3\") " pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-2pw94" Oct 30 20:53:02.419590 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:02.419425 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8603ecfb-6faf-4f18-bb0b-2aee0341b0e3-istio-envoy\") pod \"e2e-worker-w3-cb60a2-openshift-default-57db98bd79-2pw94\" (UID: \"8603ecfb-6faf-4f18-bb0b-2aee0341b0e3\") " pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-2pw94" Oct 30 20:53:02.520236 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:02.520198 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8603ecfb-6faf-4f18-bb0b-2aee0341b0e3-workload-socket\") pod \"e2e-worker-w3-cb60a2-openshift-default-57db98bd79-2pw94\" (UID: \"8603ecfb-6faf-4f18-bb0b-2aee0341b0e3\") " pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-2pw94" Oct 30 20:53:02.520236 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:02.520235 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8603ecfb-6faf-4f18-bb0b-2aee0341b0e3-istio-podinfo\") pod \"e2e-worker-w3-cb60a2-openshift-default-57db98bd79-2pw94\" (UID: \"8603ecfb-6faf-4f18-bb0b-2aee0341b0e3\") " pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-2pw94" Oct 30 20:53:02.520479 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:02.520293 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8603ecfb-6faf-4f18-bb0b-2aee0341b0e3-istio-token\") pod \"e2e-worker-w3-cb60a2-openshift-default-57db98bd79-2pw94\" (UID: \"8603ecfb-6faf-4f18-bb0b-2aee0341b0e3\") " pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-2pw94" Oct 30 20:53:02.520479 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:02.520366 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8603ecfb-6faf-4f18-bb0b-2aee0341b0e3-istiod-ca-cert\") pod \"e2e-worker-w3-cb60a2-openshift-default-57db98bd79-2pw94\" (UID: \"8603ecfb-6faf-4f18-bb0b-2aee0341b0e3\") " pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-2pw94" Oct 30 20:53:02.520479 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:02.520397 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gsjg7\" (UniqueName: \"kubernetes.io/projected/8603ecfb-6faf-4f18-bb0b-2aee0341b0e3-kube-api-access-gsjg7\") pod \"e2e-worker-w3-cb60a2-openshift-default-57db98bd79-2pw94\" (UID: \"8603ecfb-6faf-4f18-bb0b-2aee0341b0e3\") " pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-2pw94" Oct 30 20:53:02.520479 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:02.520426 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8603ecfb-6faf-4f18-bb0b-2aee0341b0e3-istio-data\") pod \"e2e-worker-w3-cb60a2-openshift-default-57db98bd79-2pw94\" (UID: \"8603ecfb-6faf-4f18-bb0b-2aee0341b0e3\") " pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-2pw94" Oct 30 20:53:02.520724 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:02.520500 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8603ecfb-6faf-4f18-bb0b-2aee0341b0e3-credential-socket\") pod \"e2e-worker-w3-cb60a2-openshift-default-57db98bd79-2pw94\" (UID: \"8603ecfb-6faf-4f18-bb0b-2aee0341b0e3\") " pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-2pw94" Oct 30 20:53:02.520724 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:02.520570 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8603ecfb-6faf-4f18-bb0b-2aee0341b0e3-workload-certs\") pod \"e2e-worker-w3-cb60a2-openshift-default-57db98bd79-2pw94\" (UID: \"8603ecfb-6faf-4f18-bb0b-2aee0341b0e3\") " pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-2pw94" Oct 30 20:53:02.520724 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:02.520691 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8603ecfb-6faf-4f18-bb0b-2aee0341b0e3-istio-envoy\") pod \"e2e-worker-w3-cb60a2-openshift-default-57db98bd79-2pw94\" (UID: \"8603ecfb-6faf-4f18-bb0b-2aee0341b0e3\") " pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-2pw94" Oct 30 20:53:02.520724 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:02.520698 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8603ecfb-6faf-4f18-bb0b-2aee0341b0e3-workload-socket\") pod \"e2e-worker-w3-cb60a2-openshift-default-57db98bd79-2pw94\" (UID: \"8603ecfb-6faf-4f18-bb0b-2aee0341b0e3\") " pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-2pw94" Oct 30 20:53:02.520936 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:02.520859 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8603ecfb-6faf-4f18-bb0b-2aee0341b0e3-istio-data\") pod \"e2e-worker-w3-cb60a2-openshift-default-57db98bd79-2pw94\" (UID: \"8603ecfb-6faf-4f18-bb0b-2aee0341b0e3\") " pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-2pw94" Oct 30 20:53:02.520936 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:02.520878 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8603ecfb-6faf-4f18-bb0b-2aee0341b0e3-credential-socket\") pod \"e2e-worker-w3-cb60a2-openshift-default-57db98bd79-2pw94\" (UID: \"8603ecfb-6faf-4f18-bb0b-2aee0341b0e3\") " pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-2pw94" Oct 30 20:53:02.521054 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:02.521038 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8603ecfb-6faf-4f18-bb0b-2aee0341b0e3-workload-certs\") pod \"e2e-worker-w3-cb60a2-openshift-default-57db98bd79-2pw94\" (UID: \"8603ecfb-6faf-4f18-bb0b-2aee0341b0e3\") " pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-2pw94" Oct 30 20:53:02.521397 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:02.521373 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8603ecfb-6faf-4f18-bb0b-2aee0341b0e3-istiod-ca-cert\") pod \"e2e-worker-w3-cb60a2-openshift-default-57db98bd79-2pw94\" (UID: \"8603ecfb-6faf-4f18-bb0b-2aee0341b0e3\") " pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-2pw94" Oct 30 20:53:02.522745 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:02.522722 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8603ecfb-6faf-4f18-bb0b-2aee0341b0e3-istio-envoy\") pod \"e2e-worker-w3-cb60a2-openshift-default-57db98bd79-2pw94\" (UID: \"8603ecfb-6faf-4f18-bb0b-2aee0341b0e3\") " pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-2pw94" Oct 30 20:53:02.522897 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:02.522771 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8603ecfb-6faf-4f18-bb0b-2aee0341b0e3-istio-podinfo\") pod \"e2e-worker-w3-cb60a2-openshift-default-57db98bd79-2pw94\" (UID: \"8603ecfb-6faf-4f18-bb0b-2aee0341b0e3\") " pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-2pw94" Oct 30 20:53:02.532580 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:02.532553 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8603ecfb-6faf-4f18-bb0b-2aee0341b0e3-istio-token\") pod \"e2e-worker-w3-cb60a2-openshift-default-57db98bd79-2pw94\" (UID: \"8603ecfb-6faf-4f18-bb0b-2aee0341b0e3\") " pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-2pw94" Oct 30 20:53:02.532799 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:02.532779 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gsjg7\" (UniqueName: \"kubernetes.io/projected/8603ecfb-6faf-4f18-bb0b-2aee0341b0e3-kube-api-access-gsjg7\") pod \"e2e-worker-w3-cb60a2-openshift-default-57db98bd79-2pw94\" (UID: \"8603ecfb-6faf-4f18-bb0b-2aee0341b0e3\") " pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-2pw94" Oct 30 20:53:02.601275 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:02.601247 4611 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-2pw94" Oct 30 20:53:02.723472 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:02.722902 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-mwkjq" event={"ID":"59bd8e32-17bd-4753-ab30-1e32a1a2085f","Type":"ContainerStarted","Data":"b2f26a42242237c9d55b800f03e076c751ae5f0f865aa72e2a225bdc99efc463"} Oct 30 20:53:02.723472 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:02.722957 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-mwkjq" event={"ID":"59bd8e32-17bd-4753-ab30-1e32a1a2085f","Type":"ContainerStarted","Data":"e089fb9ae62e8db71d7725f4dddb373e78880da2c4552048c2f57da16e09e185"} Oct 30 20:53:02.747316 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:02.747264 4611 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-mwkjq" podStartSLOduration=1.747244189 podStartE2EDuration="1.747244189s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:53:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:53:02.743661145 +0000 UTC m=+837.277687762" watchObservedRunningTime="2026-10-30 20:53:02.747244189 +0000 UTC m=+837.281270809" Oct 30 20:53:02.899558 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:02.899529 4611 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-2pw94"] Oct 30 20:53:02.900390 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:53:02.900365 4611 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8603ecfb_6faf_4f18_bb0b_2aee0341b0e3.slice/crio-4f8f9bb17315ab4f1215f5f2b7163ab03036f09fd72c75918b3bb1b63009eefa WatchSource:0}: Error finding container 4f8f9bb17315ab4f1215f5f2b7163ab03036f09fd72c75918b3bb1b63009eefa: Status 404 returned error can't find the container with id 4f8f9bb17315ab4f1215f5f2b7163ab03036f09fd72c75918b3bb1b63009eefa Oct 30 20:53:02.902924 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:02.902895 4611 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:53:02.903003 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:02.902958 4611 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:53:02.903003 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:02.902988 4611 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:53:03.017787 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:03.017763 4611 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-mwkjq" Oct 30 20:53:03.019730 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:03.019698 4611 patch_prober.go:29] interesting pod/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-mwkjq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body= Oct 30 20:53:03.019786 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:03.019747 4611 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-mwkjq" podUID="59bd8e32-17bd-4753-ab30-1e32a1a2085f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" Oct 30 20:53:03.114531 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:03.114418 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:53:03.730819 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:03.730784 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-2pw94" event={"ID":"8603ecfb-6faf-4f18-bb0b-2aee0341b0e3","Type":"ContainerStarted","Data":"02bf56353d665042d4709c90beafe218f03c0517f1183ba82c0dfba1a7f1a493"} Oct 30 20:53:03.731303 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:03.730843 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-2pw94" event={"ID":"8603ecfb-6faf-4f18-bb0b-2aee0341b0e3","Type":"ContainerStarted","Data":"4f8f9bb17315ab4f1215f5f2b7163ab03036f09fd72c75918b3bb1b63009eefa"} Oct 30 20:53:03.796531 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:03.793403 4611 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-2pw94" podStartSLOduration=1.7933656519999999 podStartE2EDuration="1.793365652s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:53:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:53:03.789906251 +0000 UTC m=+838.323932879" watchObservedRunningTime="2026-10-30 20:53:03.793365652 +0000 UTC m=+838.327392285" Oct 30 20:53:03.926986 ip-10-0-133-192 kubenswrapper[4611]: E1030 20:53:03.924587 4611 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hwzc6" containerName="extractor" Oct 30 20:53:04.017470 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:04.017392 4611 patch_prober.go:29] interesting pod/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-mwkjq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body= Oct 30 20:53:04.017470 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:04.017443 4611 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-mwkjq" podUID="59bd8e32-17bd-4753-ab30-1e32a1a2085f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" Oct 30 20:53:04.310025 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:04.309942 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:53:04.601878 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:04.601850 4611 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-2pw94" Oct 30 20:53:04.603563 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:04.603536 4611 patch_prober.go:29] interesting pod/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-2pw94 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Oct 30 20:53:04.603683 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:04.603589 4611 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-2pw94" podUID="8603ecfb-6faf-4f18-bb0b-2aee0341b0e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Oct 30 20:53:05.018256 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:05.018177 4611 patch_prober.go:29] interesting pod/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-mwkjq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body= Oct 30 20:53:05.018818 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:05.018780 4611 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-mwkjq" podUID="59bd8e32-17bd-4753-ab30-1e32a1a2085f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" Oct 30 20:53:05.404794 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:05.404757 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:53:05.601792 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:05.601759 4611 patch_prober.go:29] interesting pod/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-2pw94 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Oct 30 20:53:05.602008 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:05.601811 4611 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-2pw94" podUID="8603ecfb-6faf-4f18-bb0b-2aee0341b0e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Oct 30 20:53:06.017033 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:06.017003 4611 patch_prober.go:29] interesting pod/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-mwkjq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body= Oct 30 20:53:06.017237 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:06.017054 4611 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-mwkjq" podUID="59bd8e32-17bd-4753-ab30-1e32a1a2085f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" Oct 30 20:53:06.602189 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:06.602152 4611 patch_prober.go:29] interesting pod/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-2pw94 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Oct 30 20:53:06.602617 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:06.602219 4611 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-2pw94" podUID="8603ecfb-6faf-4f18-bb0b-2aee0341b0e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Oct 30 20:53:06.896587 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:06.896357 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:53:07.017785 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:07.017758 4611 patch_prober.go:29] interesting pod/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-mwkjq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body= Oct 30 20:53:07.018016 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:07.017988 4611 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-mwkjq" podUID="59bd8e32-17bd-4753-ab30-1e32a1a2085f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" Oct 30 20:53:07.602597 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:07.602546 4611 patch_prober.go:29] interesting pod/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-2pw94 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Oct 30 20:53:07.603083 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:07.602662 4611 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-2pw94" podUID="8603ecfb-6faf-4f18-bb0b-2aee0341b0e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Oct 30 20:53:08.017378 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:08.017299 4611 patch_prober.go:29] interesting pod/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-mwkjq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body= Oct 30 20:53:08.017378 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:08.017358 4611 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-mwkjq" podUID="59bd8e32-17bd-4753-ab30-1e32a1a2085f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" Oct 30 20:53:08.602463 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:08.602428 4611 patch_prober.go:29] interesting pod/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-2pw94 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Oct 30 20:53:08.602665 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:08.602485 4611 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-2pw94" podUID="8603ecfb-6faf-4f18-bb0b-2aee0341b0e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Oct 30 20:53:08.839254 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:08.839215 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:53:09.134030 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:09.133995 4611 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-mwkjq" Oct 30 20:53:09.135320 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:09.135297 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-mwkjq" Oct 30 20:53:09.136243 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:09.136224 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-mwkjq" Oct 30 20:53:09.601648 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:09.601618 4611 patch_prober.go:29] interesting pod/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-2pw94 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Oct 30 20:53:09.601819 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:09.601674 4611 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-2pw94" podUID="8603ecfb-6faf-4f18-bb0b-2aee0341b0e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Oct 30 20:53:10.337682 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:10.337643 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:53:10.606830 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:10.606800 4611 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-2pw94" Oct 30 20:53:10.607087 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:10.607067 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-2pw94" Oct 30 20:53:10.607757 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:10.607738 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-2pw94" Oct 30 20:53:11.105235 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:11.105199 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:53:11.901466 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:11.901429 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:53:12.697035 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:12.696707 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:53:14.772219 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:14.772183 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:53:14.965627 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:14.965583 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cb60a2/simulator-w3-cb60a2-kserve-6fd74b9d8b-vqfzf" podUID="711b167d-4e66-40d6-be75-33ede11f42c5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:53:15.074679 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:15.074581 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b8aee1/premium-w1-b8aee1-kserve-786d575599-htc8d" podUID="77368afe-edc8-4fa5-8010-d321965fb38e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded" Oct 30 20:53:15.089543 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:15.088993 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:53:15.099656 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:15.099613 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-14cca2/premium-w5-14cca2-kserve-6878777857-wpjtj" podUID="b99eb76b-d1c6-49e9-9809-c351c0f5a4de" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded" Oct 30 20:53:15.150667 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:15.148961 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad7ff0/premium-w0-ad7ff0-kserve-5485d4b948-bpmgf" podUID="53cbd692-e9f8-46ba-8fde-3e5215a38904" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded" Oct 30 20:53:15.150667 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:15.150017 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-3d909d/premium-w6-3d909d-kserve-765648946f-2qvd2" podUID="fca5d697-109d-4e90-ab41-c40ad31af874" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:53:15.251644 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:15.251595 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cb60a2/distinct-w3-cb60a2-kserve-5bb76dc74-86fds" podUID="52e301d9-f378-4e54-9515-01a6680d9fd7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:53:15.287650 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:15.287604 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-14cca2/distinct-w5-14cca2-kserve-587f6996b8-sgf6h" podUID="02b652c3-0724-4a50-9e95-2a9890cc97c4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:53:15.373637 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:15.373580 4611 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-14cca2/premium-w5-14cca2-kserve-6878777857-wpjtj" podUID="b99eb76b-d1c6-49e9-9809-c351c0f5a4de" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:53:16.114246 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:16.114206 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:53:16.791795 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:16.791751 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:53:16.957429 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:16.957397 4611 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-kb6vb"] Oct 30 20:53:16.965130 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:16.965099 4611 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-kb6vb" Oct 30 20:53:16.979649 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:16.979622 4611 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-kb6vb"] Oct 30 20:53:17.075793 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:17.075735 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41f0d1eb-188b-404a-8b31-cf9dc16b9fbc-istiod-ca-cert\") pod \"e2e-worker-w2-db534b-openshift-default-85df4d79d9-kb6vb\" (UID: \"41f0d1eb-188b-404a-8b31-cf9dc16b9fbc\") " pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-kb6vb" Oct 30 20:53:17.075968 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:17.075939 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41f0d1eb-188b-404a-8b31-cf9dc16b9fbc-istio-podinfo\") pod \"e2e-worker-w2-db534b-openshift-default-85df4d79d9-kb6vb\" (UID: \"41f0d1eb-188b-404a-8b31-cf9dc16b9fbc\") " pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-kb6vb" Oct 30 20:53:17.076043 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:17.075981 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mdcc2\" (UniqueName: \"kubernetes.io/projected/41f0d1eb-188b-404a-8b31-cf9dc16b9fbc-kube-api-access-mdcc2\") pod \"e2e-worker-w2-db534b-openshift-default-85df4d79d9-kb6vb\" (UID: \"41f0d1eb-188b-404a-8b31-cf9dc16b9fbc\") " pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-kb6vb" Oct 30 20:53:17.076043 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:17.076007 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41f0d1eb-188b-404a-8b31-cf9dc16b9fbc-credential-socket\") pod \"e2e-worker-w2-db534b-openshift-default-85df4d79d9-kb6vb\" (UID: \"41f0d1eb-188b-404a-8b31-cf9dc16b9fbc\") " pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-kb6vb" Oct 30 20:53:17.076043 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:17.076029 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41f0d1eb-188b-404a-8b31-cf9dc16b9fbc-istio-envoy\") pod \"e2e-worker-w2-db534b-openshift-default-85df4d79d9-kb6vb\" (UID: \"41f0d1eb-188b-404a-8b31-cf9dc16b9fbc\") " pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-kb6vb" Oct 30 20:53:17.076163 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:17.076128 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/41f0d1eb-188b-404a-8b31-cf9dc16b9fbc-istio-token\") pod \"e2e-worker-w2-db534b-openshift-default-85df4d79d9-kb6vb\" (UID: \"41f0d1eb-188b-404a-8b31-cf9dc16b9fbc\") " pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-kb6vb" Oct 30 20:53:17.076195 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:17.076159 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41f0d1eb-188b-404a-8b31-cf9dc16b9fbc-istio-data\") pod \"e2e-worker-w2-db534b-openshift-default-85df4d79d9-kb6vb\" (UID: \"41f0d1eb-188b-404a-8b31-cf9dc16b9fbc\") " pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-kb6vb" Oct 30 20:53:17.076260 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:17.076240 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41f0d1eb-188b-404a-8b31-cf9dc16b9fbc-workload-socket\") pod \"e2e-worker-w2-db534b-openshift-default-85df4d79d9-kb6vb\" (UID: \"41f0d1eb-188b-404a-8b31-cf9dc16b9fbc\") " pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-kb6vb" Oct 30 20:53:17.076389 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:17.076280 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41f0d1eb-188b-404a-8b31-cf9dc16b9fbc-workload-certs\") pod \"e2e-worker-w2-db534b-openshift-default-85df4d79d9-kb6vb\" (UID: \"41f0d1eb-188b-404a-8b31-cf9dc16b9fbc\") " pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-kb6vb" Oct 30 20:53:17.177781 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:17.177746 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/41f0d1eb-188b-404a-8b31-cf9dc16b9fbc-istio-token\") pod \"e2e-worker-w2-db534b-openshift-default-85df4d79d9-kb6vb\" (UID: \"41f0d1eb-188b-404a-8b31-cf9dc16b9fbc\") " pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-kb6vb" Oct 30 20:53:17.178220 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:17.177791 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41f0d1eb-188b-404a-8b31-cf9dc16b9fbc-istio-data\") pod \"e2e-worker-w2-db534b-openshift-default-85df4d79d9-kb6vb\" (UID: \"41f0d1eb-188b-404a-8b31-cf9dc16b9fbc\") " pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-kb6vb" Oct 30 20:53:17.178220 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:17.177877 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41f0d1eb-188b-404a-8b31-cf9dc16b9fbc-workload-socket\") pod \"e2e-worker-w2-db534b-openshift-default-85df4d79d9-kb6vb\" (UID: \"41f0d1eb-188b-404a-8b31-cf9dc16b9fbc\") " pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-kb6vb" Oct 30 20:53:17.178220 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:17.177900 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41f0d1eb-188b-404a-8b31-cf9dc16b9fbc-workload-certs\") pod \"e2e-worker-w2-db534b-openshift-default-85df4d79d9-kb6vb\" (UID: \"41f0d1eb-188b-404a-8b31-cf9dc16b9fbc\") " pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-kb6vb" Oct 30 20:53:17.178220 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:17.178039 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41f0d1eb-188b-404a-8b31-cf9dc16b9fbc-istiod-ca-cert\") pod \"e2e-worker-w2-db534b-openshift-default-85df4d79d9-kb6vb\" (UID: \"41f0d1eb-188b-404a-8b31-cf9dc16b9fbc\") " pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-kb6vb" Oct 30 20:53:17.178220 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:17.178179 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41f0d1eb-188b-404a-8b31-cf9dc16b9fbc-istio-podinfo\") pod \"e2e-worker-w2-db534b-openshift-default-85df4d79d9-kb6vb\" (UID: \"41f0d1eb-188b-404a-8b31-cf9dc16b9fbc\") " pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-kb6vb" Oct 30 20:53:17.178220 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:17.178207 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mdcc2\" (UniqueName: \"kubernetes.io/projected/41f0d1eb-188b-404a-8b31-cf9dc16b9fbc-kube-api-access-mdcc2\") pod \"e2e-worker-w2-db534b-openshift-default-85df4d79d9-kb6vb\" (UID: \"41f0d1eb-188b-404a-8b31-cf9dc16b9fbc\") " pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-kb6vb" Oct 30 20:53:17.178557 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:17.178234 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41f0d1eb-188b-404a-8b31-cf9dc16b9fbc-credential-socket\") pod \"e2e-worker-w2-db534b-openshift-default-85df4d79d9-kb6vb\" (UID: \"41f0d1eb-188b-404a-8b31-cf9dc16b9fbc\") " pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-kb6vb" Oct 30 20:53:17.178557 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:17.178276 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41f0d1eb-188b-404a-8b31-cf9dc16b9fbc-istio-envoy\") pod \"e2e-worker-w2-db534b-openshift-default-85df4d79d9-kb6vb\" (UID: \"41f0d1eb-188b-404a-8b31-cf9dc16b9fbc\") " pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-kb6vb" Oct 30 20:53:17.178557 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:17.178307 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41f0d1eb-188b-404a-8b31-cf9dc16b9fbc-istio-data\") pod \"e2e-worker-w2-db534b-openshift-default-85df4d79d9-kb6vb\" (UID: \"41f0d1eb-188b-404a-8b31-cf9dc16b9fbc\") " pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-kb6vb" Oct 30 20:53:17.178557 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:17.178335 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41f0d1eb-188b-404a-8b31-cf9dc16b9fbc-workload-certs\") pod \"e2e-worker-w2-db534b-openshift-default-85df4d79d9-kb6vb\" (UID: \"41f0d1eb-188b-404a-8b31-cf9dc16b9fbc\") " pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-kb6vb" Oct 30 20:53:17.178557 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:17.178485 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41f0d1eb-188b-404a-8b31-cf9dc16b9fbc-credential-socket\") pod \"e2e-worker-w2-db534b-openshift-default-85df4d79d9-kb6vb\" (UID: \"41f0d1eb-188b-404a-8b31-cf9dc16b9fbc\") " pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-kb6vb" Oct 30 20:53:17.178557 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:17.178499 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41f0d1eb-188b-404a-8b31-cf9dc16b9fbc-workload-socket\") pod \"e2e-worker-w2-db534b-openshift-default-85df4d79d9-kb6vb\" (UID: \"41f0d1eb-188b-404a-8b31-cf9dc16b9fbc\") " pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-kb6vb" Oct 30 20:53:17.178834 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:17.178744 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41f0d1eb-188b-404a-8b31-cf9dc16b9fbc-istiod-ca-cert\") pod \"e2e-worker-w2-db534b-openshift-default-85df4d79d9-kb6vb\" (UID: \"41f0d1eb-188b-404a-8b31-cf9dc16b9fbc\") " pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-kb6vb" Oct 30 20:53:17.180604 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:17.180577 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41f0d1eb-188b-404a-8b31-cf9dc16b9fbc-istio-envoy\") pod \"e2e-worker-w2-db534b-openshift-default-85df4d79d9-kb6vb\" (UID: \"41f0d1eb-188b-404a-8b31-cf9dc16b9fbc\") " pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-kb6vb" Oct 30 20:53:17.180837 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:17.180817 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41f0d1eb-188b-404a-8b31-cf9dc16b9fbc-istio-podinfo\") pod \"e2e-worker-w2-db534b-openshift-default-85df4d79d9-kb6vb\" (UID: \"41f0d1eb-188b-404a-8b31-cf9dc16b9fbc\") " pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-kb6vb" Oct 30 20:53:17.192746 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:17.192726 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/41f0d1eb-188b-404a-8b31-cf9dc16b9fbc-istio-token\") pod \"e2e-worker-w2-db534b-openshift-default-85df4d79d9-kb6vb\" (UID: \"41f0d1eb-188b-404a-8b31-cf9dc16b9fbc\") " pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-kb6vb" Oct 30 20:53:17.194403 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:17.194381 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mdcc2\" (UniqueName: \"kubernetes.io/projected/41f0d1eb-188b-404a-8b31-cf9dc16b9fbc-kube-api-access-mdcc2\") pod \"e2e-worker-w2-db534b-openshift-default-85df4d79d9-kb6vb\" (UID: \"41f0d1eb-188b-404a-8b31-cf9dc16b9fbc\") " pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-kb6vb" Oct 30 20:53:17.316686 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:17.316583 4611 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-kb6vb" Oct 30 20:53:17.781989 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:17.781963 4611 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-kb6vb"] Oct 30 20:53:17.784644 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:53:17.784618 4611 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f0d1eb_188b_404a_8b31_cf9dc16b9fbc.slice/crio-e0297d72bee10c4522c90c3fc28cbc94d68f4b484dfc3e6408808fd8cea9c66b WatchSource:0}: Error finding container e0297d72bee10c4522c90c3fc28cbc94d68f4b484dfc3e6408808fd8cea9c66b: Status 404 returned error can't find the container with id e0297d72bee10c4522c90c3fc28cbc94d68f4b484dfc3e6408808fd8cea9c66b Oct 30 20:53:17.786881 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:17.786847 4611 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:53:17.786972 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:17.786924 4611 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:53:17.787029 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:17.786967 4611 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:53:17.945666 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:17.944905 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-kb6vb" event={"ID":"41f0d1eb-188b-404a-8b31-cf9dc16b9fbc","Type":"ContainerStarted","Data":"57a88d136d133425ea062348fd5b75c500dc6d673847728ff7d889dc5e0082f2"} Oct 30 20:53:17.945666 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:17.945012 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-kb6vb" event={"ID":"41f0d1eb-188b-404a-8b31-cf9dc16b9fbc","Type":"ContainerStarted","Data":"e0297d72bee10c4522c90c3fc28cbc94d68f4b484dfc3e6408808fd8cea9c66b"} Oct 30 20:53:18.000796 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:18.000765 4611 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-f2ed1e/distinct-w4-f2ed1e-kserve-57cff9dcf8-8d5zl"] Oct 30 20:53:18.003347 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:18.003302 4611 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-kb6vb" podStartSLOduration=2.003289586 podStartE2EDuration="2.003289586s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:53:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:53:17.993327724 +0000 UTC m=+852.527354398" watchObservedRunningTime="2026-10-30 20:53:18.003289586 +0000 UTC m=+852.537316202" Oct 30 20:53:18.005011 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:18.004992 4611 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-f2ed1e/distinct-w4-f2ed1e-kserve-57cff9dcf8-8d5zl" Oct 30 20:53:18.017221 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:18.017167 4611 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-f2ed1e\"/\"kube-root-ca.crt\"" Oct 30 20:53:18.022527 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:18.022481 4611 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-f2ed1e/distinct-w4-f2ed1e-kserve-57cff9dcf8-8d5zl"] Oct 30 20:53:18.027049 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:18.027026 4611 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-f2ed1e\"/\"openshift-service-ca.crt\"" Oct 30 20:53:18.031295 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:18.031275 4611 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-f2ed1e\"/\"default-dockercfg-2nmmh\"" Oct 30 20:53:18.031695 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:18.031635 4611 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-f2ed1e\"/\"distinct-w4-f2ed1e-kserve-self-signed-certs\"" Oct 30 20:53:18.147546 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:18.147359 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:53:18.193572 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:18.192751 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/653e3ec7-2147-4813-8946-1f25cfa425f6-home\") pod \"distinct-w4-f2ed1e-kserve-57cff9dcf8-8d5zl\" (UID: \"653e3ec7-2147-4813-8946-1f25cfa425f6\") " pod="e2e-models-e2e-worker-w4-f2ed1e/distinct-w4-f2ed1e-kserve-57cff9dcf8-8d5zl" Oct 30 20:53:18.193572 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:18.192878 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/653e3ec7-2147-4813-8946-1f25cfa425f6-tls-certs\") pod \"distinct-w4-f2ed1e-kserve-57cff9dcf8-8d5zl\" (UID: \"653e3ec7-2147-4813-8946-1f25cfa425f6\") " pod="e2e-models-e2e-worker-w4-f2ed1e/distinct-w4-f2ed1e-kserve-57cff9dcf8-8d5zl" Oct 30 20:53:18.193572 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:18.192943 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j7zwk\" (UniqueName: \"kubernetes.io/projected/653e3ec7-2147-4813-8946-1f25cfa425f6-kube-api-access-j7zwk\") pod \"distinct-w4-f2ed1e-kserve-57cff9dcf8-8d5zl\" (UID: \"653e3ec7-2147-4813-8946-1f25cfa425f6\") " pod="e2e-models-e2e-worker-w4-f2ed1e/distinct-w4-f2ed1e-kserve-57cff9dcf8-8d5zl" Oct 30 20:53:18.193572 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:18.193089 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/653e3ec7-2147-4813-8946-1f25cfa425f6-dshm\") pod \"distinct-w4-f2ed1e-kserve-57cff9dcf8-8d5zl\" (UID: \"653e3ec7-2147-4813-8946-1f25cfa425f6\") " pod="e2e-models-e2e-worker-w4-f2ed1e/distinct-w4-f2ed1e-kserve-57cff9dcf8-8d5zl" Oct 30 20:53:18.193572 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:18.193122 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/653e3ec7-2147-4813-8946-1f25cfa425f6-model-cache\") pod \"distinct-w4-f2ed1e-kserve-57cff9dcf8-8d5zl\" (UID: \"653e3ec7-2147-4813-8946-1f25cfa425f6\") " pod="e2e-models-e2e-worker-w4-f2ed1e/distinct-w4-f2ed1e-kserve-57cff9dcf8-8d5zl" Oct 30 20:53:18.193572 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:18.193170 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/653e3ec7-2147-4813-8946-1f25cfa425f6-tmp-dir\") pod \"distinct-w4-f2ed1e-kserve-57cff9dcf8-8d5zl\" (UID: \"653e3ec7-2147-4813-8946-1f25cfa425f6\") " pod="e2e-models-e2e-worker-w4-f2ed1e/distinct-w4-f2ed1e-kserve-57cff9dcf8-8d5zl" Oct 30 20:53:18.299839 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:18.298434 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/653e3ec7-2147-4813-8946-1f25cfa425f6-tls-certs\") pod \"distinct-w4-f2ed1e-kserve-57cff9dcf8-8d5zl\" (UID: \"653e3ec7-2147-4813-8946-1f25cfa425f6\") " pod="e2e-models-e2e-worker-w4-f2ed1e/distinct-w4-f2ed1e-kserve-57cff9dcf8-8d5zl" Oct 30 20:53:18.299839 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:18.298490 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j7zwk\" (UniqueName: \"kubernetes.io/projected/653e3ec7-2147-4813-8946-1f25cfa425f6-kube-api-access-j7zwk\") pod \"distinct-w4-f2ed1e-kserve-57cff9dcf8-8d5zl\" (UID: \"653e3ec7-2147-4813-8946-1f25cfa425f6\") " pod="e2e-models-e2e-worker-w4-f2ed1e/distinct-w4-f2ed1e-kserve-57cff9dcf8-8d5zl" Oct 30 20:53:18.299839 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:18.298642 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/653e3ec7-2147-4813-8946-1f25cfa425f6-dshm\") pod \"distinct-w4-f2ed1e-kserve-57cff9dcf8-8d5zl\" (UID: \"653e3ec7-2147-4813-8946-1f25cfa425f6\") " pod="e2e-models-e2e-worker-w4-f2ed1e/distinct-w4-f2ed1e-kserve-57cff9dcf8-8d5zl" Oct 30 20:53:18.299839 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:18.298674 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/653e3ec7-2147-4813-8946-1f25cfa425f6-model-cache\") pod \"distinct-w4-f2ed1e-kserve-57cff9dcf8-8d5zl\" (UID: \"653e3ec7-2147-4813-8946-1f25cfa425f6\") " pod="e2e-models-e2e-worker-w4-f2ed1e/distinct-w4-f2ed1e-kserve-57cff9dcf8-8d5zl" Oct 30 20:53:18.299839 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:18.298723 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/653e3ec7-2147-4813-8946-1f25cfa425f6-tmp-dir\") pod \"distinct-w4-f2ed1e-kserve-57cff9dcf8-8d5zl\" (UID: \"653e3ec7-2147-4813-8946-1f25cfa425f6\") " pod="e2e-models-e2e-worker-w4-f2ed1e/distinct-w4-f2ed1e-kserve-57cff9dcf8-8d5zl" Oct 30 20:53:18.299839 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:18.298906 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/653e3ec7-2147-4813-8946-1f25cfa425f6-home\") pod \"distinct-w4-f2ed1e-kserve-57cff9dcf8-8d5zl\" (UID: \"653e3ec7-2147-4813-8946-1f25cfa425f6\") " pod="e2e-models-e2e-worker-w4-f2ed1e/distinct-w4-f2ed1e-kserve-57cff9dcf8-8d5zl" Oct 30 20:53:18.299839 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:18.299362 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/653e3ec7-2147-4813-8946-1f25cfa425f6-home\") pod \"distinct-w4-f2ed1e-kserve-57cff9dcf8-8d5zl\" (UID: \"653e3ec7-2147-4813-8946-1f25cfa425f6\") " pod="e2e-models-e2e-worker-w4-f2ed1e/distinct-w4-f2ed1e-kserve-57cff9dcf8-8d5zl" Oct 30 20:53:18.309572 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:18.304835 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/653e3ec7-2147-4813-8946-1f25cfa425f6-model-cache\") pod \"distinct-w4-f2ed1e-kserve-57cff9dcf8-8d5zl\" (UID: \"653e3ec7-2147-4813-8946-1f25cfa425f6\") " pod="e2e-models-e2e-worker-w4-f2ed1e/distinct-w4-f2ed1e-kserve-57cff9dcf8-8d5zl" Oct 30 20:53:18.309572 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:18.305105 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/653e3ec7-2147-4813-8946-1f25cfa425f6-tmp-dir\") pod \"distinct-w4-f2ed1e-kserve-57cff9dcf8-8d5zl\" (UID: \"653e3ec7-2147-4813-8946-1f25cfa425f6\") " pod="e2e-models-e2e-worker-w4-f2ed1e/distinct-w4-f2ed1e-kserve-57cff9dcf8-8d5zl" Oct 30 20:53:18.317155 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:18.316220 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/653e3ec7-2147-4813-8946-1f25cfa425f6-tls-certs\") pod \"distinct-w4-f2ed1e-kserve-57cff9dcf8-8d5zl\" (UID: \"653e3ec7-2147-4813-8946-1f25cfa425f6\") " pod="e2e-models-e2e-worker-w4-f2ed1e/distinct-w4-f2ed1e-kserve-57cff9dcf8-8d5zl" Oct 30 20:53:18.317825 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:18.317597 4611 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-kb6vb" Oct 30 20:53:18.319098 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:18.318749 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/653e3ec7-2147-4813-8946-1f25cfa425f6-dshm\") pod \"distinct-w4-f2ed1e-kserve-57cff9dcf8-8d5zl\" (UID: \"653e3ec7-2147-4813-8946-1f25cfa425f6\") " pod="e2e-models-e2e-worker-w4-f2ed1e/distinct-w4-f2ed1e-kserve-57cff9dcf8-8d5zl" Oct 30 20:53:18.319098 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:18.318968 4611 patch_prober.go:29] interesting pod/e2e-worker-w2-db534b-openshift-default-85df4d79d9-kb6vb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body= Oct 30 20:53:18.319098 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:18.319040 4611 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-kb6vb" podUID="41f0d1eb-188b-404a-8b31-cf9dc16b9fbc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" Oct 30 20:53:18.321277 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:18.321249 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j7zwk\" (UniqueName: \"kubernetes.io/projected/653e3ec7-2147-4813-8946-1f25cfa425f6-kube-api-access-j7zwk\") pod \"distinct-w4-f2ed1e-kserve-57cff9dcf8-8d5zl\" (UID: \"653e3ec7-2147-4813-8946-1f25cfa425f6\") " pod="e2e-models-e2e-worker-w4-f2ed1e/distinct-w4-f2ed1e-kserve-57cff9dcf8-8d5zl" Oct 30 20:53:18.328970 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:18.328849 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:53:18.332950 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:18.332928 4611 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-f2ed1e/distinct-w4-f2ed1e-kserve-57cff9dcf8-8d5zl" Oct 30 20:53:18.672715 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:18.672623 4611 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-f2ed1e/distinct-w4-f2ed1e-kserve-57cff9dcf8-8d5zl"] Oct 30 20:53:18.979275 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:18.979176 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-f2ed1e/distinct-w4-f2ed1e-kserve-57cff9dcf8-8d5zl" event={"ID":"653e3ec7-2147-4813-8946-1f25cfa425f6","Type":"ContainerStarted","Data":"880b13fdc8d0d02d089a94a05e01562d0e2b604022bb39fe0938b3ef30ccee2a"} Oct 30 20:53:18.979275 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:18.979228 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-f2ed1e/distinct-w4-f2ed1e-kserve-57cff9dcf8-8d5zl" event={"ID":"653e3ec7-2147-4813-8946-1f25cfa425f6","Type":"ContainerStarted","Data":"5616fdc6af3f8b43a6b4dafef0901dc015f7baa74ecd3dab04422331604d1b7b"} Oct 30 20:53:18.998616 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:18.998568 4611 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-f2ed1e/distinct-w4-f2ed1e-kserve-57cff9dcf8-8d5zl" podStartSLOduration=1.9985553889999998 podStartE2EDuration="1.998555389s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:53:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:53:18.997014899 +0000 UTC m=+853.531041513" watchObservedRunningTime="2026-10-30 20:53:18.998555389 +0000 UTC m=+853.532582015" Oct 30 20:53:19.133600 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:19.133565 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:53:19.317810 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:19.317731 4611 patch_prober.go:29] interesting pod/e2e-worker-w2-db534b-openshift-default-85df4d79d9-kb6vb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body= Oct 30 20:53:19.317810 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:19.317782 4611 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-kb6vb" podUID="41f0d1eb-188b-404a-8b31-cf9dc16b9fbc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" Oct 30 20:53:19.333123 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:19.333092 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-f2ed1e/distinct-w4-f2ed1e-kserve-57cff9dcf8-8d5zl" Oct 30 20:53:20.168392 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:20.168348 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:53:20.317288 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:20.317256 4611 patch_prober.go:29] interesting pod/e2e-worker-w2-db534b-openshift-default-85df4d79d9-kb6vb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body= Oct 30 20:53:20.317462 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:20.317311 4611 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-kb6vb" podUID="41f0d1eb-188b-404a-8b31-cf9dc16b9fbc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" Oct 30 20:53:20.796602 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:20.796568 4611 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-b8aee1/distinct-w1-b8aee1-kserve-5b857c55cd-x5b7b"] Oct 30 20:53:20.800788 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:20.800757 4611 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-b8aee1/distinct-w1-b8aee1-kserve-5b857c55cd-x5b7b" Oct 30 20:53:20.808633 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:20.807798 4611 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-b8aee1\"/\"distinct-w1-b8aee1-kserve-self-signed-certs\"" Oct 30 20:53:20.812675 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:20.812652 4611 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-b8aee1/distinct-w1-b8aee1-kserve-5b857c55cd-x5b7b"] Oct 30 20:53:20.832284 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:20.832251 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5q5dm\" (UniqueName: \"kubernetes.io/projected/1599e588-def8-42e1-b81d-ad0aaf119818-kube-api-access-5q5dm\") pod \"distinct-w1-b8aee1-kserve-5b857c55cd-x5b7b\" (UID: \"1599e588-def8-42e1-b81d-ad0aaf119818\") " pod="e2e-models-e2e-worker-w1-b8aee1/distinct-w1-b8aee1-kserve-5b857c55cd-x5b7b" Oct 30 20:53:20.832401 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:20.832306 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1599e588-def8-42e1-b81d-ad0aaf119818-tmp-dir\") pod \"distinct-w1-b8aee1-kserve-5b857c55cd-x5b7b\" (UID: \"1599e588-def8-42e1-b81d-ad0aaf119818\") " pod="e2e-models-e2e-worker-w1-b8aee1/distinct-w1-b8aee1-kserve-5b857c55cd-x5b7b" Oct 30 20:53:20.832445 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:20.832394 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1599e588-def8-42e1-b81d-ad0aaf119818-dshm\") pod \"distinct-w1-b8aee1-kserve-5b857c55cd-x5b7b\" (UID: \"1599e588-def8-42e1-b81d-ad0aaf119818\") " pod="e2e-models-e2e-worker-w1-b8aee1/distinct-w1-b8aee1-kserve-5b857c55cd-x5b7b" Oct 30 20:53:20.832445 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:20.832421 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1599e588-def8-42e1-b81d-ad0aaf119818-home\") pod \"distinct-w1-b8aee1-kserve-5b857c55cd-x5b7b\" (UID: \"1599e588-def8-42e1-b81d-ad0aaf119818\") " pod="e2e-models-e2e-worker-w1-b8aee1/distinct-w1-b8aee1-kserve-5b857c55cd-x5b7b" Oct 30 20:53:20.832572 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:20.832558 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1599e588-def8-42e1-b81d-ad0aaf119818-tls-certs\") pod \"distinct-w1-b8aee1-kserve-5b857c55cd-x5b7b\" (UID: \"1599e588-def8-42e1-b81d-ad0aaf119818\") " pod="e2e-models-e2e-worker-w1-b8aee1/distinct-w1-b8aee1-kserve-5b857c55cd-x5b7b" Oct 30 20:53:20.832686 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:20.832667 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1599e588-def8-42e1-b81d-ad0aaf119818-model-cache\") pod \"distinct-w1-b8aee1-kserve-5b857c55cd-x5b7b\" (UID: \"1599e588-def8-42e1-b81d-ad0aaf119818\") " pod="e2e-models-e2e-worker-w1-b8aee1/distinct-w1-b8aee1-kserve-5b857c55cd-x5b7b" Oct 30 20:53:20.897665 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:20.897633 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:53:20.933899 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:20.933869 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1599e588-def8-42e1-b81d-ad0aaf119818-model-cache\") pod \"distinct-w1-b8aee1-kserve-5b857c55cd-x5b7b\" (UID: \"1599e588-def8-42e1-b81d-ad0aaf119818\") " pod="e2e-models-e2e-worker-w1-b8aee1/distinct-w1-b8aee1-kserve-5b857c55cd-x5b7b" Oct 30 20:53:20.934033 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:20.933986 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5q5dm\" (UniqueName: \"kubernetes.io/projected/1599e588-def8-42e1-b81d-ad0aaf119818-kube-api-access-5q5dm\") pod \"distinct-w1-b8aee1-kserve-5b857c55cd-x5b7b\" (UID: \"1599e588-def8-42e1-b81d-ad0aaf119818\") " pod="e2e-models-e2e-worker-w1-b8aee1/distinct-w1-b8aee1-kserve-5b857c55cd-x5b7b" Oct 30 20:53:20.934033 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:20.934024 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1599e588-def8-42e1-b81d-ad0aaf119818-tmp-dir\") pod \"distinct-w1-b8aee1-kserve-5b857c55cd-x5b7b\" (UID: \"1599e588-def8-42e1-b81d-ad0aaf119818\") " pod="e2e-models-e2e-worker-w1-b8aee1/distinct-w1-b8aee1-kserve-5b857c55cd-x5b7b" Oct 30 20:53:20.934121 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:20.934064 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1599e588-def8-42e1-b81d-ad0aaf119818-dshm\") pod \"distinct-w1-b8aee1-kserve-5b857c55cd-x5b7b\" (UID: \"1599e588-def8-42e1-b81d-ad0aaf119818\") " pod="e2e-models-e2e-worker-w1-b8aee1/distinct-w1-b8aee1-kserve-5b857c55cd-x5b7b" Oct 30 20:53:20.934121 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:20.934091 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1599e588-def8-42e1-b81d-ad0aaf119818-home\") pod \"distinct-w1-b8aee1-kserve-5b857c55cd-x5b7b\" (UID: \"1599e588-def8-42e1-b81d-ad0aaf119818\") " pod="e2e-models-e2e-worker-w1-b8aee1/distinct-w1-b8aee1-kserve-5b857c55cd-x5b7b" Oct 30 20:53:20.934268 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:20.934245 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1599e588-def8-42e1-b81d-ad0aaf119818-tls-certs\") pod \"distinct-w1-b8aee1-kserve-5b857c55cd-x5b7b\" (UID: \"1599e588-def8-42e1-b81d-ad0aaf119818\") " pod="e2e-models-e2e-worker-w1-b8aee1/distinct-w1-b8aee1-kserve-5b857c55cd-x5b7b" Oct 30 20:53:20.934373 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:20.934356 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1599e588-def8-42e1-b81d-ad0aaf119818-model-cache\") pod \"distinct-w1-b8aee1-kserve-5b857c55cd-x5b7b\" (UID: \"1599e588-def8-42e1-b81d-ad0aaf119818\") " pod="e2e-models-e2e-worker-w1-b8aee1/distinct-w1-b8aee1-kserve-5b857c55cd-x5b7b" Oct 30 20:53:20.934544 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:20.934493 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1599e588-def8-42e1-b81d-ad0aaf119818-home\") pod \"distinct-w1-b8aee1-kserve-5b857c55cd-x5b7b\" (UID: \"1599e588-def8-42e1-b81d-ad0aaf119818\") " pod="e2e-models-e2e-worker-w1-b8aee1/distinct-w1-b8aee1-kserve-5b857c55cd-x5b7b" Oct 30 20:53:20.934691 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:20.934675 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1599e588-def8-42e1-b81d-ad0aaf119818-tmp-dir\") pod \"distinct-w1-b8aee1-kserve-5b857c55cd-x5b7b\" (UID: \"1599e588-def8-42e1-b81d-ad0aaf119818\") " pod="e2e-models-e2e-worker-w1-b8aee1/distinct-w1-b8aee1-kserve-5b857c55cd-x5b7b" Oct 30 20:53:20.936288 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:20.936261 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1599e588-def8-42e1-b81d-ad0aaf119818-dshm\") pod \"distinct-w1-b8aee1-kserve-5b857c55cd-x5b7b\" (UID: \"1599e588-def8-42e1-b81d-ad0aaf119818\") " pod="e2e-models-e2e-worker-w1-b8aee1/distinct-w1-b8aee1-kserve-5b857c55cd-x5b7b" Oct 30 20:53:20.936617 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:20.936598 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1599e588-def8-42e1-b81d-ad0aaf119818-tls-certs\") pod \"distinct-w1-b8aee1-kserve-5b857c55cd-x5b7b\" (UID: \"1599e588-def8-42e1-b81d-ad0aaf119818\") " pod="e2e-models-e2e-worker-w1-b8aee1/distinct-w1-b8aee1-kserve-5b857c55cd-x5b7b" Oct 30 20:53:20.945184 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:20.945156 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5q5dm\" (UniqueName: \"kubernetes.io/projected/1599e588-def8-42e1-b81d-ad0aaf119818-kube-api-access-5q5dm\") pod \"distinct-w1-b8aee1-kserve-5b857c55cd-x5b7b\" (UID: \"1599e588-def8-42e1-b81d-ad0aaf119818\") " pod="e2e-models-e2e-worker-w1-b8aee1/distinct-w1-b8aee1-kserve-5b857c55cd-x5b7b" Oct 30 20:53:21.118476 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:21.118425 4611 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-b8aee1/distinct-w1-b8aee1-kserve-5b857c55cd-x5b7b" Oct 30 20:53:21.317822 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:21.317789 4611 patch_prober.go:29] interesting pod/e2e-worker-w2-db534b-openshift-default-85df4d79d9-kb6vb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body= Oct 30 20:53:21.318003 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:21.317836 4611 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-kb6vb" podUID="41f0d1eb-188b-404a-8b31-cf9dc16b9fbc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" Oct 30 20:53:21.475379 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:21.475347 4611 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-b8aee1/distinct-w1-b8aee1-kserve-5b857c55cd-x5b7b"] Oct 30 20:53:21.577531 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:21.577319 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:53:22.001141 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:22.001102 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-b8aee1/distinct-w1-b8aee1-kserve-5b857c55cd-x5b7b" event={"ID":"1599e588-def8-42e1-b81d-ad0aaf119818","Type":"ContainerStarted","Data":"01d7ae87a3fb5dd04f1360b41c674bdf52a66813eea06d10c78c39bc5f491c60"} Oct 30 20:53:22.001571 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:22.001152 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-b8aee1/distinct-w1-b8aee1-kserve-5b857c55cd-x5b7b" event={"ID":"1599e588-def8-42e1-b81d-ad0aaf119818","Type":"ContainerStarted","Data":"9eb14d5ee13e6b3fc9579fc92ac26c699431277b1f570912ac5a28b4480ac055"} Oct 30 20:53:22.020453 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:22.020410 4611 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-b8aee1/distinct-w1-b8aee1-kserve-5b857c55cd-x5b7b" podStartSLOduration=2.020396025 podStartE2EDuration="2.020396025s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:53:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:53:22.018932612 +0000 UTC m=+856.552959230" watchObservedRunningTime="2026-10-30 20:53:22.020396025 +0000 UTC m=+856.554422640" Oct 30 20:53:22.118972 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:22.118944 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b8aee1/distinct-w1-b8aee1-kserve-5b857c55cd-x5b7b" Oct 30 20:53:22.317756 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:22.317670 4611 patch_prober.go:29] interesting pod/e2e-worker-w2-db534b-openshift-default-85df4d79d9-kb6vb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body= Oct 30 20:53:22.317756 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:22.317727 4611 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-kb6vb" podUID="41f0d1eb-188b-404a-8b31-cf9dc16b9fbc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" Oct 30 20:53:22.721160 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:22.721116 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:53:23.317153 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:23.317117 4611 patch_prober.go:29] interesting pod/e2e-worker-w2-db534b-openshift-default-85df4d79d9-kb6vb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body= Oct 30 20:53:23.317606 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:23.317171 4611 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-kb6vb" podUID="41f0d1eb-188b-404a-8b31-cf9dc16b9fbc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" Oct 30 20:53:23.679658 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:23.679623 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:53:24.317722 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:24.317694 4611 patch_prober.go:29] interesting pod/e2e-worker-w2-db534b-openshift-default-85df4d79d9-kb6vb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body= Oct 30 20:53:24.318112 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:24.317741 4611 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-kb6vb" podUID="41f0d1eb-188b-404a-8b31-cf9dc16b9fbc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" Oct 30 20:53:24.609129 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:24.609083 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:53:25.321726 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:25.321696 4611 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-kb6vb" Oct 30 20:53:25.322158 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:25.322136 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-kb6vb" Oct 30 20:53:25.322880 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:25.322857 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-kb6vb" Oct 30 20:53:25.429053 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:25.429017 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:53:26.068221 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:26.068185 4611 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-db534b/distinct-2-w2-db534b-kserve-8594c876-g92mh"] Oct 30 20:53:26.073180 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:26.073156 4611 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-db534b/distinct-2-w2-db534b-kserve-8594c876-g92mh" Oct 30 20:53:26.080333 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:26.080309 4611 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-db534b\"/\"default-dockercfg-z8rc4\"" Oct 30 20:53:26.080670 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:26.080651 4611 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-db534b\"/\"kube-root-ca.crt\"" Oct 30 20:53:26.080869 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:26.080851 4611 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-db534b\"/\"openshift-service-ca.crt\"" Oct 30 20:53:26.081161 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:26.081136 4611 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-db534b\"/\"distinct-2-w2-db534b-kserve-self-signed-certs\"" Oct 30 20:53:26.083028 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:26.083003 4611 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-db534b/distinct-2-w2-db534b-kserve-8594c876-g92mh"] Oct 30 20:53:26.157142 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:26.157115 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/955316d6-a9fd-4b75-a8ab-bbf382b776e2-dshm\") pod \"distinct-2-w2-db534b-kserve-8594c876-g92mh\" (UID: \"955316d6-a9fd-4b75-a8ab-bbf382b776e2\") " pod="e2e-models-e2e-worker-w2-db534b/distinct-2-w2-db534b-kserve-8594c876-g92mh" Oct 30 20:53:26.157288 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:26.157269 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/955316d6-a9fd-4b75-a8ab-bbf382b776e2-home\") pod \"distinct-2-w2-db534b-kserve-8594c876-g92mh\" (UID: \"955316d6-a9fd-4b75-a8ab-bbf382b776e2\") " pod="e2e-models-e2e-worker-w2-db534b/distinct-2-w2-db534b-kserve-8594c876-g92mh" Oct 30 20:53:26.157328 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:26.157303 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/955316d6-a9fd-4b75-a8ab-bbf382b776e2-model-cache\") pod \"distinct-2-w2-db534b-kserve-8594c876-g92mh\" (UID: \"955316d6-a9fd-4b75-a8ab-bbf382b776e2\") " pod="e2e-models-e2e-worker-w2-db534b/distinct-2-w2-db534b-kserve-8594c876-g92mh" Oct 30 20:53:26.157535 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:26.157482 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/955316d6-a9fd-4b75-a8ab-bbf382b776e2-tmp-dir\") pod \"distinct-2-w2-db534b-kserve-8594c876-g92mh\" (UID: \"955316d6-a9fd-4b75-a8ab-bbf382b776e2\") " pod="e2e-models-e2e-worker-w2-db534b/distinct-2-w2-db534b-kserve-8594c876-g92mh" Oct 30 20:53:26.157674 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:26.157592 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pghmd\" (UniqueName: \"kubernetes.io/projected/955316d6-a9fd-4b75-a8ab-bbf382b776e2-kube-api-access-pghmd\") pod \"distinct-2-w2-db534b-kserve-8594c876-g92mh\" (UID: \"955316d6-a9fd-4b75-a8ab-bbf382b776e2\") " pod="e2e-models-e2e-worker-w2-db534b/distinct-2-w2-db534b-kserve-8594c876-g92mh" Oct 30 20:53:26.157674 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:26.157634 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/955316d6-a9fd-4b75-a8ab-bbf382b776e2-tls-certs\") pod \"distinct-2-w2-db534b-kserve-8594c876-g92mh\" (UID: \"955316d6-a9fd-4b75-a8ab-bbf382b776e2\") " pod="e2e-models-e2e-worker-w2-db534b/distinct-2-w2-db534b-kserve-8594c876-g92mh" Oct 30 20:53:26.159894 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:26.159845 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:53:26.258708 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:26.258680 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/955316d6-a9fd-4b75-a8ab-bbf382b776e2-tmp-dir\") pod \"distinct-2-w2-db534b-kserve-8594c876-g92mh\" (UID: \"955316d6-a9fd-4b75-a8ab-bbf382b776e2\") " pod="e2e-models-e2e-worker-w2-db534b/distinct-2-w2-db534b-kserve-8594c876-g92mh" Oct 30 20:53:26.258801 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:26.258730 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pghmd\" (UniqueName: \"kubernetes.io/projected/955316d6-a9fd-4b75-a8ab-bbf382b776e2-kube-api-access-pghmd\") pod \"distinct-2-w2-db534b-kserve-8594c876-g92mh\" (UID: \"955316d6-a9fd-4b75-a8ab-bbf382b776e2\") " pod="e2e-models-e2e-worker-w2-db534b/distinct-2-w2-db534b-kserve-8594c876-g92mh" Oct 30 20:53:26.258801 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:26.258757 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/955316d6-a9fd-4b75-a8ab-bbf382b776e2-tls-certs\") pod \"distinct-2-w2-db534b-kserve-8594c876-g92mh\" (UID: \"955316d6-a9fd-4b75-a8ab-bbf382b776e2\") " pod="e2e-models-e2e-worker-w2-db534b/distinct-2-w2-db534b-kserve-8594c876-g92mh" Oct 30 20:53:26.258801 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:26.258790 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/955316d6-a9fd-4b75-a8ab-bbf382b776e2-dshm\") pod \"distinct-2-w2-db534b-kserve-8594c876-g92mh\" (UID: \"955316d6-a9fd-4b75-a8ab-bbf382b776e2\") " pod="e2e-models-e2e-worker-w2-db534b/distinct-2-w2-db534b-kserve-8594c876-g92mh" Oct 30 20:53:26.258927 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:26.258890 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/955316d6-a9fd-4b75-a8ab-bbf382b776e2-home\") pod \"distinct-2-w2-db534b-kserve-8594c876-g92mh\" (UID: \"955316d6-a9fd-4b75-a8ab-bbf382b776e2\") " pod="e2e-models-e2e-worker-w2-db534b/distinct-2-w2-db534b-kserve-8594c876-g92mh" Oct 30 20:53:26.258927 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:26.258921 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/955316d6-a9fd-4b75-a8ab-bbf382b776e2-model-cache\") pod \"distinct-2-w2-db534b-kserve-8594c876-g92mh\" (UID: \"955316d6-a9fd-4b75-a8ab-bbf382b776e2\") " pod="e2e-models-e2e-worker-w2-db534b/distinct-2-w2-db534b-kserve-8594c876-g92mh" Oct 30 20:53:26.259081 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:26.259058 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/955316d6-a9fd-4b75-a8ab-bbf382b776e2-tmp-dir\") pod \"distinct-2-w2-db534b-kserve-8594c876-g92mh\" (UID: \"955316d6-a9fd-4b75-a8ab-bbf382b776e2\") " pod="e2e-models-e2e-worker-w2-db534b/distinct-2-w2-db534b-kserve-8594c876-g92mh" Oct 30 20:53:26.259317 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:26.259295 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/955316d6-a9fd-4b75-a8ab-bbf382b776e2-home\") pod \"distinct-2-w2-db534b-kserve-8594c876-g92mh\" (UID: \"955316d6-a9fd-4b75-a8ab-bbf382b776e2\") " pod="e2e-models-e2e-worker-w2-db534b/distinct-2-w2-db534b-kserve-8594c876-g92mh" Oct 30 20:53:26.259388 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:26.259295 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/955316d6-a9fd-4b75-a8ab-bbf382b776e2-model-cache\") pod \"distinct-2-w2-db534b-kserve-8594c876-g92mh\" (UID: \"955316d6-a9fd-4b75-a8ab-bbf382b776e2\") " pod="e2e-models-e2e-worker-w2-db534b/distinct-2-w2-db534b-kserve-8594c876-g92mh" Oct 30 20:53:26.261076 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:26.261056 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/955316d6-a9fd-4b75-a8ab-bbf382b776e2-dshm\") pod \"distinct-2-w2-db534b-kserve-8594c876-g92mh\" (UID: \"955316d6-a9fd-4b75-a8ab-bbf382b776e2\") " pod="e2e-models-e2e-worker-w2-db534b/distinct-2-w2-db534b-kserve-8594c876-g92mh" Oct 30 20:53:26.261459 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:26.261438 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/955316d6-a9fd-4b75-a8ab-bbf382b776e2-tls-certs\") pod \"distinct-2-w2-db534b-kserve-8594c876-g92mh\" (UID: \"955316d6-a9fd-4b75-a8ab-bbf382b776e2\") " pod="e2e-models-e2e-worker-w2-db534b/distinct-2-w2-db534b-kserve-8594c876-g92mh" Oct 30 20:53:26.270112 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:26.270093 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pghmd\" (UniqueName: \"kubernetes.io/projected/955316d6-a9fd-4b75-a8ab-bbf382b776e2-kube-api-access-pghmd\") pod \"distinct-2-w2-db534b-kserve-8594c876-g92mh\" (UID: \"955316d6-a9fd-4b75-a8ab-bbf382b776e2\") " pod="e2e-models-e2e-worker-w2-db534b/distinct-2-w2-db534b-kserve-8594c876-g92mh" Oct 30 20:53:26.305977 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:26.305931 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:53:26.391761 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:26.391716 4611 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-db534b/distinct-2-w2-db534b-kserve-8594c876-g92mh" Oct 30 20:53:26.752779 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:26.752749 4611 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-db534b/distinct-2-w2-db534b-kserve-8594c876-g92mh"] Oct 30 20:53:26.753644 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:53:26.753603 4611 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod955316d6_a9fd_4b75_a8ab_bbf382b776e2.slice/crio-577337b5f2ab13ba375a980ecd8e69f45abc26b876911385314607b4663f77a3 WatchSource:0}: Error finding container 577337b5f2ab13ba375a980ecd8e69f45abc26b876911385314607b4663f77a3: Status 404 returned error can't find the container with id 577337b5f2ab13ba375a980ecd8e69f45abc26b876911385314607b4663f77a3 Oct 30 20:53:27.033867 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:27.033775 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-db534b/distinct-2-w2-db534b-kserve-8594c876-g92mh" event={"ID":"955316d6-a9fd-4b75-a8ab-bbf382b776e2","Type":"ContainerStarted","Data":"697d51feb2e4144a1e3439ee261d8df6a1f9e97c03d30e623fedfa108d15479c"} Oct 30 20:53:27.033867 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:27.033819 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-db534b/distinct-2-w2-db534b-kserve-8594c876-g92mh" event={"ID":"955316d6-a9fd-4b75-a8ab-bbf382b776e2","Type":"ContainerStarted","Data":"577337b5f2ab13ba375a980ecd8e69f45abc26b876911385314607b4663f77a3"} Oct 30 20:53:27.054335 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:27.054285 4611 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-db534b/distinct-2-w2-db534b-kserve-8594c876-g92mh" podStartSLOduration=1.054270978 podStartE2EDuration="1.054270978s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:53:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:53:27.051784719 +0000 UTC m=+861.585811336" watchObservedRunningTime="2026-10-30 20:53:27.054270978 +0000 UTC m=+861.588297594" Oct 30 20:53:27.194105 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:27.194066 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:53:27.392235 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:27.392209 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-db534b/distinct-2-w2-db534b-kserve-8594c876-g92mh" Oct 30 20:53:28.020565 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:28.020523 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:53:28.333878 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:28.333804 4611 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-f2ed1e/distinct-w4-f2ed1e-kserve-57cff9dcf8-8d5zl" Oct 30 20:53:28.346362 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:28.346340 4611 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-f2ed1e/distinct-w4-f2ed1e-kserve-57cff9dcf8-8d5zl" Oct 30 20:53:28.715396 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:28.715360 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:53:29.058965 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:29.058893 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-f2ed1e/distinct-w4-f2ed1e-kserve-57cff9dcf8-8d5zl" Oct 30 20:53:29.409470 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:29.409435 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:53:30.091467 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:30.091422 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:53:30.785055 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:30.785027 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:53:31.118979 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:31.118949 4611 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-b8aee1/distinct-w1-b8aee1-kserve-5b857c55cd-x5b7b" Oct 30 20:53:31.132655 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:31.132628 4611 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-b8aee1/distinct-w1-b8aee1-kserve-5b857c55cd-x5b7b" Oct 30 20:53:31.532595 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:31.532519 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:53:32.092014 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:32.091986 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b8aee1/distinct-w1-b8aee1-kserve-5b857c55cd-x5b7b" Oct 30 20:53:32.153470 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:32.153435 4611 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-ccszm"] Oct 30 20:53:32.168974 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:32.168943 4611 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-ccszm" Oct 30 20:53:32.302570 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:32.302533 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69e69ba0-b2fd-4201-8c22-5ecac952d4bb-istiod-ca-cert\") pod \"e2e-worker-w5-14cca2-openshift-default-d7f87db59-ccszm\" (UID: \"69e69ba0-b2fd-4201-8c22-5ecac952d4bb\") " pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-ccszm" Oct 30 20:53:32.303003 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:32.302979 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69e69ba0-b2fd-4201-8c22-5ecac952d4bb-workload-socket\") pod \"e2e-worker-w5-14cca2-openshift-default-d7f87db59-ccszm\" (UID: \"69e69ba0-b2fd-4201-8c22-5ecac952d4bb\") " pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-ccszm" Oct 30 20:53:32.303206 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:32.303187 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69e69ba0-b2fd-4201-8c22-5ecac952d4bb-credential-socket\") pod \"e2e-worker-w5-14cca2-openshift-default-d7f87db59-ccszm\" (UID: \"69e69ba0-b2fd-4201-8c22-5ecac952d4bb\") " pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-ccszm" Oct 30 20:53:32.304475 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:32.304451 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j965r\" (UniqueName: \"kubernetes.io/projected/69e69ba0-b2fd-4201-8c22-5ecac952d4bb-kube-api-access-j965r\") pod \"e2e-worker-w5-14cca2-openshift-default-d7f87db59-ccszm\" (UID: \"69e69ba0-b2fd-4201-8c22-5ecac952d4bb\") " pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-ccszm" Oct 30 20:53:32.304652 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:32.304633 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69e69ba0-b2fd-4201-8c22-5ecac952d4bb-istio-token\") pod \"e2e-worker-w5-14cca2-openshift-default-d7f87db59-ccszm\" (UID: \"69e69ba0-b2fd-4201-8c22-5ecac952d4bb\") " pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-ccszm" Oct 30 20:53:32.305299 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:32.305277 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69e69ba0-b2fd-4201-8c22-5ecac952d4bb-istio-data\") pod \"e2e-worker-w5-14cca2-openshift-default-d7f87db59-ccszm\" (UID: \"69e69ba0-b2fd-4201-8c22-5ecac952d4bb\") " pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-ccszm" Oct 30 20:53:32.305438 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:32.305420 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69e69ba0-b2fd-4201-8c22-5ecac952d4bb-workload-certs\") pod \"e2e-worker-w5-14cca2-openshift-default-d7f87db59-ccszm\" (UID: \"69e69ba0-b2fd-4201-8c22-5ecac952d4bb\") " pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-ccszm" Oct 30 20:53:32.305639 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:32.305618 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69e69ba0-b2fd-4201-8c22-5ecac952d4bb-istio-envoy\") pod \"e2e-worker-w5-14cca2-openshift-default-d7f87db59-ccszm\" (UID: \"69e69ba0-b2fd-4201-8c22-5ecac952d4bb\") " pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-ccszm" Oct 30 20:53:32.306068 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:32.306047 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69e69ba0-b2fd-4201-8c22-5ecac952d4bb-istio-podinfo\") pod \"e2e-worker-w5-14cca2-openshift-default-d7f87db59-ccszm\" (UID: \"69e69ba0-b2fd-4201-8c22-5ecac952d4bb\") " pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-ccszm" Oct 30 20:53:32.308903 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:32.308884 4611 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-ccszm"] Oct 30 20:53:32.409533 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:32.408928 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:53:32.415536 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:32.415197 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69e69ba0-b2fd-4201-8c22-5ecac952d4bb-istiod-ca-cert\") pod \"e2e-worker-w5-14cca2-openshift-default-d7f87db59-ccszm\" (UID: \"69e69ba0-b2fd-4201-8c22-5ecac952d4bb\") " pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-ccszm" Oct 30 20:53:32.415536 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:32.415356 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69e69ba0-b2fd-4201-8c22-5ecac952d4bb-workload-socket\") pod \"e2e-worker-w5-14cca2-openshift-default-d7f87db59-ccszm\" (UID: \"69e69ba0-b2fd-4201-8c22-5ecac952d4bb\") " pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-ccszm" Oct 30 20:53:32.415536 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:32.415420 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69e69ba0-b2fd-4201-8c22-5ecac952d4bb-credential-socket\") pod \"e2e-worker-w5-14cca2-openshift-default-d7f87db59-ccszm\" (UID: \"69e69ba0-b2fd-4201-8c22-5ecac952d4bb\") " pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-ccszm" Oct 30 20:53:32.415837 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:32.415545 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j965r\" (UniqueName: \"kubernetes.io/projected/69e69ba0-b2fd-4201-8c22-5ecac952d4bb-kube-api-access-j965r\") pod \"e2e-worker-w5-14cca2-openshift-default-d7f87db59-ccszm\" (UID: \"69e69ba0-b2fd-4201-8c22-5ecac952d4bb\") " pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-ccszm" Oct 30 20:53:32.415837 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:32.415572 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69e69ba0-b2fd-4201-8c22-5ecac952d4bb-istio-token\") pod \"e2e-worker-w5-14cca2-openshift-default-d7f87db59-ccszm\" (UID: \"69e69ba0-b2fd-4201-8c22-5ecac952d4bb\") " pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-ccszm" Oct 30 20:53:32.415837 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:32.415618 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69e69ba0-b2fd-4201-8c22-5ecac952d4bb-istio-data\") pod \"e2e-worker-w5-14cca2-openshift-default-d7f87db59-ccszm\" (UID: \"69e69ba0-b2fd-4201-8c22-5ecac952d4bb\") " pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-ccszm" Oct 30 20:53:32.415837 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:32.415643 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69e69ba0-b2fd-4201-8c22-5ecac952d4bb-workload-certs\") pod \"e2e-worker-w5-14cca2-openshift-default-d7f87db59-ccszm\" (UID: \"69e69ba0-b2fd-4201-8c22-5ecac952d4bb\") " pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-ccszm" Oct 30 20:53:32.415837 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:32.415666 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69e69ba0-b2fd-4201-8c22-5ecac952d4bb-istio-envoy\") pod \"e2e-worker-w5-14cca2-openshift-default-d7f87db59-ccszm\" (UID: \"69e69ba0-b2fd-4201-8c22-5ecac952d4bb\") " pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-ccszm" Oct 30 20:53:32.415837 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:32.415705 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69e69ba0-b2fd-4201-8c22-5ecac952d4bb-istio-podinfo\") pod \"e2e-worker-w5-14cca2-openshift-default-d7f87db59-ccszm\" (UID: \"69e69ba0-b2fd-4201-8c22-5ecac952d4bb\") " pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-ccszm" Oct 30 20:53:32.418978 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:32.417896 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69e69ba0-b2fd-4201-8c22-5ecac952d4bb-istiod-ca-cert\") pod \"e2e-worker-w5-14cca2-openshift-default-d7f87db59-ccszm\" (UID: \"69e69ba0-b2fd-4201-8c22-5ecac952d4bb\") " pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-ccszm" Oct 30 20:53:32.418978 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:32.418202 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69e69ba0-b2fd-4201-8c22-5ecac952d4bb-workload-socket\") pod \"e2e-worker-w5-14cca2-openshift-default-d7f87db59-ccszm\" (UID: \"69e69ba0-b2fd-4201-8c22-5ecac952d4bb\") " pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-ccszm" Oct 30 20:53:32.418978 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:32.418550 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69e69ba0-b2fd-4201-8c22-5ecac952d4bb-credential-socket\") pod \"e2e-worker-w5-14cca2-openshift-default-d7f87db59-ccszm\" (UID: \"69e69ba0-b2fd-4201-8c22-5ecac952d4bb\") " pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-ccszm" Oct 30 20:53:32.418978 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:32.418801 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69e69ba0-b2fd-4201-8c22-5ecac952d4bb-istio-data\") pod \"e2e-worker-w5-14cca2-openshift-default-d7f87db59-ccszm\" (UID: \"69e69ba0-b2fd-4201-8c22-5ecac952d4bb\") " pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-ccszm" Oct 30 20:53:32.420042 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:32.419744 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69e69ba0-b2fd-4201-8c22-5ecac952d4bb-workload-certs\") pod \"e2e-worker-w5-14cca2-openshift-default-d7f87db59-ccszm\" (UID: \"69e69ba0-b2fd-4201-8c22-5ecac952d4bb\") " pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-ccszm" Oct 30 20:53:32.427603 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:32.427551 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69e69ba0-b2fd-4201-8c22-5ecac952d4bb-istio-podinfo\") pod \"e2e-worker-w5-14cca2-openshift-default-d7f87db59-ccszm\" (UID: \"69e69ba0-b2fd-4201-8c22-5ecac952d4bb\") " pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-ccszm" Oct 30 20:53:32.435883 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:32.435827 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69e69ba0-b2fd-4201-8c22-5ecac952d4bb-istio-envoy\") pod \"e2e-worker-w5-14cca2-openshift-default-d7f87db59-ccszm\" (UID: \"69e69ba0-b2fd-4201-8c22-5ecac952d4bb\") " pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-ccszm" Oct 30 20:53:32.440819 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:32.440765 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69e69ba0-b2fd-4201-8c22-5ecac952d4bb-istio-token\") pod \"e2e-worker-w5-14cca2-openshift-default-d7f87db59-ccszm\" (UID: \"69e69ba0-b2fd-4201-8c22-5ecac952d4bb\") " pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-ccszm" Oct 30 20:53:32.441423 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:32.441399 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j965r\" (UniqueName: \"kubernetes.io/projected/69e69ba0-b2fd-4201-8c22-5ecac952d4bb-kube-api-access-j965r\") pod \"e2e-worker-w5-14cca2-openshift-default-d7f87db59-ccszm\" (UID: \"69e69ba0-b2fd-4201-8c22-5ecac952d4bb\") " pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-ccszm" Oct 30 20:53:32.606530 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:32.604243 4611 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-ccszm" Oct 30 20:53:33.145863 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:33.145820 4611 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-ccszm"] Oct 30 20:53:33.145863 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:53:33.145837 4611 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69e69ba0_b2fd_4201_8c22_5ecac952d4bb.slice/crio-49f62b300d3c8db8bd1a4e53cf63735443b25131368d849b608f6628dc8c83e2 WatchSource:0}: Error finding container 49f62b300d3c8db8bd1a4e53cf63735443b25131368d849b608f6628dc8c83e2: Status 404 returned error can't find the container with id 49f62b300d3c8db8bd1a4e53cf63735443b25131368d849b608f6628dc8c83e2 Oct 30 20:53:33.148901 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:33.148862 4611 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:53:33.148981 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:33.148944 4611 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:53:33.149017 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:33.148989 4611 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:53:33.302566 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:33.302521 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:53:33.917473 ip-10-0-133-192 kubenswrapper[4611]: E1030 20:53:33.917440 4611 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hwzc6" containerName="extractor" Oct 30 20:53:34.015213 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:34.015164 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:53:34.102969 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:34.102935 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-ccszm" event={"ID":"69e69ba0-b2fd-4201-8c22-5ecac952d4bb","Type":"ContainerStarted","Data":"9aab1ebf1ca04f57868eba0f42a0d2ede0cdd21b941a5105b9671a0dbc642a80"} Oct 30 20:53:34.103159 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:34.102991 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-ccszm" event={"ID":"69e69ba0-b2fd-4201-8c22-5ecac952d4bb","Type":"ContainerStarted","Data":"49f62b300d3c8db8bd1a4e53cf63735443b25131368d849b608f6628dc8c83e2"} Oct 30 20:53:34.128292 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:34.128234 4611 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-ccszm" podStartSLOduration=2.128220675 podStartE2EDuration="2.128220675s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:53:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:53:34.124700541 +0000 UTC m=+868.658727157" watchObservedRunningTime="2026-10-30 20:53:34.128220675 +0000 UTC m=+868.662247291" Oct 30 20:53:34.541810 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:34.541769 4611 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-3d909d/distinct-2-w6-3d909d-kserve-59c5d5c595-cjg2m"] Oct 30 20:53:34.568563 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:34.554806 4611 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-3d909d/distinct-2-w6-3d909d-kserve-59c5d5c595-cjg2m" Oct 30 20:53:34.580941 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:34.580908 4611 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-3d909d/distinct-2-w6-3d909d-kserve-59c5d5c595-cjg2m"] Oct 30 20:53:34.583523 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:34.583482 4611 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-3d909d\"/\"distinct-2-w6-3d909d-kserve-self-signed-certs\"" Oct 30 20:53:34.584010 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:34.583981 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e2b4c221-ee94-4131-abd8-2313647c641e-home\") pod \"distinct-2-w6-3d909d-kserve-59c5d5c595-cjg2m\" (UID: \"e2b4c221-ee94-4131-abd8-2313647c641e\") " pod="e2e-models-e2e-worker-w6-3d909d/distinct-2-w6-3d909d-kserve-59c5d5c595-cjg2m" Oct 30 20:53:34.584132 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:34.584020 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e2b4c221-ee94-4131-abd8-2313647c641e-tmp-dir\") pod \"distinct-2-w6-3d909d-kserve-59c5d5c595-cjg2m\" (UID: \"e2b4c221-ee94-4131-abd8-2313647c641e\") " pod="e2e-models-e2e-worker-w6-3d909d/distinct-2-w6-3d909d-kserve-59c5d5c595-cjg2m" Oct 30 20:53:34.584132 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:34.584100 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gn9vx\" (UniqueName: \"kubernetes.io/projected/e2b4c221-ee94-4131-abd8-2313647c641e-kube-api-access-gn9vx\") pod \"distinct-2-w6-3d909d-kserve-59c5d5c595-cjg2m\" (UID: \"e2b4c221-ee94-4131-abd8-2313647c641e\") " pod="e2e-models-e2e-worker-w6-3d909d/distinct-2-w6-3d909d-kserve-59c5d5c595-cjg2m" Oct 30 20:53:34.584289 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:34.584267 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e2b4c221-ee94-4131-abd8-2313647c641e-tls-certs\") pod \"distinct-2-w6-3d909d-kserve-59c5d5c595-cjg2m\" (UID: \"e2b4c221-ee94-4131-abd8-2313647c641e\") " pod="e2e-models-e2e-worker-w6-3d909d/distinct-2-w6-3d909d-kserve-59c5d5c595-cjg2m" Oct 30 20:53:34.584346 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:34.584312 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e2b4c221-ee94-4131-abd8-2313647c641e-model-cache\") pod \"distinct-2-w6-3d909d-kserve-59c5d5c595-cjg2m\" (UID: \"e2b4c221-ee94-4131-abd8-2313647c641e\") " pod="e2e-models-e2e-worker-w6-3d909d/distinct-2-w6-3d909d-kserve-59c5d5c595-cjg2m" Oct 30 20:53:34.584410 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:34.584364 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e2b4c221-ee94-4131-abd8-2313647c641e-dshm\") pod \"distinct-2-w6-3d909d-kserve-59c5d5c595-cjg2m\" (UID: \"e2b4c221-ee94-4131-abd8-2313647c641e\") " pod="e2e-models-e2e-worker-w6-3d909d/distinct-2-w6-3d909d-kserve-59c5d5c595-cjg2m" Oct 30 20:53:34.605171 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:34.605153 4611 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-ccszm" Oct 30 20:53:34.606221 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:34.606200 4611 patch_prober.go:29] interesting pod/e2e-worker-w5-14cca2-openshift-default-d7f87db59-ccszm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Oct 30 20:53:34.606320 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:34.606249 4611 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-ccszm" podUID="69e69ba0-b2fd-4201-8c22-5ecac952d4bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Oct 30 20:53:34.636472 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:34.636448 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:53:34.685657 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:34.685618 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e2b4c221-ee94-4131-abd8-2313647c641e-tls-certs\") pod \"distinct-2-w6-3d909d-kserve-59c5d5c595-cjg2m\" (UID: \"e2b4c221-ee94-4131-abd8-2313647c641e\") " pod="e2e-models-e2e-worker-w6-3d909d/distinct-2-w6-3d909d-kserve-59c5d5c595-cjg2m" Oct 30 20:53:34.689766 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:34.689737 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e2b4c221-ee94-4131-abd8-2313647c641e-model-cache\") pod \"distinct-2-w6-3d909d-kserve-59c5d5c595-cjg2m\" (UID: \"e2b4c221-ee94-4131-abd8-2313647c641e\") " pod="e2e-models-e2e-worker-w6-3d909d/distinct-2-w6-3d909d-kserve-59c5d5c595-cjg2m" Oct 30 20:53:34.689766 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:34.689109 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e2b4c221-ee94-4131-abd8-2313647c641e-tls-certs\") pod \"distinct-2-w6-3d909d-kserve-59c5d5c595-cjg2m\" (UID: \"e2b4c221-ee94-4131-abd8-2313647c641e\") " pod="e2e-models-e2e-worker-w6-3d909d/distinct-2-w6-3d909d-kserve-59c5d5c595-cjg2m" Oct 30 20:53:34.689992 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:34.689833 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e2b4c221-ee94-4131-abd8-2313647c641e-dshm\") pod \"distinct-2-w6-3d909d-kserve-59c5d5c595-cjg2m\" (UID: \"e2b4c221-ee94-4131-abd8-2313647c641e\") " pod="e2e-models-e2e-worker-w6-3d909d/distinct-2-w6-3d909d-kserve-59c5d5c595-cjg2m" Oct 30 20:53:34.690086 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:34.690067 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e2b4c221-ee94-4131-abd8-2313647c641e-home\") pod \"distinct-2-w6-3d909d-kserve-59c5d5c595-cjg2m\" (UID: \"e2b4c221-ee94-4131-abd8-2313647c641e\") " pod="e2e-models-e2e-worker-w6-3d909d/distinct-2-w6-3d909d-kserve-59c5d5c595-cjg2m" Oct 30 20:53:34.690186 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:34.690095 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e2b4c221-ee94-4131-abd8-2313647c641e-model-cache\") pod \"distinct-2-w6-3d909d-kserve-59c5d5c595-cjg2m\" (UID: \"e2b4c221-ee94-4131-abd8-2313647c641e\") " pod="e2e-models-e2e-worker-w6-3d909d/distinct-2-w6-3d909d-kserve-59c5d5c595-cjg2m" Oct 30 20:53:34.690186 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:34.690098 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e2b4c221-ee94-4131-abd8-2313647c641e-tmp-dir\") pod \"distinct-2-w6-3d909d-kserve-59c5d5c595-cjg2m\" (UID: \"e2b4c221-ee94-4131-abd8-2313647c641e\") " pod="e2e-models-e2e-worker-w6-3d909d/distinct-2-w6-3d909d-kserve-59c5d5c595-cjg2m" Oct 30 20:53:34.690186 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:34.690159 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gn9vx\" (UniqueName: \"kubernetes.io/projected/e2b4c221-ee94-4131-abd8-2313647c641e-kube-api-access-gn9vx\") pod \"distinct-2-w6-3d909d-kserve-59c5d5c595-cjg2m\" (UID: \"e2b4c221-ee94-4131-abd8-2313647c641e\") " pod="e2e-models-e2e-worker-w6-3d909d/distinct-2-w6-3d909d-kserve-59c5d5c595-cjg2m" Oct 30 20:53:34.690588 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:34.690425 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e2b4c221-ee94-4131-abd8-2313647c641e-tmp-dir\") pod \"distinct-2-w6-3d909d-kserve-59c5d5c595-cjg2m\" (UID: \"e2b4c221-ee94-4131-abd8-2313647c641e\") " pod="e2e-models-e2e-worker-w6-3d909d/distinct-2-w6-3d909d-kserve-59c5d5c595-cjg2m" Oct 30 20:53:34.698388 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:34.690844 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e2b4c221-ee94-4131-abd8-2313647c641e-home\") pod \"distinct-2-w6-3d909d-kserve-59c5d5c595-cjg2m\" (UID: \"e2b4c221-ee94-4131-abd8-2313647c641e\") " pod="e2e-models-e2e-worker-w6-3d909d/distinct-2-w6-3d909d-kserve-59c5d5c595-cjg2m" Oct 30 20:53:34.698388 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:34.692690 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e2b4c221-ee94-4131-abd8-2313647c641e-dshm\") pod \"distinct-2-w6-3d909d-kserve-59c5d5c595-cjg2m\" (UID: \"e2b4c221-ee94-4131-abd8-2313647c641e\") " pod="e2e-models-e2e-worker-w6-3d909d/distinct-2-w6-3d909d-kserve-59c5d5c595-cjg2m" Oct 30 20:53:34.709611 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:34.709585 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gn9vx\" (UniqueName: \"kubernetes.io/projected/e2b4c221-ee94-4131-abd8-2313647c641e-kube-api-access-gn9vx\") pod \"distinct-2-w6-3d909d-kserve-59c5d5c595-cjg2m\" (UID: \"e2b4c221-ee94-4131-abd8-2313647c641e\") " pod="e2e-models-e2e-worker-w6-3d909d/distinct-2-w6-3d909d-kserve-59c5d5c595-cjg2m" Oct 30 20:53:34.891043 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:34.891004 4611 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-3d909d/distinct-2-w6-3d909d-kserve-59c5d5c595-cjg2m" Oct 30 20:53:35.040972 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:35.040935 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:53:35.195320 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:35.195297 4611 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-3d909d/distinct-2-w6-3d909d-kserve-59c5d5c595-cjg2m"] Oct 30 20:53:35.196538 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:53:35.196491 4611 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode2b4c221_ee94_4131_abd8_2313647c641e.slice/crio-fb3e9e314a5c4e05c7182b2f08f0759eda66de483ad0291abda45713839f68e8 WatchSource:0}: Error finding container fb3e9e314a5c4e05c7182b2f08f0759eda66de483ad0291abda45713839f68e8: Status 404 returned error can't find the container with id fb3e9e314a5c4e05c7182b2f08f0759eda66de483ad0291abda45713839f68e8 Oct 30 20:53:35.605274 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:35.605231 4611 patch_prober.go:29] interesting pod/e2e-worker-w5-14cca2-openshift-default-d7f87db59-ccszm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Oct 30 20:53:35.605760 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:35.605291 4611 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-ccszm" podUID="69e69ba0-b2fd-4201-8c22-5ecac952d4bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Oct 30 20:53:35.818553 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:35.818487 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:53:36.125096 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:36.125063 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-3d909d/distinct-2-w6-3d909d-kserve-59c5d5c595-cjg2m" event={"ID":"e2b4c221-ee94-4131-abd8-2313647c641e","Type":"ContainerStarted","Data":"3028a7d3b7a568f38784c3208775409d12f97dba2e31bd2a99bfde0275f82168"} Oct 30 20:53:36.125249 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:36.125110 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-3d909d/distinct-2-w6-3d909d-kserve-59c5d5c595-cjg2m" event={"ID":"e2b4c221-ee94-4131-abd8-2313647c641e","Type":"ContainerStarted","Data":"fb3e9e314a5c4e05c7182b2f08f0759eda66de483ad0291abda45713839f68e8"} Oct 30 20:53:36.174610 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:36.174230 4611 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-3d909d/distinct-2-w6-3d909d-kserve-59c5d5c595-cjg2m" podStartSLOduration=2.174211763 podStartE2EDuration="2.174211763s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:53:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:53:36.163488846 +0000 UTC m=+870.697515463" watchObservedRunningTime="2026-10-30 20:53:36.174211763 +0000 UTC m=+870.708238380" Oct 30 20:53:36.392552 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:36.392457 4611 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-db534b/distinct-2-w2-db534b-kserve-8594c876-g92mh" Oct 30 20:53:36.408153 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:36.408122 4611 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-db534b/distinct-2-w2-db534b-kserve-8594c876-g92mh" Oct 30 20:53:36.581518 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:36.581457 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:53:36.605727 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:36.605695 4611 patch_prober.go:29] interesting pod/e2e-worker-w5-14cca2-openshift-default-d7f87db59-ccszm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Oct 30 20:53:36.607317 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:36.607270 4611 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-ccszm" podUID="69e69ba0-b2fd-4201-8c22-5ecac952d4bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Oct 30 20:53:36.891411 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:36.891372 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-3d909d/distinct-2-w6-3d909d-kserve-59c5d5c595-cjg2m" Oct 30 20:53:37.143415 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:37.143339 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-db534b/distinct-2-w2-db534b-kserve-8594c876-g92mh" Oct 30 20:53:37.376532 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:37.374177 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:53:37.526344 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:37.526235 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:53:37.605003 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:37.604975 4611 patch_prober.go:29] interesting pod/e2e-worker-w5-14cca2-openshift-default-d7f87db59-ccszm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Oct 30 20:53:37.605158 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:37.605023 4611 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-ccszm" podUID="69e69ba0-b2fd-4201-8c22-5ecac952d4bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Oct 30 20:53:38.116132 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:38.116074 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:53:38.605193 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:38.605167 4611 patch_prober.go:29] interesting pod/e2e-worker-w5-14cca2-openshift-default-d7f87db59-ccszm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Oct 30 20:53:38.605348 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:38.605219 4611 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-ccszm" podUID="69e69ba0-b2fd-4201-8c22-5ecac952d4bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Oct 30 20:53:38.874345 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:38.874260 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:53:39.605718 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:39.605685 4611 patch_prober.go:29] interesting pod/e2e-worker-w5-14cca2-openshift-default-d7f87db59-ccszm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Oct 30 20:53:39.606110 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:39.605736 4611 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-ccszm" podUID="69e69ba0-b2fd-4201-8c22-5ecac952d4bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Oct 30 20:53:39.656567 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:39.656528 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:53:40.295942 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:40.295900 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:53:40.610079 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:40.610049 4611 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-ccszm" Oct 30 20:53:40.610583 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:40.610451 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-ccszm" Oct 30 20:53:40.611157 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:40.611136 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-ccszm" Oct 30 20:53:40.982627 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:40.982536 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:53:41.802310 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:41.802268 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:53:42.401629 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:42.401593 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:53:43.100017 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:43.099973 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:53:43.747427 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:43.747387 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:53:44.427179 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:44.427140 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:53:44.891583 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:44.891549 4611 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-3d909d/distinct-2-w6-3d909d-kserve-59c5d5c595-cjg2m" Oct 30 20:53:44.910002 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:44.909970 4611 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-3d909d/distinct-2-w6-3d909d-kserve-59c5d5c595-cjg2m" Oct 30 20:53:45.093647 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:45.093616 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:53:45.220840 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:45.220763 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-3d909d/distinct-2-w6-3d909d-kserve-59c5d5c595-cjg2m" Oct 30 20:53:45.797995 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:45.797963 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:53:46.380364 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:46.380326 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:53:46.478257 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:46.478220 4611 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-vpbzr"] Oct 30 20:53:46.483953 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:46.483927 4611 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-vpbzr" Oct 30 20:53:46.496715 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:46.496690 4611 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-vpbzr"] Oct 30 20:53:46.576755 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:46.576723 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6fdc9f9-8be5-4dfa-9390-3ca43a74b6be-credential-socket\") pod \"e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-vpbzr\" (UID: \"b6fdc9f9-8be5-4dfa-9390-3ca43a74b6be\") " pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-vpbzr" Oct 30 20:53:46.576937 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:46.576769 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6fdc9f9-8be5-4dfa-9390-3ca43a74b6be-workload-socket\") pod \"e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-vpbzr\" (UID: \"b6fdc9f9-8be5-4dfa-9390-3ca43a74b6be\") " pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-vpbzr" Oct 30 20:53:46.576937 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:46.576799 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b6fdc9f9-8be5-4dfa-9390-3ca43a74b6be-workload-certs\") pod \"e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-vpbzr\" (UID: \"b6fdc9f9-8be5-4dfa-9390-3ca43a74b6be\") " pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-vpbzr" Oct 30 20:53:46.577044 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:46.577013 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b6fdc9f9-8be5-4dfa-9390-3ca43a74b6be-istio-podinfo\") pod \"e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-vpbzr\" (UID: \"b6fdc9f9-8be5-4dfa-9390-3ca43a74b6be\") " pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-vpbzr" Oct 30 20:53:46.577126 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:46.577109 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b6fdc9f9-8be5-4dfa-9390-3ca43a74b6be-istio-envoy\") pod \"e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-vpbzr\" (UID: \"b6fdc9f9-8be5-4dfa-9390-3ca43a74b6be\") " pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-vpbzr" Oct 30 20:53:46.577207 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:46.577192 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b6fdc9f9-8be5-4dfa-9390-3ca43a74b6be-istio-data\") pod \"e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-vpbzr\" (UID: \"b6fdc9f9-8be5-4dfa-9390-3ca43a74b6be\") " pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-vpbzr" Oct 30 20:53:46.577283 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:46.577263 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b6fdc9f9-8be5-4dfa-9390-3ca43a74b6be-istiod-ca-cert\") pod \"e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-vpbzr\" (UID: \"b6fdc9f9-8be5-4dfa-9390-3ca43a74b6be\") " pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-vpbzr" Oct 30 20:53:46.577326 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:46.577298 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-72mrx\" (UniqueName: \"kubernetes.io/projected/b6fdc9f9-8be5-4dfa-9390-3ca43a74b6be-kube-api-access-72mrx\") pod \"e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-vpbzr\" (UID: \"b6fdc9f9-8be5-4dfa-9390-3ca43a74b6be\") " pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-vpbzr" Oct 30 20:53:46.577409 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:46.577397 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b6fdc9f9-8be5-4dfa-9390-3ca43a74b6be-istio-token\") pod \"e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-vpbzr\" (UID: \"b6fdc9f9-8be5-4dfa-9390-3ca43a74b6be\") " pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-vpbzr" Oct 30 20:53:46.678985 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:46.678922 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b6fdc9f9-8be5-4dfa-9390-3ca43a74b6be-istio-token\") pod \"e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-vpbzr\" (UID: \"b6fdc9f9-8be5-4dfa-9390-3ca43a74b6be\") " pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-vpbzr" Oct 30 20:53:46.678985 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:46.678973 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6fdc9f9-8be5-4dfa-9390-3ca43a74b6be-credential-socket\") pod \"e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-vpbzr\" (UID: \"b6fdc9f9-8be5-4dfa-9390-3ca43a74b6be\") " pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-vpbzr" Oct 30 20:53:46.679160 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:46.678997 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6fdc9f9-8be5-4dfa-9390-3ca43a74b6be-workload-socket\") pod \"e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-vpbzr\" (UID: \"b6fdc9f9-8be5-4dfa-9390-3ca43a74b6be\") " pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-vpbzr" Oct 30 20:53:46.679160 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:46.679022 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b6fdc9f9-8be5-4dfa-9390-3ca43a74b6be-workload-certs\") pod \"e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-vpbzr\" (UID: \"b6fdc9f9-8be5-4dfa-9390-3ca43a74b6be\") " pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-vpbzr" Oct 30 20:53:46.679160 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:46.679092 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b6fdc9f9-8be5-4dfa-9390-3ca43a74b6be-istio-podinfo\") pod \"e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-vpbzr\" (UID: \"b6fdc9f9-8be5-4dfa-9390-3ca43a74b6be\") " pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-vpbzr" Oct 30 20:53:46.679160 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:46.679136 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b6fdc9f9-8be5-4dfa-9390-3ca43a74b6be-istio-envoy\") pod \"e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-vpbzr\" (UID: \"b6fdc9f9-8be5-4dfa-9390-3ca43a74b6be\") " pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-vpbzr" Oct 30 20:53:46.679376 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:46.679192 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b6fdc9f9-8be5-4dfa-9390-3ca43a74b6be-istio-data\") pod \"e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-vpbzr\" (UID: \"b6fdc9f9-8be5-4dfa-9390-3ca43a74b6be\") " pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-vpbzr" Oct 30 20:53:46.679376 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:46.679261 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b6fdc9f9-8be5-4dfa-9390-3ca43a74b6be-istiod-ca-cert\") pod \"e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-vpbzr\" (UID: \"b6fdc9f9-8be5-4dfa-9390-3ca43a74b6be\") " pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-vpbzr" Oct 30 20:53:46.679376 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:46.679304 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-72mrx\" (UniqueName: \"kubernetes.io/projected/b6fdc9f9-8be5-4dfa-9390-3ca43a74b6be-kube-api-access-72mrx\") pod \"e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-vpbzr\" (UID: \"b6fdc9f9-8be5-4dfa-9390-3ca43a74b6be\") " pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-vpbzr" Oct 30 20:53:46.679555 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:46.679475 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6fdc9f9-8be5-4dfa-9390-3ca43a74b6be-workload-socket\") pod \"e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-vpbzr\" (UID: \"b6fdc9f9-8be5-4dfa-9390-3ca43a74b6be\") " pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-vpbzr" Oct 30 20:53:46.679618 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:46.679551 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b6fdc9f9-8be5-4dfa-9390-3ca43a74b6be-workload-certs\") pod \"e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-vpbzr\" (UID: \"b6fdc9f9-8be5-4dfa-9390-3ca43a74b6be\") " pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-vpbzr" Oct 30 20:53:46.679668 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:46.679616 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b6fdc9f9-8be5-4dfa-9390-3ca43a74b6be-istio-data\") pod \"e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-vpbzr\" (UID: \"b6fdc9f9-8be5-4dfa-9390-3ca43a74b6be\") " pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-vpbzr" Oct 30 20:53:46.679721 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:46.679685 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6fdc9f9-8be5-4dfa-9390-3ca43a74b6be-credential-socket\") pod \"e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-vpbzr\" (UID: \"b6fdc9f9-8be5-4dfa-9390-3ca43a74b6be\") " pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-vpbzr" Oct 30 20:53:46.680137 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:46.680115 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b6fdc9f9-8be5-4dfa-9390-3ca43a74b6be-istiod-ca-cert\") pod \"e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-vpbzr\" (UID: \"b6fdc9f9-8be5-4dfa-9390-3ca43a74b6be\") " pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-vpbzr" Oct 30 20:53:46.681497 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:46.681477 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b6fdc9f9-8be5-4dfa-9390-3ca43a74b6be-istio-envoy\") pod \"e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-vpbzr\" (UID: \"b6fdc9f9-8be5-4dfa-9390-3ca43a74b6be\") " pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-vpbzr" Oct 30 20:53:46.681654 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:46.681634 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b6fdc9f9-8be5-4dfa-9390-3ca43a74b6be-istio-podinfo\") pod \"e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-vpbzr\" (UID: \"b6fdc9f9-8be5-4dfa-9390-3ca43a74b6be\") " pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-vpbzr" Oct 30 20:53:46.697395 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:46.697376 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b6fdc9f9-8be5-4dfa-9390-3ca43a74b6be-istio-token\") pod \"e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-vpbzr\" (UID: \"b6fdc9f9-8be5-4dfa-9390-3ca43a74b6be\") " pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-vpbzr" Oct 30 20:53:46.697620 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:46.697599 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-72mrx\" (UniqueName: \"kubernetes.io/projected/b6fdc9f9-8be5-4dfa-9390-3ca43a74b6be-kube-api-access-72mrx\") pod \"e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-vpbzr\" (UID: \"b6fdc9f9-8be5-4dfa-9390-3ca43a74b6be\") " pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-vpbzr" Oct 30 20:53:46.809859 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:46.809792 4611 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-vpbzr" Oct 30 20:53:46.823467 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:46.823441 4611 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-g2gnd"] Oct 30 20:53:46.828240 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:46.828221 4611 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-g2gnd" Oct 30 20:53:46.840323 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:46.840300 4611 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-g2gnd"] Oct 30 20:53:46.982017 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:46.981943 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef59c099-cbfc-48c7-9d10-1da01281d7f9-istio-data\") pod \"e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-g2gnd\" (UID: \"ef59c099-cbfc-48c7-9d10-1da01281d7f9\") " pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-g2gnd" Oct 30 20:53:46.982148 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:46.982075 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef59c099-cbfc-48c7-9d10-1da01281d7f9-workload-certs\") pod \"e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-g2gnd\" (UID: \"ef59c099-cbfc-48c7-9d10-1da01281d7f9\") " pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-g2gnd" Oct 30 20:53:46.982148 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:46.982109 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef59c099-cbfc-48c7-9d10-1da01281d7f9-istio-envoy\") pod \"e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-g2gnd\" (UID: \"ef59c099-cbfc-48c7-9d10-1da01281d7f9\") " pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-g2gnd" Oct 30 20:53:46.982218 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:46.982156 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef59c099-cbfc-48c7-9d10-1da01281d7f9-istiod-ca-cert\") pod \"e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-g2gnd\" (UID: \"ef59c099-cbfc-48c7-9d10-1da01281d7f9\") " pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-g2gnd" Oct 30 20:53:46.982293 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:46.982274 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef59c099-cbfc-48c7-9d10-1da01281d7f9-workload-socket\") pod \"e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-g2gnd\" (UID: \"ef59c099-cbfc-48c7-9d10-1da01281d7f9\") " pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-g2gnd" Oct 30 20:53:46.982328 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:46.982320 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef59c099-cbfc-48c7-9d10-1da01281d7f9-credential-socket\") pod \"e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-g2gnd\" (UID: \"ef59c099-cbfc-48c7-9d10-1da01281d7f9\") " pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-g2gnd" Oct 30 20:53:46.982399 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:46.982385 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef59c099-cbfc-48c7-9d10-1da01281d7f9-istio-podinfo\") pod \"e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-g2gnd\" (UID: \"ef59c099-cbfc-48c7-9d10-1da01281d7f9\") " pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-g2gnd" Oct 30 20:53:46.982472 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:46.982460 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ef59c099-cbfc-48c7-9d10-1da01281d7f9-istio-token\") pod \"e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-g2gnd\" (UID: \"ef59c099-cbfc-48c7-9d10-1da01281d7f9\") " pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-g2gnd" Oct 30 20:53:46.982534 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:46.982485 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vrtg7\" (UniqueName: \"kubernetes.io/projected/ef59c099-cbfc-48c7-9d10-1da01281d7f9-kube-api-access-vrtg7\") pod \"e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-g2gnd\" (UID: \"ef59c099-cbfc-48c7-9d10-1da01281d7f9\") " pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-g2gnd" Oct 30 20:53:47.084118 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:47.083903 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef59c099-cbfc-48c7-9d10-1da01281d7f9-workload-certs\") pod \"e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-g2gnd\" (UID: \"ef59c099-cbfc-48c7-9d10-1da01281d7f9\") " pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-g2gnd" Oct 30 20:53:47.084118 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:47.083499 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef59c099-cbfc-48c7-9d10-1da01281d7f9-workload-certs\") pod \"e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-g2gnd\" (UID: \"ef59c099-cbfc-48c7-9d10-1da01281d7f9\") " pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-g2gnd" Oct 30 20:53:47.084118 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:47.083991 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef59c099-cbfc-48c7-9d10-1da01281d7f9-istio-envoy\") pod \"e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-g2gnd\" (UID: \"ef59c099-cbfc-48c7-9d10-1da01281d7f9\") " pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-g2gnd" Oct 30 20:53:47.084118 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:47.084054 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef59c099-cbfc-48c7-9d10-1da01281d7f9-istiod-ca-cert\") pod \"e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-g2gnd\" (UID: \"ef59c099-cbfc-48c7-9d10-1da01281d7f9\") " pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-g2gnd" Oct 30 20:53:47.084118 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:47.084103 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef59c099-cbfc-48c7-9d10-1da01281d7f9-workload-socket\") pod \"e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-g2gnd\" (UID: \"ef59c099-cbfc-48c7-9d10-1da01281d7f9\") " pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-g2gnd" Oct 30 20:53:47.084480 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:47.084143 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef59c099-cbfc-48c7-9d10-1da01281d7f9-credential-socket\") pod \"e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-g2gnd\" (UID: \"ef59c099-cbfc-48c7-9d10-1da01281d7f9\") " pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-g2gnd" Oct 30 20:53:47.084480 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:47.084180 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef59c099-cbfc-48c7-9d10-1da01281d7f9-istio-podinfo\") pod \"e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-g2gnd\" (UID: \"ef59c099-cbfc-48c7-9d10-1da01281d7f9\") " pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-g2gnd" Oct 30 20:53:47.084480 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:47.084286 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ef59c099-cbfc-48c7-9d10-1da01281d7f9-istio-token\") pod \"e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-g2gnd\" (UID: \"ef59c099-cbfc-48c7-9d10-1da01281d7f9\") " pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-g2gnd" Oct 30 20:53:47.084480 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:47.084323 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vrtg7\" (UniqueName: \"kubernetes.io/projected/ef59c099-cbfc-48c7-9d10-1da01281d7f9-kube-api-access-vrtg7\") pod \"e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-g2gnd\" (UID: \"ef59c099-cbfc-48c7-9d10-1da01281d7f9\") " pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-g2gnd" Oct 30 20:53:47.084480 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:47.084394 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef59c099-cbfc-48c7-9d10-1da01281d7f9-istio-data\") pod \"e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-g2gnd\" (UID: \"ef59c099-cbfc-48c7-9d10-1da01281d7f9\") " pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-g2gnd" Oct 30 20:53:47.085473 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:47.084822 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef59c099-cbfc-48c7-9d10-1da01281d7f9-credential-socket\") pod \"e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-g2gnd\" (UID: \"ef59c099-cbfc-48c7-9d10-1da01281d7f9\") " pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-g2gnd" Oct 30 20:53:47.085473 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:47.085140 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef59c099-cbfc-48c7-9d10-1da01281d7f9-workload-socket\") pod \"e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-g2gnd\" (UID: \"ef59c099-cbfc-48c7-9d10-1da01281d7f9\") " pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-g2gnd" Oct 30 20:53:47.085473 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:47.085166 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef59c099-cbfc-48c7-9d10-1da01281d7f9-istio-data\") pod \"e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-g2gnd\" (UID: \"ef59c099-cbfc-48c7-9d10-1da01281d7f9\") " pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-g2gnd" Oct 30 20:53:47.085473 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:47.085307 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef59c099-cbfc-48c7-9d10-1da01281d7f9-istiod-ca-cert\") pod \"e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-g2gnd\" (UID: \"ef59c099-cbfc-48c7-9d10-1da01281d7f9\") " pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-g2gnd" Oct 30 20:53:47.086851 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:47.086824 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef59c099-cbfc-48c7-9d10-1da01281d7f9-istio-envoy\") pod \"e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-g2gnd\" (UID: \"ef59c099-cbfc-48c7-9d10-1da01281d7f9\") " pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-g2gnd" Oct 30 20:53:47.087018 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:47.086998 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef59c099-cbfc-48c7-9d10-1da01281d7f9-istio-podinfo\") pod \"e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-g2gnd\" (UID: \"ef59c099-cbfc-48c7-9d10-1da01281d7f9\") " pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-g2gnd" Oct 30 20:53:47.117908 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:47.117884 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ef59c099-cbfc-48c7-9d10-1da01281d7f9-istio-token\") pod \"e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-g2gnd\" (UID: \"ef59c099-cbfc-48c7-9d10-1da01281d7f9\") " pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-g2gnd" Oct 30 20:53:47.119299 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:47.119282 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vrtg7\" (UniqueName: \"kubernetes.io/projected/ef59c099-cbfc-48c7-9d10-1da01281d7f9-kube-api-access-vrtg7\") pod \"e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-g2gnd\" (UID: \"ef59c099-cbfc-48c7-9d10-1da01281d7f9\") " pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-g2gnd" Oct 30 20:53:47.141583 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:47.141561 4611 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-vpbzr"] Oct 30 20:53:47.142654 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:53:47.142627 4611 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6fdc9f9_8be5_4dfa_9390_3ca43a74b6be.slice/crio-e90c4d6b384e65bdb28d8389ea63c41a6760d1a9cf1dc486dcbdd84ae41b3b0e WatchSource:0}: Error finding container e90c4d6b384e65bdb28d8389ea63c41a6760d1a9cf1dc486dcbdd84ae41b3b0e: Status 404 returned error can't find the container with id e90c4d6b384e65bdb28d8389ea63c41a6760d1a9cf1dc486dcbdd84ae41b3b0e Oct 30 20:53:47.146741 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:47.146707 4611 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:53:47.146823 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:47.146803 4611 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:53:47.146872 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:47.146846 4611 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:53:47.169603 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:47.169581 4611 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-g2gnd" Oct 30 20:53:47.236927 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:47.236733 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-vpbzr" event={"ID":"b6fdc9f9-8be5-4dfa-9390-3ca43a74b6be","Type":"ContainerStarted","Data":"e90c4d6b384e65bdb28d8389ea63c41a6760d1a9cf1dc486dcbdd84ae41b3b0e"} Oct 30 20:53:47.276214 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:47.276051 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:53:47.474287 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:53:47.474254 4611 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef59c099_cbfc_48c7_9d10_1da01281d7f9.slice/crio-ab015277a223f742645055afdb399ed5ad9b596d287ef730245f0a9a13f31da7 WatchSource:0}: Error finding container ab015277a223f742645055afdb399ed5ad9b596d287ef730245f0a9a13f31da7: Status 404 returned error can't find the container with id ab015277a223f742645055afdb399ed5ad9b596d287ef730245f0a9a13f31da7 Oct 30 20:53:47.475570 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:47.475540 4611 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-g2gnd"] Oct 30 20:53:47.477205 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:47.477167 4611 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:53:47.477296 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:47.477249 4611 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:53:47.477335 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:47.477297 4611 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:53:47.979170 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:47.979137 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:53:48.244844 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:48.244747 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-g2gnd" event={"ID":"ef59c099-cbfc-48c7-9d10-1da01281d7f9","Type":"ContainerStarted","Data":"f339e661941f927be58d34084d6e8d704e06f578b798e5d4e9a8c5e5f6bd0596"} Oct 30 20:53:48.244988 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:48.244845 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-g2gnd" event={"ID":"ef59c099-cbfc-48c7-9d10-1da01281d7f9","Type":"ContainerStarted","Data":"ab015277a223f742645055afdb399ed5ad9b596d287ef730245f0a9a13f31da7"} Oct 30 20:53:48.254981 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:48.254055 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-vpbzr" event={"ID":"b6fdc9f9-8be5-4dfa-9390-3ca43a74b6be","Type":"ContainerStarted","Data":"2bc814836d17daacf5ccf26f67b858b82e8a583d3005c1ab3df30b51d13ef359"} Oct 30 20:53:48.271888 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:48.271814 4611 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-g2gnd" podStartSLOduration=2.27179536 podStartE2EDuration="2.27179536s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:53:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:53:48.270786455 +0000 UTC m=+882.804813085" watchObservedRunningTime="2026-10-30 20:53:48.27179536 +0000 UTC m=+882.805821978" Oct 30 20:53:48.295568 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:48.295523 4611 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-vpbzr" podStartSLOduration=2.295492866 podStartE2EDuration="2.295492866s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:53:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:53:48.293330565 +0000 UTC m=+882.827357193" watchObservedRunningTime="2026-10-30 20:53:48.295492866 +0000 UTC m=+882.829519482" Oct 30 20:53:48.500672 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:48.499579 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:53:48.696088 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:48.696046 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:53:48.814600 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:48.813075 4611 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-vpbzr" Oct 30 20:53:48.819630 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:48.817876 4611 patch_prober.go:29] interesting pod/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-vpbzr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Oct 30 20:53:48.819630 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:48.817936 4611 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-vpbzr" podUID="b6fdc9f9-8be5-4dfa-9390-3ca43a74b6be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Oct 30 20:53:48.902544 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:48.900429 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:53:49.203646 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:49.201212 4611 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-g2gnd" Oct 30 20:53:49.203646 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:49.203399 4611 patch_prober.go:29] interesting pod/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-g2gnd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Oct 30 20:53:49.203646 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:49.203441 4611 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-g2gnd" podUID="ef59c099-cbfc-48c7-9d10-1da01281d7f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Oct 30 20:53:49.708531 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:49.685364 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:53:49.811099 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:49.811060 4611 patch_prober.go:29] interesting pod/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-vpbzr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Oct 30 20:53:49.811302 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:49.811144 4611 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-vpbzr" podUID="b6fdc9f9-8be5-4dfa-9390-3ca43a74b6be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Oct 30 20:53:50.180532 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:50.176611 4611 patch_prober.go:29] interesting pod/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-g2gnd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Oct 30 20:53:50.180532 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:50.176677 4611 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-g2gnd" podUID="ef59c099-cbfc-48c7-9d10-1da01281d7f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Oct 30 20:53:50.514499 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:50.514408 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:53:50.811145 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:50.811046 4611 patch_prober.go:29] interesting pod/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-vpbzr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Oct 30 20:53:50.811145 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:50.811103 4611 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-vpbzr" podUID="b6fdc9f9-8be5-4dfa-9390-3ca43a74b6be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Oct 30 20:53:51.183533 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:51.182571 4611 patch_prober.go:29] interesting pod/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-g2gnd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Oct 30 20:53:51.183533 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:51.182631 4611 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-g2gnd" podUID="ef59c099-cbfc-48c7-9d10-1da01281d7f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Oct 30 20:53:51.442218 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:51.442109 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:53:51.810314 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:51.810231 4611 patch_prober.go:29] interesting pod/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-vpbzr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Oct 30 20:53:51.810314 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:51.810301 4611 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-vpbzr" podUID="b6fdc9f9-8be5-4dfa-9390-3ca43a74b6be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Oct 30 20:53:52.180533 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:52.177161 4611 patch_prober.go:29] interesting pod/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-g2gnd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Oct 30 20:53:52.180533 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:52.177245 4611 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-g2gnd" podUID="ef59c099-cbfc-48c7-9d10-1da01281d7f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Oct 30 20:53:52.262007 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:52.261078 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:53:52.810663 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:52.810628 4611 patch_prober.go:29] interesting pod/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-vpbzr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Oct 30 20:53:52.811063 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:52.810707 4611 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-vpbzr" podUID="b6fdc9f9-8be5-4dfa-9390-3ca43a74b6be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Oct 30 20:53:53.180533 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:53.179877 4611 patch_prober.go:29] interesting pod/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-g2gnd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Oct 30 20:53:53.180533 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:53.179938 4611 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-g2gnd" podUID="ef59c099-cbfc-48c7-9d10-1da01281d7f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Oct 30 20:53:53.231464 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:53.231432 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:53:53.810979 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:53.810940 4611 patch_prober.go:29] interesting pod/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-vpbzr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Oct 30 20:53:53.811489 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:53.810991 4611 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-vpbzr" podUID="b6fdc9f9-8be5-4dfa-9390-3ca43a74b6be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Oct 30 20:53:54.005980 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:54.005947 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:53:54.148270 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:54.148232 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:53:54.170649 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:54.170618 4611 patch_prober.go:29] interesting pod/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-g2gnd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Oct 30 20:53:54.170816 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:54.170673 4611 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-g2gnd" podUID="ef59c099-cbfc-48c7-9d10-1da01281d7f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Oct 30 20:53:54.528068 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:54.527975 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:53:54.820604 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:54.818809 4611 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-vpbzr" Oct 30 20:53:54.820604 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:54.819819 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-vpbzr" Oct 30 20:53:54.823227 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:54.823203 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-vpbzr" Oct 30 20:53:55.022815 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:55.022754 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:53:55.175635 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:55.175606 4611 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-g2gnd" Oct 30 20:53:55.176101 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:55.176070 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-g2gnd" Oct 30 20:53:55.176839 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:55.176817 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-g2gnd" Oct 30 20:53:55.741025 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:55.740971 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:53:56.379656 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:56.379623 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:53:57.113116 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:57.113082 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:53:58.090049 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:58.090014 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:53:59.976535 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:53:59.976170 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cb60a2/simulator-w3-cb60a2-kserve-6fd74b9d8b-vqfzf" podUID="711b167d-4e66-40d6-be75-33ede11f42c5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:54:00.084532 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:00.074969 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b8aee1/premium-w1-b8aee1-kserve-786d575599-htc8d" podUID="77368afe-edc8-4fa5-8010-d321965fb38e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded" Oct 30 20:54:00.120533 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:00.105956 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-14cca2/premium-w5-14cca2-kserve-6878777857-wpjtj" podUID="b99eb76b-d1c6-49e9-9809-c351c0f5a4de" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded" Oct 30 20:54:00.234599 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:00.221627 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db534b/distinct-2-w2-db534b-kserve-8594c876-g92mh" podUID="955316d6-a9fd-4b75-a8ab-bbf382b776e2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:54:00.310897 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:00.304187 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad7ff0/premium-w0-ad7ff0-kserve-5485d4b948-bpmgf" podUID="53cbd692-e9f8-46ba-8fde-3e5215a38904" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded" Oct 30 20:54:00.310897 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:00.304835 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-3d909d/premium-w6-3d909d-kserve-765648946f-2qvd2" podUID="fca5d697-109d-4e90-ab41-c40ad31af874" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded" Oct 30 20:54:00.327641 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:00.315911 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:54:00.327641 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:00.316378 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-3d909d/distinct-2-w6-3d909d-kserve-59c5d5c595-cjg2m" podUID="e2b4c221-ee94-4131-abd8-2313647c641e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:54:00.327641 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:00.317742 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cb60a2/distinct-w3-cb60a2-kserve-5bb76dc74-86fds" podUID="52e301d9-f378-4e54-9515-01a6680d9fd7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded" Oct 30 20:54:00.380540 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:00.355598 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-14cca2/distinct-w5-14cca2-kserve-587f6996b8-sgf6h" podUID="02b652c3-0724-4a50-9e95-2a9890cc97c4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 20:54:00.978799 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:00.978753 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cb60a2/simulator-w3-cb60a2-kserve-6fd74b9d8b-vqfzf" podUID="711b167d-4e66-40d6-be75-33ede11f42c5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:54:00.980097 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:00.980073 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-cb60a2/simulator-w3-cb60a2-kserve-6fd74b9d8b-vqfzf" Oct 30 20:54:01.112012 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:01.110089 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b8aee1/distinct-w1-b8aee1-kserve-5b857c55cd-x5b7b" podUID="1599e588-def8-42e1-b81d-ad0aaf119818" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:54:01.112012 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:01.110424 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f2ed1e/distinct-w4-f2ed1e-kserve-57cff9dcf8-8d5zl" podUID="653e3ec7-2147-4813-8946-1f25cfa425f6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded" Oct 30 20:54:01.256594 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:01.223588 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b8aee1/premium-w1-b8aee1-kserve-786d575599-htc8d" podUID="77368afe-edc8-4fa5-8010-d321965fb38e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:54:01.256594 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:01.223955 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db534b/distinct-2-w2-db534b-kserve-8594c876-g92mh" podUID="955316d6-a9fd-4b75-a8ab-bbf382b776e2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": context deadline exceeded" Oct 30 20:54:01.256594 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:01.224615 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b8aee1/premium-w1-b8aee1-kserve-786d575599-htc8d" Oct 30 20:54:01.256594 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:01.224671 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-db534b/distinct-2-w2-db534b-kserve-8594c876-g92mh" Oct 30 20:54:01.455165 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:01.407598 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-14cca2/premium-w5-14cca2-kserve-6878777857-wpjtj" podUID="b99eb76b-d1c6-49e9-9809-c351c0f5a4de" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded" Oct 30 20:54:01.455165 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:01.408041 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-14cca2/premium-w5-14cca2-kserve-6878777857-wpjtj" Oct 30 20:54:01.455165 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:01.409160 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cb60a2/distinct-w3-cb60a2-kserve-5bb76dc74-86fds" podUID="52e301d9-f378-4e54-9515-01a6680d9fd7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded" Oct 30 20:54:01.455165 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:01.410113 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-cb60a2/distinct-w3-cb60a2-kserve-5bb76dc74-86fds" Oct 30 20:54:01.455165 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:01.438115 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-14cca2/distinct-w5-14cca2-kserve-587f6996b8-sgf6h" podUID="02b652c3-0724-4a50-9e95-2a9890cc97c4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:54:01.455165 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:01.442041 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-14cca2/distinct-w5-14cca2-kserve-587f6996b8-sgf6h" Oct 30 20:54:01.455165 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:01.445363 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-db534b/distinct-2-w2-db534b-kserve-8594c876-g92mh" Oct 30 20:54:01.463392 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:01.461393 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad7ff0/premium-w0-ad7ff0-kserve-5485d4b948-bpmgf" podUID="53cbd692-e9f8-46ba-8fde-3e5215a38904" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:54:01.463910 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:01.463884 4611 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-b8aee1/premium-w1-b8aee1-kserve-786d575599-htc8d" podUID="77368afe-edc8-4fa5-8010-d321965fb38e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/health\": context deadline exceeded" Oct 30 20:54:01.464104 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:01.464084 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-3d909d/premium-w6-3d909d-kserve-765648946f-2qvd2" podUID="fca5d697-109d-4e90-ab41-c40ad31af874" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded" Oct 30 20:54:01.465400 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:01.465377 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-3d909d/distinct-2-w6-3d909d-kserve-59c5d5c595-cjg2m" podUID="e2b4c221-ee94-4131-abd8-2313647c641e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": context deadline exceeded" Oct 30 20:54:01.465861 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:01.465843 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ad7ff0/premium-w0-ad7ff0-kserve-5485d4b948-bpmgf" Oct 30 20:54:01.466030 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:01.466016 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-3d909d/premium-w6-3d909d-kserve-765648946f-2qvd2" Oct 30 20:54:01.466171 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:01.466157 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-3d909d/distinct-2-w6-3d909d-kserve-59c5d5c595-cjg2m" Oct 30 20:54:01.867537 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:01.867243 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-cb60a2/simulator-w3-cb60a2-kserve-6fd74b9d8b-vqfzf" Oct 30 20:54:01.867537 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:01.867302 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ad7ff0/premium-w0-ad7ff0-kserve-5485d4b948-bpmgf" Oct 30 20:54:01.903557 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:01.894682 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-cb60a2/distinct-w3-cb60a2-kserve-5bb76dc74-86fds" Oct 30 20:54:01.983749 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:01.983700 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-3d909d/premium-w6-3d909d-kserve-765648946f-2qvd2" Oct 30 20:54:01.985849 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:01.985819 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b8aee1/premium-w1-b8aee1-kserve-786d575599-htc8d" Oct 30 20:54:02.082537 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:02.079722 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-3d909d/distinct-2-w6-3d909d-kserve-59c5d5c595-cjg2m" Oct 30 20:54:02.082813 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:02.082793 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-14cca2/distinct-w5-14cca2-kserve-587f6996b8-sgf6h" Oct 30 20:54:02.091181 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:02.091160 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-14cca2/premium-w5-14cca2-kserve-6878777857-wpjtj" Oct 30 20:54:02.125936 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:02.125854 4611 patch_prober.go:29] interesting pod/e2e-worker-w6-3d909d-openshift-default-77787695b7-trngw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:54:02.136561 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:02.130866 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-trngw" podUID="5ba753c3-5979-4263-8b90-2741747c8592" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:54:02.185331 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:02.185199 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:54:02.579865 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:02.579783 4611 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-b8aee1/distinct-2-w1-b8aee1-kserve-6bcdd59797-cx6hr"] Oct 30 20:54:02.586752 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:02.586724 4611 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-b8aee1/distinct-2-w1-b8aee1-kserve-6bcdd59797-cx6hr"] Oct 30 20:54:02.586936 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:02.586846 4611 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-b8aee1/distinct-2-w1-b8aee1-kserve-6bcdd59797-cx6hr" Oct 30 20:54:02.589995 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:02.589968 4611 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-b8aee1\"/\"distinct-2-w1-b8aee1-kserve-self-signed-certs\"" Oct 30 20:54:02.616064 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:02.616028 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/286eab7a-81a7-4768-a7f6-be6b3720028a-model-cache\") pod \"distinct-2-w1-b8aee1-kserve-6bcdd59797-cx6hr\" (UID: \"286eab7a-81a7-4768-a7f6-be6b3720028a\") " pod="e2e-models-e2e-worker-w1-b8aee1/distinct-2-w1-b8aee1-kserve-6bcdd59797-cx6hr" Oct 30 20:54:02.616302 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:02.616274 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vphzb\" (UniqueName: \"kubernetes.io/projected/286eab7a-81a7-4768-a7f6-be6b3720028a-kube-api-access-vphzb\") pod \"distinct-2-w1-b8aee1-kserve-6bcdd59797-cx6hr\" (UID: \"286eab7a-81a7-4768-a7f6-be6b3720028a\") " pod="e2e-models-e2e-worker-w1-b8aee1/distinct-2-w1-b8aee1-kserve-6bcdd59797-cx6hr" Oct 30 20:54:02.616430 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:02.616372 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/286eab7a-81a7-4768-a7f6-be6b3720028a-dshm\") pod \"distinct-2-w1-b8aee1-kserve-6bcdd59797-cx6hr\" (UID: \"286eab7a-81a7-4768-a7f6-be6b3720028a\") " pod="e2e-models-e2e-worker-w1-b8aee1/distinct-2-w1-b8aee1-kserve-6bcdd59797-cx6hr" Oct 30 20:54:02.616565 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:02.616549 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/286eab7a-81a7-4768-a7f6-be6b3720028a-home\") pod \"distinct-2-w1-b8aee1-kserve-6bcdd59797-cx6hr\" (UID: \"286eab7a-81a7-4768-a7f6-be6b3720028a\") " pod="e2e-models-e2e-worker-w1-b8aee1/distinct-2-w1-b8aee1-kserve-6bcdd59797-cx6hr" Oct 30 20:54:02.616621 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:02.616580 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/286eab7a-81a7-4768-a7f6-be6b3720028a-tls-certs\") pod \"distinct-2-w1-b8aee1-kserve-6bcdd59797-cx6hr\" (UID: \"286eab7a-81a7-4768-a7f6-be6b3720028a\") " pod="e2e-models-e2e-worker-w1-b8aee1/distinct-2-w1-b8aee1-kserve-6bcdd59797-cx6hr" Oct 30 20:54:02.616680 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:02.616661 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/286eab7a-81a7-4768-a7f6-be6b3720028a-tmp-dir\") pod \"distinct-2-w1-b8aee1-kserve-6bcdd59797-cx6hr\" (UID: \"286eab7a-81a7-4768-a7f6-be6b3720028a\") " pod="e2e-models-e2e-worker-w1-b8aee1/distinct-2-w1-b8aee1-kserve-6bcdd59797-cx6hr" Oct 30 20:54:02.653533 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:02.653482 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:54:02.718308 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:02.718281 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/286eab7a-81a7-4768-a7f6-be6b3720028a-model-cache\") pod \"distinct-2-w1-b8aee1-kserve-6bcdd59797-cx6hr\" (UID: \"286eab7a-81a7-4768-a7f6-be6b3720028a\") " pod="e2e-models-e2e-worker-w1-b8aee1/distinct-2-w1-b8aee1-kserve-6bcdd59797-cx6hr" Oct 30 20:54:02.718465 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:02.718324 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vphzb\" (UniqueName: \"kubernetes.io/projected/286eab7a-81a7-4768-a7f6-be6b3720028a-kube-api-access-vphzb\") pod \"distinct-2-w1-b8aee1-kserve-6bcdd59797-cx6hr\" (UID: \"286eab7a-81a7-4768-a7f6-be6b3720028a\") " pod="e2e-models-e2e-worker-w1-b8aee1/distinct-2-w1-b8aee1-kserve-6bcdd59797-cx6hr" Oct 30 20:54:02.718465 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:02.718423 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/286eab7a-81a7-4768-a7f6-be6b3720028a-dshm\") pod \"distinct-2-w1-b8aee1-kserve-6bcdd59797-cx6hr\" (UID: \"286eab7a-81a7-4768-a7f6-be6b3720028a\") " pod="e2e-models-e2e-worker-w1-b8aee1/distinct-2-w1-b8aee1-kserve-6bcdd59797-cx6hr" Oct 30 20:54:02.718606 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:02.718589 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/286eab7a-81a7-4768-a7f6-be6b3720028a-home\") pod \"distinct-2-w1-b8aee1-kserve-6bcdd59797-cx6hr\" (UID: \"286eab7a-81a7-4768-a7f6-be6b3720028a\") " pod="e2e-models-e2e-worker-w1-b8aee1/distinct-2-w1-b8aee1-kserve-6bcdd59797-cx6hr" Oct 30 20:54:02.718660 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:02.718619 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/286eab7a-81a7-4768-a7f6-be6b3720028a-tls-certs\") pod \"distinct-2-w1-b8aee1-kserve-6bcdd59797-cx6hr\" (UID: \"286eab7a-81a7-4768-a7f6-be6b3720028a\") " pod="e2e-models-e2e-worker-w1-b8aee1/distinct-2-w1-b8aee1-kserve-6bcdd59797-cx6hr" Oct 30 20:54:02.718716 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:02.718698 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/286eab7a-81a7-4768-a7f6-be6b3720028a-tmp-dir\") pod \"distinct-2-w1-b8aee1-kserve-6bcdd59797-cx6hr\" (UID: \"286eab7a-81a7-4768-a7f6-be6b3720028a\") " pod="e2e-models-e2e-worker-w1-b8aee1/distinct-2-w1-b8aee1-kserve-6bcdd59797-cx6hr" Oct 30 20:54:02.718814 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:02.718788 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/286eab7a-81a7-4768-a7f6-be6b3720028a-model-cache\") pod \"distinct-2-w1-b8aee1-kserve-6bcdd59797-cx6hr\" (UID: \"286eab7a-81a7-4768-a7f6-be6b3720028a\") " pod="e2e-models-e2e-worker-w1-b8aee1/distinct-2-w1-b8aee1-kserve-6bcdd59797-cx6hr" Oct 30 20:54:02.718969 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:02.718947 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/286eab7a-81a7-4768-a7f6-be6b3720028a-home\") pod \"distinct-2-w1-b8aee1-kserve-6bcdd59797-cx6hr\" (UID: \"286eab7a-81a7-4768-a7f6-be6b3720028a\") " pod="e2e-models-e2e-worker-w1-b8aee1/distinct-2-w1-b8aee1-kserve-6bcdd59797-cx6hr" Oct 30 20:54:02.719050 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:02.719028 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/286eab7a-81a7-4768-a7f6-be6b3720028a-tmp-dir\") pod \"distinct-2-w1-b8aee1-kserve-6bcdd59797-cx6hr\" (UID: \"286eab7a-81a7-4768-a7f6-be6b3720028a\") " pod="e2e-models-e2e-worker-w1-b8aee1/distinct-2-w1-b8aee1-kserve-6bcdd59797-cx6hr" Oct 30 20:54:02.720707 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:02.720685 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/286eab7a-81a7-4768-a7f6-be6b3720028a-dshm\") pod \"distinct-2-w1-b8aee1-kserve-6bcdd59797-cx6hr\" (UID: \"286eab7a-81a7-4768-a7f6-be6b3720028a\") " pod="e2e-models-e2e-worker-w1-b8aee1/distinct-2-w1-b8aee1-kserve-6bcdd59797-cx6hr" Oct 30 20:54:02.720997 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:02.720975 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/286eab7a-81a7-4768-a7f6-be6b3720028a-tls-certs\") pod \"distinct-2-w1-b8aee1-kserve-6bcdd59797-cx6hr\" (UID: \"286eab7a-81a7-4768-a7f6-be6b3720028a\") " pod="e2e-models-e2e-worker-w1-b8aee1/distinct-2-w1-b8aee1-kserve-6bcdd59797-cx6hr" Oct 30 20:54:02.733370 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:02.733347 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vphzb\" (UniqueName: \"kubernetes.io/projected/286eab7a-81a7-4768-a7f6-be6b3720028a-kube-api-access-vphzb\") pod \"distinct-2-w1-b8aee1-kserve-6bcdd59797-cx6hr\" (UID: \"286eab7a-81a7-4768-a7f6-be6b3720028a\") " pod="e2e-models-e2e-worker-w1-b8aee1/distinct-2-w1-b8aee1-kserve-6bcdd59797-cx6hr" Oct 30 20:54:02.903722 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:02.903687 4611 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-b8aee1/distinct-2-w1-b8aee1-kserve-6bcdd59797-cx6hr" Oct 30 20:54:03.088779 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:03.088215 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:54:03.213464 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:03.213434 4611 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-b8aee1/distinct-2-w1-b8aee1-kserve-6bcdd59797-cx6hr"] Oct 30 20:54:03.214434 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:54:03.214410 4611 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod286eab7a_81a7_4768_a7f6_be6b3720028a.slice/crio-d932e2d6505e925f627b181ec40f12af3f1d3ddec68dad3d5bc0d329f0fb3adb WatchSource:0}: Error finding container d932e2d6505e925f627b181ec40f12af3f1d3ddec68dad3d5bc0d329f0fb3adb: Status 404 returned error can't find the container with id d932e2d6505e925f627b181ec40f12af3f1d3ddec68dad3d5bc0d329f0fb3adb Oct 30 20:54:03.641911 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:03.641876 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-b8aee1/distinct-2-w1-b8aee1-kserve-6bcdd59797-cx6hr" event={"ID":"286eab7a-81a7-4768-a7f6-be6b3720028a","Type":"ContainerStarted","Data":"ff8f2e91da6d2bf856f2e45b8edd50279f8c3f04439f63dc404b0e50e5d54a99"} Oct 30 20:54:03.642164 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:03.641922 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-b8aee1/distinct-2-w1-b8aee1-kserve-6bcdd59797-cx6hr" event={"ID":"286eab7a-81a7-4768-a7f6-be6b3720028a","Type":"ContainerStarted","Data":"d932e2d6505e925f627b181ec40f12af3f1d3ddec68dad3d5bc0d329f0fb3adb"} Oct 30 20:54:03.662653 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:03.662607 4611 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-b8aee1/distinct-2-w1-b8aee1-kserve-6bcdd59797-cx6hr" podStartSLOduration=1.662594028 podStartE2EDuration="1.662594028s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:54:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:54:03.659093992 +0000 UTC m=+898.193120614" watchObservedRunningTime="2026-10-30 20:54:03.662594028 +0000 UTC m=+898.196620646" Oct 30 20:54:03.904774 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:03.904705 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b8aee1/distinct-2-w1-b8aee1-kserve-6bcdd59797-cx6hr" Oct 30 20:54:03.913291 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:03.913257 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:54:03.918833 ip-10-0-133-192 kubenswrapper[4611]: E1030 20:54:03.918807 4611 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hwzc6" containerName="extractor" Oct 30 20:54:04.685027 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:04.684988 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:54:05.469362 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:05.469327 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:54:05.729142 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:05.729077 4611 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7w5wz_7e2574c1-b59c-453c-9501-25ec13a1ccad/ovn-acl-logging/0.log" Oct 30 20:54:05.729142 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:05.729077 4611 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7w5wz_7e2574c1-b59c-453c-9501-25ec13a1ccad/ovn-acl-logging/0.log" Oct 30 20:54:06.308105 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:06.308059 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:54:06.979857 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:06.979825 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:54:07.606930 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:07.606896 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:54:08.255955 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:08.255917 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:54:08.883919 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:08.883877 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:54:09.598064 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:09.598023 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:54:10.289272 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:10.289237 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:54:11.096195 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:11.096132 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:54:11.791374 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:11.791339 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:54:12.514118 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:12.514073 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:54:12.904643 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:12.904609 4611 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-b8aee1/distinct-2-w1-b8aee1-kserve-6bcdd59797-cx6hr" Oct 30 20:54:12.917940 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:12.917912 4611 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-b8aee1/distinct-2-w1-b8aee1-kserve-6bcdd59797-cx6hr" Oct 30 20:54:13.250125 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:13.250030 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:54:13.729961 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:13.729929 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b8aee1/distinct-2-w1-b8aee1-kserve-6bcdd59797-cx6hr" Oct 30 20:54:14.026171 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:14.026083 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:54:14.696917 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:14.696878 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:54:15.391204 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:15.391165 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:54:16.110322 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:16.110277 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:54:16.864542 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:16.864434 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:54:17.057638 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:17.057601 4611 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-lmpng"] Oct 30 20:54:17.063852 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:17.063829 4611 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-lmpng" Oct 30 20:54:17.077135 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:17.077105 4611 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-lmpng"] Oct 30 20:54:17.147586 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:17.147463 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61477918-6577-4957-807b-ebfa7a01f9d2-istio-envoy\") pod \"e2e-worker-w2-db534b-openshift-default-85df4d79d9-lmpng\" (UID: \"61477918-6577-4957-807b-ebfa7a01f9d2\") " pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-lmpng" Oct 30 20:54:17.147761 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:17.147644 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61477918-6577-4957-807b-ebfa7a01f9d2-istio-data\") pod \"e2e-worker-w2-db534b-openshift-default-85df4d79d9-lmpng\" (UID: \"61477918-6577-4957-807b-ebfa7a01f9d2\") " pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-lmpng" Oct 30 20:54:17.147761 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:17.147736 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61477918-6577-4957-807b-ebfa7a01f9d2-istio-token\") pod \"e2e-worker-w2-db534b-openshift-default-85df4d79d9-lmpng\" (UID: \"61477918-6577-4957-807b-ebfa7a01f9d2\") " pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-lmpng" Oct 30 20:54:17.147884 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:17.147859 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61477918-6577-4957-807b-ebfa7a01f9d2-istiod-ca-cert\") pod \"e2e-worker-w2-db534b-openshift-default-85df4d79d9-lmpng\" (UID: \"61477918-6577-4957-807b-ebfa7a01f9d2\") " pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-lmpng" Oct 30 20:54:17.147998 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:17.147976 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61477918-6577-4957-807b-ebfa7a01f9d2-workload-certs\") pod \"e2e-worker-w2-db534b-openshift-default-85df4d79d9-lmpng\" (UID: \"61477918-6577-4957-807b-ebfa7a01f9d2\") " pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-lmpng" Oct 30 20:54:17.148097 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:17.148072 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-klg4q\" (UniqueName: \"kubernetes.io/projected/61477918-6577-4957-807b-ebfa7a01f9d2-kube-api-access-klg4q\") pod \"e2e-worker-w2-db534b-openshift-default-85df4d79d9-lmpng\" (UID: \"61477918-6577-4957-807b-ebfa7a01f9d2\") " pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-lmpng" Oct 30 20:54:17.148243 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:17.148226 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61477918-6577-4957-807b-ebfa7a01f9d2-credential-socket\") pod \"e2e-worker-w2-db534b-openshift-default-85df4d79d9-lmpng\" (UID: \"61477918-6577-4957-807b-ebfa7a01f9d2\") " pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-lmpng" Oct 30 20:54:17.148309 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:17.148293 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61477918-6577-4957-807b-ebfa7a01f9d2-istio-podinfo\") pod \"e2e-worker-w2-db534b-openshift-default-85df4d79d9-lmpng\" (UID: \"61477918-6577-4957-807b-ebfa7a01f9d2\") " pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-lmpng" Oct 30 20:54:17.148381 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:17.148365 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61477918-6577-4957-807b-ebfa7a01f9d2-workload-socket\") pod \"e2e-worker-w2-db534b-openshift-default-85df4d79d9-lmpng\" (UID: \"61477918-6577-4957-807b-ebfa7a01f9d2\") " pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-lmpng" Oct 30 20:54:17.249923 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:17.249884 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61477918-6577-4957-807b-ebfa7a01f9d2-credential-socket\") pod \"e2e-worker-w2-db534b-openshift-default-85df4d79d9-lmpng\" (UID: \"61477918-6577-4957-807b-ebfa7a01f9d2\") " pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-lmpng" Oct 30 20:54:17.250084 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:17.249948 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61477918-6577-4957-807b-ebfa7a01f9d2-istio-podinfo\") pod \"e2e-worker-w2-db534b-openshift-default-85df4d79d9-lmpng\" (UID: \"61477918-6577-4957-807b-ebfa7a01f9d2\") " pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-lmpng" Oct 30 20:54:17.250084 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:17.249979 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61477918-6577-4957-807b-ebfa7a01f9d2-workload-socket\") pod \"e2e-worker-w2-db534b-openshift-default-85df4d79d9-lmpng\" (UID: \"61477918-6577-4957-807b-ebfa7a01f9d2\") " pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-lmpng" Oct 30 20:54:17.250084 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:17.250021 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61477918-6577-4957-807b-ebfa7a01f9d2-istio-envoy\") pod \"e2e-worker-w2-db534b-openshift-default-85df4d79d9-lmpng\" (UID: \"61477918-6577-4957-807b-ebfa7a01f9d2\") " pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-lmpng" Oct 30 20:54:17.250084 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:17.250074 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61477918-6577-4957-807b-ebfa7a01f9d2-istio-data\") pod \"e2e-worker-w2-db534b-openshift-default-85df4d79d9-lmpng\" (UID: \"61477918-6577-4957-807b-ebfa7a01f9d2\") " pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-lmpng" Oct 30 20:54:17.250302 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:17.250140 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61477918-6577-4957-807b-ebfa7a01f9d2-istio-token\") pod \"e2e-worker-w2-db534b-openshift-default-85df4d79d9-lmpng\" (UID: \"61477918-6577-4957-807b-ebfa7a01f9d2\") " pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-lmpng" Oct 30 20:54:17.250302 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:17.250194 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61477918-6577-4957-807b-ebfa7a01f9d2-istiod-ca-cert\") pod \"e2e-worker-w2-db534b-openshift-default-85df4d79d9-lmpng\" (UID: \"61477918-6577-4957-807b-ebfa7a01f9d2\") " pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-lmpng" Oct 30 20:54:17.250302 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:17.250262 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61477918-6577-4957-807b-ebfa7a01f9d2-workload-certs\") pod \"e2e-worker-w2-db534b-openshift-default-85df4d79d9-lmpng\" (UID: \"61477918-6577-4957-807b-ebfa7a01f9d2\") " pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-lmpng" Oct 30 20:54:17.250302 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:17.250300 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-klg4q\" (UniqueName: \"kubernetes.io/projected/61477918-6577-4957-807b-ebfa7a01f9d2-kube-api-access-klg4q\") pod \"e2e-worker-w2-db534b-openshift-default-85df4d79d9-lmpng\" (UID: \"61477918-6577-4957-807b-ebfa7a01f9d2\") " pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-lmpng" Oct 30 20:54:17.250498 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:17.250317 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61477918-6577-4957-807b-ebfa7a01f9d2-credential-socket\") pod \"e2e-worker-w2-db534b-openshift-default-85df4d79d9-lmpng\" (UID: \"61477918-6577-4957-807b-ebfa7a01f9d2\") " pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-lmpng" Oct 30 20:54:17.250498 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:17.250347 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61477918-6577-4957-807b-ebfa7a01f9d2-workload-socket\") pod \"e2e-worker-w2-db534b-openshift-default-85df4d79d9-lmpng\" (UID: \"61477918-6577-4957-807b-ebfa7a01f9d2\") " pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-lmpng" Oct 30 20:54:17.250749 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:17.250721 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61477918-6577-4957-807b-ebfa7a01f9d2-istio-data\") pod \"e2e-worker-w2-db534b-openshift-default-85df4d79d9-lmpng\" (UID: \"61477918-6577-4957-807b-ebfa7a01f9d2\") " pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-lmpng" Oct 30 20:54:17.250871 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:17.250783 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61477918-6577-4957-807b-ebfa7a01f9d2-workload-certs\") pod \"e2e-worker-w2-db534b-openshift-default-85df4d79d9-lmpng\" (UID: \"61477918-6577-4957-807b-ebfa7a01f9d2\") " pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-lmpng" Oct 30 20:54:17.251024 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:17.251006 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61477918-6577-4957-807b-ebfa7a01f9d2-istiod-ca-cert\") pod \"e2e-worker-w2-db534b-openshift-default-85df4d79d9-lmpng\" (UID: \"61477918-6577-4957-807b-ebfa7a01f9d2\") " pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-lmpng" Oct 30 20:54:17.252678 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:17.252650 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61477918-6577-4957-807b-ebfa7a01f9d2-istio-podinfo\") pod \"e2e-worker-w2-db534b-openshift-default-85df4d79d9-lmpng\" (UID: \"61477918-6577-4957-807b-ebfa7a01f9d2\") " pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-lmpng" Oct 30 20:54:17.252791 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:17.252687 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61477918-6577-4957-807b-ebfa7a01f9d2-istio-envoy\") pod \"e2e-worker-w2-db534b-openshift-default-85df4d79d9-lmpng\" (UID: \"61477918-6577-4957-807b-ebfa7a01f9d2\") " pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-lmpng" Oct 30 20:54:17.262230 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:17.262202 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61477918-6577-4957-807b-ebfa7a01f9d2-istio-token\") pod \"e2e-worker-w2-db534b-openshift-default-85df4d79d9-lmpng\" (UID: \"61477918-6577-4957-807b-ebfa7a01f9d2\") " pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-lmpng" Oct 30 20:54:17.263038 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:17.263018 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-klg4q\" (UniqueName: \"kubernetes.io/projected/61477918-6577-4957-807b-ebfa7a01f9d2-kube-api-access-klg4q\") pod \"e2e-worker-w2-db534b-openshift-default-85df4d79d9-lmpng\" (UID: \"61477918-6577-4957-807b-ebfa7a01f9d2\") " pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-lmpng" Oct 30 20:54:17.322747 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:17.322714 4611 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-wf8qg"] Oct 30 20:54:17.328758 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:17.328733 4611 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-wf8qg" Oct 30 20:54:17.346455 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:17.346428 4611 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-wf8qg"] Oct 30 20:54:17.380453 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:17.380414 4611 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-lmpng" Oct 30 20:54:17.473076 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:17.472114 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c468f3c6-7d73-4fd1-ae34-1212e4d1b387-istio-envoy\") pod \"e2e-worker-w6-3d909d-openshift-default-77787695b7-wf8qg\" (UID: \"c468f3c6-7d73-4fd1-ae34-1212e4d1b387\") " pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-wf8qg" Oct 30 20:54:17.473076 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:17.472197 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c468f3c6-7d73-4fd1-ae34-1212e4d1b387-workload-socket\") pod \"e2e-worker-w6-3d909d-openshift-default-77787695b7-wf8qg\" (UID: \"c468f3c6-7d73-4fd1-ae34-1212e4d1b387\") " pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-wf8qg" Oct 30 20:54:17.473076 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:17.472262 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c468f3c6-7d73-4fd1-ae34-1212e4d1b387-istiod-ca-cert\") pod \"e2e-worker-w6-3d909d-openshift-default-77787695b7-wf8qg\" (UID: \"c468f3c6-7d73-4fd1-ae34-1212e4d1b387\") " pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-wf8qg" Oct 30 20:54:17.473076 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:17.472372 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mm94t\" (UniqueName: \"kubernetes.io/projected/c468f3c6-7d73-4fd1-ae34-1212e4d1b387-kube-api-access-mm94t\") pod \"e2e-worker-w6-3d909d-openshift-default-77787695b7-wf8qg\" (UID: \"c468f3c6-7d73-4fd1-ae34-1212e4d1b387\") " pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-wf8qg" Oct 30 20:54:17.473076 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:17.472434 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c468f3c6-7d73-4fd1-ae34-1212e4d1b387-istio-token\") pod \"e2e-worker-w6-3d909d-openshift-default-77787695b7-wf8qg\" (UID: \"c468f3c6-7d73-4fd1-ae34-1212e4d1b387\") " pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-wf8qg" Oct 30 20:54:17.473076 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:17.472473 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c468f3c6-7d73-4fd1-ae34-1212e4d1b387-workload-certs\") pod \"e2e-worker-w6-3d909d-openshift-default-77787695b7-wf8qg\" (UID: \"c468f3c6-7d73-4fd1-ae34-1212e4d1b387\") " pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-wf8qg" Oct 30 20:54:17.473076 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:17.472698 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c468f3c6-7d73-4fd1-ae34-1212e4d1b387-istio-podinfo\") pod \"e2e-worker-w6-3d909d-openshift-default-77787695b7-wf8qg\" (UID: \"c468f3c6-7d73-4fd1-ae34-1212e4d1b387\") " pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-wf8qg" Oct 30 20:54:17.473076 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:17.472791 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c468f3c6-7d73-4fd1-ae34-1212e4d1b387-istio-data\") pod \"e2e-worker-w6-3d909d-openshift-default-77787695b7-wf8qg\" (UID: \"c468f3c6-7d73-4fd1-ae34-1212e4d1b387\") " pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-wf8qg" Oct 30 20:54:17.473076 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:17.472952 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c468f3c6-7d73-4fd1-ae34-1212e4d1b387-credential-socket\") pod \"e2e-worker-w6-3d909d-openshift-default-77787695b7-wf8qg\" (UID: \"c468f3c6-7d73-4fd1-ae34-1212e4d1b387\") " pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-wf8qg" Oct 30 20:54:17.507559 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:17.505240 4611 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-9d6hc"] Oct 30 20:54:17.511329 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:17.511304 4611 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-9d6hc" Oct 30 20:54:17.528541 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:17.528489 4611 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-9d6hc"] Oct 30 20:54:17.574587 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:17.574554 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mm94t\" (UniqueName: \"kubernetes.io/projected/c468f3c6-7d73-4fd1-ae34-1212e4d1b387-kube-api-access-mm94t\") pod \"e2e-worker-w6-3d909d-openshift-default-77787695b7-wf8qg\" (UID: \"c468f3c6-7d73-4fd1-ae34-1212e4d1b387\") " pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-wf8qg" Oct 30 20:54:17.574778 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:17.574602 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1f5526b-53f1-4c2b-a801-49ee75e16f9c-workload-socket\") pod \"e2e-worker-w3-cb60a2-openshift-default-57db98bd79-9d6hc\" (UID: \"f1f5526b-53f1-4c2b-a801-49ee75e16f9c\") " pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-9d6hc" Oct 30 20:54:17.574778 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:17.574647 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c468f3c6-7d73-4fd1-ae34-1212e4d1b387-istio-token\") pod \"e2e-worker-w6-3d909d-openshift-default-77787695b7-wf8qg\" (UID: \"c468f3c6-7d73-4fd1-ae34-1212e4d1b387\") " pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-wf8qg" Oct 30 20:54:17.574778 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:17.574678 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c468f3c6-7d73-4fd1-ae34-1212e4d1b387-workload-certs\") pod \"e2e-worker-w6-3d909d-openshift-default-77787695b7-wf8qg\" (UID: \"c468f3c6-7d73-4fd1-ae34-1212e4d1b387\") " pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-wf8qg" Oct 30 20:54:17.574778 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:17.574702 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1f5526b-53f1-4c2b-a801-49ee75e16f9c-workload-certs\") pod \"e2e-worker-w3-cb60a2-openshift-default-57db98bd79-9d6hc\" (UID: \"f1f5526b-53f1-4c2b-a801-49ee75e16f9c\") " pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-9d6hc" Oct 30 20:54:17.574778 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:17.574755 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hd265\" (UniqueName: \"kubernetes.io/projected/f1f5526b-53f1-4c2b-a801-49ee75e16f9c-kube-api-access-hd265\") pod \"e2e-worker-w3-cb60a2-openshift-default-57db98bd79-9d6hc\" (UID: \"f1f5526b-53f1-4c2b-a801-49ee75e16f9c\") " pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-9d6hc" Oct 30 20:54:17.575063 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:17.574859 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c468f3c6-7d73-4fd1-ae34-1212e4d1b387-istio-podinfo\") pod \"e2e-worker-w6-3d909d-openshift-default-77787695b7-wf8qg\" (UID: \"c468f3c6-7d73-4fd1-ae34-1212e4d1b387\") " pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-wf8qg" Oct 30 20:54:17.575063 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:17.574909 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c468f3c6-7d73-4fd1-ae34-1212e4d1b387-istio-data\") pod \"e2e-worker-w6-3d909d-openshift-default-77787695b7-wf8qg\" (UID: \"c468f3c6-7d73-4fd1-ae34-1212e4d1b387\") " pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-wf8qg" Oct 30 20:54:17.575063 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:17.574939 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f1f5526b-53f1-4c2b-a801-49ee75e16f9c-istio-token\") pod \"e2e-worker-w3-cb60a2-openshift-default-57db98bd79-9d6hc\" (UID: \"f1f5526b-53f1-4c2b-a801-49ee75e16f9c\") " pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-9d6hc" Oct 30 20:54:17.575063 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:17.575003 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c468f3c6-7d73-4fd1-ae34-1212e4d1b387-credential-socket\") pod \"e2e-worker-w6-3d909d-openshift-default-77787695b7-wf8qg\" (UID: \"c468f3c6-7d73-4fd1-ae34-1212e4d1b387\") " pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-wf8qg" Oct 30 20:54:17.575063 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:17.575031 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1f5526b-53f1-4c2b-a801-49ee75e16f9c-istio-envoy\") pod \"e2e-worker-w3-cb60a2-openshift-default-57db98bd79-9d6hc\" (UID: \"f1f5526b-53f1-4c2b-a801-49ee75e16f9c\") " pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-9d6hc" Oct 30 20:54:17.575316 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:17.575067 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1f5526b-53f1-4c2b-a801-49ee75e16f9c-credential-socket\") pod \"e2e-worker-w3-cb60a2-openshift-default-57db98bd79-9d6hc\" (UID: \"f1f5526b-53f1-4c2b-a801-49ee75e16f9c\") " pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-9d6hc" Oct 30 20:54:17.575316 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:17.575096 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c468f3c6-7d73-4fd1-ae34-1212e4d1b387-istio-envoy\") pod \"e2e-worker-w6-3d909d-openshift-default-77787695b7-wf8qg\" (UID: \"c468f3c6-7d73-4fd1-ae34-1212e4d1b387\") " pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-wf8qg" Oct 30 20:54:17.575316 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:17.575134 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c468f3c6-7d73-4fd1-ae34-1212e4d1b387-workload-socket\") pod \"e2e-worker-w6-3d909d-openshift-default-77787695b7-wf8qg\" (UID: \"c468f3c6-7d73-4fd1-ae34-1212e4d1b387\") " pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-wf8qg" Oct 30 20:54:17.575316 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:17.575166 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c468f3c6-7d73-4fd1-ae34-1212e4d1b387-istiod-ca-cert\") pod \"e2e-worker-w6-3d909d-openshift-default-77787695b7-wf8qg\" (UID: \"c468f3c6-7d73-4fd1-ae34-1212e4d1b387\") " pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-wf8qg" Oct 30 20:54:17.575316 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:17.575191 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1f5526b-53f1-4c2b-a801-49ee75e16f9c-istiod-ca-cert\") pod \"e2e-worker-w3-cb60a2-openshift-default-57db98bd79-9d6hc\" (UID: \"f1f5526b-53f1-4c2b-a801-49ee75e16f9c\") " pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-9d6hc" Oct 30 20:54:17.575316 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:17.575234 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1f5526b-53f1-4c2b-a801-49ee75e16f9c-istio-podinfo\") pod \"e2e-worker-w3-cb60a2-openshift-default-57db98bd79-9d6hc\" (UID: \"f1f5526b-53f1-4c2b-a801-49ee75e16f9c\") " pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-9d6hc" Oct 30 20:54:17.575316 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:17.575268 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1f5526b-53f1-4c2b-a801-49ee75e16f9c-istio-data\") pod \"e2e-worker-w3-cb60a2-openshift-default-57db98bd79-9d6hc\" (UID: \"f1f5526b-53f1-4c2b-a801-49ee75e16f9c\") " pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-9d6hc" Oct 30 20:54:17.576501 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:17.576133 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c468f3c6-7d73-4fd1-ae34-1212e4d1b387-workload-socket\") pod \"e2e-worker-w6-3d909d-openshift-default-77787695b7-wf8qg\" (UID: \"c468f3c6-7d73-4fd1-ae34-1212e4d1b387\") " pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-wf8qg" Oct 30 20:54:17.576501 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:17.576331 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c468f3c6-7d73-4fd1-ae34-1212e4d1b387-workload-certs\") pod \"e2e-worker-w6-3d909d-openshift-default-77787695b7-wf8qg\" (UID: \"c468f3c6-7d73-4fd1-ae34-1212e4d1b387\") " pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-wf8qg" Oct 30 20:54:17.576695 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:17.576500 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c468f3c6-7d73-4fd1-ae34-1212e4d1b387-istiod-ca-cert\") pod \"e2e-worker-w6-3d909d-openshift-default-77787695b7-wf8qg\" (UID: \"c468f3c6-7d73-4fd1-ae34-1212e4d1b387\") " pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-wf8qg" Oct 30 20:54:17.576695 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:17.576643 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c468f3c6-7d73-4fd1-ae34-1212e4d1b387-istio-data\") pod \"e2e-worker-w6-3d909d-openshift-default-77787695b7-wf8qg\" (UID: \"c468f3c6-7d73-4fd1-ae34-1212e4d1b387\") " pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-wf8qg" Oct 30 20:54:17.576909 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:17.576891 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c468f3c6-7d73-4fd1-ae34-1212e4d1b387-credential-socket\") pod \"e2e-worker-w6-3d909d-openshift-default-77787695b7-wf8qg\" (UID: \"c468f3c6-7d73-4fd1-ae34-1212e4d1b387\") " pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-wf8qg" Oct 30 20:54:17.580224 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:17.580197 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c468f3c6-7d73-4fd1-ae34-1212e4d1b387-istio-envoy\") pod \"e2e-worker-w6-3d909d-openshift-default-77787695b7-wf8qg\" (UID: \"c468f3c6-7d73-4fd1-ae34-1212e4d1b387\") " pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-wf8qg" Oct 30 20:54:17.580690 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:17.580670 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c468f3c6-7d73-4fd1-ae34-1212e4d1b387-istio-podinfo\") pod \"e2e-worker-w6-3d909d-openshift-default-77787695b7-wf8qg\" (UID: \"c468f3c6-7d73-4fd1-ae34-1212e4d1b387\") " pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-wf8qg" Oct 30 20:54:17.587299 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:17.587268 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c468f3c6-7d73-4fd1-ae34-1212e4d1b387-istio-token\") pod \"e2e-worker-w6-3d909d-openshift-default-77787695b7-wf8qg\" (UID: \"c468f3c6-7d73-4fd1-ae34-1212e4d1b387\") " pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-wf8qg" Oct 30 20:54:17.589062 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:17.589038 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mm94t\" (UniqueName: \"kubernetes.io/projected/c468f3c6-7d73-4fd1-ae34-1212e4d1b387-kube-api-access-mm94t\") pod \"e2e-worker-w6-3d909d-openshift-default-77787695b7-wf8qg\" (UID: \"c468f3c6-7d73-4fd1-ae34-1212e4d1b387\") " pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-wf8qg" Oct 30 20:54:17.647344 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:17.647305 4611 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-wf8qg" Oct 30 20:54:17.676380 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:17.676342 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hd265\" (UniqueName: \"kubernetes.io/projected/f1f5526b-53f1-4c2b-a801-49ee75e16f9c-kube-api-access-hd265\") pod \"e2e-worker-w3-cb60a2-openshift-default-57db98bd79-9d6hc\" (UID: \"f1f5526b-53f1-4c2b-a801-49ee75e16f9c\") " pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-9d6hc" Oct 30 20:54:17.676529 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:17.676486 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f1f5526b-53f1-4c2b-a801-49ee75e16f9c-istio-token\") pod \"e2e-worker-w3-cb60a2-openshift-default-57db98bd79-9d6hc\" (UID: \"f1f5526b-53f1-4c2b-a801-49ee75e16f9c\") " pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-9d6hc" Oct 30 20:54:17.677697 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:17.676612 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1f5526b-53f1-4c2b-a801-49ee75e16f9c-istio-envoy\") pod \"e2e-worker-w3-cb60a2-openshift-default-57db98bd79-9d6hc\" (UID: \"f1f5526b-53f1-4c2b-a801-49ee75e16f9c\") " pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-9d6hc" Oct 30 20:54:17.677697 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:17.676759 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1f5526b-53f1-4c2b-a801-49ee75e16f9c-credential-socket\") pod \"e2e-worker-w3-cb60a2-openshift-default-57db98bd79-9d6hc\" (UID: \"f1f5526b-53f1-4c2b-a801-49ee75e16f9c\") " pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-9d6hc" Oct 30 20:54:17.677697 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:17.676808 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1f5526b-53f1-4c2b-a801-49ee75e16f9c-istiod-ca-cert\") pod \"e2e-worker-w3-cb60a2-openshift-default-57db98bd79-9d6hc\" (UID: \"f1f5526b-53f1-4c2b-a801-49ee75e16f9c\") " pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-9d6hc" Oct 30 20:54:17.677697 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:17.676840 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1f5526b-53f1-4c2b-a801-49ee75e16f9c-istio-podinfo\") pod \"e2e-worker-w3-cb60a2-openshift-default-57db98bd79-9d6hc\" (UID: \"f1f5526b-53f1-4c2b-a801-49ee75e16f9c\") " pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-9d6hc" Oct 30 20:54:17.677697 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:17.676866 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1f5526b-53f1-4c2b-a801-49ee75e16f9c-istio-data\") pod \"e2e-worker-w3-cb60a2-openshift-default-57db98bd79-9d6hc\" (UID: \"f1f5526b-53f1-4c2b-a801-49ee75e16f9c\") " pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-9d6hc" Oct 30 20:54:17.677697 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:17.676953 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1f5526b-53f1-4c2b-a801-49ee75e16f9c-workload-socket\") pod \"e2e-worker-w3-cb60a2-openshift-default-57db98bd79-9d6hc\" (UID: \"f1f5526b-53f1-4c2b-a801-49ee75e16f9c\") " pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-9d6hc" Oct 30 20:54:17.677697 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:17.677027 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1f5526b-53f1-4c2b-a801-49ee75e16f9c-workload-certs\") pod \"e2e-worker-w3-cb60a2-openshift-default-57db98bd79-9d6hc\" (UID: \"f1f5526b-53f1-4c2b-a801-49ee75e16f9c\") " pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-9d6hc" Oct 30 20:54:17.677697 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:17.677089 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1f5526b-53f1-4c2b-a801-49ee75e16f9c-credential-socket\") pod \"e2e-worker-w3-cb60a2-openshift-default-57db98bd79-9d6hc\" (UID: \"f1f5526b-53f1-4c2b-a801-49ee75e16f9c\") " pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-9d6hc" Oct 30 20:54:17.677697 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:17.677173 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1f5526b-53f1-4c2b-a801-49ee75e16f9c-istio-data\") pod \"e2e-worker-w3-cb60a2-openshift-default-57db98bd79-9d6hc\" (UID: \"f1f5526b-53f1-4c2b-a801-49ee75e16f9c\") " pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-9d6hc" Oct 30 20:54:17.677697 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:17.677358 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1f5526b-53f1-4c2b-a801-49ee75e16f9c-workload-certs\") pod \"e2e-worker-w3-cb60a2-openshift-default-57db98bd79-9d6hc\" (UID: \"f1f5526b-53f1-4c2b-a801-49ee75e16f9c\") " pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-9d6hc" Oct 30 20:54:17.677697 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:17.677615 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1f5526b-53f1-4c2b-a801-49ee75e16f9c-workload-socket\") pod \"e2e-worker-w3-cb60a2-openshift-default-57db98bd79-9d6hc\" (UID: \"f1f5526b-53f1-4c2b-a801-49ee75e16f9c\") " pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-9d6hc" Oct 30 20:54:17.678465 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:17.677735 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1f5526b-53f1-4c2b-a801-49ee75e16f9c-istiod-ca-cert\") pod \"e2e-worker-w3-cb60a2-openshift-default-57db98bd79-9d6hc\" (UID: \"f1f5526b-53f1-4c2b-a801-49ee75e16f9c\") " pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-9d6hc" Oct 30 20:54:17.679228 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:17.679201 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1f5526b-53f1-4c2b-a801-49ee75e16f9c-istio-envoy\") pod \"e2e-worker-w3-cb60a2-openshift-default-57db98bd79-9d6hc\" (UID: \"f1f5526b-53f1-4c2b-a801-49ee75e16f9c\") " pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-9d6hc" Oct 30 20:54:17.680135 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:17.680112 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1f5526b-53f1-4c2b-a801-49ee75e16f9c-istio-podinfo\") pod \"e2e-worker-w3-cb60a2-openshift-default-57db98bd79-9d6hc\" (UID: \"f1f5526b-53f1-4c2b-a801-49ee75e16f9c\") " pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-9d6hc" Oct 30 20:54:17.690491 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:17.690467 4611 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-lmpng"] Oct 30 20:54:17.692934 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:54:17.692903 4611 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61477918_6577_4957_807b_ebfa7a01f9d2.slice/crio-4e57136caa11544abcd79158f21983af8deca0802beeae1b8a55216f17d9c412 WatchSource:0}: Error finding container 4e57136caa11544abcd79158f21983af8deca0802beeae1b8a55216f17d9c412: Status 404 returned error can't find the container with id 4e57136caa11544abcd79158f21983af8deca0802beeae1b8a55216f17d9c412 Oct 30 20:54:17.693330 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:17.693310 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f1f5526b-53f1-4c2b-a801-49ee75e16f9c-istio-token\") pod \"e2e-worker-w3-cb60a2-openshift-default-57db98bd79-9d6hc\" (UID: \"f1f5526b-53f1-4c2b-a801-49ee75e16f9c\") " pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-9d6hc" Oct 30 20:54:17.694869 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:17.694826 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hd265\" (UniqueName: \"kubernetes.io/projected/f1f5526b-53f1-4c2b-a801-49ee75e16f9c-kube-api-access-hd265\") pod \"e2e-worker-w3-cb60a2-openshift-default-57db98bd79-9d6hc\" (UID: \"f1f5526b-53f1-4c2b-a801-49ee75e16f9c\") " pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-9d6hc" Oct 30 20:54:17.697665 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:17.697242 4611 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:54:17.697665 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:17.697333 4611 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:54:17.697665 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:17.697374 4611 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:54:17.758109 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:17.758072 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-lmpng" event={"ID":"61477918-6577-4957-807b-ebfa7a01f9d2","Type":"ContainerStarted","Data":"4e57136caa11544abcd79158f21983af8deca0802beeae1b8a55216f17d9c412"} Oct 30 20:54:17.808640 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:17.808615 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:54:17.839104 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:17.839078 4611 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-9d6hc" Oct 30 20:54:18.010761 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:18.010402 4611 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-wf8qg"] Oct 30 20:54:18.040908 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:18.038805 4611 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:54:18.040908 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:18.038900 4611 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:54:18.040908 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:18.038946 4611 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:54:18.242594 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:18.242406 4611 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-9d6hc"] Oct 30 20:54:18.310796 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:18.310751 4611 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:54:18.311031 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:18.311003 4611 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:54:18.311201 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:18.311178 4611 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:54:18.798535 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:18.797983 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-9d6hc" event={"ID":"f1f5526b-53f1-4c2b-a801-49ee75e16f9c","Type":"ContainerStarted","Data":"f1034ebd6de0aa7d70504195eb408bf9c78e55bc0caeffefdc776f925bdba0c2"} Oct 30 20:54:18.798535 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:18.798070 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-9d6hc" event={"ID":"f1f5526b-53f1-4c2b-a801-49ee75e16f9c","Type":"ContainerStarted","Data":"ac2c235cc3b9199171c242828d7bc92bae609e69629c035b155ed64c47c177ae"} Oct 30 20:54:18.835537 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:18.834641 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-wf8qg" event={"ID":"c468f3c6-7d73-4fd1-ae34-1212e4d1b387","Type":"ContainerStarted","Data":"1a52628c1464f529ef397549f13e488f2d96482f8fda373851190beef6458016"} Oct 30 20:54:18.835537 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:18.834712 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-wf8qg" event={"ID":"c468f3c6-7d73-4fd1-ae34-1212e4d1b387","Type":"ContainerStarted","Data":"7a15a8bf690a62f4bda18a6c63b9fa6998ac704d367ccd92af87e66dd6ebc002"} Oct 30 20:54:18.843531 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:18.839580 4611 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-9d6hc" Oct 30 20:54:18.879268 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:18.879101 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-lmpng" event={"ID":"61477918-6577-4957-807b-ebfa7a01f9d2","Type":"ContainerStarted","Data":"c2b4cfd5065653286408300dd2991b494221273438708cc131102f583d6f4b91"} Oct 30 20:54:18.913534 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:18.881877 4611 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-9d6hc" podStartSLOduration=1.881860485 podStartE2EDuration="1.881860485s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:54:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:54:18.867393572 +0000 UTC m=+913.401420189" watchObservedRunningTime="2026-10-30 20:54:18.881860485 +0000 UTC m=+913.415887102" Oct 30 20:54:19.201533 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:19.189852 4611 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-wf8qg" podStartSLOduration=2.189830308 podStartE2EDuration="2.189830308s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:54:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:54:18.895988918 +0000 UTC m=+913.430015591" watchObservedRunningTime="2026-10-30 20:54:19.189830308 +0000 UTC m=+913.723856925" Oct 30 20:54:19.294534 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:19.293714 4611 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-lmpng" podStartSLOduration=2.293691404 podStartE2EDuration="2.293691404s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:54:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:54:18.921533429 +0000 UTC m=+913.455560046" watchObservedRunningTime="2026-10-30 20:54:19.293691404 +0000 UTC m=+913.827718021" Oct 30 20:54:19.397556 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:19.395377 4611 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-lmpng" Oct 30 20:54:19.437533 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:19.437336 4611 patch_prober.go:29] interesting pod/e2e-worker-w2-db534b-openshift-default-85df4d79d9-lmpng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Oct 30 20:54:19.437533 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:19.437402 4611 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-lmpng" podUID="61477918-6577-4957-807b-ebfa7a01f9d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Oct 30 20:54:19.671231 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:19.670616 4611 patch_prober.go:29] interesting pod/e2e-worker-w6-3d909d-openshift-default-77787695b7-wf8qg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Oct 30 20:54:19.671231 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:19.670675 4611 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-wf8qg" podUID="c468f3c6-7d73-4fd1-ae34-1212e4d1b387" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Oct 30 20:54:19.681841 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:19.675818 4611 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-wf8qg" Oct 30 20:54:19.844620 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:19.844586 4611 patch_prober.go:29] interesting pod/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-9d6hc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body= Oct 30 20:54:19.844887 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:19.844859 4611 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-9d6hc" podUID="f1f5526b-53f1-4c2b-a801-49ee75e16f9c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" Oct 30 20:54:19.898850 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:19.898810 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cb60a2/distinct-w3-cb60a2-kserve-5bb76dc74-86fds" podUID="52e301d9-f378-4e54-9515-01a6680d9fd7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:54:19.994205 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:19.994111 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b8aee1/premium-w1-b8aee1-kserve-786d575599-htc8d" podUID="77368afe-edc8-4fa5-8010-d321965fb38e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:54:19.995148 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:19.994439 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-3d909d/premium-w6-3d909d-kserve-765648946f-2qvd2" podUID="fca5d697-109d-4e90-ab41-c40ad31af874" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:54:20.082941 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:20.082897 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f2ed1e/distinct-w4-f2ed1e-kserve-57cff9dcf8-8d5zl" podUID="653e3ec7-2147-4813-8946-1f25cfa425f6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:54:20.084226 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:20.084196 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-14cca2/distinct-w5-14cca2-kserve-587f6996b8-sgf6h" podUID="02b652c3-0724-4a50-9e95-2a9890cc97c4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded" Oct 30 20:54:20.084707 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:20.084670 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b8aee1/distinct-w1-b8aee1-kserve-5b857c55cd-x5b7b" podUID="1599e588-def8-42e1-b81d-ad0aaf119818" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": context deadline exceeded" Oct 30 20:54:20.085070 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:20.085049 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-3d909d/distinct-2-w6-3d909d-kserve-59c5d5c595-cjg2m" podUID="e2b4c221-ee94-4131-abd8-2313647c641e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": context deadline exceeded" Oct 30 20:54:20.096528 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:20.094566 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-14cca2/premium-w5-14cca2-kserve-6878777857-wpjtj" podUID="b99eb76b-d1c6-49e9-9809-c351c0f5a4de" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:54:20.384534 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:20.383791 4611 patch_prober.go:29] interesting pod/e2e-worker-w2-db534b-openshift-default-85df4d79d9-lmpng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Oct 30 20:54:20.384534 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:20.383860 4611 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-lmpng" podUID="61477918-6577-4957-807b-ebfa7a01f9d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Oct 30 20:54:20.501634 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:20.501588 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db534b/distinct-2-w2-db534b-kserve-8594c876-g92mh" podUID="955316d6-a9fd-4b75-a8ab-bbf382b776e2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": context deadline exceeded" Oct 30 20:54:20.746589 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:20.738309 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b8aee1/distinct-2-w1-b8aee1-kserve-6bcdd59797-cx6hr" podUID="286eab7a-81a7-4768-a7f6-be6b3720028a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:54:20.805531 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:20.764344 4611 patch_prober.go:29] interesting pod/e2e-worker-w6-3d909d-openshift-default-77787695b7-wf8qg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Oct 30 20:54:20.805531 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:20.776595 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cb60a2/simulator-w3-cb60a2-kserve-6fd74b9d8b-vqfzf" podUID="711b167d-4e66-40d6-be75-33ede11f42c5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:54:20.805531 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:20.796002 4611 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-wf8qg" podUID="c468f3c6-7d73-4fd1-ae34-1212e4d1b387" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Oct 30 20:54:20.927833 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:20.828754 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad7ff0/premium-w0-ad7ff0-kserve-5485d4b948-bpmgf" podUID="53cbd692-e9f8-46ba-8fde-3e5215a38904" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:54:20.927833 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:20.901060 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cb60a2/distinct-w3-cb60a2-kserve-5bb76dc74-86fds" podUID="52e301d9-f378-4e54-9515-01a6680d9fd7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:54:20.927833 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:20.901620 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-cb60a2/distinct-w3-cb60a2-kserve-5bb76dc74-86fds" Oct 30 20:54:21.009592 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:20.928790 4611 patch_prober.go:29] interesting pod/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-9d6hc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body= Oct 30 20:54:21.009592 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:20.928850 4611 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-9d6hc" podUID="f1f5526b-53f1-4c2b-a801-49ee75e16f9c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" Oct 30 20:54:21.044534 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:21.024314 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b8aee1/premium-w1-b8aee1-kserve-786d575599-htc8d" podUID="77368afe-edc8-4fa5-8010-d321965fb38e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded" Oct 30 20:54:21.044534 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:21.035872 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-3d909d/premium-w6-3d909d-kserve-765648946f-2qvd2" podUID="fca5d697-109d-4e90-ab41-c40ad31af874" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:54:21.044534 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:21.036618 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b8aee1/premium-w1-b8aee1-kserve-786d575599-htc8d" Oct 30 20:54:21.044534 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:21.037887 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-3d909d/premium-w6-3d909d-kserve-765648946f-2qvd2" Oct 30 20:54:21.099007 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:21.097644 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-14cca2/premium-w5-14cca2-kserve-6878777857-wpjtj" podUID="b99eb76b-d1c6-49e9-9809-c351c0f5a4de" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded" Oct 30 20:54:21.099007 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:21.097684 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-14cca2/distinct-w5-14cca2-kserve-587f6996b8-sgf6h" podUID="02b652c3-0724-4a50-9e95-2a9890cc97c4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded" Oct 30 20:54:21.099007 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:21.097648 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f2ed1e/distinct-w4-f2ed1e-kserve-57cff9dcf8-8d5zl" podUID="653e3ec7-2147-4813-8946-1f25cfa425f6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded" Oct 30 20:54:21.100398 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:21.099375 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b8aee1/distinct-w1-b8aee1-kserve-5b857c55cd-x5b7b" podUID="1599e588-def8-42e1-b81d-ad0aaf119818" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": context deadline exceeded" Oct 30 20:54:21.100398 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:21.100069 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-3d909d/distinct-2-w6-3d909d-kserve-59c5d5c595-cjg2m" podUID="e2b4c221-ee94-4131-abd8-2313647c641e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": context deadline exceeded" Oct 30 20:54:21.113533 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:21.105736 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-14cca2/distinct-w5-14cca2-kserve-587f6996b8-sgf6h" Oct 30 20:54:21.147780 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:21.147742 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-14cca2/premium-w5-14cca2-kserve-6878777857-wpjtj" Oct 30 20:54:21.147968 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:21.147863 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-f2ed1e/distinct-w4-f2ed1e-kserve-57cff9dcf8-8d5zl" Oct 30 20:54:21.147968 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:21.147910 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b8aee1/distinct-w1-b8aee1-kserve-5b857c55cd-x5b7b" Oct 30 20:54:21.148100 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:21.147969 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-3d909d/distinct-2-w6-3d909d-kserve-59c5d5c595-cjg2m" Oct 30 20:54:21.148541 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:21.148492 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:54:21.266586 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:21.234116 4611 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-b8aee1/premium-w1-b8aee1-kserve-786d575599-htc8d" podUID="77368afe-edc8-4fa5-8010-d321965fb38e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:54:21.479551 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:21.469599 4611 patch_prober.go:29] interesting pod/e2e-worker-w2-db534b-openshift-default-85df4d79d9-lmpng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Oct 30 20:54:21.479551 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:21.469661 4611 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-lmpng" podUID="61477918-6577-4957-807b-ebfa7a01f9d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Oct 30 20:54:21.514971 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:21.514930 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db534b/distinct-2-w2-db534b-kserve-8594c876-g92mh" podUID="955316d6-a9fd-4b75-a8ab-bbf382b776e2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": context deadline exceeded" Oct 30 20:54:21.516164 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:21.516141 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-db534b/distinct-2-w2-db534b-kserve-8594c876-g92mh" Oct 30 20:54:21.750598 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:21.746299 4611 patch_prober.go:29] interesting pod/e2e-worker-w6-3d909d-openshift-default-77787695b7-wf8qg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Oct 30 20:54:21.750598 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:21.746448 4611 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-wf8qg" podUID="c468f3c6-7d73-4fd1-ae34-1212e4d1b387" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Oct 30 20:54:21.798761 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:21.798701 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cb60a2/simulator-w3-cb60a2-kserve-6fd74b9d8b-vqfzf" podUID="711b167d-4e66-40d6-be75-33ede11f42c5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:54:21.875594 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:21.841774 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad7ff0/premium-w0-ad7ff0-kserve-5485d4b948-bpmgf" podUID="53cbd692-e9f8-46ba-8fde-3e5215a38904" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded" Oct 30 20:54:21.875594 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:21.842777 4611 patch_prober.go:29] interesting pod/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-9d6hc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body= Oct 30 20:54:21.875594 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:21.842821 4611 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-9d6hc" podUID="f1f5526b-53f1-4c2b-a801-49ee75e16f9c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" Oct 30 20:54:21.875594 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:21.855908 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b8aee1/distinct-2-w1-b8aee1-kserve-6bcdd59797-cx6hr" podUID="286eab7a-81a7-4768-a7f6-be6b3720028a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": context deadline exceeded" Oct 30 20:54:22.031585 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:22.003655 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-smdl6" podUID="31d263b9-b3b3-4db4-9026-da84d454dc34" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.133.0.22:6080/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:54:22.606535 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:22.550590 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-14cca2/premium-w5-14cca2-kserve-6878777857-wpjtj" podUID="b99eb76b-d1c6-49e9-9809-c351c0f5a4de" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:54:22.864764 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:22.864653 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-3d909d/premium-w6-3d909d-kserve-765648946f-2qvd2" podUID="fca5d697-109d-4e90-ab41-c40ad31af874" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:54:22.985619 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:22.559800 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db534b/distinct-2-w2-db534b-kserve-8594c876-g92mh" podUID="955316d6-a9fd-4b75-a8ab-bbf382b776e2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:54:22.989776 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:22.989565 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b8aee1/distinct-2-w1-b8aee1-kserve-6bcdd59797-cx6hr" podUID="286eab7a-81a7-4768-a7f6-be6b3720028a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:54:22.993645 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:22.659988 4611 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-b8aee1/distinct-w1-b8aee1-kserve-5b857c55cd-x5b7b" podUID="1599e588-def8-42e1-b81d-ad0aaf119818" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/health\": context deadline exceeded" Oct 30 20:54:22.997532 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:22.747685 4611 patch_prober.go:29] interesting pod/e2e-worker-w2-db534b-openshift-default-85df4d79d9-lmpng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Oct 30 20:54:22.997783 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:22.997748 4611 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-lmpng" podUID="61477918-6577-4957-807b-ebfa7a01f9d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Oct 30 20:54:22.997930 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:22.024928 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b8aee1/premium-w1-b8aee1-kserve-786d575599-htc8d" podUID="77368afe-edc8-4fa5-8010-d321965fb38e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded" Oct 30 20:54:23.028610 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:22.823935 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-14cca2/distinct-w5-14cca2-kserve-587f6996b8-sgf6h" podUID="02b652c3-0724-4a50-9e95-2a9890cc97c4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:54:23.030411 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:22.027845 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cb60a2/distinct-w3-cb60a2-kserve-5bb76dc74-86fds" podUID="52e301d9-f378-4e54-9515-01a6680d9fd7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded" Oct 30 20:54:23.034788 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:22.975093 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f2ed1e/distinct-w4-f2ed1e-kserve-57cff9dcf8-8d5zl" podUID="653e3ec7-2147-4813-8946-1f25cfa425f6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded" Oct 30 20:54:23.084533 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:23.080715 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-dq6vs" podUID="f23cb637-ada7-4e6d-897f-83978d56ac7d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:54:23.162589 ip-10-0-133-192 kubenswrapper[4611]: E1030 20:54:23.146890 4611 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.502s" Oct 30 20:54:23.162589 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:23.147042 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-cb60a2/simulator-w3-cb60a2-kserve-6fd74b9d8b-vqfzf" Oct 30 20:54:23.162589 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:23.147138 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ad7ff0/premium-w0-ad7ff0-kserve-5485d4b948-bpmgf" Oct 30 20:54:23.162589 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:23.147336 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b8aee1/distinct-2-w1-b8aee1-kserve-6bcdd59797-cx6hr" Oct 30 20:54:23.186642 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:23.186592 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b8aee1/distinct-w1-b8aee1-kserve-5b857c55cd-x5b7b" podUID="1599e588-def8-42e1-b81d-ad0aaf119818" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": context deadline exceeded" Oct 30 20:54:23.262543 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:22.560212 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-3d909d/distinct-2-w6-3d909d-kserve-59c5d5c595-cjg2m" podUID="e2b4c221-ee94-4131-abd8-2313647c641e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": context deadline exceeded" Oct 30 20:54:23.331800 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:23.260031 4611 patch_prober.go:29] interesting pod/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-9d6hc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body= Oct 30 20:54:23.334083 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:23.334046 4611 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-9d6hc" podUID="f1f5526b-53f1-4c2b-a801-49ee75e16f9c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" Oct 30 20:54:23.415823 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:23.415724 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:54:23.438570 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:23.438537 4611 patch_prober.go:29] interesting pod/e2e-worker-w6-3d909d-openshift-default-77787695b7-wf8qg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Oct 30 20:54:23.448538 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:23.438825 4611 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-wf8qg" podUID="c468f3c6-7d73-4fd1-ae34-1212e4d1b387" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Oct 30 20:54:23.719810 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:23.719604 4611 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-cb60a2/simulator-w3-cb60a2-kserve-6fd74b9d8b-vqfzf" podUID="711b167d-4e66-40d6-be75-33ede11f42c5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:54:23.750838 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:23.750643 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cb60a2/simulator-w3-cb60a2-kserve-6fd74b9d8b-vqfzf" podUID="711b167d-4e66-40d6-be75-33ede11f42c5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded" Oct 30 20:54:23.766543 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:23.764606 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad7ff0/premium-w0-ad7ff0-kserve-5485d4b948-bpmgf" podUID="53cbd692-e9f8-46ba-8fde-3e5215a38904" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded" Oct 30 20:54:23.782333 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:23.772103 4611 patch_prober.go:29] interesting pod/e2e-worker-w2-db534b-openshift-default-85df4d79d9-lmpng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Oct 30 20:54:23.811809 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:23.811766 4611 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-lmpng" podUID="61477918-6577-4957-807b-ebfa7a01f9d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Oct 30 20:54:23.963532 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:23.962636 4611 patch_prober.go:29] interesting pod/e2e-worker-w6-3d909d-openshift-default-77787695b7-wf8qg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Oct 30 20:54:23.963532 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:23.962703 4611 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-wf8qg" podUID="c468f3c6-7d73-4fd1-ae34-1212e4d1b387" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Oct 30 20:54:24.030717 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:24.030608 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-14cca2/distinct-w5-14cca2-kserve-587f6996b8-sgf6h" podUID="02b652c3-0724-4a50-9e95-2a9890cc97c4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:54:24.031102 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:24.031070 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cb60a2/distinct-w3-cb60a2-kserve-5bb76dc74-86fds" podUID="52e301d9-f378-4e54-9515-01a6680d9fd7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:54:24.034657 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:24.034629 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b8aee1/distinct-2-w1-b8aee1-kserve-6bcdd59797-cx6hr" podUID="286eab7a-81a7-4768-a7f6-be6b3720028a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": context deadline exceeded" Oct 30 20:54:24.058575 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:24.050649 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f2ed1e/distinct-w4-f2ed1e-kserve-57cff9dcf8-8d5zl" podUID="653e3ec7-2147-4813-8946-1f25cfa425f6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:54:24.058575 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:24.050784 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b8aee1/premium-w1-b8aee1-kserve-786d575599-htc8d" podUID="77368afe-edc8-4fa5-8010-d321965fb38e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded" Oct 30 20:54:24.150578 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:24.104428 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:54:24.150578 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:24.118975 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-3d909d/premium-w6-3d909d-kserve-765648946f-2qvd2" podUID="fca5d697-109d-4e90-ab41-c40ad31af874" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded" Oct 30 20:54:24.150578 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:24.140612 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-14cca2/premium-w5-14cca2-kserve-6878777857-wpjtj" podUID="b99eb76b-d1c6-49e9-9809-c351c0f5a4de" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded" Oct 30 20:54:24.159673 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:24.159638 4611 patch_prober.go:29] interesting pod/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-9d6hc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body= Oct 30 20:54:24.163622 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:24.163578 4611 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-9d6hc" podUID="f1f5526b-53f1-4c2b-a801-49ee75e16f9c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" Oct 30 20:54:24.295964 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:24.191931 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b8aee1/distinct-w1-b8aee1-kserve-5b857c55cd-x5b7b" Oct 30 20:54:24.406539 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:24.212861 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db534b/distinct-2-w2-db534b-kserve-8594c876-g92mh" podUID="955316d6-a9fd-4b75-a8ab-bbf382b776e2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:54:24.406539 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:24.217820 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-s9l7p" podUID="a86251b2-e065-4aaa-9b7b-b8b7bee34bc2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.32:8000/ready\": context deadline exceeded" Oct 30 20:54:24.451625 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:24.229454 4611 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-b8aee1/distinct-2-w1-b8aee1-kserve-6bcdd59797-cx6hr" podUID="286eab7a-81a7-4768-a7f6-be6b3720028a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/health\": context deadline exceeded" Oct 30 20:54:24.451625 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:24.254753 4611 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-14cca2/distinct-w5-14cca2-kserve-587f6996b8-sgf6h" podUID="02b652c3-0724-4a50-9e95-2a9890cc97c4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:54:24.451625 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:24.299199 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-3d909d/distinct-2-w6-3d909d-kserve-59c5d5c595-cjg2m" podUID="e2b4c221-ee94-4131-abd8-2313647c641e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:54:24.451625 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:24.413263 4611 patch_prober.go:29] interesting pod/e2e-worker-w2-db534b-openshift-default-85df4d79d9-lmpng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Oct 30 20:54:24.451625 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:24.413315 4611 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-lmpng" podUID="61477918-6577-4957-807b-ebfa7a01f9d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Oct 30 20:54:24.676533 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:24.676100 4611 patch_prober.go:29] interesting pod/e2e-worker-w6-3d909d-openshift-default-77787695b7-wf8qg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Oct 30 20:54:24.676533 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:24.676158 4611 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-wf8qg" podUID="c468f3c6-7d73-4fd1-ae34-1212e4d1b387" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Oct 30 20:54:24.769538 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:24.769010 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b8aee1/distinct-w1-b8aee1-kserve-5b857c55cd-x5b7b" podUID="1599e588-def8-42e1-b81d-ad0aaf119818" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:54:24.769538 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:24.769458 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cb60a2/simulator-w3-cb60a2-kserve-6fd74b9d8b-vqfzf" podUID="711b167d-4e66-40d6-be75-33ede11f42c5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded" Oct 30 20:54:24.770103 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:24.769943 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad7ff0/premium-w0-ad7ff0-kserve-5485d4b948-bpmgf" podUID="53cbd692-e9f8-46ba-8fde-3e5215a38904" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:54:24.911532 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:24.882470 4611 patch_prober.go:29] interesting pod/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-9d6hc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body= Oct 30 20:54:24.911842 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:24.911807 4611 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-9d6hc" podUID="f1f5526b-53f1-4c2b-a801-49ee75e16f9c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" Oct 30 20:54:25.001590 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:24.994249 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-3d909d/distinct-2-w6-3d909d-kserve-59c5d5c595-cjg2m" Oct 30 20:54:25.033425 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:25.033389 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cb60a2/distinct-w3-cb60a2-kserve-5bb76dc74-86fds" podUID="52e301d9-f378-4e54-9515-01a6680d9fd7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:54:25.038089 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:25.037716 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-14cca2/distinct-w5-14cca2-kserve-587f6996b8-sgf6h" podUID="02b652c3-0724-4a50-9e95-2a9890cc97c4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:54:25.038485 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:25.037761 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b8aee1/distinct-2-w1-b8aee1-kserve-6bcdd59797-cx6hr" podUID="286eab7a-81a7-4768-a7f6-be6b3720028a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:54:25.107646 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:25.107597 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f2ed1e/distinct-w4-f2ed1e-kserve-57cff9dcf8-8d5zl" podUID="653e3ec7-2147-4813-8946-1f25cfa425f6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:54:25.197729 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:25.197671 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-14cca2/premium-w5-14cca2-kserve-6878777857-wpjtj" podUID="b99eb76b-d1c6-49e9-9809-c351c0f5a4de" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:54:25.289591 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:25.272804 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b8aee1/premium-w1-b8aee1-kserve-786d575599-htc8d" podUID="77368afe-edc8-4fa5-8010-d321965fb38e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded" Oct 30 20:54:25.305529 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:25.274343 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:54:25.351538 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:25.274385 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-3d909d/premium-w6-3d909d-kserve-765648946f-2qvd2" podUID="fca5d697-109d-4e90-ab41-c40ad31af874" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:54:25.351538 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:25.332580 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-cb60a2/simulator-w3-cb60a2-kserve-6fd74b9d8b-vqfzf" Oct 30 20:54:25.408522 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:25.388618 4611 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-14cca2/premium-w5-14cca2-kserve-6878777857-wpjtj" podUID="b99eb76b-d1c6-49e9-9809-c351c0f5a4de" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/health\": context deadline exceeded" Oct 30 20:54:25.492549 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:25.388856 4611 patch_prober.go:29] interesting pod/e2e-worker-w2-db534b-openshift-default-85df4d79d9-lmpng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Oct 30 20:54:25.492549 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:25.467176 4611 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-lmpng" podUID="61477918-6577-4957-807b-ebfa7a01f9d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Oct 30 20:54:25.492549 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:25.411086 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db534b/distinct-2-w2-db534b-kserve-8594c876-g92mh" podUID="955316d6-a9fd-4b75-a8ab-bbf382b776e2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:54:25.838928 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:25.784331 4611 patch_prober.go:29] interesting pod/e2e-worker-w6-3d909d-openshift-default-77787695b7-wf8qg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Oct 30 20:54:25.838928 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:25.838810 4611 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-wf8qg" podUID="c468f3c6-7d73-4fd1-ae34-1212e4d1b387" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Oct 30 20:54:25.838928 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:25.784875 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad7ff0/premium-w0-ad7ff0-kserve-5485d4b948-bpmgf" podUID="53cbd692-e9f8-46ba-8fde-3e5215a38904" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:54:25.920649 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:25.920597 4611 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-3d909d/distinct-2-w6-3d909d-kserve-59c5d5c595-cjg2m" podUID="e2b4c221-ee94-4131-abd8-2313647c641e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/health\": context deadline exceeded" Oct 30 20:54:25.965944 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:25.965898 4611 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-cb60a2/distinct-w3-cb60a2-kserve-5bb76dc74-86fds" podUID="52e301d9-f378-4e54-9515-01a6680d9fd7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 20:54:25.968426 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:25.968402 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b8aee1/distinct-2-w1-b8aee1-kserve-6bcdd59797-cx6hr" Oct 30 20:54:26.138625 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:26.020486 4611 patch_prober.go:29] interesting pod/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-9d6hc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body= Oct 30 20:54:26.165540 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:26.146583 4611 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-9d6hc" podUID="f1f5526b-53f1-4c2b-a801-49ee75e16f9c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" Oct 30 20:54:26.320426 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:26.099384 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:54:26.320426 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:26.305670 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cb60a2/simulator-w3-cb60a2-kserve-6fd74b9d8b-vqfzf" podUID="711b167d-4e66-40d6-be75-33ede11f42c5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:54:26.320426 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:26.305739 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f2ed1e/distinct-w4-f2ed1e-kserve-57cff9dcf8-8d5zl" podUID="653e3ec7-2147-4813-8946-1f25cfa425f6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:54:26.336006 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:26.335967 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-3d909d/premium-w6-3d909d-kserve-765648946f-2qvd2" podUID="fca5d697-109d-4e90-ab41-c40ad31af874" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:54:26.345453 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:26.095624 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-14cca2/distinct-w5-14cca2-kserve-587f6996b8-sgf6h" podUID="02b652c3-0724-4a50-9e95-2a9890cc97c4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 20:54:26.572281 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:26.571089 4611 patch_prober.go:29] interesting pod/e2e-worker-w2-db534b-openshift-default-85df4d79d9-lmpng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Oct 30 20:54:26.572281 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:26.571144 4611 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-lmpng" podUID="61477918-6577-4957-807b-ebfa7a01f9d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Oct 30 20:54:26.640150 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:26.640120 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b8aee1/premium-w1-b8aee1-kserve-786d575599-htc8d" Oct 30 20:54:26.674542 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:26.668334 4611 patch_prober.go:29] interesting pod/e2e-worker-w6-3d909d-openshift-default-77787695b7-wf8qg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Oct 30 20:54:26.674542 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:26.668394 4611 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-wf8qg" podUID="c468f3c6-7d73-4fd1-ae34-1212e4d1b387" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Oct 30 20:54:26.675941 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:26.675921 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-14cca2/premium-w5-14cca2-kserve-6878777857-wpjtj" Oct 30 20:54:26.703736 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:26.703706 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-cb60a2/distinct-w3-cb60a2-kserve-5bb76dc74-86fds" Oct 30 20:54:26.705473 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:26.705454 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-db534b/distinct-2-w2-db534b-kserve-8594c876-g92mh" Oct 30 20:54:26.706817 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:26.706799 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ad7ff0/premium-w0-ad7ff0-kserve-5485d4b948-bpmgf" Oct 30 20:54:26.841890 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:26.841805 4611 patch_prober.go:29] interesting pod/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-9d6hc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body= Oct 30 20:54:26.842424 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:26.842394 4611 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-9d6hc" podUID="f1f5526b-53f1-4c2b-a801-49ee75e16f9c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" Oct 30 20:54:26.845763 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:26.845740 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-3d909d/premium-w6-3d909d-kserve-765648946f-2qvd2" Oct 30 20:54:26.892255 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:26.892225 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-14cca2/distinct-w5-14cca2-kserve-587f6996b8-sgf6h" Oct 30 20:54:26.958561 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:26.930758 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:54:26.958561 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:26.942629 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-f2ed1e/distinct-w4-f2ed1e-kserve-57cff9dcf8-8d5zl" Oct 30 20:54:27.405738 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:27.405686 4611 patch_prober.go:29] interesting pod/e2e-worker-w2-db534b-openshift-default-85df4d79d9-lmpng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Oct 30 20:54:27.406049 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:27.406005 4611 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-lmpng" podUID="61477918-6577-4957-807b-ebfa7a01f9d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Oct 30 20:54:27.658590 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:27.655424 4611 patch_prober.go:29] interesting pod/e2e-worker-w6-3d909d-openshift-default-77787695b7-wf8qg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Oct 30 20:54:27.658590 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:27.655488 4611 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-wf8qg" podUID="c468f3c6-7d73-4fd1-ae34-1212e4d1b387" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Oct 30 20:54:27.841033 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:27.841003 4611 patch_prober.go:29] interesting pod/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-9d6hc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body= Oct 30 20:54:27.841204 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:27.841053 4611 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-9d6hc" podUID="f1f5526b-53f1-4c2b-a801-49ee75e16f9c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" Oct 30 20:54:27.990728 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:27.990632 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:54:28.391532 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:28.380906 4611 patch_prober.go:29] interesting pod/e2e-worker-w2-db534b-openshift-default-85df4d79d9-lmpng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Oct 30 20:54:28.391532 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:28.380980 4611 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-lmpng" podUID="61477918-6577-4957-807b-ebfa7a01f9d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Oct 30 20:54:28.668293 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:28.668206 4611 patch_prober.go:29] interesting pod/e2e-worker-w6-3d909d-openshift-default-77787695b7-wf8qg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Oct 30 20:54:28.668562 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:28.668532 4611 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-wf8qg" podUID="c468f3c6-7d73-4fd1-ae34-1212e4d1b387" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Oct 30 20:54:28.839868 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:28.839829 4611 patch_prober.go:29] interesting pod/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-9d6hc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body= Oct 30 20:54:28.840149 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:28.840119 4611 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-9d6hc" podUID="f1f5526b-53f1-4c2b-a801-49ee75e16f9c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" Oct 30 20:54:28.902895 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:28.902216 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:54:29.385531 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:29.384169 4611 patch_prober.go:29] interesting pod/e2e-worker-w2-db534b-openshift-default-85df4d79d9-lmpng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Oct 30 20:54:29.385531 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:29.384228 4611 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-lmpng" podUID="61477918-6577-4957-807b-ebfa7a01f9d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Oct 30 20:54:29.520531 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:29.516302 4611 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-czzct"] Oct 30 20:54:29.532882 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:29.529937 4611 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-czzct" Oct 30 20:54:29.596599 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:29.556437 4611 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-czzct"] Oct 30 20:54:29.596599 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:29.562140 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jq8hn\" (UniqueName: \"kubernetes.io/projected/6e28fa18-8c44-4272-b3a6-9448e957b02a-kube-api-access-jq8hn\") pod \"istiod-openshift-gateway-7ddd9cfdc8-czzct\" (UID: \"6e28fa18-8c44-4272-b3a6-9448e957b02a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-czzct" Oct 30 20:54:29.596599 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:29.594913 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/6e28fa18-8c44-4272-b3a6-9448e957b02a-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-czzct\" (UID: \"6e28fa18-8c44-4272-b3a6-9448e957b02a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-czzct" Oct 30 20:54:29.596599 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:29.594962 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/6e28fa18-8c44-4272-b3a6-9448e957b02a-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-czzct\" (UID: \"6e28fa18-8c44-4272-b3a6-9448e957b02a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-czzct" Oct 30 20:54:29.596599 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:29.595138 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/6e28fa18-8c44-4272-b3a6-9448e957b02a-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-czzct\" (UID: \"6e28fa18-8c44-4272-b3a6-9448e957b02a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-czzct" Oct 30 20:54:29.596599 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:29.595196 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/6e28fa18-8c44-4272-b3a6-9448e957b02a-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-czzct\" (UID: \"6e28fa18-8c44-4272-b3a6-9448e957b02a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-czzct" Oct 30 20:54:29.596599 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:29.595366 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/6e28fa18-8c44-4272-b3a6-9448e957b02a-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-czzct\" (UID: \"6e28fa18-8c44-4272-b3a6-9448e957b02a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-czzct" Oct 30 20:54:29.596599 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:29.595476 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6e28fa18-8c44-4272-b3a6-9448e957b02a-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-czzct\" (UID: \"6e28fa18-8c44-4272-b3a6-9448e957b02a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-czzct" Oct 30 20:54:29.698644 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:29.683870 4611 patch_prober.go:29] interesting pod/e2e-worker-w6-3d909d-openshift-default-77787695b7-wf8qg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Oct 30 20:54:29.698644 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:29.683925 4611 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-wf8qg" podUID="c468f3c6-7d73-4fd1-ae34-1212e4d1b387" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Oct 30 20:54:29.698644 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:29.685034 4611 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Oct 30 20:54:29.698644 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:29.685092 4611 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Oct 30 20:54:29.698644 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:29.685799 4611 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Oct 30 20:54:29.698644 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:29.685959 4611 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-n56bm\"" Oct 30 20:54:29.700480 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:29.700447 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6e28fa18-8c44-4272-b3a6-9448e957b02a-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-czzct\" (UID: \"6e28fa18-8c44-4272-b3a6-9448e957b02a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-czzct" Oct 30 20:54:29.700716 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:29.700696 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jq8hn\" (UniqueName: \"kubernetes.io/projected/6e28fa18-8c44-4272-b3a6-9448e957b02a-kube-api-access-jq8hn\") pod \"istiod-openshift-gateway-7ddd9cfdc8-czzct\" (UID: \"6e28fa18-8c44-4272-b3a6-9448e957b02a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-czzct" Oct 30 20:54:29.700838 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:29.700824 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/6e28fa18-8c44-4272-b3a6-9448e957b02a-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-czzct\" (UID: \"6e28fa18-8c44-4272-b3a6-9448e957b02a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-czzct" Oct 30 20:54:29.700948 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:29.700935 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/6e28fa18-8c44-4272-b3a6-9448e957b02a-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-czzct\" (UID: \"6e28fa18-8c44-4272-b3a6-9448e957b02a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-czzct" Oct 30 20:54:29.701164 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:29.701150 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/6e28fa18-8c44-4272-b3a6-9448e957b02a-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-czzct\" (UID: \"6e28fa18-8c44-4272-b3a6-9448e957b02a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-czzct" Oct 30 20:54:29.701869 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:29.701850 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/6e28fa18-8c44-4272-b3a6-9448e957b02a-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-czzct\" (UID: \"6e28fa18-8c44-4272-b3a6-9448e957b02a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-czzct" Oct 30 20:54:29.702119 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:29.702104 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/6e28fa18-8c44-4272-b3a6-9448e957b02a-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-czzct\" (UID: \"6e28fa18-8c44-4272-b3a6-9448e957b02a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-czzct" Oct 30 20:54:29.718573 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:29.701668 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:54:29.728572 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:29.719453 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/6e28fa18-8c44-4272-b3a6-9448e957b02a-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-czzct\" (UID: \"6e28fa18-8c44-4272-b3a6-9448e957b02a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-czzct" Oct 30 20:54:29.858530 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:29.839814 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jq8hn\" (UniqueName: \"kubernetes.io/projected/6e28fa18-8c44-4272-b3a6-9448e957b02a-kube-api-access-jq8hn\") pod \"istiod-openshift-gateway-7ddd9cfdc8-czzct\" (UID: \"6e28fa18-8c44-4272-b3a6-9448e957b02a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-czzct" Oct 30 20:54:29.860129 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:29.860048 4611 patch_prober.go:29] interesting pod/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-9d6hc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body= Oct 30 20:54:29.860129 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:29.860112 4611 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-9d6hc" podUID="f1f5526b-53f1-4c2b-a801-49ee75e16f9c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" Oct 30 20:54:29.860821 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:29.860796 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6e28fa18-8c44-4272-b3a6-9448e957b02a-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-czzct\" (UID: \"6e28fa18-8c44-4272-b3a6-9448e957b02a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-czzct" Oct 30 20:54:29.860980 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:29.860826 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/6e28fa18-8c44-4272-b3a6-9448e957b02a-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-czzct\" (UID: \"6e28fa18-8c44-4272-b3a6-9448e957b02a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-czzct" Oct 30 20:54:29.861127 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:29.861101 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/6e28fa18-8c44-4272-b3a6-9448e957b02a-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-czzct\" (UID: \"6e28fa18-8c44-4272-b3a6-9448e957b02a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-czzct" Oct 30 20:54:29.894085 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:29.894045 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/6e28fa18-8c44-4272-b3a6-9448e957b02a-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-czzct\" (UID: \"6e28fa18-8c44-4272-b3a6-9448e957b02a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-czzct" Oct 30 20:54:29.966570 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:29.966422 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/6e28fa18-8c44-4272-b3a6-9448e957b02a-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-czzct\" (UID: \"6e28fa18-8c44-4272-b3a6-9448e957b02a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-czzct" Oct 30 20:54:30.164785 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:30.164747 4611 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-czzct" Oct 30 20:54:30.391823 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:30.391778 4611 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-lmpng" Oct 30 20:54:30.393571 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:30.393499 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-lmpng" Oct 30 20:54:30.426956 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:30.426635 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:54:30.446822 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:30.446790 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-lmpng" Oct 30 20:54:30.594034 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:30.593997 4611 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-czzct"] Oct 30 20:54:31.018451 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:31.018421 4611 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-9d6hc" Oct 30 20:54:31.020031 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:31.020011 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-9d6hc" Oct 30 20:54:31.021018 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:31.020998 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-9d6hc" Oct 30 20:54:31.053527 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:31.052190 4611 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-wf8qg" Oct 30 20:54:31.053710 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:31.053580 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-wf8qg" Oct 30 20:54:31.056566 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:31.055125 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-wf8qg" Oct 30 20:54:31.195323 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:31.195285 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:54:31.380437 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:31.380399 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-czzct" event={"ID":"6e28fa18-8c44-4272-b3a6-9448e957b02a","Type":"ContainerStarted","Data":"38f1edf942d90f86a3db980d0800e3da251d31d4bd138a57ede1f35e03c26f20"} Oct 30 20:54:31.887775 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:31.887735 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:54:32.376748 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:32.376714 4611 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-qkwks"] Oct 30 20:54:32.382380 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:32.382352 4611 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-qkwks" Oct 30 20:54:32.393264 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:32.393123 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db1a832a-0df2-4a93-bee2-40c9b99218d9-workload-socket\") pod \"e2e-worker-w6-3d909d-openshift-default-77787695b7-qkwks\" (UID: \"db1a832a-0df2-4a93-bee2-40c9b99218d9\") " pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-qkwks" Oct 30 20:54:32.393264 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:32.393182 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db1a832a-0df2-4a93-bee2-40c9b99218d9-workload-certs\") pod \"e2e-worker-w6-3d909d-openshift-default-77787695b7-qkwks\" (UID: \"db1a832a-0df2-4a93-bee2-40c9b99218d9\") " pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-qkwks" Oct 30 20:54:32.393264 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:32.393208 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db1a832a-0df2-4a93-bee2-40c9b99218d9-istio-data\") pod \"e2e-worker-w6-3d909d-openshift-default-77787695b7-qkwks\" (UID: \"db1a832a-0df2-4a93-bee2-40c9b99218d9\") " pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-qkwks" Oct 30 20:54:32.393546 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:32.393286 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db1a832a-0df2-4a93-bee2-40c9b99218d9-istio-envoy\") pod \"e2e-worker-w6-3d909d-openshift-default-77787695b7-qkwks\" (UID: \"db1a832a-0df2-4a93-bee2-40c9b99218d9\") " pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-qkwks" Oct 30 20:54:32.393546 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:32.393340 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mnf2z\" (UniqueName: \"kubernetes.io/projected/db1a832a-0df2-4a93-bee2-40c9b99218d9-kube-api-access-mnf2z\") pod \"e2e-worker-w6-3d909d-openshift-default-77787695b7-qkwks\" (UID: \"db1a832a-0df2-4a93-bee2-40c9b99218d9\") " pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-qkwks" Oct 30 20:54:32.393653 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:32.393585 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db1a832a-0df2-4a93-bee2-40c9b99218d9-istiod-ca-cert\") pod \"e2e-worker-w6-3d909d-openshift-default-77787695b7-qkwks\" (UID: \"db1a832a-0df2-4a93-bee2-40c9b99218d9\") " pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-qkwks" Oct 30 20:54:32.393715 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:32.393661 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db1a832a-0df2-4a93-bee2-40c9b99218d9-istio-token\") pod \"e2e-worker-w6-3d909d-openshift-default-77787695b7-qkwks\" (UID: \"db1a832a-0df2-4a93-bee2-40c9b99218d9\") " pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-qkwks" Oct 30 20:54:32.393715 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:32.393697 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db1a832a-0df2-4a93-bee2-40c9b99218d9-credential-socket\") pod \"e2e-worker-w6-3d909d-openshift-default-77787695b7-qkwks\" (UID: \"db1a832a-0df2-4a93-bee2-40c9b99218d9\") " pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-qkwks" Oct 30 20:54:32.393824 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:32.393732 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db1a832a-0df2-4a93-bee2-40c9b99218d9-istio-podinfo\") pod \"e2e-worker-w6-3d909d-openshift-default-77787695b7-qkwks\" (UID: \"db1a832a-0df2-4a93-bee2-40c9b99218d9\") " pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-qkwks" Oct 30 20:54:32.399276 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:32.399254 4611 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-qkwks"] Oct 30 20:54:32.495288 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:32.495244 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db1a832a-0df2-4a93-bee2-40c9b99218d9-workload-socket\") pod \"e2e-worker-w6-3d909d-openshift-default-77787695b7-qkwks\" (UID: \"db1a832a-0df2-4a93-bee2-40c9b99218d9\") " pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-qkwks" Oct 30 20:54:32.495589 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:32.495561 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db1a832a-0df2-4a93-bee2-40c9b99218d9-workload-certs\") pod \"e2e-worker-w6-3d909d-openshift-default-77787695b7-qkwks\" (UID: \"db1a832a-0df2-4a93-bee2-40c9b99218d9\") " pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-qkwks" Oct 30 20:54:32.495680 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:32.495606 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db1a832a-0df2-4a93-bee2-40c9b99218d9-istio-data\") pod \"e2e-worker-w6-3d909d-openshift-default-77787695b7-qkwks\" (UID: \"db1a832a-0df2-4a93-bee2-40c9b99218d9\") " pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-qkwks" Oct 30 20:54:32.495738 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:32.495662 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db1a832a-0df2-4a93-bee2-40c9b99218d9-workload-socket\") pod \"e2e-worker-w6-3d909d-openshift-default-77787695b7-qkwks\" (UID: \"db1a832a-0df2-4a93-bee2-40c9b99218d9\") " pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-qkwks" Oct 30 20:54:32.495799 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:32.495746 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db1a832a-0df2-4a93-bee2-40c9b99218d9-istio-envoy\") pod \"e2e-worker-w6-3d909d-openshift-default-77787695b7-qkwks\" (UID: \"db1a832a-0df2-4a93-bee2-40c9b99218d9\") " pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-qkwks" Oct 30 20:54:32.495859 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:32.495823 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mnf2z\" (UniqueName: \"kubernetes.io/projected/db1a832a-0df2-4a93-bee2-40c9b99218d9-kube-api-access-mnf2z\") pod \"e2e-worker-w6-3d909d-openshift-default-77787695b7-qkwks\" (UID: \"db1a832a-0df2-4a93-bee2-40c9b99218d9\") " pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-qkwks" Oct 30 20:54:32.495921 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:32.495871 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db1a832a-0df2-4a93-bee2-40c9b99218d9-workload-certs\") pod \"e2e-worker-w6-3d909d-openshift-default-77787695b7-qkwks\" (UID: \"db1a832a-0df2-4a93-bee2-40c9b99218d9\") " pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-qkwks" Oct 30 20:54:32.495998 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:32.495977 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db1a832a-0df2-4a93-bee2-40c9b99218d9-istio-data\") pod \"e2e-worker-w6-3d909d-openshift-default-77787695b7-qkwks\" (UID: \"db1a832a-0df2-4a93-bee2-40c9b99218d9\") " pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-qkwks" Oct 30 20:54:32.496136 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:32.496117 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db1a832a-0df2-4a93-bee2-40c9b99218d9-istiod-ca-cert\") pod \"e2e-worker-w6-3d909d-openshift-default-77787695b7-qkwks\" (UID: \"db1a832a-0df2-4a93-bee2-40c9b99218d9\") " pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-qkwks" Oct 30 20:54:32.496237 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:32.496222 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db1a832a-0df2-4a93-bee2-40c9b99218d9-istio-token\") pod \"e2e-worker-w6-3d909d-openshift-default-77787695b7-qkwks\" (UID: \"db1a832a-0df2-4a93-bee2-40c9b99218d9\") " pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-qkwks" Oct 30 20:54:32.496356 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:32.496334 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db1a832a-0df2-4a93-bee2-40c9b99218d9-credential-socket\") pod \"e2e-worker-w6-3d909d-openshift-default-77787695b7-qkwks\" (UID: \"db1a832a-0df2-4a93-bee2-40c9b99218d9\") " pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-qkwks" Oct 30 20:54:32.496485 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:32.496385 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db1a832a-0df2-4a93-bee2-40c9b99218d9-istio-podinfo\") pod \"e2e-worker-w6-3d909d-openshift-default-77787695b7-qkwks\" (UID: \"db1a832a-0df2-4a93-bee2-40c9b99218d9\") " pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-qkwks" Oct 30 20:54:32.497076 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:32.497053 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db1a832a-0df2-4a93-bee2-40c9b99218d9-credential-socket\") pod \"e2e-worker-w6-3d909d-openshift-default-77787695b7-qkwks\" (UID: \"db1a832a-0df2-4a93-bee2-40c9b99218d9\") " pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-qkwks" Oct 30 20:54:32.497274 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:32.497224 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db1a832a-0df2-4a93-bee2-40c9b99218d9-istiod-ca-cert\") pod \"e2e-worker-w6-3d909d-openshift-default-77787695b7-qkwks\" (UID: \"db1a832a-0df2-4a93-bee2-40c9b99218d9\") " pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-qkwks" Oct 30 20:54:32.498655 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:32.498630 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db1a832a-0df2-4a93-bee2-40c9b99218d9-istio-envoy\") pod \"e2e-worker-w6-3d909d-openshift-default-77787695b7-qkwks\" (UID: \"db1a832a-0df2-4a93-bee2-40c9b99218d9\") " pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-qkwks" Oct 30 20:54:32.499272 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:32.499250 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db1a832a-0df2-4a93-bee2-40c9b99218d9-istio-podinfo\") pod \"e2e-worker-w6-3d909d-openshift-default-77787695b7-qkwks\" (UID: \"db1a832a-0df2-4a93-bee2-40c9b99218d9\") " pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-qkwks" Oct 30 20:54:32.510471 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:32.510444 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db1a832a-0df2-4a93-bee2-40c9b99218d9-istio-token\") pod \"e2e-worker-w6-3d909d-openshift-default-77787695b7-qkwks\" (UID: \"db1a832a-0df2-4a93-bee2-40c9b99218d9\") " pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-qkwks" Oct 30 20:54:32.510989 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:32.510964 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mnf2z\" (UniqueName: \"kubernetes.io/projected/db1a832a-0df2-4a93-bee2-40c9b99218d9-kube-api-access-mnf2z\") pod \"e2e-worker-w6-3d909d-openshift-default-77787695b7-qkwks\" (UID: \"db1a832a-0df2-4a93-bee2-40c9b99218d9\") " pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-qkwks" Oct 30 20:54:32.699337 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:32.699243 4611 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-qkwks" Oct 30 20:54:32.701593 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:32.701307 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:54:33.017609 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:33.017573 4611 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-qkwks"] Oct 30 20:54:33.018283 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:54:33.018257 4611 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb1a832a_0df2_4a93_bee2_40c9b99218d9.slice/crio-0e26a9f9931834fb2fabf8784ba3ff1ca5f57fc3f16abe5e2eda23b9a3d2d0d3 WatchSource:0}: Error finding container 0e26a9f9931834fb2fabf8784ba3ff1ca5f57fc3f16abe5e2eda23b9a3d2d0d3: Status 404 returned error can't find the container with id 0e26a9f9931834fb2fabf8784ba3ff1ca5f57fc3f16abe5e2eda23b9a3d2d0d3 Oct 30 20:54:33.021751 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:33.021714 4611 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:54:33.021845 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:33.021799 4611 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:54:33.021889 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:33.021840 4611 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:54:33.371279 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:33.371237 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:54:33.406763 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:33.405591 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-qkwks" event={"ID":"db1a832a-0df2-4a93-bee2-40c9b99218d9","Type":"ContainerStarted","Data":"00b4462f302792be026e0808d35c51a456553937259897e483726e2dfb2fbed4"} Oct 30 20:54:33.406763 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:33.405675 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-qkwks" event={"ID":"db1a832a-0df2-4a93-bee2-40c9b99218d9","Type":"ContainerStarted","Data":"0e26a9f9931834fb2fabf8784ba3ff1ca5f57fc3f16abe5e2eda23b9a3d2d0d3"} Oct 30 20:54:33.433920 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:33.430005 4611 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-qkwks" podStartSLOduration=1.429986274 podStartE2EDuration="1.429986274s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:54:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:54:33.429370296 +0000 UTC m=+927.963396931" watchObservedRunningTime="2026-10-30 20:54:33.429986274 +0000 UTC m=+927.964012891" Oct 30 20:54:33.699445 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:33.699357 4611 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-qkwks" Oct 30 20:54:33.922266 ip-10-0-133-192 kubenswrapper[4611]: E1030 20:54:33.922234 4611 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hwzc6" containerName="extractor" Oct 30 20:54:34.410921 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:34.410876 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:54:34.702360 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:34.702282 4611 patch_prober.go:29] interesting pod/e2e-worker-w6-3d909d-openshift-default-77787695b7-qkwks container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Oct 30 20:54:34.702360 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:34.702343 4611 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-qkwks" podUID="db1a832a-0df2-4a93-bee2-40c9b99218d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Oct 30 20:54:34.949554 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:34.945973 4611 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:54:34.949554 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:34.946057 4611 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:54:35.114886 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:35.114845 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:54:35.482530 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:35.481575 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-czzct" event={"ID":"6e28fa18-8c44-4272-b3a6-9448e957b02a","Type":"ContainerStarted","Data":"5d9cb080123e0e5bb89149b2b258f90deebe74499536f29788f136cb15125472"} Oct 30 20:54:35.482530 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:35.481664 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-czzct" Oct 30 20:54:35.514912 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:35.514854 4611 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-czzct" podStartSLOduration=2.167170231 podStartE2EDuration="6.514833845s" totalImagesPullingTime="4.347663614s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:54:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:54:35.512005158 +0000 UTC m=+930.046031800" watchObservedRunningTime="2026-10-30 20:54:35.514833845 +0000 UTC m=+930.048860462" Oct 30 20:54:35.701536 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:35.699806 4611 patch_prober.go:29] interesting pod/e2e-worker-w6-3d909d-openshift-default-77787695b7-qkwks container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Oct 30 20:54:35.701536 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:35.699882 4611 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-qkwks" podUID="db1a832a-0df2-4a93-bee2-40c9b99218d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Oct 30 20:54:35.990845 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:35.990808 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:54:36.489199 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:36.489171 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-czzct" Oct 30 20:54:36.699960 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:36.699923 4611 patch_prober.go:29] interesting pod/e2e-worker-w6-3d909d-openshift-default-77787695b7-qkwks container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Oct 30 20:54:36.700122 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:36.699978 4611 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-qkwks" podUID="db1a832a-0df2-4a93-bee2-40c9b99218d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Oct 30 20:54:36.730206 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:36.730168 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:54:37.480584 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:37.480502 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:54:37.700359 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:37.700320 4611 patch_prober.go:29] interesting pod/e2e-worker-w6-3d909d-openshift-default-77787695b7-qkwks container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Oct 30 20:54:37.700780 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:37.700394 4611 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-qkwks" podUID="db1a832a-0df2-4a93-bee2-40c9b99218d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Oct 30 20:54:38.218636 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:38.218594 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:54:38.700443 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:38.700410 4611 patch_prober.go:29] interesting pod/e2e-worker-w6-3d909d-openshift-default-77787695b7-qkwks container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Oct 30 20:54:38.700845 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:38.700474 4611 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-qkwks" podUID="db1a832a-0df2-4a93-bee2-40c9b99218d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Oct 30 20:54:39.073876 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:39.073765 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:54:39.700038 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:39.700005 4611 patch_prober.go:29] interesting pod/e2e-worker-w6-3d909d-openshift-default-77787695b7-qkwks container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Oct 30 20:54:39.700216 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:39.700065 4611 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-qkwks" podUID="db1a832a-0df2-4a93-bee2-40c9b99218d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Oct 30 20:54:39.929274 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:39.929240 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:54:40.676321 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:40.676287 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:54:40.704654 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:40.704625 4611 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-qkwks" Oct 30 20:54:40.705149 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:40.705114 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-qkwks" Oct 30 20:54:40.706055 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:40.706035 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-qkwks" Oct 30 20:54:41.771997 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:41.771962 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:54:42.395871 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:42.395830 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:54:42.594880 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:42.594843 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:54:43.195614 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:43.195577 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:54:43.850549 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:43.850490 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:54:43.878359 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:43.878323 4611 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7e8fa042-8xjrj"] Oct 30 20:54:43.883309 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:43.883285 4611 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e8fa042-8xjrj" Oct 30 20:54:43.887186 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:43.887163 4611 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:54:43.893075 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:43.892860 4611 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7hf8r\"" Oct 30 20:54:43.897604 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:43.897125 4611 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7e8fa042-8xjrj"] Oct 30 20:54:43.981822 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:43.981779 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23c4eefd-503e-4988-8777-cd4f6f91247b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7e8fa042-8xjrj\" (UID: \"23c4eefd-503e-4988-8777-cd4f6f91247b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e8fa042-8xjrj" Oct 30 20:54:43.982018 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:43.981844 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23c4eefd-503e-4988-8777-cd4f6f91247b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7e8fa042-8xjrj\" (UID: \"23c4eefd-503e-4988-8777-cd4f6f91247b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e8fa042-8xjrj" Oct 30 20:54:43.982094 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:43.982006 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jmpcx\" (UniqueName: \"kubernetes.io/projected/23c4eefd-503e-4988-8777-cd4f6f91247b-kube-api-access-jmpcx\") pod \"maas-api-revoke-sub-7e8fa042-8xjrj\" (UID: \"23c4eefd-503e-4988-8777-cd4f6f91247b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e8fa042-8xjrj" Oct 30 20:54:44.083607 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:44.083575 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23c4eefd-503e-4988-8777-cd4f6f91247b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7e8fa042-8xjrj\" (UID: \"23c4eefd-503e-4988-8777-cd4f6f91247b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e8fa042-8xjrj" Oct 30 20:54:44.083944 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:44.083693 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23c4eefd-503e-4988-8777-cd4f6f91247b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7e8fa042-8xjrj\" (UID: \"23c4eefd-503e-4988-8777-cd4f6f91247b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e8fa042-8xjrj" Oct 30 20:54:44.083944 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:44.083754 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jmpcx\" (UniqueName: \"kubernetes.io/projected/23c4eefd-503e-4988-8777-cd4f6f91247b-kube-api-access-jmpcx\") pod \"maas-api-revoke-sub-7e8fa042-8xjrj\" (UID: \"23c4eefd-503e-4988-8777-cd4f6f91247b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e8fa042-8xjrj" Oct 30 20:54:44.084308 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:44.084282 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23c4eefd-503e-4988-8777-cd4f6f91247b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7e8fa042-8xjrj\" (UID: \"23c4eefd-503e-4988-8777-cd4f6f91247b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e8fa042-8xjrj" Oct 30 20:54:44.088134 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:44.088105 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23c4eefd-503e-4988-8777-cd4f6f91247b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7e8fa042-8xjrj\" (UID: \"23c4eefd-503e-4988-8777-cd4f6f91247b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e8fa042-8xjrj" Oct 30 20:54:44.096489 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:44.096462 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jmpcx\" (UniqueName: \"kubernetes.io/projected/23c4eefd-503e-4988-8777-cd4f6f91247b-kube-api-access-jmpcx\") pod \"maas-api-revoke-sub-7e8fa042-8xjrj\" (UID: \"23c4eefd-503e-4988-8777-cd4f6f91247b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e8fa042-8xjrj" Oct 30 20:54:44.217106 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:44.217069 4611 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e8fa042-8xjrj" Oct 30 20:54:44.510572 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:44.510539 4611 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7e8fa042-8xjrj"] Oct 30 20:54:44.511082 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:54:44.511061 4611 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23c4eefd_503e_4988_8777_cd4f6f91247b.slice/crio-d759f19f10a4ef15627b41f5671373705449e598bdb9abe657d0130a9a375de5 WatchSource:0}: Error finding container d759f19f10a4ef15627b41f5671373705449e598bdb9abe657d0130a9a375de5: Status 404 returned error can't find the container with id d759f19f10a4ef15627b41f5671373705449e598bdb9abe657d0130a9a375de5 Oct 30 20:54:44.551393 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:44.551365 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e8fa042-8xjrj" event={"ID":"23c4eefd-503e-4988-8777-cd4f6f91247b","Type":"ContainerStarted","Data":"d759f19f10a4ef15627b41f5671373705449e598bdb9abe657d0130a9a375de5"} Oct 30 20:54:44.814297 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:44.814258 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:54:45.503223 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:45.503189 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:54:45.586455 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:45.586416 4611 generic.go:356] "Generic (PLEG): container finished" podID="23c4eefd-503e-4988-8777-cd4f6f91247b" containerID="3b11347347b19a06bec0fbed9f23c6395be0c1630fc7ffd7d8ae1f4716093e23" exitCode=0 Oct 30 20:54:45.586699 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:45.586528 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e8fa042-8xjrj" event={"ID":"23c4eefd-503e-4988-8777-cd4f6f91247b","Type":"ContainerDied","Data":"3b11347347b19a06bec0fbed9f23c6395be0c1630fc7ffd7d8ae1f4716093e23"} Oct 30 20:54:46.186568 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:46.186490 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:54:46.779694 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:46.779671 4611 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e8fa042-8xjrj" Oct 30 20:54:46.803134 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:46.803100 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:54:46.947567 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:46.938238 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/23c4eefd-503e-4988-8777-cd4f6f91247b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23c4eefd-503e-4988-8777-cd4f6f91247b-maas-api-openshift-ca\") pod \"23c4eefd-503e-4988-8777-cd4f6f91247b\" (UID: \"23c4eefd-503e-4988-8777-cd4f6f91247b\") " Oct 30 20:54:46.947567 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:46.938320 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/23c4eefd-503e-4988-8777-cd4f6f91247b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23c4eefd-503e-4988-8777-cd4f6f91247b-maas-api-xks-ca\") pod \"23c4eefd-503e-4988-8777-cd4f6f91247b\" (UID: \"23c4eefd-503e-4988-8777-cd4f6f91247b\") " Oct 30 20:54:46.947567 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:46.938439 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/23c4eefd-503e-4988-8777-cd4f6f91247b-kube-api-access-jmpcx\" (UniqueName: \"kubernetes.io/projected/23c4eefd-503e-4988-8777-cd4f6f91247b-kube-api-access-jmpcx\") pod \"23c4eefd-503e-4988-8777-cd4f6f91247b\" (UID: \"23c4eefd-503e-4988-8777-cd4f6f91247b\") " Oct 30 20:54:46.947567 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:46.939670 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23c4eefd-503e-4988-8777-cd4f6f91247b-maas-api-openshift-ca" pod "23c4eefd-503e-4988-8777-cd4f6f91247b" (UID: "23c4eefd-503e-4988-8777-cd4f6f91247b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:54:46.949987 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:46.949947 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/23c4eefd-503e-4988-8777-cd4f6f91247b-maas-api-xks-ca" pod "23c4eefd-503e-4988-8777-cd4f6f91247b" (UID: "23c4eefd-503e-4988-8777-cd4f6f91247b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:54:46.950162 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:46.949989 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23c4eefd-503e-4988-8777-cd4f6f91247b-kube-api-access-jmpcx" pod "23c4eefd-503e-4988-8777-cd4f6f91247b" (UID: "23c4eefd-503e-4988-8777-cd4f6f91247b"). InnerVolumeSpecName "kube-api-access-jmpcx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:54:47.011736 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:47.011702 4611 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-td2hl"] Oct 30 20:54:47.012761 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:47.012741 4611 cpu_manager.go:414] "containerMap: removing container" podUID="23c4eefd-503e-4988-8777-cd4f6f91247b" containerName="revoke-keys" Oct 30 20:54:47.012817 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:47.012764 4611 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="23c4eefd-503e-4988-8777-cd4f6f91247b" containerName="revoke-keys" Oct 30 20:54:47.013000 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:47.012976 4611 memory_manager.go:365] "RemoveStaleState removing state" podUID="23c4eefd-503e-4988-8777-cd4f6f91247b" containerName="revoke-keys" Oct 30 20:54:47.016556 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:47.016540 4611 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-td2hl" Oct 30 20:54:47.029962 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:47.029681 4611 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-td2hl"] Oct 30 20:54:47.046768 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:47.046737 4611 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23c4eefd-503e-4988-8777-cd4f6f91247b-maas-api-openshift-ca\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:54:47.046889 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:47.046772 4611 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23c4eefd-503e-4988-8777-cd4f6f91247b-maas-api-xks-ca\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:54:47.046889 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:47.046788 4611 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jmpcx\" (UniqueName: \"kubernetes.io/projected/23c4eefd-503e-4988-8777-cd4f6f91247b-kube-api-access-jmpcx\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:54:47.148020 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:47.147989 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a08fd3b8-493c-47db-9839-462bcc598e92-istio-envoy\") pod \"e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-td2hl\" (UID: \"a08fd3b8-493c-47db-9839-462bcc598e92\") " pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-td2hl" Oct 30 20:54:47.148020 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:47.148023 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a08fd3b8-493c-47db-9839-462bcc598e92-istio-podinfo\") pod \"e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-td2hl\" (UID: \"a08fd3b8-493c-47db-9839-462bcc598e92\") " pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-td2hl" Oct 30 20:54:47.148277 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:47.148161 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a08fd3b8-493c-47db-9839-462bcc598e92-workload-socket\") pod \"e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-td2hl\" (UID: \"a08fd3b8-493c-47db-9839-462bcc598e92\") " pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-td2hl" Oct 30 20:54:47.148277 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:47.148204 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a08fd3b8-493c-47db-9839-462bcc598e92-istiod-ca-cert\") pod \"e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-td2hl\" (UID: \"a08fd3b8-493c-47db-9839-462bcc598e92\") " pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-td2hl" Oct 30 20:54:47.148357 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:47.148295 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a08fd3b8-493c-47db-9839-462bcc598e92-istio-token\") pod \"e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-td2hl\" (UID: \"a08fd3b8-493c-47db-9839-462bcc598e92\") " pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-td2hl" Oct 30 20:54:47.148391 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:47.148352 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d2blq\" (UniqueName: \"kubernetes.io/projected/a08fd3b8-493c-47db-9839-462bcc598e92-kube-api-access-d2blq\") pod \"e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-td2hl\" (UID: \"a08fd3b8-493c-47db-9839-462bcc598e92\") " pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-td2hl" Oct 30 20:54:47.148495 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:47.148467 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a08fd3b8-493c-47db-9839-462bcc598e92-credential-socket\") pod \"e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-td2hl\" (UID: \"a08fd3b8-493c-47db-9839-462bcc598e92\") " pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-td2hl" Oct 30 20:54:47.148567 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:47.148558 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a08fd3b8-493c-47db-9839-462bcc598e92-workload-certs\") pod \"e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-td2hl\" (UID: \"a08fd3b8-493c-47db-9839-462bcc598e92\") " pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-td2hl" Oct 30 20:54:47.148603 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:47.148580 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a08fd3b8-493c-47db-9839-462bcc598e92-istio-data\") pod \"e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-td2hl\" (UID: \"a08fd3b8-493c-47db-9839-462bcc598e92\") " pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-td2hl" Oct 30 20:54:47.250418 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:47.250337 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a08fd3b8-493c-47db-9839-462bcc598e92-istio-token\") pod \"e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-td2hl\" (UID: \"a08fd3b8-493c-47db-9839-462bcc598e92\") " pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-td2hl" Oct 30 20:54:47.250641 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:47.250429 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d2blq\" (UniqueName: \"kubernetes.io/projected/a08fd3b8-493c-47db-9839-462bcc598e92-kube-api-access-d2blq\") pod \"e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-td2hl\" (UID: \"a08fd3b8-493c-47db-9839-462bcc598e92\") " pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-td2hl" Oct 30 20:54:47.250641 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:47.250596 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a08fd3b8-493c-47db-9839-462bcc598e92-credential-socket\") pod \"e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-td2hl\" (UID: \"a08fd3b8-493c-47db-9839-462bcc598e92\") " pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-td2hl" Oct 30 20:54:47.250774 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:47.250647 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a08fd3b8-493c-47db-9839-462bcc598e92-workload-certs\") pod \"e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-td2hl\" (UID: \"a08fd3b8-493c-47db-9839-462bcc598e92\") " pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-td2hl" Oct 30 20:54:47.250774 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:47.250672 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a08fd3b8-493c-47db-9839-462bcc598e92-istio-data\") pod \"e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-td2hl\" (UID: \"a08fd3b8-493c-47db-9839-462bcc598e92\") " pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-td2hl" Oct 30 20:54:47.250879 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:47.250844 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a08fd3b8-493c-47db-9839-462bcc598e92-istio-envoy\") pod \"e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-td2hl\" (UID: \"a08fd3b8-493c-47db-9839-462bcc598e92\") " pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-td2hl" Oct 30 20:54:47.250879 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:47.250870 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a08fd3b8-493c-47db-9839-462bcc598e92-istio-podinfo\") pod \"e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-td2hl\" (UID: \"a08fd3b8-493c-47db-9839-462bcc598e92\") " pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-td2hl" Oct 30 20:54:47.250999 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:47.250984 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a08fd3b8-493c-47db-9839-462bcc598e92-workload-socket\") pod \"e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-td2hl\" (UID: \"a08fd3b8-493c-47db-9839-462bcc598e92\") " pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-td2hl" Oct 30 20:54:47.251058 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:47.251022 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a08fd3b8-493c-47db-9839-462bcc598e92-istiod-ca-cert\") pod \"e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-td2hl\" (UID: \"a08fd3b8-493c-47db-9839-462bcc598e92\") " pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-td2hl" Oct 30 20:54:47.251118 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:47.251059 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a08fd3b8-493c-47db-9839-462bcc598e92-credential-socket\") pod \"e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-td2hl\" (UID: \"a08fd3b8-493c-47db-9839-462bcc598e92\") " pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-td2hl" Oct 30 20:54:47.251172 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:47.251123 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a08fd3b8-493c-47db-9839-462bcc598e92-workload-certs\") pod \"e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-td2hl\" (UID: \"a08fd3b8-493c-47db-9839-462bcc598e92\") " pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-td2hl" Oct 30 20:54:47.251272 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:47.251246 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a08fd3b8-493c-47db-9839-462bcc598e92-istio-data\") pod \"e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-td2hl\" (UID: \"a08fd3b8-493c-47db-9839-462bcc598e92\") " pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-td2hl" Oct 30 20:54:47.251476 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:47.251436 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a08fd3b8-493c-47db-9839-462bcc598e92-workload-socket\") pod \"e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-td2hl\" (UID: \"a08fd3b8-493c-47db-9839-462bcc598e92\") " pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-td2hl" Oct 30 20:54:47.251965 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:47.251941 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a08fd3b8-493c-47db-9839-462bcc598e92-istiod-ca-cert\") pod \"e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-td2hl\" (UID: \"a08fd3b8-493c-47db-9839-462bcc598e92\") " pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-td2hl" Oct 30 20:54:47.253204 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:47.253183 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a08fd3b8-493c-47db-9839-462bcc598e92-istio-envoy\") pod \"e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-td2hl\" (UID: \"a08fd3b8-493c-47db-9839-462bcc598e92\") " pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-td2hl" Oct 30 20:54:47.253888 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:47.253868 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a08fd3b8-493c-47db-9839-462bcc598e92-istio-podinfo\") pod \"e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-td2hl\" (UID: \"a08fd3b8-493c-47db-9839-462bcc598e92\") " pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-td2hl" Oct 30 20:54:47.266255 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:47.266230 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a08fd3b8-493c-47db-9839-462bcc598e92-istio-token\") pod \"e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-td2hl\" (UID: \"a08fd3b8-493c-47db-9839-462bcc598e92\") " pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-td2hl" Oct 30 20:54:47.266450 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:47.266428 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d2blq\" (UniqueName: \"kubernetes.io/projected/a08fd3b8-493c-47db-9839-462bcc598e92-kube-api-access-d2blq\") pod \"e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-td2hl\" (UID: \"a08fd3b8-493c-47db-9839-462bcc598e92\") " pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-td2hl" Oct 30 20:54:47.335370 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:47.335342 4611 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-td2hl" Oct 30 20:54:47.591697 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:47.591607 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:54:47.605774 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:47.605744 4611 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e8fa042-8xjrj" Oct 30 20:54:47.617587 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:47.617543 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e8fa042-8xjrj" event={"ID":"23c4eefd-503e-4988-8777-cd4f6f91247b","Type":"ContainerDied","Data":"d759f19f10a4ef15627b41f5671373705449e598bdb9abe657d0130a9a375de5"} Oct 30 20:54:47.617773 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:47.617611 4611 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d759f19f10a4ef15627b41f5671373705449e598bdb9abe657d0130a9a375de5" Oct 30 20:54:47.652561 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:47.651351 4611 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-td2hl"] Oct 30 20:54:47.661909 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:54:47.661872 4611 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda08fd3b8_493c_47db_9839_462bcc598e92.slice/crio-fc75d9673d8dc9d24c18bd287a7fc8a4ad5c5b716a5df461475b0219c8b6484c WatchSource:0}: Error finding container fc75d9673d8dc9d24c18bd287a7fc8a4ad5c5b716a5df461475b0219c8b6484c: Status 404 returned error can't find the container with id fc75d9673d8dc9d24c18bd287a7fc8a4ad5c5b716a5df461475b0219c8b6484c Oct 30 20:54:47.677908 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:47.677853 4611 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:54:47.678113 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:47.677960 4611 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:54:47.678113 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:47.678007 4611 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:54:48.301585 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:48.301487 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:54:48.609606 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:48.609573 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:54:48.614457 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:48.614431 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-td2hl" event={"ID":"a08fd3b8-493c-47db-9839-462bcc598e92","Type":"ContainerStarted","Data":"c3b88cbb89aaf4136d6f91ab75097911dbefa29c114d75768b822fb9f72bb325"} Oct 30 20:54:48.614610 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:48.614482 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-td2hl" event={"ID":"a08fd3b8-493c-47db-9839-462bcc598e92","Type":"ContainerStarted","Data":"fc75d9673d8dc9d24c18bd287a7fc8a4ad5c5b716a5df461475b0219c8b6484c"} Oct 30 20:54:48.639013 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:48.638964 4611 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-td2hl" podStartSLOduration=2.638952413 podStartE2EDuration="2.638952413s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:54:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:54:48.636758719 +0000 UTC m=+943.170785336" watchObservedRunningTime="2026-10-30 20:54:48.638952413 +0000 UTC m=+943.172979078" Oct 30 20:54:49.214320 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:49.214286 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:54:49.335564 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:49.335530 4611 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-td2hl" Oct 30 20:54:49.336953 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:49.336927 4611 patch_prober.go:29] interesting pod/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-td2hl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body= Oct 30 20:54:49.337076 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:49.336975 4611 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-td2hl" podUID="a08fd3b8-493c-47db-9839-462bcc598e92" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" Oct 30 20:54:50.077416 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:50.077377 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:54:50.336147 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:50.336057 4611 patch_prober.go:29] interesting pod/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-td2hl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body= Oct 30 20:54:50.336147 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:50.336132 4611 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-td2hl" podUID="a08fd3b8-493c-47db-9839-462bcc598e92" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" Oct 30 20:54:50.929498 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:50.929458 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:54:51.337984 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:51.337903 4611 patch_prober.go:29] interesting pod/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-td2hl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body= Oct 30 20:54:51.338369 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:51.338004 4611 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-td2hl" podUID="a08fd3b8-493c-47db-9839-462bcc598e92" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" Oct 30 20:54:52.335685 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:52.335644 4611 patch_prober.go:29] interesting pod/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-td2hl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body= Oct 30 20:54:52.335862 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:52.335704 4611 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-td2hl" podUID="a08fd3b8-493c-47db-9839-462bcc598e92" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" Oct 30 20:54:52.416723 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:52.416611 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:54:53.290731 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:53.290680 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:54:53.336156 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:53.336121 4611 patch_prober.go:29] interesting pod/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-td2hl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body= Oct 30 20:54:53.336303 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:53.336177 4611 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-td2hl" podUID="a08fd3b8-493c-47db-9839-462bcc598e92" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" Oct 30 20:54:54.336797 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:54.336763 4611 patch_prober.go:29] interesting pod/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-td2hl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body= Oct 30 20:54:54.337302 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:54.336837 4611 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-td2hl" podUID="a08fd3b8-493c-47db-9839-462bcc598e92" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" Oct 30 20:54:54.527559 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:54.527488 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:54:55.389205 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:55.389167 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:54:55.411474 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:55.411443 4611 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-td2hl" Oct 30 20:54:55.411945 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:55.411918 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-td2hl" Oct 30 20:54:55.412784 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:55.412766 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-td2hl" Oct 30 20:54:56.466281 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:56.466242 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:54:57.112633 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:57.112602 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:54:57.993483 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:57.993450 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:54:58.893255 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:58.893198 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:54:59.599545 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:54:59.599493 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:55:01.398724 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:01.398659 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cb60a2/simulator-w3-cb60a2-kserve-6fd74b9d8b-vqfzf" podUID="711b167d-4e66-40d6-be75-33ede11f42c5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded" Oct 30 20:55:01.781585 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:01.780629 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad7ff0/premium-w0-ad7ff0-kserve-5485d4b948-bpmgf" podUID="53cbd692-e9f8-46ba-8fde-3e5215a38904" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded" Oct 30 20:55:01.846882 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:01.801169 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b8aee1/distinct-w1-b8aee1-kserve-5b857c55cd-x5b7b" podUID="1599e588-def8-42e1-b81d-ad0aaf119818" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": context deadline exceeded" Oct 30 20:55:01.878025 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:01.877912 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:55:01.995576 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:01.983946 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-14cca2/premium-w5-14cca2-kserve-6878777857-wpjtj" podUID="b99eb76b-d1c6-49e9-9809-c351c0f5a4de" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:01.999540 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:01.997871 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-3d909d/distinct-2-w6-3d909d-kserve-59c5d5c595-cjg2m" podUID="e2b4c221-ee94-4131-abd8-2313647c641e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:02.016546 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:02.011780 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b8aee1/premium-w1-b8aee1-kserve-786d575599-htc8d" podUID="77368afe-edc8-4fa5-8010-d321965fb38e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:02.039610 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:02.035828 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f2ed1e/distinct-w4-f2ed1e-kserve-57cff9dcf8-8d5zl" podUID="653e3ec7-2147-4813-8946-1f25cfa425f6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:02.039610 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:02.037364 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cb60a2/distinct-w3-cb60a2-kserve-5bb76dc74-86fds" podUID="52e301d9-f378-4e54-9515-01a6680d9fd7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded" Oct 30 20:55:02.039940 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:02.039904 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-14cca2/distinct-w5-14cca2-kserve-587f6996b8-sgf6h" podUID="02b652c3-0724-4a50-9e95-2a9890cc97c4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded" Oct 30 20:55:02.099557 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:02.043303 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b8aee1/distinct-2-w1-b8aee1-kserve-6bcdd59797-cx6hr" podUID="286eab7a-81a7-4768-a7f6-be6b3720028a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": context deadline exceeded" Oct 30 20:55:02.129544 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:02.126854 4611 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-b8aee1/distinct-w1-b8aee1-kserve-5b857c55cd-x5b7b" podUID="1599e588-def8-42e1-b81d-ad0aaf119818" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:02.152800 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:02.152748 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-3d909d/premium-w6-3d909d-kserve-765648946f-2qvd2" podUID="fca5d697-109d-4e90-ab41-c40ad31af874" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded" Oct 30 20:55:02.459872 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:02.459831 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cb60a2/simulator-w3-cb60a2-kserve-6fd74b9d8b-vqfzf" podUID="711b167d-4e66-40d6-be75-33ede11f42c5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:02.487416 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:02.477997 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db534b/distinct-2-w2-db534b-kserve-8594c876-g92mh" podUID="955316d6-a9fd-4b75-a8ab-bbf382b776e2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": context deadline exceeded" Oct 30 20:55:02.511742 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:02.490412 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-cb60a2/simulator-w3-cb60a2-kserve-6fd74b9d8b-vqfzf" Oct 30 20:55:02.786907 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:02.786805 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad7ff0/premium-w0-ad7ff0-kserve-5485d4b948-bpmgf" podUID="53cbd692-e9f8-46ba-8fde-3e5215a38904" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded" Oct 30 20:55:02.860534 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:02.820182 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ad7ff0/premium-w0-ad7ff0-kserve-5485d4b948-bpmgf" Oct 30 20:55:02.998542 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:02.997462 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-14cca2/premium-w5-14cca2-kserve-6878777857-wpjtj" podUID="b99eb76b-d1c6-49e9-9809-c351c0f5a4de" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:03.033530 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:03.026094 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-14cca2/premium-w5-14cca2-kserve-6878777857-wpjtj" Oct 30 20:55:03.033530 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:03.027179 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b8aee1/premium-w1-b8aee1-kserve-786d575599-htc8d" podUID="77368afe-edc8-4fa5-8010-d321965fb38e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded" Oct 30 20:55:03.033530 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:03.027575 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b8aee1/premium-w1-b8aee1-kserve-786d575599-htc8d" Oct 30 20:55:03.033530 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:03.028421 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-3d909d/distinct-2-w6-3d909d-kserve-59c5d5c595-cjg2m" podUID="e2b4c221-ee94-4131-abd8-2313647c641e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": context deadline exceeded" Oct 30 20:55:03.033530 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:03.028610 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-3d909d/distinct-2-w6-3d909d-kserve-59c5d5c595-cjg2m" Oct 30 20:55:03.053587 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:03.041689 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cb60a2/distinct-w3-cb60a2-kserve-5bb76dc74-86fds" podUID="52e301d9-f378-4e54-9515-01a6680d9fd7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded" Oct 30 20:55:03.053587 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:03.042038 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f2ed1e/distinct-w4-f2ed1e-kserve-57cff9dcf8-8d5zl" podUID="653e3ec7-2147-4813-8946-1f25cfa425f6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded" Oct 30 20:55:03.053587 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:03.042348 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-cb60a2/distinct-w3-cb60a2-kserve-5bb76dc74-86fds" Oct 30 20:55:03.053587 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:03.042402 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-f2ed1e/distinct-w4-f2ed1e-kserve-57cff9dcf8-8d5zl" Oct 30 20:55:03.125532 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:03.057590 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-14cca2/distinct-w5-14cca2-kserve-587f6996b8-sgf6h" podUID="02b652c3-0724-4a50-9e95-2a9890cc97c4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded" Oct 30 20:55:03.125532 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:03.057996 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-14cca2/distinct-w5-14cca2-kserve-587f6996b8-sgf6h" Oct 30 20:55:03.125532 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:03.058956 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b8aee1/distinct-2-w1-b8aee1-kserve-6bcdd59797-cx6hr" podUID="286eab7a-81a7-4768-a7f6-be6b3720028a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": context deadline exceeded" Oct 30 20:55:03.125532 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:03.059285 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b8aee1/distinct-2-w1-b8aee1-kserve-6bcdd59797-cx6hr" Oct 30 20:55:03.183533 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:03.182785 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-3d909d/premium-w6-3d909d-kserve-765648946f-2qvd2" podUID="fca5d697-109d-4e90-ab41-c40ad31af874" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:03.183533 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:03.183178 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-3d909d/premium-w6-3d909d-kserve-765648946f-2qvd2" Oct 30 20:55:03.340252 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:03.340046 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b8aee1/distinct-w1-b8aee1-kserve-5b857c55cd-x5b7b" podUID="1599e588-def8-42e1-b81d-ad0aaf119818" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:03.391140 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:03.391097 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b8aee1/distinct-w1-b8aee1-kserve-5b857c55cd-x5b7b" Oct 30 20:55:03.525535 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:03.513000 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db534b/distinct-2-w2-db534b-kserve-8594c876-g92mh" podUID="955316d6-a9fd-4b75-a8ab-bbf382b776e2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:03.654534 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:03.567193 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cb60a2/simulator-w3-cb60a2-kserve-6fd74b9d8b-vqfzf" podUID="711b167d-4e66-40d6-be75-33ede11f42c5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded" Oct 30 20:55:03.660656 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:03.660609 4611 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-cb60a2/simulator-w3-cb60a2-kserve-6fd74b9d8b-vqfzf" podUID="711b167d-4e66-40d6-be75-33ede11f42c5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:03.951673 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:03.864151 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad7ff0/premium-w0-ad7ff0-kserve-5485d4b948-bpmgf" podUID="53cbd692-e9f8-46ba-8fde-3e5215a38904" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:03.976535 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:03.910918 4611 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-b8aee1/distinct-2-w1-b8aee1-kserve-6bcdd59797-cx6hr" podUID="286eab7a-81a7-4768-a7f6-be6b3720028a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:04.018853 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:04.018796 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-14cca2/premium-w5-14cca2-kserve-6878777857-wpjtj" podUID="b99eb76b-d1c6-49e9-9809-c351c0f5a4de" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:04.038796 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:04.038753 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-3d909d/distinct-2-w6-3d909d-kserve-59c5d5c595-cjg2m" podUID="e2b4c221-ee94-4131-abd8-2313647c641e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": context deadline exceeded" Oct 30 20:55:04.048060 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:04.041679 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b8aee1/premium-w1-b8aee1-kserve-786d575599-htc8d" podUID="77368afe-edc8-4fa5-8010-d321965fb38e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:04.049276 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:04.041818 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cb60a2/distinct-w3-cb60a2-kserve-5bb76dc74-86fds" podUID="52e301d9-f378-4e54-9515-01a6680d9fd7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:04.049695 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:04.042818 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f2ed1e/distinct-w4-f2ed1e-kserve-57cff9dcf8-8d5zl" podUID="653e3ec7-2147-4813-8946-1f25cfa425f6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:04.057826 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:04.057740 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-14cca2/distinct-w5-14cca2-kserve-587f6996b8-sgf6h" podUID="02b652c3-0724-4a50-9e95-2a9890cc97c4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:04.060753 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:04.060468 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-db534b/distinct-2-w2-db534b-kserve-8594c876-g92mh" Oct 30 20:55:04.265794 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:04.105205 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b8aee1/distinct-2-w1-b8aee1-kserve-6bcdd59797-cx6hr" podUID="286eab7a-81a7-4768-a7f6-be6b3720028a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:04.266218 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:04.105250 4611 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-14cca2/distinct-w5-14cca2-kserve-587f6996b8-sgf6h" podUID="02b652c3-0724-4a50-9e95-2a9890cc97c4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/health\": context deadline exceeded" Oct 30 20:55:04.266365 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:04.264002 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-3d909d/premium-w6-3d909d-kserve-765648946f-2qvd2" podUID="fca5d697-109d-4e90-ab41-c40ad31af874" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:04.423534 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:04.421655 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b8aee1/distinct-w1-b8aee1-kserve-5b857c55cd-x5b7b" podUID="1599e588-def8-42e1-b81d-ad0aaf119818" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:04.637535 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:04.633498 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db534b/distinct-2-w2-db534b-kserve-8594c876-g92mh" podUID="955316d6-a9fd-4b75-a8ab-bbf382b776e2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": context deadline exceeded" Oct 30 20:55:04.661896 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:04.661604 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cb60a2/simulator-w3-cb60a2-kserve-6fd74b9d8b-vqfzf" podUID="711b167d-4e66-40d6-be75-33ede11f42c5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded" Oct 30 20:55:05.048589 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:05.040731 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad7ff0/premium-w0-ad7ff0-kserve-5485d4b948-bpmgf" podUID="53cbd692-e9f8-46ba-8fde-3e5215a38904" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded" Oct 30 20:55:05.074534 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:05.049637 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cb60a2/distinct-w3-cb60a2-kserve-5bb76dc74-86fds" podUID="52e301d9-f378-4e54-9515-01a6680d9fd7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:05.086533 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:05.075152 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f2ed1e/distinct-w4-f2ed1e-kserve-57cff9dcf8-8d5zl" podUID="653e3ec7-2147-4813-8946-1f25cfa425f6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:05.231556 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:05.053840 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-14cca2/premium-w5-14cca2-kserve-6878777857-wpjtj" podUID="b99eb76b-d1c6-49e9-9809-c351c0f5a4de" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded" Oct 30 20:55:05.231556 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:05.053943 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-3d909d/distinct-2-w6-3d909d-kserve-59c5d5c595-cjg2m" podUID="e2b4c221-ee94-4131-abd8-2313647c641e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": context deadline exceeded" Oct 30 20:55:05.231556 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:05.096275 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-14cca2/distinct-w5-14cca2-kserve-587f6996b8-sgf6h" podUID="02b652c3-0724-4a50-9e95-2a9890cc97c4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded" Oct 30 20:55:05.231556 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:05.052850 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b8aee1/premium-w1-b8aee1-kserve-786d575599-htc8d" podUID="77368afe-edc8-4fa5-8010-d321965fb38e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded" Oct 30 20:55:05.231556 ip-10-0-133-192 kubenswrapper[4611]: E1030 20:55:05.144490 4611 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hwzc6" containerName="extractor" Oct 30 20:55:05.283993 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:05.283953 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-3d909d/premium-w6-3d909d-kserve-765648946f-2qvd2" podUID="fca5d697-109d-4e90-ab41-c40ad31af874" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:05.284668 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:05.284637 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b8aee1/distinct-2-w1-b8aee1-kserve-6bcdd59797-cx6hr" podUID="286eab7a-81a7-4768-a7f6-be6b3720028a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": context deadline exceeded" Oct 30 20:55:05.398591 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:05.398250 4611 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-14cca2/premium-w5-14cca2-kserve-6878777857-wpjtj" podUID="b99eb76b-d1c6-49e9-9809-c351c0f5a4de" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:05.476534 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:05.472657 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b8aee1/distinct-w1-b8aee1-kserve-5b857c55cd-x5b7b" podUID="1599e588-def8-42e1-b81d-ad0aaf119818" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:05.737615 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:05.729838 4611 patch_prober.go:29] interesting pod/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-6xtqj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:55:05.746645 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:05.746598 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-6xtqj" podUID="63d294fd-5293-4752-9116-0aafa81d5e4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:05.781076 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:05.781029 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cb60a2/simulator-w3-cb60a2-kserve-6fd74b9d8b-vqfzf" podUID="711b167d-4e66-40d6-be75-33ede11f42c5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:05.834675 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:05.834611 4611 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-cb60a2/distinct-w3-cb60a2-kserve-5bb76dc74-86fds" podUID="52e301d9-f378-4e54-9515-01a6680d9fd7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:05.846565 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:05.835111 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db534b/distinct-2-w2-db534b-kserve-8594c876-g92mh" podUID="955316d6-a9fd-4b75-a8ab-bbf382b776e2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": context deadline exceeded" Oct 30 20:55:06.271532 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:06.270473 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cb60a2/distinct-w3-cb60a2-kserve-5bb76dc74-86fds" podUID="52e301d9-f378-4e54-9515-01a6680d9fd7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded" Oct 30 20:55:06.279229 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:06.279183 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f2ed1e/distinct-w4-f2ed1e-kserve-57cff9dcf8-8d5zl" podUID="653e3ec7-2147-4813-8946-1f25cfa425f6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded" Oct 30 20:55:06.282971 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:06.282923 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-14cca2/premium-w5-14cca2-kserve-6878777857-wpjtj" podUID="b99eb76b-d1c6-49e9-9809-c351c0f5a4de" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded" Oct 30 20:55:06.419802 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:06.287577 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b8aee1/distinct-2-w1-b8aee1-kserve-6bcdd59797-cx6hr" podUID="286eab7a-81a7-4768-a7f6-be6b3720028a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:06.506538 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:06.287613 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-3d909d/distinct-2-w6-3d909d-kserve-59c5d5c595-cjg2m" podUID="e2b4c221-ee94-4131-abd8-2313647c641e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": context deadline exceeded" Oct 30 20:55:06.540917 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:06.287635 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-14cca2/distinct-w5-14cca2-kserve-587f6996b8-sgf6h" podUID="02b652c3-0724-4a50-9e95-2a9890cc97c4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded" Oct 30 20:55:06.540917 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:06.287672 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-3d909d/premium-w6-3d909d-kserve-765648946f-2qvd2" podUID="fca5d697-109d-4e90-ab41-c40ad31af874" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded" Oct 30 20:55:06.540917 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:06.287692 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b8aee1/premium-w1-b8aee1-kserve-786d575599-htc8d" podUID="77368afe-edc8-4fa5-8010-d321965fb38e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:06.540917 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:06.342602 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad7ff0/premium-w0-ad7ff0-kserve-5485d4b948-bpmgf" podUID="53cbd692-e9f8-46ba-8fde-3e5215a38904" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded" Oct 30 20:55:06.540917 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:06.454255 4611 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-3d909d/distinct-2-w6-3d909d-kserve-59c5d5c595-cjg2m" podUID="e2b4c221-ee94-4131-abd8-2313647c641e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:06.541430 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:06.541394 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b8aee1/distinct-w1-b8aee1-kserve-5b857c55cd-x5b7b" podUID="1599e588-def8-42e1-b81d-ad0aaf119818" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:06.665534 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:06.661634 4611 prober.go:122] "Probe failed" probeType="Liveness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-smdl6" podUID="31d263b9-b3b3-4db4-9026-da84d454dc34" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.133.0.22:6080/livez\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:06.812754 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:06.812662 4611 patch_prober.go:29] interesting pod/prometheus-operator-admission-webhook-6489d8bf89-fgskx container/prometheus-operator-admission-webhook namespace/openshift-monitoring: Readiness probe status=failure output="Get \"https://10.133.0.10:8443/healthz\": context deadline exceeded" start-of-body= Oct 30 20:55:06.813334 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:06.813300 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-fgskx" podUID="312370de-5cc7-438b-b70a-49b8dad611ec" containerName="prometheus-operator-admission-webhook" probeResult="failure" output="Get \"https://10.133.0.10:8443/healthz\": context deadline exceeded" Oct 30 20:55:06.813525 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:06.813486 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cb60a2/simulator-w3-cb60a2-kserve-6fd74b9d8b-vqfzf" podUID="711b167d-4e66-40d6-be75-33ede11f42c5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded" Oct 30 20:55:06.905052 ip-10-0-133-192 kubenswrapper[4611]: E1030 20:55:06.905017 4611 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.287s" Oct 30 20:55:06.971862 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:06.935959 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db534b/distinct-2-w2-db534b-kserve-8594c876-g92mh" podUID="955316d6-a9fd-4b75-a8ab-bbf382b776e2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:06.989276 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:06.989235 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-smdl6" podUID="31d263b9-b3b3-4db4-9026-da84d454dc34" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.133.0.22:6080/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:07.273651 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:07.273602 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cb60a2/distinct-w3-cb60a2-kserve-5bb76dc74-86fds" podUID="52e301d9-f378-4e54-9515-01a6680d9fd7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:07.292623 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:07.292564 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-14cca2/premium-w5-14cca2-kserve-6878777857-wpjtj" podUID="b99eb76b-d1c6-49e9-9809-c351c0f5a4de" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:07.294017 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:07.293450 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f2ed1e/distinct-w4-f2ed1e-kserve-57cff9dcf8-8d5zl" podUID="653e3ec7-2147-4813-8946-1f25cfa425f6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded" Oct 30 20:55:07.495534 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:07.470638 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b8aee1/distinct-2-w1-b8aee1-kserve-6bcdd59797-cx6hr" podUID="286eab7a-81a7-4768-a7f6-be6b3720028a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:07.495534 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:07.471621 4611 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-db534b/distinct-2-w2-db534b-kserve-8594c876-g92mh" podUID="955316d6-a9fd-4b75-a8ab-bbf382b776e2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/health\": context deadline exceeded" Oct 30 20:55:07.545584 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:07.513784 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-3d909d/distinct-2-w6-3d909d-kserve-59c5d5c595-cjg2m" podUID="e2b4c221-ee94-4131-abd8-2313647c641e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": context deadline exceeded" Oct 30 20:55:07.548370 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:07.543705 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad7ff0/premium-w0-ad7ff0-kserve-5485d4b948-bpmgf" podUID="53cbd692-e9f8-46ba-8fde-3e5215a38904" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:07.573660 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:07.543746 4611 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-ad7ff0/premium-w0-ad7ff0-kserve-5485d4b948-bpmgf" podUID="53cbd692-e9f8-46ba-8fde-3e5215a38904" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/health\": context deadline exceeded" Oct 30 20:55:07.573660 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:07.543773 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-3d909d/premium-w6-3d909d-kserve-765648946f-2qvd2" podUID="fca5d697-109d-4e90-ab41-c40ad31af874" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded" Oct 30 20:55:07.573660 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:07.543796 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b8aee1/premium-w1-b8aee1-kserve-786d575599-htc8d" podUID="77368afe-edc8-4fa5-8010-d321965fb38e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded" Oct 30 20:55:07.573660 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:07.545847 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b8aee1/distinct-w1-b8aee1-kserve-5b857c55cd-x5b7b" podUID="1599e588-def8-42e1-b81d-ad0aaf119818" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:07.605541 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:07.576162 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-14cca2/distinct-w5-14cca2-kserve-587f6996b8-sgf6h" podUID="02b652c3-0724-4a50-9e95-2a9890cc97c4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:07.833589 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:07.828325 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cb60a2/simulator-w3-cb60a2-kserve-6fd74b9d8b-vqfzf" podUID="711b167d-4e66-40d6-be75-33ede11f42c5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded" Oct 30 20:55:07.997637 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:07.987724 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db534b/distinct-2-w2-db534b-kserve-8594c876-g92mh" podUID="955316d6-a9fd-4b75-a8ab-bbf382b776e2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": context deadline exceeded" Oct 30 20:55:08.257696 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:08.239610 4611 patch_prober.go:29] interesting pod/e2e-worker-w2-db534b-openshift-default-85df4d79d9-5f78z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:55:08.257696 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:08.239681 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-5f78z" podUID="06064c28-fd6d-4765-8635-8690665a9b83" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:08.307549 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:08.277448 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cb60a2/distinct-w3-cb60a2-kserve-5bb76dc74-86fds" podUID="52e301d9-f378-4e54-9515-01a6680d9fd7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:08.307549 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:08.296012 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f2ed1e/distinct-w4-f2ed1e-kserve-57cff9dcf8-8d5zl" podUID="653e3ec7-2147-4813-8946-1f25cfa425f6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:08.309707 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:08.309034 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-14cca2/premium-w5-14cca2-kserve-6878777857-wpjtj" podUID="b99eb76b-d1c6-49e9-9809-c351c0f5a4de" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:08.387838 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:08.387790 4611 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-94228e47-nfh6n"] Oct 30 20:55:08.481534 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:08.481017 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b8aee1/distinct-2-w1-b8aee1-kserve-6bcdd59797-cx6hr" podUID="286eab7a-81a7-4768-a7f6-be6b3720028a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:08.481744 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:08.481542 4611 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-94228e47-nfh6n" Oct 30 20:55:08.553536 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:08.547374 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-3d909d/distinct-2-w6-3d909d-kserve-59c5d5c595-cjg2m" podUID="e2b4c221-ee94-4131-abd8-2313647c641e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:08.553536 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:08.548455 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad7ff0/premium-w0-ad7ff0-kserve-5485d4b948-bpmgf" podUID="53cbd692-e9f8-46ba-8fde-3e5215a38904" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:08.575657 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:08.566224 4611 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7hf8r\"" Oct 30 20:55:08.575657 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:08.566325 4611 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:55:08.577795 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:08.577762 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b8aee1/distinct-w1-b8aee1-kserve-5b857c55cd-x5b7b" podUID="1599e588-def8-42e1-b81d-ad0aaf119818" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:08.578356 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:08.577793 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b8aee1/premium-w1-b8aee1-kserve-786d575599-htc8d" podUID="77368afe-edc8-4fa5-8010-d321965fb38e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded" Oct 30 20:55:08.578858 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:08.577827 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-3d909d/premium-w6-3d909d-kserve-765648946f-2qvd2" podUID="fca5d697-109d-4e90-ab41-c40ad31af874" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded" Oct 30 20:55:08.655546 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:08.651707 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dgb9k\" (UniqueName: \"kubernetes.io/projected/fb626b8f-b455-48d5-b436-3f0d9a2f0498-kube-api-access-dgb9k\") pod \"maas-api-revoke-sub-94228e47-nfh6n\" (UID: \"fb626b8f-b455-48d5-b436-3f0d9a2f0498\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94228e47-nfh6n" Oct 30 20:55:08.860374 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:08.860331 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb626b8f-b455-48d5-b436-3f0d9a2f0498-maas-api-xks-ca\") pod \"maas-api-revoke-sub-94228e47-nfh6n\" (UID: \"fb626b8f-b455-48d5-b436-3f0d9a2f0498\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94228e47-nfh6n" Oct 30 20:55:08.907431 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:08.907385 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb626b8f-b455-48d5-b436-3f0d9a2f0498-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-94228e47-nfh6n\" (UID: \"fb626b8f-b455-48d5-b436-3f0d9a2f0498\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94228e47-nfh6n" Oct 30 20:55:08.984540 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:08.859268 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cb60a2/simulator-w3-cb60a2-kserve-6fd74b9d8b-vqfzf" podUID="711b167d-4e66-40d6-be75-33ede11f42c5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:09.293585 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:08.905946 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-14cca2/distinct-w5-14cca2-kserve-587f6996b8-sgf6h" podUID="02b652c3-0724-4a50-9e95-2a9890cc97c4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded" Oct 30 20:55:09.293585 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:08.861142 4611 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-94228e47-nfh6n"] Oct 30 20:55:09.293585 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:09.052638 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db534b/distinct-2-w2-db534b-kserve-8594c876-g92mh" podUID="955316d6-a9fd-4b75-a8ab-bbf382b776e2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": context deadline exceeded" Oct 30 20:55:09.293585 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:09.129204 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb626b8f-b455-48d5-b436-3f0d9a2f0498-maas-api-xks-ca\") pod \"maas-api-revoke-sub-94228e47-nfh6n\" (UID: \"fb626b8f-b455-48d5-b436-3f0d9a2f0498\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94228e47-nfh6n" Oct 30 20:55:09.293585 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:09.292252 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb626b8f-b455-48d5-b436-3f0d9a2f0498-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-94228e47-nfh6n\" (UID: \"fb626b8f-b455-48d5-b436-3f0d9a2f0498\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94228e47-nfh6n" Oct 30 20:55:09.293585 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:09.292524 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dgb9k\" (UniqueName: \"kubernetes.io/projected/fb626b8f-b455-48d5-b436-3f0d9a2f0498-kube-api-access-dgb9k\") pod \"maas-api-revoke-sub-94228e47-nfh6n\" (UID: \"fb626b8f-b455-48d5-b436-3f0d9a2f0498\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94228e47-nfh6n" Oct 30 20:55:09.293585 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:09.222698 4611 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-3d909d/premium-w6-3d909d-kserve-765648946f-2qvd2" podUID="fca5d697-109d-4e90-ab41-c40ad31af874" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:09.293585 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:09.156851 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:55:09.302831 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:09.293810 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb626b8f-b455-48d5-b436-3f0d9a2f0498-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-94228e47-nfh6n\" (UID: \"fb626b8f-b455-48d5-b436-3f0d9a2f0498\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94228e47-nfh6n" Oct 30 20:55:09.302831 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:09.173456 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb626b8f-b455-48d5-b436-3f0d9a2f0498-maas-api-xks-ca\") pod \"maas-api-revoke-sub-94228e47-nfh6n\" (UID: \"fb626b8f-b455-48d5-b436-3f0d9a2f0498\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94228e47-nfh6n" Oct 30 20:55:09.302831 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:09.294200 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cb60a2/distinct-w3-cb60a2-kserve-5bb76dc74-86fds" podUID="52e301d9-f378-4e54-9515-01a6680d9fd7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:09.327531 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:09.319670 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-14cca2/premium-w5-14cca2-kserve-6878777857-wpjtj" podUID="b99eb76b-d1c6-49e9-9809-c351c0f5a4de" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:09.327531 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:09.320118 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f2ed1e/distinct-w4-f2ed1e-kserve-57cff9dcf8-8d5zl" podUID="653e3ec7-2147-4813-8946-1f25cfa425f6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:09.389535 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:09.378799 4611 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-f2ed1e/distinct-w4-f2ed1e-kserve-57cff9dcf8-8d5zl" podUID="653e3ec7-2147-4813-8946-1f25cfa425f6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:09.450805 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:09.407429 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dgb9k\" (UniqueName: \"kubernetes.io/projected/fb626b8f-b455-48d5-b436-3f0d9a2f0498-kube-api-access-dgb9k\") pod \"maas-api-revoke-sub-94228e47-nfh6n\" (UID: \"fb626b8f-b455-48d5-b436-3f0d9a2f0498\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94228e47-nfh6n" Oct 30 20:55:09.484783 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:09.484751 4611 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-94228e47-nfh6n" Oct 30 20:55:09.498392 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:09.497356 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b8aee1/distinct-2-w1-b8aee1-kserve-6bcdd59797-cx6hr" podUID="286eab7a-81a7-4768-a7f6-be6b3720028a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:09.508116 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:09.506819 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:55:09.553070 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:09.552971 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad7ff0/premium-w0-ad7ff0-kserve-5485d4b948-bpmgf" podUID="53cbd692-e9f8-46ba-8fde-3e5215a38904" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded" Oct 30 20:55:09.553711 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:09.553682 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-3d909d/distinct-2-w6-3d909d-kserve-59c5d5c595-cjg2m" podUID="e2b4c221-ee94-4131-abd8-2313647c641e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": context deadline exceeded" Oct 30 20:55:09.599573 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:09.581319 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-3d909d/premium-w6-3d909d-kserve-765648946f-2qvd2" podUID="fca5d697-109d-4e90-ab41-c40ad31af874" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:09.599573 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:09.581630 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b8aee1/distinct-w1-b8aee1-kserve-5b857c55cd-x5b7b" podUID="1599e588-def8-42e1-b81d-ad0aaf119818" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": context deadline exceeded" Oct 30 20:55:09.599573 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:09.581867 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b8aee1/premium-w1-b8aee1-kserve-786d575599-htc8d" podUID="77368afe-edc8-4fa5-8010-d321965fb38e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:09.747538 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:09.746981 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:55:09.812632 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:09.801887 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b8aee1/distinct-2-w1-b8aee1-kserve-6bcdd59797-cx6hr" Oct 30 20:55:09.812632 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:09.801948 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-cb60a2/simulator-w3-cb60a2-kserve-6fd74b9d8b-vqfzf" Oct 30 20:55:09.812632 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:09.801982 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-14cca2/distinct-w5-14cca2-kserve-587f6996b8-sgf6h" Oct 30 20:55:09.814795 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:09.814761 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-14cca2/premium-w5-14cca2-kserve-6878777857-wpjtj" Oct 30 20:55:09.822841 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:09.822464 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-cb60a2/distinct-w3-cb60a2-kserve-5bb76dc74-86fds" Oct 30 20:55:09.965637 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:09.964606 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b8aee1/premium-w1-b8aee1-kserve-786d575599-htc8d" Oct 30 20:55:09.989905 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:09.989873 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-f2ed1e/distinct-w4-f2ed1e-kserve-57cff9dcf8-8d5zl" Oct 30 20:55:09.994140 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:09.994114 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b8aee1/distinct-w1-b8aee1-kserve-5b857c55cd-x5b7b" Oct 30 20:55:10.139351 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:10.139321 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ad7ff0/premium-w0-ad7ff0-kserve-5485d4b948-bpmgf" Oct 30 20:55:10.172805 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:10.172781 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-3d909d/distinct-2-w6-3d909d-kserve-59c5d5c595-cjg2m" Oct 30 20:55:10.178533 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:10.177253 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-3d909d/premium-w6-3d909d-kserve-765648946f-2qvd2" Oct 30 20:55:10.183340 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:10.183316 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-db534b/distinct-2-w2-db534b-kserve-8594c876-g92mh" Oct 30 20:55:10.297232 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:10.297203 4611 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-94228e47-nfh6n"] Oct 30 20:55:10.406347 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:10.406304 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:55:10.803829 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:10.803793 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-94228e47-nfh6n" event={"ID":"fb626b8f-b455-48d5-b436-3f0d9a2f0498","Type":"ContainerStarted","Data":"2185b35e2920d87e34091a797b813937841f9c29840ea72fb395f40bce822aec"} Oct 30 20:55:10.804009 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:10.803857 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-94228e47-nfh6n" event={"ID":"fb626b8f-b455-48d5-b436-3f0d9a2f0498","Type":"ContainerStarted","Data":"7466286751d745d6f6f77c59b96d7256ae2861c1705493d2f6076cc2d0f8a313"} Oct 30 20:55:10.827527 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:10.827468 4611 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-94228e47-nfh6n" podStartSLOduration=3.827455242 podStartE2EDuration="3.827455242s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:55:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:55:10.826778413 +0000 UTC m=+965.360805041" watchObservedRunningTime="2026-10-30 20:55:10.827455242 +0000 UTC m=+965.361481888" Oct 30 20:55:11.312841 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:11.312807 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:55:11.813888 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:11.813848 4611 generic.go:356] "Generic (PLEG): container finished" podID="fb626b8f-b455-48d5-b436-3f0d9a2f0498" containerID="2185b35e2920d87e34091a797b813937841f9c29840ea72fb395f40bce822aec" exitCode=0 Oct 30 20:55:11.814062 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:11.813928 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-94228e47-nfh6n" event={"ID":"fb626b8f-b455-48d5-b436-3f0d9a2f0498","Type":"ContainerDied","Data":"2185b35e2920d87e34091a797b813937841f9c29840ea72fb395f40bce822aec"} Oct 30 20:55:11.905044 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:11.905007 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:55:12.610203 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:12.610159 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:55:13.023983 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:13.023949 4611 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-94228e47-nfh6n" Oct 30 20:55:13.136831 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:13.136744 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fb626b8f-b455-48d5-b436-3f0d9a2f0498-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb626b8f-b455-48d5-b436-3f0d9a2f0498-maas-api-xks-ca\") pod \"fb626b8f-b455-48d5-b436-3f0d9a2f0498\" (UID: \"fb626b8f-b455-48d5-b436-3f0d9a2f0498\") " Oct 30 20:55:13.136992 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:13.136881 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fb626b8f-b455-48d5-b436-3f0d9a2f0498-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb626b8f-b455-48d5-b436-3f0d9a2f0498-maas-api-openshift-ca\") pod \"fb626b8f-b455-48d5-b436-3f0d9a2f0498\" (UID: \"fb626b8f-b455-48d5-b436-3f0d9a2f0498\") " Oct 30 20:55:13.137063 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:13.136993 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb626b8f-b455-48d5-b436-3f0d9a2f0498-kube-api-access-dgb9k\" (UniqueName: \"kubernetes.io/projected/fb626b8f-b455-48d5-b436-3f0d9a2f0498-kube-api-access-dgb9k\") pod \"fb626b8f-b455-48d5-b436-3f0d9a2f0498\" (UID: \"fb626b8f-b455-48d5-b436-3f0d9a2f0498\") " Oct 30 20:55:13.137232 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:13.137208 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb626b8f-b455-48d5-b436-3f0d9a2f0498-maas-api-openshift-ca" pod "fb626b8f-b455-48d5-b436-3f0d9a2f0498" (UID: "fb626b8f-b455-48d5-b436-3f0d9a2f0498"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:55:13.137690 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:13.137667 4611 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb626b8f-b455-48d5-b436-3f0d9a2f0498-maas-api-openshift-ca\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:55:13.138862 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:13.138831 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb626b8f-b455-48d5-b436-3f0d9a2f0498-maas-api-xks-ca" pod "fb626b8f-b455-48d5-b436-3f0d9a2f0498" (UID: "fb626b8f-b455-48d5-b436-3f0d9a2f0498"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:55:13.138960 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:13.138931 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb626b8f-b455-48d5-b436-3f0d9a2f0498-kube-api-access-dgb9k" pod "fb626b8f-b455-48d5-b436-3f0d9a2f0498" (UID: "fb626b8f-b455-48d5-b436-3f0d9a2f0498"). InnerVolumeSpecName "kube-api-access-dgb9k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:55:13.239381 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:13.239356 4611 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb626b8f-b455-48d5-b436-3f0d9a2f0498-maas-api-xks-ca\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:55:13.239381 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:13.239379 4611 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dgb9k\" (UniqueName: \"kubernetes.io/projected/fb626b8f-b455-48d5-b436-3f0d9a2f0498-kube-api-access-dgb9k\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:55:13.385100 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:13.385064 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:55:13.832152 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:13.832125 4611 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-94228e47-nfh6n" Oct 30 20:55:13.832559 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:13.832123 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-94228e47-nfh6n" event={"ID":"fb626b8f-b455-48d5-b436-3f0d9a2f0498","Type":"ContainerDied","Data":"7466286751d745d6f6f77c59b96d7256ae2861c1705493d2f6076cc2d0f8a313"} Oct 30 20:55:13.832559 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:13.832277 4611 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7466286751d745d6f6f77c59b96d7256ae2861c1705493d2f6076cc2d0f8a313" Oct 30 20:55:14.491365 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:14.491310 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:55:15.496214 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:15.496174 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:55:16.302270 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:16.302227 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:55:17.066628 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:17.066591 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:55:17.997036 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:17.996979 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:55:18.944952 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:18.944903 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:55:19.682377 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:19.682343 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:55:20.378436 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:20.378392 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:55:21.200491 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:21.200447 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:55:21.907985 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:21.907947 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:55:22.766054 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:22.766007 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:55:23.858312 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:23.858276 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:55:24.648658 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:24.648483 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:55:25.411611 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:25.411564 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:55:26.202462 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:26.202427 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:55:26.906701 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:26.906667 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:55:27.588659 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:27.588617 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:55:27.746555 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:27.746501 4611 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1ffb1833-hm2pr"] Oct 30 20:55:27.747161 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:27.747148 4611 cpu_manager.go:414] "containerMap: removing container" podUID="fb626b8f-b455-48d5-b436-3f0d9a2f0498" containerName="revoke-keys" Oct 30 20:55:27.747208 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:27.747162 4611 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb626b8f-b455-48d5-b436-3f0d9a2f0498" containerName="revoke-keys" Oct 30 20:55:27.747322 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:27.747312 4611 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb626b8f-b455-48d5-b436-3f0d9a2f0498" containerName="revoke-keys" Oct 30 20:55:27.750849 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:27.750828 4611 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ffb1833-hm2pr" Oct 30 20:55:27.756009 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:27.755990 4611 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7hf8r\"" Oct 30 20:55:27.756158 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:27.756003 4611 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:55:27.761444 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:27.761421 4611 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1ffb1833-hm2pr"] Oct 30 20:55:27.888297 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:27.888262 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d22ba0fe-de2a-4ee9-b71e-68d1ee087da5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1ffb1833-hm2pr\" (UID: \"d22ba0fe-de2a-4ee9-b71e-68d1ee087da5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ffb1833-hm2pr" Oct 30 20:55:27.888483 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:27.888320 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d22ba0fe-de2a-4ee9-b71e-68d1ee087da5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1ffb1833-hm2pr\" (UID: \"d22ba0fe-de2a-4ee9-b71e-68d1ee087da5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ffb1833-hm2pr" Oct 30 20:55:27.888483 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:27.888341 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lz7tt\" (UniqueName: \"kubernetes.io/projected/d22ba0fe-de2a-4ee9-b71e-68d1ee087da5-kube-api-access-lz7tt\") pod \"maas-api-revoke-sub-1ffb1833-hm2pr\" (UID: \"d22ba0fe-de2a-4ee9-b71e-68d1ee087da5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ffb1833-hm2pr" Oct 30 20:55:27.990659 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:27.990626 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d22ba0fe-de2a-4ee9-b71e-68d1ee087da5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1ffb1833-hm2pr\" (UID: \"d22ba0fe-de2a-4ee9-b71e-68d1ee087da5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ffb1833-hm2pr" Oct 30 20:55:27.991041 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:27.990691 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d22ba0fe-de2a-4ee9-b71e-68d1ee087da5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1ffb1833-hm2pr\" (UID: \"d22ba0fe-de2a-4ee9-b71e-68d1ee087da5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ffb1833-hm2pr" Oct 30 20:55:27.991041 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:27.990718 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lz7tt\" (UniqueName: \"kubernetes.io/projected/d22ba0fe-de2a-4ee9-b71e-68d1ee087da5-kube-api-access-lz7tt\") pod \"maas-api-revoke-sub-1ffb1833-hm2pr\" (UID: \"d22ba0fe-de2a-4ee9-b71e-68d1ee087da5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ffb1833-hm2pr" Oct 30 20:55:27.991448 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:27.991429 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d22ba0fe-de2a-4ee9-b71e-68d1ee087da5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1ffb1833-hm2pr\" (UID: \"d22ba0fe-de2a-4ee9-b71e-68d1ee087da5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ffb1833-hm2pr" Oct 30 20:55:27.993065 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:27.993046 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d22ba0fe-de2a-4ee9-b71e-68d1ee087da5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1ffb1833-hm2pr\" (UID: \"d22ba0fe-de2a-4ee9-b71e-68d1ee087da5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ffb1833-hm2pr" Oct 30 20:55:28.002076 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:28.002051 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lz7tt\" (UniqueName: \"kubernetes.io/projected/d22ba0fe-de2a-4ee9-b71e-68d1ee087da5-kube-api-access-lz7tt\") pod \"maas-api-revoke-sub-1ffb1833-hm2pr\" (UID: \"d22ba0fe-de2a-4ee9-b71e-68d1ee087da5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ffb1833-hm2pr" Oct 30 20:55:28.067474 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:28.067451 4611 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ffb1833-hm2pr" Oct 30 20:55:28.293176 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:28.293060 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:55:28.357723 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:28.357688 4611 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1ffb1833-hm2pr"] Oct 30 20:55:28.710438 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:28.710404 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:55:28.958933 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:28.958893 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ffb1833-hm2pr" event={"ID":"d22ba0fe-de2a-4ee9-b71e-68d1ee087da5","Type":"ContainerStarted","Data":"b5bc767e383fe4a05db6bd506dab86d9d93437887e05cad2be11d894736f71cb"} Oct 30 20:55:28.959101 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:28.958958 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ffb1833-hm2pr" event={"ID":"d22ba0fe-de2a-4ee9-b71e-68d1ee087da5","Type":"ContainerStarted","Data":"56bd6e7feefdcd0e89ea107c56d951438f1fd6ea5b4cd4d755ee4d83b9800b1d"} Oct 30 20:55:28.978712 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:28.978666 4611 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ffb1833-hm2pr" podStartSLOduration=1.978652311 podStartE2EDuration="1.978652311s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:55:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:55:28.976665118 +0000 UTC m=+983.510691735" watchObservedRunningTime="2026-10-30 20:55:28.978652311 +0000 UTC m=+983.512678981" Oct 30 20:55:29.193883 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:29.193673 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:55:29.684770 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:29.684705 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:55:29.971933 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:29.968323 4611 generic.go:356] "Generic (PLEG): container finished" podID="d22ba0fe-de2a-4ee9-b71e-68d1ee087da5" containerID="b5bc767e383fe4a05db6bd506dab86d9d93437887e05cad2be11d894736f71cb" exitCode=0 Oct 30 20:55:29.971933 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:29.968452 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ffb1833-hm2pr" event={"ID":"d22ba0fe-de2a-4ee9-b71e-68d1ee087da5","Type":"ContainerDied","Data":"b5bc767e383fe4a05db6bd506dab86d9d93437887e05cad2be11d894736f71cb"} Oct 30 20:55:29.971933 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:29.969425 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:55:30.717829 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:30.717792 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:55:31.145182 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:31.145154 4611 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ffb1833-hm2pr" Oct 30 20:55:31.247339 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:31.247265 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d22ba0fe-de2a-4ee9-b71e-68d1ee087da5-kube-api-access-lz7tt\" (UniqueName: \"kubernetes.io/projected/d22ba0fe-de2a-4ee9-b71e-68d1ee087da5-kube-api-access-lz7tt\") pod \"d22ba0fe-de2a-4ee9-b71e-68d1ee087da5\" (UID: \"d22ba0fe-de2a-4ee9-b71e-68d1ee087da5\") " Oct 30 20:55:31.247544 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:31.247501 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d22ba0fe-de2a-4ee9-b71e-68d1ee087da5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d22ba0fe-de2a-4ee9-b71e-68d1ee087da5-maas-api-openshift-ca\") pod \"d22ba0fe-de2a-4ee9-b71e-68d1ee087da5\" (UID: \"d22ba0fe-de2a-4ee9-b71e-68d1ee087da5\") " Oct 30 20:55:31.247658 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:31.247641 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d22ba0fe-de2a-4ee9-b71e-68d1ee087da5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d22ba0fe-de2a-4ee9-b71e-68d1ee087da5-maas-api-xks-ca\") pod \"d22ba0fe-de2a-4ee9-b71e-68d1ee087da5\" (UID: \"d22ba0fe-de2a-4ee9-b71e-68d1ee087da5\") " Oct 30 20:55:31.247910 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:31.247888 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d22ba0fe-de2a-4ee9-b71e-68d1ee087da5-maas-api-openshift-ca" pod "d22ba0fe-de2a-4ee9-b71e-68d1ee087da5" (UID: "d22ba0fe-de2a-4ee9-b71e-68d1ee087da5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:55:31.248204 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:31.248190 4611 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d22ba0fe-de2a-4ee9-b71e-68d1ee087da5-maas-api-openshift-ca\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:55:31.249437 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:31.249413 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d22ba0fe-de2a-4ee9-b71e-68d1ee087da5-kube-api-access-lz7tt" pod "d22ba0fe-de2a-4ee9-b71e-68d1ee087da5" (UID: "d22ba0fe-de2a-4ee9-b71e-68d1ee087da5"). InnerVolumeSpecName "kube-api-access-lz7tt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:55:31.249613 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:31.249597 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d22ba0fe-de2a-4ee9-b71e-68d1ee087da5-maas-api-xks-ca" pod "d22ba0fe-de2a-4ee9-b71e-68d1ee087da5" (UID: "d22ba0fe-de2a-4ee9-b71e-68d1ee087da5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:55:31.349256 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:31.349225 4611 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lz7tt\" (UniqueName: \"kubernetes.io/projected/d22ba0fe-de2a-4ee9-b71e-68d1ee087da5-kube-api-access-lz7tt\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:55:31.349256 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:31.349253 4611 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d22ba0fe-de2a-4ee9-b71e-68d1ee087da5-maas-api-xks-ca\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:55:31.482641 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:31.482601 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:55:31.983550 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:31.983447 4611 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ffb1833-hm2pr" Oct 30 20:55:31.983550 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:31.983496 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ffb1833-hm2pr" event={"ID":"d22ba0fe-de2a-4ee9-b71e-68d1ee087da5","Type":"ContainerDied","Data":"56bd6e7feefdcd0e89ea107c56d951438f1fd6ea5b4cd4d755ee4d83b9800b1d"} Oct 30 20:55:31.983993 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:31.983585 4611 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="56bd6e7feefdcd0e89ea107c56d951438f1fd6ea5b4cd4d755ee4d83b9800b1d" Oct 30 20:55:32.097795 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:32.097741 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:55:32.536561 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:32.536533 4611 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-453cda0b-2krm4"] Oct 30 20:55:32.537204 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:32.537189 4611 cpu_manager.go:414] "containerMap: removing container" podUID="d22ba0fe-de2a-4ee9-b71e-68d1ee087da5" containerName="revoke-keys" Oct 30 20:55:32.537250 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:32.537206 4611 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d22ba0fe-de2a-4ee9-b71e-68d1ee087da5" containerName="revoke-keys" Oct 30 20:55:32.537312 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:32.537302 4611 memory_manager.go:365] "RemoveStaleState removing state" podUID="d22ba0fe-de2a-4ee9-b71e-68d1ee087da5" containerName="revoke-keys" Oct 30 20:55:32.541037 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:32.541019 4611 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-453cda0b-2krm4" Oct 30 20:55:32.544503 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:32.544481 4611 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:55:32.544648 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:32.544636 4611 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7hf8r\"" Oct 30 20:55:32.550793 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:32.550630 4611 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-453cda0b-2krm4"] Oct 30 20:55:32.566091 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:32.566064 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66bfd97d-e97b-41ad-9f4c-dc55ae2cc4c3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-453cda0b-2krm4\" (UID: \"66bfd97d-e97b-41ad-9f4c-dc55ae2cc4c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-453cda0b-2krm4" Oct 30 20:55:32.566225 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:32.566108 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pcmlq\" (UniqueName: \"kubernetes.io/projected/66bfd97d-e97b-41ad-9f4c-dc55ae2cc4c3-kube-api-access-pcmlq\") pod \"maas-api-revoke-sub-453cda0b-2krm4\" (UID: \"66bfd97d-e97b-41ad-9f4c-dc55ae2cc4c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-453cda0b-2krm4" Oct 30 20:55:32.566388 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:32.566367 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66bfd97d-e97b-41ad-9f4c-dc55ae2cc4c3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-453cda0b-2krm4\" (UID: \"66bfd97d-e97b-41ad-9f4c-dc55ae2cc4c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-453cda0b-2krm4" Oct 30 20:55:32.634369 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:32.634338 4611 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-ztxdx"] Oct 30 20:55:32.639720 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:32.639688 4611 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-ztxdx" Oct 30 20:55:32.667841 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:32.667817 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a93b27f-f269-471c-8be2-523f373bca08-istio-podinfo\") pod \"e2e-worker-w6-3d909d-openshift-default-77787695b7-ztxdx\" (UID: \"5a93b27f-f269-471c-8be2-523f373bca08\") " pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-ztxdx" Oct 30 20:55:32.667979 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:32.667855 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66bfd97d-e97b-41ad-9f4c-dc55ae2cc4c3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-453cda0b-2krm4\" (UID: \"66bfd97d-e97b-41ad-9f4c-dc55ae2cc4c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-453cda0b-2krm4" Oct 30 20:55:32.668026 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:32.668003 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66bfd97d-e97b-41ad-9f4c-dc55ae2cc4c3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-453cda0b-2krm4\" (UID: \"66bfd97d-e97b-41ad-9f4c-dc55ae2cc4c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-453cda0b-2krm4" Oct 30 20:55:32.668068 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:32.668031 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a93b27f-f269-471c-8be2-523f373bca08-workload-certs\") pod \"e2e-worker-w6-3d909d-openshift-default-77787695b7-ztxdx\" (UID: \"5a93b27f-f269-471c-8be2-523f373bca08\") " pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-ztxdx" Oct 30 20:55:32.668068 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:32.668057 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pcmlq\" (UniqueName: \"kubernetes.io/projected/66bfd97d-e97b-41ad-9f4c-dc55ae2cc4c3-kube-api-access-pcmlq\") pod \"maas-api-revoke-sub-453cda0b-2krm4\" (UID: \"66bfd97d-e97b-41ad-9f4c-dc55ae2cc4c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-453cda0b-2krm4" Oct 30 20:55:32.668143 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:32.668074 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zdj5h\" (UniqueName: \"kubernetes.io/projected/5a93b27f-f269-471c-8be2-523f373bca08-kube-api-access-zdj5h\") pod \"e2e-worker-w6-3d909d-openshift-default-77787695b7-ztxdx\" (UID: \"5a93b27f-f269-471c-8be2-523f373bca08\") " pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-ztxdx" Oct 30 20:55:32.668143 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:32.668107 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a93b27f-f269-471c-8be2-523f373bca08-istio-envoy\") pod \"e2e-worker-w6-3d909d-openshift-default-77787695b7-ztxdx\" (UID: \"5a93b27f-f269-471c-8be2-523f373bca08\") " pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-ztxdx" Oct 30 20:55:32.668218 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:32.668145 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a93b27f-f269-471c-8be2-523f373bca08-istio-token\") pod \"e2e-worker-w6-3d909d-openshift-default-77787695b7-ztxdx\" (UID: \"5a93b27f-f269-471c-8be2-523f373bca08\") " pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-ztxdx" Oct 30 20:55:32.668218 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:32.668169 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a93b27f-f269-471c-8be2-523f373bca08-workload-socket\") pod \"e2e-worker-w6-3d909d-openshift-default-77787695b7-ztxdx\" (UID: \"5a93b27f-f269-471c-8be2-523f373bca08\") " pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-ztxdx" Oct 30 20:55:32.668289 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:32.668217 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a93b27f-f269-471c-8be2-523f373bca08-istio-data\") pod \"e2e-worker-w6-3d909d-openshift-default-77787695b7-ztxdx\" (UID: \"5a93b27f-f269-471c-8be2-523f373bca08\") " pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-ztxdx" Oct 30 20:55:32.668289 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:32.668266 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a93b27f-f269-471c-8be2-523f373bca08-credential-socket\") pod \"e2e-worker-w6-3d909d-openshift-default-77787695b7-ztxdx\" (UID: \"5a93b27f-f269-471c-8be2-523f373bca08\") " pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-ztxdx" Oct 30 20:55:32.668368 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:32.668310 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a93b27f-f269-471c-8be2-523f373bca08-istiod-ca-cert\") pod \"e2e-worker-w6-3d909d-openshift-default-77787695b7-ztxdx\" (UID: \"5a93b27f-f269-471c-8be2-523f373bca08\") " pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-ztxdx" Oct 30 20:55:32.669163 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:32.669134 4611 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-ztxdx"] Oct 30 20:55:32.669318 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:32.669285 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66bfd97d-e97b-41ad-9f4c-dc55ae2cc4c3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-453cda0b-2krm4\" (UID: \"66bfd97d-e97b-41ad-9f4c-dc55ae2cc4c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-453cda0b-2krm4" Oct 30 20:55:32.670990 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:32.670971 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66bfd97d-e97b-41ad-9f4c-dc55ae2cc4c3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-453cda0b-2krm4\" (UID: \"66bfd97d-e97b-41ad-9f4c-dc55ae2cc4c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-453cda0b-2krm4" Oct 30 20:55:32.683621 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:32.683593 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pcmlq\" (UniqueName: \"kubernetes.io/projected/66bfd97d-e97b-41ad-9f4c-dc55ae2cc4c3-kube-api-access-pcmlq\") pod \"maas-api-revoke-sub-453cda0b-2krm4\" (UID: \"66bfd97d-e97b-41ad-9f4c-dc55ae2cc4c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-453cda0b-2krm4" Oct 30 20:55:32.769595 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:32.769566 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a93b27f-f269-471c-8be2-523f373bca08-workload-certs\") pod \"e2e-worker-w6-3d909d-openshift-default-77787695b7-ztxdx\" (UID: \"5a93b27f-f269-471c-8be2-523f373bca08\") " pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-ztxdx" Oct 30 20:55:32.769716 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:32.769610 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zdj5h\" (UniqueName: \"kubernetes.io/projected/5a93b27f-f269-471c-8be2-523f373bca08-kube-api-access-zdj5h\") pod \"e2e-worker-w6-3d909d-openshift-default-77787695b7-ztxdx\" (UID: \"5a93b27f-f269-471c-8be2-523f373bca08\") " pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-ztxdx" Oct 30 20:55:32.769716 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:32.769647 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a93b27f-f269-471c-8be2-523f373bca08-istio-envoy\") pod \"e2e-worker-w6-3d909d-openshift-default-77787695b7-ztxdx\" (UID: \"5a93b27f-f269-471c-8be2-523f373bca08\") " pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-ztxdx" Oct 30 20:55:32.769716 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:32.769692 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a93b27f-f269-471c-8be2-523f373bca08-istio-token\") pod \"e2e-worker-w6-3d909d-openshift-default-77787695b7-ztxdx\" (UID: \"5a93b27f-f269-471c-8be2-523f373bca08\") " pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-ztxdx" Oct 30 20:55:32.769895 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:32.769716 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a93b27f-f269-471c-8be2-523f373bca08-workload-socket\") pod \"e2e-worker-w6-3d909d-openshift-default-77787695b7-ztxdx\" (UID: \"5a93b27f-f269-471c-8be2-523f373bca08\") " pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-ztxdx" Oct 30 20:55:32.769895 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:32.769780 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a93b27f-f269-471c-8be2-523f373bca08-istio-data\") pod \"e2e-worker-w6-3d909d-openshift-default-77787695b7-ztxdx\" (UID: \"5a93b27f-f269-471c-8be2-523f373bca08\") " pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-ztxdx" Oct 30 20:55:32.769895 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:32.769864 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a93b27f-f269-471c-8be2-523f373bca08-credential-socket\") pod \"e2e-worker-w6-3d909d-openshift-default-77787695b7-ztxdx\" (UID: \"5a93b27f-f269-471c-8be2-523f373bca08\") " pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-ztxdx" Oct 30 20:55:32.770063 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:32.769906 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a93b27f-f269-471c-8be2-523f373bca08-istiod-ca-cert\") pod \"e2e-worker-w6-3d909d-openshift-default-77787695b7-ztxdx\" (UID: \"5a93b27f-f269-471c-8be2-523f373bca08\") " pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-ztxdx" Oct 30 20:55:32.770063 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:32.769947 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a93b27f-f269-471c-8be2-523f373bca08-istio-podinfo\") pod \"e2e-worker-w6-3d909d-openshift-default-77787695b7-ztxdx\" (UID: \"5a93b27f-f269-471c-8be2-523f373bca08\") " pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-ztxdx" Oct 30 20:55:32.770063 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:32.769945 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a93b27f-f269-471c-8be2-523f373bca08-workload-certs\") pod \"e2e-worker-w6-3d909d-openshift-default-77787695b7-ztxdx\" (UID: \"5a93b27f-f269-471c-8be2-523f373bca08\") " pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-ztxdx" Oct 30 20:55:32.770244 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:32.770223 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a93b27f-f269-471c-8be2-523f373bca08-workload-socket\") pod \"e2e-worker-w6-3d909d-openshift-default-77787695b7-ztxdx\" (UID: \"5a93b27f-f269-471c-8be2-523f373bca08\") " pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-ztxdx" Oct 30 20:55:32.770368 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:32.770347 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a93b27f-f269-471c-8be2-523f373bca08-istio-data\") pod \"e2e-worker-w6-3d909d-openshift-default-77787695b7-ztxdx\" (UID: \"5a93b27f-f269-471c-8be2-523f373bca08\") " pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-ztxdx" Oct 30 20:55:32.770430 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:32.770369 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a93b27f-f269-471c-8be2-523f373bca08-credential-socket\") pod \"e2e-worker-w6-3d909d-openshift-default-77787695b7-ztxdx\" (UID: \"5a93b27f-f269-471c-8be2-523f373bca08\") " pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-ztxdx" Oct 30 20:55:32.770677 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:32.770653 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a93b27f-f269-471c-8be2-523f373bca08-istiod-ca-cert\") pod \"e2e-worker-w6-3d909d-openshift-default-77787695b7-ztxdx\" (UID: \"5a93b27f-f269-471c-8be2-523f373bca08\") " pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-ztxdx" Oct 30 20:55:32.771898 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:32.771874 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a93b27f-f269-471c-8be2-523f373bca08-istio-envoy\") pod \"e2e-worker-w6-3d909d-openshift-default-77787695b7-ztxdx\" (UID: \"5a93b27f-f269-471c-8be2-523f373bca08\") " pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-ztxdx" Oct 30 20:55:32.772307 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:32.772289 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a93b27f-f269-471c-8be2-523f373bca08-istio-podinfo\") pod \"e2e-worker-w6-3d909d-openshift-default-77787695b7-ztxdx\" (UID: \"5a93b27f-f269-471c-8be2-523f373bca08\") " pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-ztxdx" Oct 30 20:55:32.780700 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:32.780678 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a93b27f-f269-471c-8be2-523f373bca08-istio-token\") pod \"e2e-worker-w6-3d909d-openshift-default-77787695b7-ztxdx\" (UID: \"5a93b27f-f269-471c-8be2-523f373bca08\") " pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-ztxdx" Oct 30 20:55:32.781005 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:32.780989 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zdj5h\" (UniqueName: \"kubernetes.io/projected/5a93b27f-f269-471c-8be2-523f373bca08-kube-api-access-zdj5h\") pod \"e2e-worker-w6-3d909d-openshift-default-77787695b7-ztxdx\" (UID: \"5a93b27f-f269-471c-8be2-523f373bca08\") " pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-ztxdx" Oct 30 20:55:32.857873 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:32.857852 4611 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-453cda0b-2krm4" Oct 30 20:55:32.960756 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:32.960725 4611 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-ztxdx" Oct 30 20:55:33.000011 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:32.999969 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:55:33.152805 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:33.152774 4611 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-453cda0b-2krm4"] Oct 30 20:55:33.156609 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:55:33.156577 4611 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66bfd97d_e97b_41ad_9f4c_dc55ae2cc4c3.slice/crio-ae17a7a5e8189206785b5b354b564537e41a71f456436a013d323a1ced9c498f WatchSource:0}: Error finding container ae17a7a5e8189206785b5b354b564537e41a71f456436a013d323a1ced9c498f: Status 404 returned error can't find the container with id ae17a7a5e8189206785b5b354b564537e41a71f456436a013d323a1ced9c498f Oct 30 20:55:33.272216 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:33.272181 4611 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-ztxdx"] Oct 30 20:55:33.272635 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:55:33.272610 4611 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a93b27f_f269_471c_8be2_523f373bca08.slice/crio-0c597908494eba6e21902a88e018b44f15bb5d1a9da340e64c5d0fd2257671ff WatchSource:0}: Error finding container 0c597908494eba6e21902a88e018b44f15bb5d1a9da340e64c5d0fd2257671ff: Status 404 returned error can't find the container with id 0c597908494eba6e21902a88e018b44f15bb5d1a9da340e64c5d0fd2257671ff Oct 30 20:55:33.275769 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:33.275727 4611 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:55:33.275871 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:33.275818 4611 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:55:33.275932 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:33.275868 4611 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:55:33.693692 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:33.693657 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:55:33.920042 ip-10-0-133-192 kubenswrapper[4611]: E1030 20:55:33.920007 4611 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hwzc6" containerName="extractor" Oct 30 20:55:34.002524 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:34.002478 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-ztxdx" event={"ID":"5a93b27f-f269-471c-8be2-523f373bca08","Type":"ContainerStarted","Data":"f8d503a52fd7f6a1f40bb16219178d4eadb387455d394cd382f26d0f645f0dd5"} Oct 30 20:55:34.002969 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:34.002551 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-ztxdx" event={"ID":"5a93b27f-f269-471c-8be2-523f373bca08","Type":"ContainerStarted","Data":"0c597908494eba6e21902a88e018b44f15bb5d1a9da340e64c5d0fd2257671ff"} Oct 30 20:55:34.004291 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:34.004259 4611 generic.go:356] "Generic (PLEG): container finished" podID="66bfd97d-e97b-41ad-9f4c-dc55ae2cc4c3" containerID="0263dc0f6eeecd38f12c665ad11fb9022ddb078ff69fd9d2198a264d2f900478" exitCode=0 Oct 30 20:55:34.004413 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:34.004385 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-453cda0b-2krm4" event={"ID":"66bfd97d-e97b-41ad-9f4c-dc55ae2cc4c3","Type":"ContainerDied","Data":"0263dc0f6eeecd38f12c665ad11fb9022ddb078ff69fd9d2198a264d2f900478"} Oct 30 20:55:34.004455 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:34.004414 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-453cda0b-2krm4" event={"ID":"66bfd97d-e97b-41ad-9f4c-dc55ae2cc4c3","Type":"ContainerStarted","Data":"ae17a7a5e8189206785b5b354b564537e41a71f456436a013d323a1ced9c498f"} Oct 30 20:55:34.028812 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:34.028760 4611 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-ztxdx" podStartSLOduration=2.028746465 podStartE2EDuration="2.028746465s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:55:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:55:34.028180759 +0000 UTC m=+988.562207374" watchObservedRunningTime="2026-10-30 20:55:34.028746465 +0000 UTC m=+988.562773081" Oct 30 20:55:34.582131 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:34.582089 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:55:34.970850 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:34.970818 4611 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-ztxdx" Oct 30 20:55:34.987274 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:34.986940 4611 patch_prober.go:29] interesting pod/e2e-worker-w6-3d909d-openshift-default-77787695b7-ztxdx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body= Oct 30 20:55:34.987274 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:34.987010 4611 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-ztxdx" podUID="5a93b27f-f269-471c-8be2-523f373bca08" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" Oct 30 20:55:35.217189 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:35.217159 4611 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-453cda0b-2krm4" Oct 30 20:55:35.313962 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:35.313882 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/66bfd97d-e97b-41ad-9f4c-dc55ae2cc4c3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66bfd97d-e97b-41ad-9f4c-dc55ae2cc4c3-maas-api-xks-ca\") pod \"66bfd97d-e97b-41ad-9f4c-dc55ae2cc4c3\" (UID: \"66bfd97d-e97b-41ad-9f4c-dc55ae2cc4c3\") " Oct 30 20:55:35.313962 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:35.313927 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/66bfd97d-e97b-41ad-9f4c-dc55ae2cc4c3-kube-api-access-pcmlq\" (UniqueName: \"kubernetes.io/projected/66bfd97d-e97b-41ad-9f4c-dc55ae2cc4c3-kube-api-access-pcmlq\") pod \"66bfd97d-e97b-41ad-9f4c-dc55ae2cc4c3\" (UID: \"66bfd97d-e97b-41ad-9f4c-dc55ae2cc4c3\") " Oct 30 20:55:35.314260 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:35.314016 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/66bfd97d-e97b-41ad-9f4c-dc55ae2cc4c3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66bfd97d-e97b-41ad-9f4c-dc55ae2cc4c3-maas-api-openshift-ca\") pod \"66bfd97d-e97b-41ad-9f4c-dc55ae2cc4c3\" (UID: \"66bfd97d-e97b-41ad-9f4c-dc55ae2cc4c3\") " Oct 30 20:55:35.314434 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:35.314412 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66bfd97d-e97b-41ad-9f4c-dc55ae2cc4c3-maas-api-openshift-ca" pod "66bfd97d-e97b-41ad-9f4c-dc55ae2cc4c3" (UID: "66bfd97d-e97b-41ad-9f4c-dc55ae2cc4c3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:55:35.314731 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:35.314709 4611 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66bfd97d-e97b-41ad-9f4c-dc55ae2cc4c3-maas-api-openshift-ca\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:55:35.316088 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:35.316065 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66bfd97d-e97b-41ad-9f4c-dc55ae2cc4c3-kube-api-access-pcmlq" pod "66bfd97d-e97b-41ad-9f4c-dc55ae2cc4c3" (UID: "66bfd97d-e97b-41ad-9f4c-dc55ae2cc4c3"). InnerVolumeSpecName "kube-api-access-pcmlq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:55:35.316453 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:35.316433 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/66bfd97d-e97b-41ad-9f4c-dc55ae2cc4c3-maas-api-xks-ca" pod "66bfd97d-e97b-41ad-9f4c-dc55ae2cc4c3" (UID: "66bfd97d-e97b-41ad-9f4c-dc55ae2cc4c3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:55:35.415782 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:35.415740 4611 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66bfd97d-e97b-41ad-9f4c-dc55ae2cc4c3-maas-api-xks-ca\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:55:35.415782 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:35.415778 4611 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pcmlq\" (UniqueName: \"kubernetes.io/projected/66bfd97d-e97b-41ad-9f4c-dc55ae2cc4c3-kube-api-access-pcmlq\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:55:35.541336 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:35.541295 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:55:35.970908 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:35.970697 4611 patch_prober.go:29] interesting pod/e2e-worker-w6-3d909d-openshift-default-77787695b7-ztxdx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body= Oct 30 20:55:35.970908 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:35.970755 4611 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-ztxdx" podUID="5a93b27f-f269-471c-8be2-523f373bca08" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" Oct 30 20:55:36.024116 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:36.024074 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-453cda0b-2krm4" event={"ID":"66bfd97d-e97b-41ad-9f4c-dc55ae2cc4c3","Type":"ContainerDied","Data":"ae17a7a5e8189206785b5b354b564537e41a71f456436a013d323a1ced9c498f"} Oct 30 20:55:36.024375 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:36.024342 4611 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ae17a7a5e8189206785b5b354b564537e41a71f456436a013d323a1ced9c498f" Oct 30 20:55:36.024375 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:36.024107 4611 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-453cda0b-2krm4" Oct 30 20:55:36.530822 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:36.530788 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:55:36.912887 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:36.912853 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:55:36.964890 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:36.964853 4611 patch_prober.go:29] interesting pod/e2e-worker-w6-3d909d-openshift-default-77787695b7-ztxdx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body= Oct 30 20:55:36.966114 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:36.966082 4611 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-ztxdx" podUID="5a93b27f-f269-471c-8be2-523f373bca08" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" Oct 30 20:55:37.071165 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:37.071115 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:55:37.525003 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:37.524966 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:55:37.970601 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:37.967869 4611 patch_prober.go:29] interesting pod/e2e-worker-w6-3d909d-openshift-default-77787695b7-ztxdx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body= Oct 30 20:55:37.970601 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:37.967928 4611 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-ztxdx" podUID="5a93b27f-f269-471c-8be2-523f373bca08" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" Oct 30 20:55:38.612990 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:38.612951 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:55:38.973588 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:38.972146 4611 patch_prober.go:29] interesting pod/e2e-worker-w6-3d909d-openshift-default-77787695b7-ztxdx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body= Oct 30 20:55:38.973588 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:38.972211 4611 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-ztxdx" podUID="5a93b27f-f269-471c-8be2-523f373bca08" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" Oct 30 20:55:39.968305 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:39.968269 4611 patch_prober.go:29] interesting pod/e2e-worker-w6-3d909d-openshift-default-77787695b7-ztxdx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body= Oct 30 20:55:39.968587 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:39.968556 4611 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-ztxdx" podUID="5a93b27f-f269-471c-8be2-523f373bca08" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" Oct 30 20:55:40.986535 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:40.974138 4611 patch_prober.go:29] interesting pod/e2e-worker-w6-3d909d-openshift-default-77787695b7-ztxdx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body= Oct 30 20:55:40.986535 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:40.974205 4611 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-ztxdx" podUID="5a93b27f-f269-471c-8be2-523f373bca08" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" Oct 30 20:55:41.057966 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:41.053657 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b8aee1/premium-w1-b8aee1-kserve-786d575599-htc8d" podUID="77368afe-edc8-4fa5-8010-d321965fb38e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded" Oct 30 20:55:41.057966 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:41.054976 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cb60a2/distinct-w3-cb60a2-kserve-5bb76dc74-86fds" podUID="52e301d9-f378-4e54-9515-01a6680d9fd7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:41.057966 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:41.055991 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-14cca2/premium-w5-14cca2-kserve-6878777857-wpjtj" podUID="b99eb76b-d1c6-49e9-9809-c351c0f5a4de" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded" Oct 30 20:55:41.091093 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:41.090998 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:55:41.169786 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:41.169741 4611 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-b8aee1/premium-w1-b8aee1-kserve-786d575599-htc8d" podUID="77368afe-edc8-4fa5-8010-d321965fb38e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:41.235529 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:41.235102 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f2ed1e/distinct-w4-f2ed1e-kserve-57cff9dcf8-8d5zl" podUID="653e3ec7-2147-4813-8946-1f25cfa425f6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:41.281695 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:41.273775 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b8aee1/distinct-2-w1-b8aee1-kserve-6bcdd59797-cx6hr" podUID="286eab7a-81a7-4768-a7f6-be6b3720028a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": context deadline exceeded" Oct 30 20:55:41.281695 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:41.277275 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-14cca2/distinct-w5-14cca2-kserve-587f6996b8-sgf6h" podUID="02b652c3-0724-4a50-9e95-2a9890cc97c4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:41.381535 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:41.378969 4611 patch_prober.go:29] interesting pod/e2e-worker-w2-db534b-openshift-default-85df4d79d9-kb6vb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:55:41.381535 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:41.379036 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-kb6vb" podUID="41f0d1eb-188b-404a-8b31-cf9dc16b9fbc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:41.443535 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:41.427685 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b8aee1/distinct-w1-b8aee1-kserve-5b857c55cd-x5b7b" podUID="1599e588-def8-42e1-b81d-ad0aaf119818" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:41.764530 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:41.760827 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db534b/distinct-2-w2-db534b-kserve-8594c876-g92mh" podUID="955316d6-a9fd-4b75-a8ab-bbf382b776e2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": context deadline exceeded" Oct 30 20:55:41.765928 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:41.765891 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-3d909d/distinct-2-w6-3d909d-kserve-59c5d5c595-cjg2m" podUID="e2b4c221-ee94-4131-abd8-2313647c641e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": context deadline exceeded" Oct 30 20:55:41.890534 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:41.766145 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-3d909d/premium-w6-3d909d-kserve-765648946f-2qvd2" podUID="fca5d697-109d-4e90-ab41-c40ad31af874" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:41.890534 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:41.827914 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cb60a2/simulator-w3-cb60a2-kserve-6fd74b9d8b-vqfzf" podUID="711b167d-4e66-40d6-be75-33ede11f42c5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:41.990022 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:41.989967 4611 patch_prober.go:29] interesting pod/e2e-worker-w6-3d909d-openshift-default-77787695b7-ztxdx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body= Oct 30 20:55:41.990609 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:41.990576 4611 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-ztxdx" podUID="5a93b27f-f269-471c-8be2-523f373bca08" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" Oct 30 20:55:41.991130 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:41.991102 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad7ff0/premium-w0-ad7ff0-kserve-5485d4b948-bpmgf" podUID="53cbd692-e9f8-46ba-8fde-3e5215a38904" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:42.104635 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:42.054646 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b8aee1/premium-w1-b8aee1-kserve-786d575599-htc8d" podUID="77368afe-edc8-4fa5-8010-d321965fb38e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:42.226617 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:42.056687 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-14cca2/premium-w5-14cca2-kserve-6878777857-wpjtj" podUID="b99eb76b-d1c6-49e9-9809-c351c0f5a4de" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:42.227210 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:42.056723 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cb60a2/distinct-w3-cb60a2-kserve-5bb76dc74-86fds" podUID="52e301d9-f378-4e54-9515-01a6680d9fd7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded" Oct 30 20:55:42.227660 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:42.132651 4611 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-b8aee1/distinct-w1-b8aee1-kserve-5b857c55cd-x5b7b" podUID="1599e588-def8-42e1-b81d-ad0aaf119818" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:42.256864 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:42.256807 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f2ed1e/distinct-w4-f2ed1e-kserve-57cff9dcf8-8d5zl" podUID="653e3ec7-2147-4813-8946-1f25cfa425f6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:42.258053 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:42.258031 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b8aee1/premium-w1-b8aee1-kserve-786d575599-htc8d" Oct 30 20:55:42.281295 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:42.281267 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-14cca2/premium-w5-14cca2-kserve-6878777857-wpjtj" Oct 30 20:55:42.281578 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:42.281561 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-cb60a2/distinct-w3-cb60a2-kserve-5bb76dc74-86fds" Oct 30 20:55:42.281783 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:42.281758 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-f2ed1e/distinct-w4-f2ed1e-kserve-57cff9dcf8-8d5zl" Oct 30 20:55:42.365657 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:42.344708 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-14cca2/distinct-w5-14cca2-kserve-587f6996b8-sgf6h" podUID="02b652c3-0724-4a50-9e95-2a9890cc97c4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:42.365657 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:42.345097 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-14cca2/distinct-w5-14cca2-kserve-587f6996b8-sgf6h" Oct 30 20:55:42.445746 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:42.445685 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b8aee1/distinct-w1-b8aee1-kserve-5b857c55cd-x5b7b" podUID="1599e588-def8-42e1-b81d-ad0aaf119818" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": context deadline exceeded" Oct 30 20:55:42.525573 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:42.446434 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b8aee1/distinct-w1-b8aee1-kserve-5b857c55cd-x5b7b" Oct 30 20:55:42.799207 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:42.799102 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db534b/distinct-2-w2-db534b-kserve-8594c876-g92mh" podUID="955316d6-a9fd-4b75-a8ab-bbf382b776e2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": context deadline exceeded" Oct 30 20:55:42.830232 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:42.830203 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-db534b/distinct-2-w2-db534b-kserve-8594c876-g92mh" Oct 30 20:55:42.882719 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:42.882664 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-3d909d/distinct-2-w6-3d909d-kserve-59c5d5c595-cjg2m" podUID="e2b4c221-ee94-4131-abd8-2313647c641e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": context deadline exceeded" Oct 30 20:55:42.914534 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:42.882864 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cb60a2/simulator-w3-cb60a2-kserve-6fd74b9d8b-vqfzf" podUID="711b167d-4e66-40d6-be75-33ede11f42c5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:42.914534 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:42.882892 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-3d909d/premium-w6-3d909d-kserve-765648946f-2qvd2" podUID="fca5d697-109d-4e90-ab41-c40ad31af874" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded" Oct 30 20:55:42.952674 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:42.914888 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-3d909d/distinct-2-w6-3d909d-kserve-59c5d5c595-cjg2m" Oct 30 20:55:42.952674 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:42.914968 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-cb60a2/simulator-w3-cb60a2-kserve-6fd74b9d8b-vqfzf" Oct 30 20:55:42.952674 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:42.915007 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-3d909d/premium-w6-3d909d-kserve-765648946f-2qvd2" Oct 30 20:55:42.953347 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:42.953310 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b8aee1/distinct-2-w1-b8aee1-kserve-6bcdd59797-cx6hr" podUID="286eab7a-81a7-4768-a7f6-be6b3720028a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": context deadline exceeded" Oct 30 20:55:42.953849 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:42.953816 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b8aee1/distinct-2-w1-b8aee1-kserve-6bcdd59797-cx6hr" Oct 30 20:55:43.051062 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:43.003002 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad7ff0/premium-w0-ad7ff0-kserve-5485d4b948-bpmgf" podUID="53cbd692-e9f8-46ba-8fde-3e5215a38904" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded" Oct 30 20:55:43.145567 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:43.098181 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ad7ff0/premium-w0-ad7ff0-kserve-5485d4b948-bpmgf" Oct 30 20:55:43.232655 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:43.232593 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cb60a2/distinct-w3-cb60a2-kserve-5bb76dc74-86fds" podUID="52e301d9-f378-4e54-9515-01a6680d9fd7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded" Oct 30 20:55:43.248771 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:43.248716 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-14cca2/premium-w5-14cca2-kserve-6878777857-wpjtj" podUID="b99eb76b-d1c6-49e9-9809-c351c0f5a4de" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded" Oct 30 20:55:43.250778 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:43.250738 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:55:43.284798 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:43.266325 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f2ed1e/distinct-w4-f2ed1e-kserve-57cff9dcf8-8d5zl" podUID="653e3ec7-2147-4813-8946-1f25cfa425f6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:43.353591 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:43.351026 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-14cca2/distinct-w5-14cca2-kserve-587f6996b8-sgf6h" podUID="02b652c3-0724-4a50-9e95-2a9890cc97c4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded" Oct 30 20:55:43.461535 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:43.448621 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b8aee1/distinct-w1-b8aee1-kserve-5b857c55cd-x5b7b" podUID="1599e588-def8-42e1-b81d-ad0aaf119818" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:43.582530 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:43.559553 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:55:43.735610 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:43.735492 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b8aee1/premium-w1-b8aee1-kserve-786d575599-htc8d" podUID="77368afe-edc8-4fa5-8010-d321965fb38e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:43.785532 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:43.762608 4611 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-cb60a2/simulator-w3-cb60a2-kserve-6fd74b9d8b-vqfzf" podUID="711b167d-4e66-40d6-be75-33ede11f42c5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:43.836797 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:43.836753 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db534b/distinct-2-w2-db534b-kserve-8594c876-g92mh" podUID="955316d6-a9fd-4b75-a8ab-bbf382b776e2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:43.914011 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:43.913964 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-3d909d/premium-w6-3d909d-kserve-765648946f-2qvd2" podUID="fca5d697-109d-4e90-ab41-c40ad31af874" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:43.936130 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:43.925654 4611 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-b8aee1/distinct-2-w1-b8aee1-kserve-6bcdd59797-cx6hr" podUID="286eab7a-81a7-4768-a7f6-be6b3720028a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:43.936130 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:43.925745 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cb60a2/simulator-w3-cb60a2-kserve-6fd74b9d8b-vqfzf" podUID="711b167d-4e66-40d6-be75-33ede11f42c5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded" Oct 30 20:55:43.963263 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:43.963125 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-3d909d/distinct-2-w6-3d909d-kserve-59c5d5c595-cjg2m" podUID="e2b4c221-ee94-4131-abd8-2313647c641e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:43.981951 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:43.968501 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b8aee1/distinct-2-w1-b8aee1-kserve-6bcdd59797-cx6hr" podUID="286eab7a-81a7-4768-a7f6-be6b3720028a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:44.104362 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:44.103682 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad7ff0/premium-w0-ad7ff0-kserve-5485d4b948-bpmgf" podUID="53cbd692-e9f8-46ba-8fde-3e5215a38904" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:44.214680 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:44.214631 4611 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-14cca2/distinct-w5-14cca2-kserve-587f6996b8-sgf6h" podUID="02b652c3-0724-4a50-9e95-2a9890cc97c4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:44.262642 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:44.262356 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-14cca2/premium-w5-14cca2-kserve-6878777857-wpjtj" podUID="b99eb76b-d1c6-49e9-9809-c351c0f5a4de" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded" Oct 30 20:55:44.324020 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:44.323974 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cb60a2/distinct-w3-cb60a2-kserve-5bb76dc74-86fds" podUID="52e301d9-f378-4e54-9515-01a6680d9fd7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:44.325289 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:44.325258 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:55:44.437424 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:44.430785 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-14cca2/distinct-w5-14cca2-kserve-587f6996b8-sgf6h" podUID="02b652c3-0724-4a50-9e95-2a9890cc97c4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:44.437424 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:44.433039 4611 patch_prober.go:29] interesting pod/e2e-worker-w6-3d909d-openshift-default-77787695b7-ztxdx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:55:44.437424 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:44.433105 4611 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-ztxdx" podUID="5a93b27f-f269-471c-8be2-523f373bca08" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:44.440911 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:44.440885 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-f2ed1e/distinct-w4-f2ed1e-kserve-57cff9dcf8-8d5zl" Oct 30 20:55:44.443258 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:44.441343 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b8aee1/distinct-2-w1-b8aee1-kserve-6bcdd59797-cx6hr" Oct 30 20:55:44.443258 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:44.441414 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b8aee1/distinct-w1-b8aee1-kserve-5b857c55cd-x5b7b" Oct 30 20:55:44.443258 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:44.441465 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-3d909d/distinct-2-w6-3d909d-kserve-59c5d5c595-cjg2m" Oct 30 20:55:44.443258 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:44.441547 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-cb60a2/simulator-w3-cb60a2-kserve-6fd74b9d8b-vqfzf" Oct 30 20:55:44.443258 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:44.441597 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-3d909d/premium-w6-3d909d-kserve-765648946f-2qvd2" Oct 30 20:55:44.443258 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:44.441648 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b8aee1/premium-w1-b8aee1-kserve-786d575599-htc8d" Oct 30 20:55:44.568561 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:44.565052 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-14cca2/premium-w5-14cca2-kserve-6878777857-wpjtj" Oct 30 20:55:44.569235 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:44.569217 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ad7ff0/premium-w0-ad7ff0-kserve-5485d4b948-bpmgf" Oct 30 20:55:44.570426 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:44.570409 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-cb60a2/distinct-w3-cb60a2-kserve-5bb76dc74-86fds" Oct 30 20:55:44.597631 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:44.597595 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-db534b/distinct-2-w2-db534b-kserve-8594c876-g92mh" Oct 30 20:55:44.602152 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:44.602130 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-14cca2/distinct-w5-14cca2-kserve-587f6996b8-sgf6h" Oct 30 20:55:44.921679 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:44.921490 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:55:45.474655 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:45.474613 4611 patch_prober.go:29] interesting pod/e2e-worker-w6-3d909d-openshift-default-77787695b7-ztxdx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:55:45.475169 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:45.474684 4611 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-ztxdx" podUID="5a93b27f-f269-471c-8be2-523f373bca08" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:45.698571 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:45.698539 4611 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-ztxdx" Oct 30 20:55:45.699114 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:45.699088 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-ztxdx" Oct 30 20:55:45.700088 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:45.700060 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-ztxdx" Oct 30 20:55:46.021756 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:46.021714 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:55:46.481333 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:46.481290 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:55:46.850538 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:46.850444 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:55:47.074767 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:47.074727 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:55:47.310605 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:47.310560 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:55:47.697737 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:47.697704 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:55:48.388967 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:48.388928 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:55:48.607567 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:48.607528 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:55:49.198756 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:49.198701 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:55:49.878388 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:49.878354 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:55:50.694280 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:50.694179 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:55:50.843183 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:50.843043 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:55:51.381753 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:51.381711 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:55:51.821136 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:51.821037 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:55:52.078650 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:52.078571 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:55:52.515528 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:52.515466 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:55:53.015926 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:53.015887 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:55:53.339019 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:53.338950 4611 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-a8daf029-openshift-default-79cb595b95-qqft5"] Oct 30 20:55:53.339647 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:53.339621 4611 cpu_manager.go:414] "containerMap: removing container" podUID="66bfd97d-e97b-41ad-9f4c-dc55ae2cc4c3" containerName="revoke-keys" Oct 30 20:55:53.339647 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:53.339643 4611 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66bfd97d-e97b-41ad-9f4c-dc55ae2cc4c3" containerName="revoke-keys" Oct 30 20:55:53.339933 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:53.339899 4611 memory_manager.go:365] "RemoveStaleState removing state" podUID="66bfd97d-e97b-41ad-9f4c-dc55ae2cc4c3" containerName="revoke-keys" Oct 30 20:55:53.343989 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:53.343958 4611 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-a8daf029-openshift-default-79cb595b95-qqft5" Oct 30 20:55:53.358048 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:53.358024 4611 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-a8daf029-openshift-default-dockercfg-gnt9p\"" Oct 30 20:55:53.362096 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:53.362070 4611 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-a8daf029-openshift-default-79cb595b95-qqft5"] Oct 30 20:55:53.403379 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:53.403350 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8632241a-3073-4493-9fab-298607317a96-istio-data\") pod \"e2e-ait-a8daf029-openshift-default-79cb595b95-qqft5\" (UID: \"8632241a-3073-4493-9fab-298607317a96\") " pod="openshift-ingress/e2e-ait-a8daf029-openshift-default-79cb595b95-qqft5" Oct 30 20:55:53.403558 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:53.403499 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8632241a-3073-4493-9fab-298607317a96-istio-token\") pod \"e2e-ait-a8daf029-openshift-default-79cb595b95-qqft5\" (UID: \"8632241a-3073-4493-9fab-298607317a96\") " pod="openshift-ingress/e2e-ait-a8daf029-openshift-default-79cb595b95-qqft5" Oct 30 20:55:53.403639 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:53.403582 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8632241a-3073-4493-9fab-298607317a96-workload-certs\") pod \"e2e-ait-a8daf029-openshift-default-79cb595b95-qqft5\" (UID: \"8632241a-3073-4493-9fab-298607317a96\") " pod="openshift-ingress/e2e-ait-a8daf029-openshift-default-79cb595b95-qqft5" Oct 30 20:55:53.403700 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:53.403654 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8632241a-3073-4493-9fab-298607317a96-workload-socket\") pod \"e2e-ait-a8daf029-openshift-default-79cb595b95-qqft5\" (UID: \"8632241a-3073-4493-9fab-298607317a96\") " pod="openshift-ingress/e2e-ait-a8daf029-openshift-default-79cb595b95-qqft5" Oct 30 20:55:53.403700 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:53.403692 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jjxzb\" (UniqueName: \"kubernetes.io/projected/8632241a-3073-4493-9fab-298607317a96-kube-api-access-jjxzb\") pod \"e2e-ait-a8daf029-openshift-default-79cb595b95-qqft5\" (UID: \"8632241a-3073-4493-9fab-298607317a96\") " pod="openshift-ingress/e2e-ait-a8daf029-openshift-default-79cb595b95-qqft5" Oct 30 20:55:53.403813 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:53.403716 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8632241a-3073-4493-9fab-298607317a96-istiod-ca-cert\") pod \"e2e-ait-a8daf029-openshift-default-79cb595b95-qqft5\" (UID: \"8632241a-3073-4493-9fab-298607317a96\") " pod="openshift-ingress/e2e-ait-a8daf029-openshift-default-79cb595b95-qqft5" Oct 30 20:55:53.403874 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:53.403830 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8632241a-3073-4493-9fab-298607317a96-istio-podinfo\") pod \"e2e-ait-a8daf029-openshift-default-79cb595b95-qqft5\" (UID: \"8632241a-3073-4493-9fab-298607317a96\") " pod="openshift-ingress/e2e-ait-a8daf029-openshift-default-79cb595b95-qqft5" Oct 30 20:55:53.403874 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:53.403861 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8632241a-3073-4493-9fab-298607317a96-istio-envoy\") pod \"e2e-ait-a8daf029-openshift-default-79cb595b95-qqft5\" (UID: \"8632241a-3073-4493-9fab-298607317a96\") " pod="openshift-ingress/e2e-ait-a8daf029-openshift-default-79cb595b95-qqft5" Oct 30 20:55:53.403979 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:53.403900 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8632241a-3073-4493-9fab-298607317a96-credential-socket\") pod \"e2e-ait-a8daf029-openshift-default-79cb595b95-qqft5\" (UID: \"8632241a-3073-4493-9fab-298607317a96\") " pod="openshift-ingress/e2e-ait-a8daf029-openshift-default-79cb595b95-qqft5" Oct 30 20:55:53.460178 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:53.457549 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:55:53.506666 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:53.506624 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8632241a-3073-4493-9fab-298607317a96-istio-podinfo\") pod \"e2e-ait-a8daf029-openshift-default-79cb595b95-qqft5\" (UID: \"8632241a-3073-4493-9fab-298607317a96\") " pod="openshift-ingress/e2e-ait-a8daf029-openshift-default-79cb595b95-qqft5" Oct 30 20:55:53.506974 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:53.506948 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8632241a-3073-4493-9fab-298607317a96-istio-envoy\") pod \"e2e-ait-a8daf029-openshift-default-79cb595b95-qqft5\" (UID: \"8632241a-3073-4493-9fab-298607317a96\") " pod="openshift-ingress/e2e-ait-a8daf029-openshift-default-79cb595b95-qqft5" Oct 30 20:55:53.507129 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:53.507111 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8632241a-3073-4493-9fab-298607317a96-credential-socket\") pod \"e2e-ait-a8daf029-openshift-default-79cb595b95-qqft5\" (UID: \"8632241a-3073-4493-9fab-298607317a96\") " pod="openshift-ingress/e2e-ait-a8daf029-openshift-default-79cb595b95-qqft5" Oct 30 20:55:53.507291 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:53.507277 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8632241a-3073-4493-9fab-298607317a96-istio-data\") pod \"e2e-ait-a8daf029-openshift-default-79cb595b95-qqft5\" (UID: \"8632241a-3073-4493-9fab-298607317a96\") " pod="openshift-ingress/e2e-ait-a8daf029-openshift-default-79cb595b95-qqft5" Oct 30 20:55:53.507547 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:53.507529 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8632241a-3073-4493-9fab-298607317a96-istio-token\") pod \"e2e-ait-a8daf029-openshift-default-79cb595b95-qqft5\" (UID: \"8632241a-3073-4493-9fab-298607317a96\") " pod="openshift-ingress/e2e-ait-a8daf029-openshift-default-79cb595b95-qqft5" Oct 30 20:55:53.507749 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:53.507728 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8632241a-3073-4493-9fab-298607317a96-workload-certs\") pod \"e2e-ait-a8daf029-openshift-default-79cb595b95-qqft5\" (UID: \"8632241a-3073-4493-9fab-298607317a96\") " pod="openshift-ingress/e2e-ait-a8daf029-openshift-default-79cb595b95-qqft5" Oct 30 20:55:53.507904 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:53.507888 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8632241a-3073-4493-9fab-298607317a96-workload-socket\") pod \"e2e-ait-a8daf029-openshift-default-79cb595b95-qqft5\" (UID: \"8632241a-3073-4493-9fab-298607317a96\") " pod="openshift-ingress/e2e-ait-a8daf029-openshift-default-79cb595b95-qqft5" Oct 30 20:55:53.508036 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:53.508018 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jjxzb\" (UniqueName: \"kubernetes.io/projected/8632241a-3073-4493-9fab-298607317a96-kube-api-access-jjxzb\") pod \"e2e-ait-a8daf029-openshift-default-79cb595b95-qqft5\" (UID: \"8632241a-3073-4493-9fab-298607317a96\") " pod="openshift-ingress/e2e-ait-a8daf029-openshift-default-79cb595b95-qqft5" Oct 30 20:55:53.508165 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:53.508149 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8632241a-3073-4493-9fab-298607317a96-istiod-ca-cert\") pod \"e2e-ait-a8daf029-openshift-default-79cb595b95-qqft5\" (UID: \"8632241a-3073-4493-9fab-298607317a96\") " pod="openshift-ingress/e2e-ait-a8daf029-openshift-default-79cb595b95-qqft5" Oct 30 20:55:53.517763 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:53.509145 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8632241a-3073-4493-9fab-298607317a96-istiod-ca-cert\") pod \"e2e-ait-a8daf029-openshift-default-79cb595b95-qqft5\" (UID: \"8632241a-3073-4493-9fab-298607317a96\") " pod="openshift-ingress/e2e-ait-a8daf029-openshift-default-79cb595b95-qqft5" Oct 30 20:55:53.527910 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:53.527881 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8632241a-3073-4493-9fab-298607317a96-workload-socket\") pod \"e2e-ait-a8daf029-openshift-default-79cb595b95-qqft5\" (UID: \"8632241a-3073-4493-9fab-298607317a96\") " pod="openshift-ingress/e2e-ait-a8daf029-openshift-default-79cb595b95-qqft5" Oct 30 20:55:53.529093 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:53.528361 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8632241a-3073-4493-9fab-298607317a96-workload-certs\") pod \"e2e-ait-a8daf029-openshift-default-79cb595b95-qqft5\" (UID: \"8632241a-3073-4493-9fab-298607317a96\") " pod="openshift-ingress/e2e-ait-a8daf029-openshift-default-79cb595b95-qqft5" Oct 30 20:55:53.529285 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:53.528819 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8632241a-3073-4493-9fab-298607317a96-credential-socket\") pod \"e2e-ait-a8daf029-openshift-default-79cb595b95-qqft5\" (UID: \"8632241a-3073-4493-9fab-298607317a96\") " pod="openshift-ingress/e2e-ait-a8daf029-openshift-default-79cb595b95-qqft5" Oct 30 20:55:53.529391 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:53.529013 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8632241a-3073-4493-9fab-298607317a96-istio-data\") pod \"e2e-ait-a8daf029-openshift-default-79cb595b95-qqft5\" (UID: \"8632241a-3073-4493-9fab-298607317a96\") " pod="openshift-ingress/e2e-ait-a8daf029-openshift-default-79cb595b95-qqft5" Oct 30 20:55:53.540901 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:53.540874 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8632241a-3073-4493-9fab-298607317a96-istio-envoy\") pod \"e2e-ait-a8daf029-openshift-default-79cb595b95-qqft5\" (UID: \"8632241a-3073-4493-9fab-298607317a96\") " pod="openshift-ingress/e2e-ait-a8daf029-openshift-default-79cb595b95-qqft5" Oct 30 20:55:53.577529 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:53.575795 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8632241a-3073-4493-9fab-298607317a96-istio-podinfo\") pod \"e2e-ait-a8daf029-openshift-default-79cb595b95-qqft5\" (UID: \"8632241a-3073-4493-9fab-298607317a96\") " pod="openshift-ingress/e2e-ait-a8daf029-openshift-default-79cb595b95-qqft5" Oct 30 20:55:53.605572 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:53.601085 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8632241a-3073-4493-9fab-298607317a96-istio-token\") pod \"e2e-ait-a8daf029-openshift-default-79cb595b95-qqft5\" (UID: \"8632241a-3073-4493-9fab-298607317a96\") " pod="openshift-ingress/e2e-ait-a8daf029-openshift-default-79cb595b95-qqft5" Oct 30 20:55:53.629684 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:53.629647 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jjxzb\" (UniqueName: \"kubernetes.io/projected/8632241a-3073-4493-9fab-298607317a96-kube-api-access-jjxzb\") pod \"e2e-ait-a8daf029-openshift-default-79cb595b95-qqft5\" (UID: \"8632241a-3073-4493-9fab-298607317a96\") " pod="openshift-ingress/e2e-ait-a8daf029-openshift-default-79cb595b95-qqft5" Oct 30 20:55:53.666458 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:53.666417 4611 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-a8daf029-openshift-default-79cb595b95-qqft5" Oct 30 20:55:53.679590 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:53.679553 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:55:53.836854 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:53.836816 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:55:53.981064 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:55:53.981036 4611 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8632241a_3073_4493_9fab_298607317a96.slice/crio-de5babc5927a5f7c9e5f48ffd1bd60a7388060b6dba267fecd48589c0bf637fa WatchSource:0}: Error finding container de5babc5927a5f7c9e5f48ffd1bd60a7388060b6dba267fecd48589c0bf637fa: Status 404 returned error can't find the container with id de5babc5927a5f7c9e5f48ffd1bd60a7388060b6dba267fecd48589c0bf637fa Oct 30 20:55:53.982518 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:53.982477 4611 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-a8daf029-openshift-default-79cb595b95-qqft5"] Oct 30 20:55:53.984268 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:53.984235 4611 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:55:53.984365 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:53.984316 4611 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:55:53.984365 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:53.984351 4611 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:55:54.009086 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:54.009055 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:55:54.163216 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:54.163182 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:55:54.366030 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:54.365997 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a8daf029-openshift-default-79cb595b95-qqft5" event={"ID":"8632241a-3073-4493-9fab-298607317a96","Type":"ContainerStarted","Data":"e00691c8cdcf7470c3b6bfae79ebf61a61c888085aad5613b0578654e7a980d0"} Oct 30 20:55:54.366193 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:54.366076 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a8daf029-openshift-default-79cb595b95-qqft5" event={"ID":"8632241a-3073-4493-9fab-298607317a96","Type":"ContainerStarted","Data":"de5babc5927a5f7c9e5f48ffd1bd60a7388060b6dba267fecd48589c0bf637fa"} Oct 30 20:55:54.394710 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:54.394655 4611 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-a8daf029-openshift-default-79cb595b95-qqft5" podStartSLOduration=1.39464151 podStartE2EDuration="1.39464151s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:55:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:55:54.392008758 +0000 UTC m=+1008.926035399" watchObservedRunningTime="2026-10-30 20:55:54.39464151 +0000 UTC m=+1008.928668125" Oct 30 20:55:54.460070 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:54.460027 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:55:54.666975 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:54.666883 4611 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-a8daf029-openshift-default-79cb595b95-qqft5" Oct 30 20:55:55.208977 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:55.208946 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:55:55.360534 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:55.360450 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:55:55.664984 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:55.664945 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:55:55.682193 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:55.682165 4611 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-a8daf029-openshift-default-79cb595b95-qqft5" Oct 30 20:55:56.074586 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:56.073984 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:55:56.230184 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:56.230141 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:55:56.448254 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:56.448227 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-a8daf029-openshift-default-79cb595b95-qqft5" Oct 30 20:55:56.449101 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:56.449074 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-a8daf029-openshift-default-79cb595b95-qqft5" Oct 30 20:55:56.693836 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:56.693798 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:55:57.059598 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:57.059567 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:55:57.205705 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:57.205652 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:55:57.706149 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:57.706113 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:55:58.501468 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:58.501422 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:55:59.116327 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:59.116290 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:55:59.813407 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:55:59.813372 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:56:00.554477 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:00.554438 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:56:00.696886 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:00.696828 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:56:00.835129 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:00.835017 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:56:01.128089 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:01.128052 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:56:01.456686 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:01.456585 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:56:02.300897 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:02.300856 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:56:03.091833 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:03.091801 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:56:03.247025 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:03.246975 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:56:03.388102 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:03.388071 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:56:03.944779 ip-10-0-133-192 kubenswrapper[4611]: E1030 20:56:03.944722 4611 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hwzc6" containerName="extractor" Oct 30 20:56:03.965920 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:03.965884 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:56:04.610591 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:04.610498 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:56:05.242451 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:05.240018 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:56:05.909839 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:05.909800 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:56:06.600502 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:06.599873 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:56:07.206706 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:07.206663 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:56:07.812717 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:07.812680 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:56:08.415292 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:08.415092 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:56:09.087701 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:09.087641 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:56:09.717109 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:09.717046 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:56:10.430547 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:10.430434 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:56:10.761034 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:10.760951 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:56:10.935035 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:10.935001 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:56:11.077457 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:11.077370 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:56:11.220654 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:11.220621 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:56:11.653392 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:11.653351 4611 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-2e92a84c-openshift-default-8598669b69-kq4pw"] Oct 30 20:56:11.659133 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:11.659111 4611 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-2e92a84c-openshift-default-8598669b69-kq4pw" Oct 30 20:56:11.722792 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:11.722766 4611 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-2e92a84c-openshift-default-8598669b69-kq4pw"] Oct 30 20:56:11.758341 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:11.758307 4611 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-2e92a84c-openshift-default-dockercfg-27nq9\"" Oct 30 20:56:11.778317 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:11.777576 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e828845-f5af-44d1-8db6-5737ffb966f5-workload-certs\") pod \"e2e-ait-2e92a84c-openshift-default-8598669b69-kq4pw\" (UID: \"2e828845-f5af-44d1-8db6-5737ffb966f5\") " pod="openshift-ingress/e2e-ait-2e92a84c-openshift-default-8598669b69-kq4pw" Oct 30 20:56:11.779556 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:11.779435 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e828845-f5af-44d1-8db6-5737ffb966f5-istio-envoy\") pod \"e2e-ait-2e92a84c-openshift-default-8598669b69-kq4pw\" (UID: \"2e828845-f5af-44d1-8db6-5737ffb966f5\") " pod="openshift-ingress/e2e-ait-2e92a84c-openshift-default-8598669b69-kq4pw" Oct 30 20:56:11.779841 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:11.779820 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e828845-f5af-44d1-8db6-5737ffb966f5-istio-token\") pod \"e2e-ait-2e92a84c-openshift-default-8598669b69-kq4pw\" (UID: \"2e828845-f5af-44d1-8db6-5737ffb966f5\") " pod="openshift-ingress/e2e-ait-2e92a84c-openshift-default-8598669b69-kq4pw" Oct 30 20:56:11.780081 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:11.780042 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e828845-f5af-44d1-8db6-5737ffb966f5-istio-data\") pod \"e2e-ait-2e92a84c-openshift-default-8598669b69-kq4pw\" (UID: \"2e828845-f5af-44d1-8db6-5737ffb966f5\") " pod="openshift-ingress/e2e-ait-2e92a84c-openshift-default-8598669b69-kq4pw" Oct 30 20:56:11.780287 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:11.780269 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e828845-f5af-44d1-8db6-5737ffb966f5-credential-socket\") pod \"e2e-ait-2e92a84c-openshift-default-8598669b69-kq4pw\" (UID: \"2e828845-f5af-44d1-8db6-5737ffb966f5\") " pod="openshift-ingress/e2e-ait-2e92a84c-openshift-default-8598669b69-kq4pw" Oct 30 20:56:11.780528 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:11.780496 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e828845-f5af-44d1-8db6-5737ffb966f5-workload-socket\") pod \"e2e-ait-2e92a84c-openshift-default-8598669b69-kq4pw\" (UID: \"2e828845-f5af-44d1-8db6-5737ffb966f5\") " pod="openshift-ingress/e2e-ait-2e92a84c-openshift-default-8598669b69-kq4pw" Oct 30 20:56:11.780689 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:11.780668 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e828845-f5af-44d1-8db6-5737ffb966f5-istiod-ca-cert\") pod \"e2e-ait-2e92a84c-openshift-default-8598669b69-kq4pw\" (UID: \"2e828845-f5af-44d1-8db6-5737ffb966f5\") " pod="openshift-ingress/e2e-ait-2e92a84c-openshift-default-8598669b69-kq4pw" Oct 30 20:56:11.781253 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:11.781231 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e828845-f5af-44d1-8db6-5737ffb966f5-istio-podinfo\") pod \"e2e-ait-2e92a84c-openshift-default-8598669b69-kq4pw\" (UID: \"2e828845-f5af-44d1-8db6-5737ffb966f5\") " pod="openshift-ingress/e2e-ait-2e92a84c-openshift-default-8598669b69-kq4pw" Oct 30 20:56:11.782165 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:11.782143 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wdwxc\" (UniqueName: \"kubernetes.io/projected/2e828845-f5af-44d1-8db6-5737ffb966f5-kube-api-access-wdwxc\") pod \"e2e-ait-2e92a84c-openshift-default-8598669b69-kq4pw\" (UID: \"2e828845-f5af-44d1-8db6-5737ffb966f5\") " pod="openshift-ingress/e2e-ait-2e92a84c-openshift-default-8598669b69-kq4pw" Oct 30 20:56:11.842017 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:11.841536 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:56:11.884634 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:11.884596 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e828845-f5af-44d1-8db6-5737ffb966f5-istio-data\") pod \"e2e-ait-2e92a84c-openshift-default-8598669b69-kq4pw\" (UID: \"2e828845-f5af-44d1-8db6-5737ffb966f5\") " pod="openshift-ingress/e2e-ait-2e92a84c-openshift-default-8598669b69-kq4pw" Oct 30 20:56:11.885477 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:11.885452 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e828845-f5af-44d1-8db6-5737ffb966f5-credential-socket\") pod \"e2e-ait-2e92a84c-openshift-default-8598669b69-kq4pw\" (UID: \"2e828845-f5af-44d1-8db6-5737ffb966f5\") " pod="openshift-ingress/e2e-ait-2e92a84c-openshift-default-8598669b69-kq4pw" Oct 30 20:56:11.886025 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:11.886005 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e828845-f5af-44d1-8db6-5737ffb966f5-workload-socket\") pod \"e2e-ait-2e92a84c-openshift-default-8598669b69-kq4pw\" (UID: \"2e828845-f5af-44d1-8db6-5737ffb966f5\") " pod="openshift-ingress/e2e-ait-2e92a84c-openshift-default-8598669b69-kq4pw" Oct 30 20:56:11.886641 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:11.886623 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e828845-f5af-44d1-8db6-5737ffb966f5-istiod-ca-cert\") pod \"e2e-ait-2e92a84c-openshift-default-8598669b69-kq4pw\" (UID: \"2e828845-f5af-44d1-8db6-5737ffb966f5\") " pod="openshift-ingress/e2e-ait-2e92a84c-openshift-default-8598669b69-kq4pw" Oct 30 20:56:11.887555 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:11.887535 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e828845-f5af-44d1-8db6-5737ffb966f5-istio-podinfo\") pod \"e2e-ait-2e92a84c-openshift-default-8598669b69-kq4pw\" (UID: \"2e828845-f5af-44d1-8db6-5737ffb966f5\") " pod="openshift-ingress/e2e-ait-2e92a84c-openshift-default-8598669b69-kq4pw" Oct 30 20:56:11.888252 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:11.888231 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wdwxc\" (UniqueName: \"kubernetes.io/projected/2e828845-f5af-44d1-8db6-5737ffb966f5-kube-api-access-wdwxc\") pod \"e2e-ait-2e92a84c-openshift-default-8598669b69-kq4pw\" (UID: \"2e828845-f5af-44d1-8db6-5737ffb966f5\") " pod="openshift-ingress/e2e-ait-2e92a84c-openshift-default-8598669b69-kq4pw" Oct 30 20:56:11.888501 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:11.888477 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e828845-f5af-44d1-8db6-5737ffb966f5-workload-certs\") pod \"e2e-ait-2e92a84c-openshift-default-8598669b69-kq4pw\" (UID: \"2e828845-f5af-44d1-8db6-5737ffb966f5\") " pod="openshift-ingress/e2e-ait-2e92a84c-openshift-default-8598669b69-kq4pw" Oct 30 20:56:11.888642 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:11.888545 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e828845-f5af-44d1-8db6-5737ffb966f5-istio-envoy\") pod \"e2e-ait-2e92a84c-openshift-default-8598669b69-kq4pw\" (UID: \"2e828845-f5af-44d1-8db6-5737ffb966f5\") " pod="openshift-ingress/e2e-ait-2e92a84c-openshift-default-8598669b69-kq4pw" Oct 30 20:56:11.888642 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:11.888604 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e828845-f5af-44d1-8db6-5737ffb966f5-istio-token\") pod \"e2e-ait-2e92a84c-openshift-default-8598669b69-kq4pw\" (UID: \"2e828845-f5af-44d1-8db6-5737ffb966f5\") " pod="openshift-ingress/e2e-ait-2e92a84c-openshift-default-8598669b69-kq4pw" Oct 30 20:56:11.889007 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:11.887388 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e828845-f5af-44d1-8db6-5737ffb966f5-istiod-ca-cert\") pod \"e2e-ait-2e92a84c-openshift-default-8598669b69-kq4pw\" (UID: \"2e828845-f5af-44d1-8db6-5737ffb966f5\") " pod="openshift-ingress/e2e-ait-2e92a84c-openshift-default-8598669b69-kq4pw" Oct 30 20:56:11.889209 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:11.889186 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e828845-f5af-44d1-8db6-5737ffb966f5-workload-certs\") pod \"e2e-ait-2e92a84c-openshift-default-8598669b69-kq4pw\" (UID: \"2e828845-f5af-44d1-8db6-5737ffb966f5\") " pod="openshift-ingress/e2e-ait-2e92a84c-openshift-default-8598669b69-kq4pw" Oct 30 20:56:11.889305 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:11.886566 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e828845-f5af-44d1-8db6-5737ffb966f5-workload-socket\") pod \"e2e-ait-2e92a84c-openshift-default-8598669b69-kq4pw\" (UID: \"2e828845-f5af-44d1-8db6-5737ffb966f5\") " pod="openshift-ingress/e2e-ait-2e92a84c-openshift-default-8598669b69-kq4pw" Oct 30 20:56:11.889305 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:11.885880 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e828845-f5af-44d1-8db6-5737ffb966f5-credential-socket\") pod \"e2e-ait-2e92a84c-openshift-default-8598669b69-kq4pw\" (UID: \"2e828845-f5af-44d1-8db6-5737ffb966f5\") " pod="openshift-ingress/e2e-ait-2e92a84c-openshift-default-8598669b69-kq4pw" Oct 30 20:56:11.889305 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:11.885346 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e828845-f5af-44d1-8db6-5737ffb966f5-istio-data\") pod \"e2e-ait-2e92a84c-openshift-default-8598669b69-kq4pw\" (UID: \"2e828845-f5af-44d1-8db6-5737ffb966f5\") " pod="openshift-ingress/e2e-ait-2e92a84c-openshift-default-8598669b69-kq4pw" Oct 30 20:56:11.894477 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:11.894447 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e828845-f5af-44d1-8db6-5737ffb966f5-istio-envoy\") pod \"e2e-ait-2e92a84c-openshift-default-8598669b69-kq4pw\" (UID: \"2e828845-f5af-44d1-8db6-5737ffb966f5\") " pod="openshift-ingress/e2e-ait-2e92a84c-openshift-default-8598669b69-kq4pw" Oct 30 20:56:11.895479 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:11.895438 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e828845-f5af-44d1-8db6-5737ffb966f5-istio-podinfo\") pod \"e2e-ait-2e92a84c-openshift-default-8598669b69-kq4pw\" (UID: \"2e828845-f5af-44d1-8db6-5737ffb966f5\") " pod="openshift-ingress/e2e-ait-2e92a84c-openshift-default-8598669b69-kq4pw" Oct 30 20:56:11.909331 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:11.908255 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wdwxc\" (UniqueName: \"kubernetes.io/projected/2e828845-f5af-44d1-8db6-5737ffb966f5-kube-api-access-wdwxc\") pod \"e2e-ait-2e92a84c-openshift-default-8598669b69-kq4pw\" (UID: \"2e828845-f5af-44d1-8db6-5737ffb966f5\") " pod="openshift-ingress/e2e-ait-2e92a84c-openshift-default-8598669b69-kq4pw" Oct 30 20:56:11.909331 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:11.908994 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e828845-f5af-44d1-8db6-5737ffb966f5-istio-token\") pod \"e2e-ait-2e92a84c-openshift-default-8598669b69-kq4pw\" (UID: \"2e828845-f5af-44d1-8db6-5737ffb966f5\") " pod="openshift-ingress/e2e-ait-2e92a84c-openshift-default-8598669b69-kq4pw" Oct 30 20:56:11.976332 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:11.973795 4611 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-2e92a84c-openshift-default-8598669b69-kq4pw" Oct 30 20:56:12.012181 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:12.009211 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:56:12.212905 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:12.212789 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:56:12.335060 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:12.332097 4611 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-2e92a84c-openshift-default-8598669b69-kq4pw"] Oct 30 20:56:12.355431 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:56:12.355403 4611 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e828845_f5af_44d1_8db6_5737ffb966f5.slice/crio-5085471df97d3ccd0319d629d6f2ed1ce61fc5c3cc1a24e4014e52feee40681d WatchSource:0}: Error finding container 5085471df97d3ccd0319d629d6f2ed1ce61fc5c3cc1a24e4014e52feee40681d: Status 404 returned error can't find the container with id 5085471df97d3ccd0319d629d6f2ed1ce61fc5c3cc1a24e4014e52feee40681d Oct 30 20:56:12.359465 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:12.359425 4611 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:56:12.359627 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:12.359545 4611 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:56:12.359627 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:12.359592 4611 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:56:12.389161 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:12.389128 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:56:12.567259 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:12.567169 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:56:12.673599 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:12.673546 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-2e92a84c-openshift-default-8598669b69-kq4pw" event={"ID":"2e828845-f5af-44d1-8db6-5737ffb966f5","Type":"ContainerStarted","Data":"a95bd2542908f91dbbcbadca062ba8c97680b1d829ad6e7187c877efeff2b071"} Oct 30 20:56:12.673988 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:12.673945 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-2e92a84c-openshift-default-8598669b69-kq4pw" event={"ID":"2e828845-f5af-44d1-8db6-5737ffb966f5","Type":"ContainerStarted","Data":"5085471df97d3ccd0319d629d6f2ed1ce61fc5c3cc1a24e4014e52feee40681d"} Oct 30 20:56:12.707337 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:12.707278 4611 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-2e92a84c-openshift-default-8598669b69-kq4pw" podStartSLOduration=1.707259257 podStartE2EDuration="1.707259257s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:56:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:56:12.698616751 +0000 UTC m=+1027.232643367" watchObservedRunningTime="2026-10-30 20:56:12.707259257 +0000 UTC m=+1027.241285879" Oct 30 20:56:12.974932 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:12.974902 4611 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-2e92a84c-openshift-default-8598669b69-kq4pw" Oct 30 20:56:13.401084 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:13.401041 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:56:13.981981 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:13.981947 4611 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-2e92a84c-openshift-default-8598669b69-kq4pw" Oct 30 20:56:14.774775 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:14.774746 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-2e92a84c-openshift-default-8598669b69-kq4pw" Oct 30 20:56:14.783353 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:14.783314 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-2e92a84c-openshift-default-8598669b69-kq4pw" Oct 30 20:56:14.813978 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:14.813921 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:56:14.995892 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:14.993357 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:56:15.141746 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:15.125807 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad7ff0/premium-w0-ad7ff0-kserve-5485d4b948-bpmgf" podUID="53cbd692-e9f8-46ba-8fde-3e5215a38904" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:56:15.351531 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:15.351375 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cb60a2/distinct-w3-cb60a2-kserve-5bb76dc74-86fds" podUID="52e301d9-f378-4e54-9515-01a6680d9fd7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:56:15.380533 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:15.355112 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f2ed1e/distinct-w4-f2ed1e-kserve-57cff9dcf8-8d5zl" podUID="653e3ec7-2147-4813-8946-1f25cfa425f6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:56:15.380533 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:15.357323 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-14cca2/premium-w5-14cca2-kserve-6878777857-wpjtj" podUID="b99eb76b-d1c6-49e9-9809-c351c0f5a4de" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded" Oct 30 20:56:15.381066 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:15.381024 4611 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-14cca2/premium-w5-14cca2-kserve-6878777857-wpjtj" podUID="b99eb76b-d1c6-49e9-9809-c351c0f5a4de" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:56:15.448589 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:15.431811 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-14cca2/distinct-w5-14cca2-kserve-587f6996b8-sgf6h" podUID="02b652c3-0724-4a50-9e95-2a9890cc97c4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:56:15.448589 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:15.445598 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cb60a2/simulator-w3-cb60a2-kserve-6fd74b9d8b-vqfzf" podUID="711b167d-4e66-40d6-be75-33ede11f42c5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:56:15.584534 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:15.582816 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b8aee1/distinct-w1-b8aee1-kserve-5b857c55cd-x5b7b" podUID="1599e588-def8-42e1-b81d-ad0aaf119818" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:56:15.762590 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:15.759370 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b8aee1/premium-w1-b8aee1-kserve-786d575599-htc8d" podUID="77368afe-edc8-4fa5-8010-d321965fb38e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded" Oct 30 20:56:15.851123 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:15.850824 4611 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-cb60a2/distinct-w3-cb60a2-kserve-5bb76dc74-86fds" podUID="52e301d9-f378-4e54-9515-01a6680d9fd7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:56:15.906556 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:15.878224 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db534b/distinct-2-w2-db534b-kserve-8594c876-g92mh" podUID="955316d6-a9fd-4b75-a8ab-bbf382b776e2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:56:16.119259 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:16.119208 4611 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-3d909d/distinct-2-w6-3d909d-kserve-59c5d5c595-cjg2m" podUID="e2b4c221-ee94-4131-abd8-2313647c641e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/health\": context deadline exceeded" Oct 30 20:56:16.259534 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:16.122038 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b8aee1/distinct-2-w1-b8aee1-kserve-6bcdd59797-cx6hr" podUID="286eab7a-81a7-4768-a7f6-be6b3720028a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": context deadline exceeded" Oct 30 20:56:16.260909 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:16.130945 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad7ff0/premium-w0-ad7ff0-kserve-5485d4b948-bpmgf" podUID="53cbd692-e9f8-46ba-8fde-3e5215a38904" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:56:16.291962 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:16.130978 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-3d909d/premium-w6-3d909d-kserve-765648946f-2qvd2" podUID="fca5d697-109d-4e90-ab41-c40ad31af874" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded" Oct 30 20:56:16.313534 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:16.130998 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-3d909d/distinct-2-w6-3d909d-kserve-59c5d5c595-cjg2m" podUID="e2b4c221-ee94-4131-abd8-2313647c641e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": context deadline exceeded" Oct 30 20:56:16.313534 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:16.298892 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ad7ff0/premium-w0-ad7ff0-kserve-5485d4b948-bpmgf" Oct 30 20:56:16.361859 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:16.357647 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f2ed1e/distinct-w4-f2ed1e-kserve-57cff9dcf8-8d5zl" podUID="653e3ec7-2147-4813-8946-1f25cfa425f6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded" Oct 30 20:56:16.361859 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:16.358021 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cb60a2/distinct-w3-cb60a2-kserve-5bb76dc74-86fds" podUID="52e301d9-f378-4e54-9515-01a6680d9fd7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded" Oct 30 20:56:16.361859 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:16.358337 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-f2ed1e/distinct-w4-f2ed1e-kserve-57cff9dcf8-8d5zl" Oct 30 20:56:16.361859 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:16.358395 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-cb60a2/distinct-w3-cb60a2-kserve-5bb76dc74-86fds" Oct 30 20:56:16.361859 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:16.357602 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-14cca2/premium-w5-14cca2-kserve-6878777857-wpjtj" podUID="b99eb76b-d1c6-49e9-9809-c351c0f5a4de" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:56:16.361859 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:16.360715 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-14cca2/premium-w5-14cca2-kserve-6878777857-wpjtj" Oct 30 20:56:16.452386 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:16.452288 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cb60a2/simulator-w3-cb60a2-kserve-6fd74b9d8b-vqfzf" podUID="711b167d-4e66-40d6-be75-33ede11f42c5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:56:16.472535 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:16.460403 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-cb60a2/simulator-w3-cb60a2-kserve-6fd74b9d8b-vqfzf" Oct 30 20:56:16.472535 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:16.460818 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-14cca2/distinct-w5-14cca2-kserve-587f6996b8-sgf6h" podUID="02b652c3-0724-4a50-9e95-2a9890cc97c4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded" Oct 30 20:56:16.472535 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:16.461172 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-14cca2/distinct-w5-14cca2-kserve-587f6996b8-sgf6h" Oct 30 20:56:16.694312 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:16.694267 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b8aee1/distinct-w1-b8aee1-kserve-5b857c55cd-x5b7b" podUID="1599e588-def8-42e1-b81d-ad0aaf119818" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:56:16.755585 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:16.736071 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b8aee1/distinct-w1-b8aee1-kserve-5b857c55cd-x5b7b" Oct 30 20:56:16.986533 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:16.778895 4611 prober.go:122] "Probe failed" probeType="Liveness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-smdl6" podUID="31d263b9-b3b3-4db4-9026-da84d454dc34" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.133.0.22:6080/livez\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:56:16.986533 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:16.823164 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b8aee1/premium-w1-b8aee1-kserve-786d575599-htc8d" podUID="77368afe-edc8-4fa5-8010-d321965fb38e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded" Oct 30 20:56:16.986533 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:16.959053 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b8aee1/premium-w1-b8aee1-kserve-786d575599-htc8d" Oct 30 20:56:16.986533 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:16.926602 4611 patch_prober.go:29] interesting pod/e2e-worker-w6-3d909d-openshift-default-77787695b7-ztxdx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:56:16.986533 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:16.959300 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-ztxdx" podUID="5a93b27f-f269-471c-8be2-523f373bca08" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:56:16.986533 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:16.915005 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db534b/distinct-2-w2-db534b-kserve-8594c876-g92mh" podUID="955316d6-a9fd-4b75-a8ab-bbf382b776e2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": context deadline exceeded" Oct 30 20:56:16.986533 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:16.959793 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-db534b/distinct-2-w2-db534b-kserve-8594c876-g92mh" Oct 30 20:56:17.005479 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:17.005438 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-smdl6" podUID="31d263b9-b3b3-4db4-9026-da84d454dc34" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.133.0.22:6080/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:56:17.358871 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:17.358827 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-3d909d/premium-w6-3d909d-kserve-765648946f-2qvd2" podUID="fca5d697-109d-4e90-ab41-c40ad31af874" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded" Oct 30 20:56:17.629652 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:17.617110 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad7ff0/premium-w0-ad7ff0-kserve-5485d4b948-bpmgf" podUID="53cbd692-e9f8-46ba-8fde-3e5215a38904" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded" Oct 30 20:56:18.119857 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:18.119813 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b8aee1/premium-w1-b8aee1-kserve-786d575599-htc8d" podUID="77368afe-edc8-4fa5-8010-d321965fb38e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded" Oct 30 20:56:18.128051 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:18.128008 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db534b/distinct-2-w2-db534b-kserve-8594c876-g92mh" podUID="955316d6-a9fd-4b75-a8ab-bbf382b776e2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:56:18.137357 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:17.468083 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cb60a2/distinct-w3-cb60a2-kserve-5bb76dc74-86fds" podUID="52e301d9-f378-4e54-9515-01a6680d9fd7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:56:18.150532 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:17.498239 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-3d909d/distinct-2-w6-3d909d-kserve-59c5d5c595-cjg2m" podUID="e2b4c221-ee94-4131-abd8-2313647c641e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": context deadline exceeded" Oct 30 20:56:18.150777 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:17.513367 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b8aee1/distinct-2-w1-b8aee1-kserve-6bcdd59797-cx6hr" podUID="286eab7a-81a7-4768-a7f6-be6b3720028a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:56:18.169369 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:17.552703 4611 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-ad7ff0/premium-w0-ad7ff0-kserve-5485d4b948-bpmgf" podUID="53cbd692-e9f8-46ba-8fde-3e5215a38904" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/health\": context deadline exceeded" Oct 30 20:56:18.169638 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:17.575189 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cb60a2/simulator-w3-cb60a2-kserve-6fd74b9d8b-vqfzf" podUID="711b167d-4e66-40d6-be75-33ede11f42c5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:56:18.170916 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:17.554748 4611 patch_prober.go:29] interesting pod/e2e-worker-w6-3d909d-openshift-default-77787695b7-wf8qg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:56:18.171111 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:18.171081 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-wf8qg" podUID="c468f3c6-7d73-4fd1-ae34-1212e4d1b387" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:56:18.368531 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:17.409880 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-14cca2/premium-w5-14cca2-kserve-6878777857-wpjtj" podUID="b99eb76b-d1c6-49e9-9809-c351c0f5a4de" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:56:18.623590 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:17.715716 4611 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-db534b/distinct-2-w2-db534b-kserve-8594c876-g92mh" podUID="955316d6-a9fd-4b75-a8ab-bbf382b776e2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/health\": context deadline exceeded" Oct 30 20:56:18.623590 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:17.575470 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-14cca2/distinct-w5-14cca2-kserve-587f6996b8-sgf6h" podUID="02b652c3-0724-4a50-9e95-2a9890cc97c4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded" Oct 30 20:56:18.623590 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:17.646113 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-3d909d/premium-w6-3d909d-kserve-765648946f-2qvd2" Oct 30 20:56:18.623590 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:17.555042 4611 patch_prober.go:29] interesting pod/e2e-worker-w6-3d909d-openshift-default-77787695b7-trngw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:56:18.623590 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:18.622052 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-trngw" podUID="5ba753c3-5979-4263-8b90-2741747c8592" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:56:18.623590 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:17.467937 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f2ed1e/distinct-w4-f2ed1e-kserve-57cff9dcf8-8d5zl" podUID="653e3ec7-2147-4813-8946-1f25cfa425f6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded" Oct 30 20:56:18.623590 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:18.276734 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b8aee1/distinct-w1-b8aee1-kserve-5b857c55cd-x5b7b" podUID="1599e588-def8-42e1-b81d-ad0aaf119818" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": context deadline exceeded" Oct 30 20:56:18.623590 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:18.603663 4611 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-w44qs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:56:18.623590 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:18.622279 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-w44qs" podUID="6d6fd4b2-ad0a-4ce3-b5b3-966de34bd8b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:56:18.623590 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:18.461451 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:56:18.627562 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:18.612976 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-3d909d/premium-w6-3d909d-kserve-765648946f-2qvd2" podUID="fca5d697-109d-4e90-ab41-c40ad31af874" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded" Oct 30 20:56:19.116614 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:19.116065 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad7ff0/premium-w0-ad7ff0-kserve-5485d4b948-bpmgf" podUID="53cbd692-e9f8-46ba-8fde-3e5215a38904" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:56:19.132970 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:19.132935 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:56:19.168715 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:19.168672 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b8aee1/distinct-2-w1-b8aee1-kserve-6bcdd59797-cx6hr" podUID="286eab7a-81a7-4768-a7f6-be6b3720028a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:56:19.169284 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:19.169251 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b8aee1/premium-w1-b8aee1-kserve-786d575599-htc8d" podUID="77368afe-edc8-4fa5-8010-d321965fb38e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded" Oct 30 20:56:19.169740 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:19.169716 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db534b/distinct-2-w2-db534b-kserve-8594c876-g92mh" podUID="955316d6-a9fd-4b75-a8ab-bbf382b776e2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": context deadline exceeded" Oct 30 20:56:19.170101 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:19.170077 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cb60a2/distinct-w3-cb60a2-kserve-5bb76dc74-86fds" podUID="52e301d9-f378-4e54-9515-01a6680d9fd7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded" Oct 30 20:56:19.246312 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:19.187590 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-3d909d/distinct-2-w6-3d909d-kserve-59c5d5c595-cjg2m" podUID="e2b4c221-ee94-4131-abd8-2313647c641e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": context deadline exceeded" Oct 30 20:56:19.270879 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:19.188900 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cb60a2/simulator-w3-cb60a2-kserve-6fd74b9d8b-vqfzf" podUID="711b167d-4e66-40d6-be75-33ede11f42c5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:56:19.370736 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:19.370629 4611 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-3d909d/premium-w6-3d909d-kserve-765648946f-2qvd2" podUID="fca5d697-109d-4e90-ab41-c40ad31af874" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:56:19.518752 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:19.518709 4611 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-f2ed1e/distinct-w4-f2ed1e-kserve-57cff9dcf8-8d5zl" podUID="653e3ec7-2147-4813-8946-1f25cfa425f6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:56:19.630087 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:19.629984 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-14cca2/premium-w5-14cca2-kserve-6878777857-wpjtj" podUID="b99eb76b-d1c6-49e9-9809-c351c0f5a4de" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded" Oct 30 20:56:19.632144 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:19.632085 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b8aee1/distinct-w1-b8aee1-kserve-5b857c55cd-x5b7b" podUID="1599e588-def8-42e1-b81d-ad0aaf119818" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": context deadline exceeded" Oct 30 20:56:19.667239 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:19.661631 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-3d909d/premium-w6-3d909d-kserve-765648946f-2qvd2" podUID="fca5d697-109d-4e90-ab41-c40ad31af874" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:56:19.667239 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:19.661973 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-14cca2/distinct-w5-14cca2-kserve-587f6996b8-sgf6h" podUID="02b652c3-0724-4a50-9e95-2a9890cc97c4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:56:19.667239 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:19.662178 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f2ed1e/distinct-w4-f2ed1e-kserve-57cff9dcf8-8d5zl" podUID="653e3ec7-2147-4813-8946-1f25cfa425f6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded" Oct 30 20:56:19.667239 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:19.663019 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:56:19.762539 ip-10-0-133-192 kubenswrapper[4611]: E1030 20:56:19.752928 4611 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.131s" Oct 30 20:56:19.865767 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:19.865731 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-3d909d/distinct-2-w6-3d909d-kserve-59c5d5c595-cjg2m" Oct 30 20:56:19.869621 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:19.869593 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b8aee1/distinct-2-w1-b8aee1-kserve-6bcdd59797-cx6hr" Oct 30 20:56:20.124612 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:20.124571 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad7ff0/premium-w0-ad7ff0-kserve-5485d4b948-bpmgf" podUID="53cbd692-e9f8-46ba-8fde-3e5215a38904" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:56:20.181968 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:20.181917 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db534b/distinct-2-w2-db534b-kserve-8594c876-g92mh" podUID="955316d6-a9fd-4b75-a8ab-bbf382b776e2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:56:20.182902 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:20.182859 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b8aee1/distinct-2-w1-b8aee1-kserve-6bcdd59797-cx6hr" podUID="286eab7a-81a7-4768-a7f6-be6b3720028a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:56:20.212676 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:20.212634 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b8aee1/premium-w1-b8aee1-kserve-786d575599-htc8d" podUID="77368afe-edc8-4fa5-8010-d321965fb38e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded" Oct 30 20:56:20.293888 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:20.293844 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cb60a2/simulator-w3-cb60a2-kserve-6fd74b9d8b-vqfzf" podUID="711b167d-4e66-40d6-be75-33ede11f42c5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:56:20.319164 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:20.312047 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-3d909d/distinct-2-w6-3d909d-kserve-59c5d5c595-cjg2m" podUID="e2b4c221-ee94-4131-abd8-2313647c641e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:56:20.320524 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:20.312087 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cb60a2/distinct-w3-cb60a2-kserve-5bb76dc74-86fds" podUID="52e301d9-f378-4e54-9515-01a6680d9fd7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded" Oct 30 20:56:20.635724 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:20.635683 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b8aee1/distinct-w1-b8aee1-kserve-5b857c55cd-x5b7b" podUID="1599e588-def8-42e1-b81d-ad0aaf119818" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:56:20.640639 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:20.640585 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-14cca2/premium-w5-14cca2-kserve-6878777857-wpjtj" podUID="b99eb76b-d1c6-49e9-9809-c351c0f5a4de" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:56:20.713934 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:20.713885 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-3d909d/premium-w6-3d909d-kserve-765648946f-2qvd2" podUID="fca5d697-109d-4e90-ab41-c40ad31af874" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:56:20.737790 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:20.737731 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-14cca2/distinct-w5-14cca2-kserve-587f6996b8-sgf6h" podUID="02b652c3-0724-4a50-9e95-2a9890cc97c4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:56:20.739288 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:20.739256 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f2ed1e/distinct-w4-f2ed1e-kserve-57cff9dcf8-8d5zl" podUID="653e3ec7-2147-4813-8946-1f25cfa425f6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:56:21.179596 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:21.178793 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad7ff0/premium-w0-ad7ff0-kserve-5485d4b948-bpmgf" podUID="53cbd692-e9f8-46ba-8fde-3e5215a38904" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:56:21.179596 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:21.179159 4611 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-b8aee1/premium-w1-b8aee1-kserve-786d575599-htc8d" podUID="77368afe-edc8-4fa5-8010-d321965fb38e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:56:21.187620 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:21.187581 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db534b/distinct-2-w2-db534b-kserve-8594c876-g92mh" podUID="955316d6-a9fd-4b75-a8ab-bbf382b776e2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": context deadline exceeded" Oct 30 20:56:21.192814 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:21.191785 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b8aee1/distinct-2-w1-b8aee1-kserve-6bcdd59797-cx6hr" podUID="286eab7a-81a7-4768-a7f6-be6b3720028a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:56:21.228772 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:21.222674 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b8aee1/premium-w1-b8aee1-kserve-786d575599-htc8d" podUID="77368afe-edc8-4fa5-8010-d321965fb38e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded" Oct 30 20:56:21.327740 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:21.327685 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cb60a2/simulator-w3-cb60a2-kserve-6fd74b9d8b-vqfzf" podUID="711b167d-4e66-40d6-be75-33ede11f42c5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:56:21.328260 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:21.328217 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cb60a2/distinct-w3-cb60a2-kserve-5bb76dc74-86fds" podUID="52e301d9-f378-4e54-9515-01a6680d9fd7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded" Oct 30 20:56:21.328469 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:21.328443 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-3d909d/distinct-2-w6-3d909d-kserve-59c5d5c595-cjg2m" podUID="e2b4c221-ee94-4131-abd8-2313647c641e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": context deadline exceeded" Oct 30 20:56:21.670798 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:21.670740 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-14cca2/premium-w5-14cca2-kserve-6878777857-wpjtj" podUID="b99eb76b-d1c6-49e9-9809-c351c0f5a4de" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:56:21.712430 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:21.712383 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b8aee1/distinct-w1-b8aee1-kserve-5b857c55cd-x5b7b" podUID="1599e588-def8-42e1-b81d-ad0aaf119818" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:56:21.738307 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:21.738253 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-3d909d/premium-w6-3d909d-kserve-765648946f-2qvd2" podUID="fca5d697-109d-4e90-ab41-c40ad31af874" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded" Oct 30 20:56:21.744874 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:21.744847 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-14cca2/distinct-w5-14cca2-kserve-587f6996b8-sgf6h" podUID="02b652c3-0724-4a50-9e95-2a9890cc97c4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:56:21.804068 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:21.744791 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f2ed1e/distinct-w4-f2ed1e-kserve-57cff9dcf8-8d5zl" podUID="653e3ec7-2147-4813-8946-1f25cfa425f6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:56:22.003588 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:22.001345 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-smdl6" podUID="31d263b9-b3b3-4db4-9026-da84d454dc34" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.133.0.22:6080/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:56:22.065433 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:22.065399 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-3d909d/distinct-2-w6-3d909d-kserve-59c5d5c595-cjg2m" Oct 30 20:56:22.068746 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:22.068723 4611 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-2e92a84c-748fd59c8c-hhx6n"] Oct 30 20:56:22.141485 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:22.141453 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:56:22.177533 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:22.165954 4611 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-b8aee1/distinct-w1-b8aee1-kserve-5b857c55cd-x5b7b" podUID="1599e588-def8-42e1-b81d-ad0aaf119818" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:56:22.219540 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:22.216140 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad7ff0/premium-w0-ad7ff0-kserve-5485d4b948-bpmgf" podUID="53cbd692-e9f8-46ba-8fde-3e5215a38904" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded" Oct 30 20:56:22.219818 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:22.219786 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db534b/distinct-2-w2-db534b-kserve-8594c876-g92mh" podUID="955316d6-a9fd-4b75-a8ab-bbf382b776e2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:56:22.221004 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:22.219949 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b8aee1/distinct-2-w1-b8aee1-kserve-6bcdd59797-cx6hr" podUID="286eab7a-81a7-4768-a7f6-be6b3720028a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": context deadline exceeded" Oct 30 20:56:22.249540 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:22.245651 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b8aee1/premium-w1-b8aee1-kserve-786d575599-htc8d" podUID="77368afe-edc8-4fa5-8010-d321965fb38e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:56:22.358588 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:22.329406 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cb60a2/distinct-w3-cb60a2-kserve-5bb76dc74-86fds" podUID="52e301d9-f378-4e54-9515-01a6680d9fd7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:56:22.424853 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:22.375445 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cb60a2/simulator-w3-cb60a2-kserve-6fd74b9d8b-vqfzf" podUID="711b167d-4e66-40d6-be75-33ede11f42c5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded" Oct 30 20:56:22.673496 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:22.673388 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-14cca2/premium-w5-14cca2-kserve-6878777857-wpjtj" podUID="b99eb76b-d1c6-49e9-9809-c351c0f5a4de" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:56:22.761534 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:22.751831 4611 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-2e92a84c-87c49cccd-wx9fz"] Oct 30 20:56:22.886176 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:22.886133 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b8aee1/distinct-w1-b8aee1-kserve-5b857c55cd-x5b7b" podUID="1599e588-def8-42e1-b81d-ad0aaf119818" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": context deadline exceeded" Oct 30 20:56:22.888983 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:22.888949 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-3d909d/premium-w6-3d909d-kserve-765648946f-2qvd2" podUID="fca5d697-109d-4e90-ab41-c40ad31af874" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:56:22.892259 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:22.891075 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-14cca2/distinct-w5-14cca2-kserve-587f6996b8-sgf6h" podUID="02b652c3-0724-4a50-9e95-2a9890cc97c4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded" Oct 30 20:56:22.894134 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:22.894113 4611 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-2e92a84c-748fd59c8c-hhx6n" Oct 30 20:56:22.904889 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:22.897270 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f2ed1e/distinct-w4-f2ed1e-kserve-57cff9dcf8-8d5zl" podUID="653e3ec7-2147-4813-8946-1f25cfa425f6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:56:22.914683 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:22.914664 4611 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-2e92a84c\"" Oct 30 20:56:22.922346 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:22.919827 4611 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-2e92a84c-748fd59c8c-hhx6n"] Oct 30 20:56:22.922346 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:22.919874 4611 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-2e92a84c-87c49cccd-wx9fz"] Oct 30 20:56:22.922346 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:22.920193 4611 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-2e92a84c-87c49cccd-wx9fz" Oct 30 20:56:22.923204 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:22.923184 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:56:22.938184 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:22.938126 4611 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-2e92a84c-dockercfg-ffkmb\"" Oct 30 20:56:23.019623 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:22.999499 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f7bzj\" (UniqueName: \"kubernetes.io/projected/94a3b297-c47d-4b2a-8e78-697a605c5481-kube-api-access-f7bzj\") pod \"payload-processing-e2e-ait-2e92a84c-87c49cccd-wx9fz\" (UID: \"94a3b297-c47d-4b2a-8e78-697a605c5481\") " pod="openshift-ingress/payload-processing-e2e-ait-2e92a84c-87c49cccd-wx9fz" Oct 30 20:56:23.019981 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:23.019951 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b305f151-d272-46e8-af32-5f1274ffce1c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-2e92a84c-748fd59c8c-hhx6n\" (UID: \"b305f151-d272-46e8-af32-5f1274ffce1c\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-2e92a84c-748fd59c8c-hhx6n" Oct 30 20:56:23.020574 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:23.020543 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94a3b297-c47d-4b2a-8e78-697a605c5481-plugins-config-volume\") pod \"payload-processing-e2e-ait-2e92a84c-87c49cccd-wx9fz\" (UID: \"94a3b297-c47d-4b2a-8e78-697a605c5481\") " pod="openshift-ingress/payload-processing-e2e-ait-2e92a84c-87c49cccd-wx9fz" Oct 30 20:56:23.020763 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:23.020744 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c6cxp\" (UniqueName: \"kubernetes.io/projected/b305f151-d272-46e8-af32-5f1274ffce1c-kube-api-access-c6cxp\") pod \"payload-pre-processing-e2e-ait-2e92a84c-748fd59c8c-hhx6n\" (UID: \"b305f151-d272-46e8-af32-5f1274ffce1c\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-2e92a84c-748fd59c8c-hhx6n" Oct 30 20:56:23.131539 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:23.129371 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b305f151-d272-46e8-af32-5f1274ffce1c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-2e92a84c-748fd59c8c-hhx6n\" (UID: \"b305f151-d272-46e8-af32-5f1274ffce1c\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-2e92a84c-748fd59c8c-hhx6n" Oct 30 20:56:23.131539 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:23.129766 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94a3b297-c47d-4b2a-8e78-697a605c5481-plugins-config-volume\") pod \"payload-processing-e2e-ait-2e92a84c-87c49cccd-wx9fz\" (UID: \"94a3b297-c47d-4b2a-8e78-697a605c5481\") " pod="openshift-ingress/payload-processing-e2e-ait-2e92a84c-87c49cccd-wx9fz" Oct 30 20:56:23.131539 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:23.129847 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c6cxp\" (UniqueName: \"kubernetes.io/projected/b305f151-d272-46e8-af32-5f1274ffce1c-kube-api-access-c6cxp\") pod \"payload-pre-processing-e2e-ait-2e92a84c-748fd59c8c-hhx6n\" (UID: \"b305f151-d272-46e8-af32-5f1274ffce1c\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-2e92a84c-748fd59c8c-hhx6n" Oct 30 20:56:23.131539 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:23.129927 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f7bzj\" (UniqueName: \"kubernetes.io/projected/94a3b297-c47d-4b2a-8e78-697a605c5481-kube-api-access-f7bzj\") pod \"payload-processing-e2e-ait-2e92a84c-87c49cccd-wx9fz\" (UID: \"94a3b297-c47d-4b2a-8e78-697a605c5481\") " pod="openshift-ingress/payload-processing-e2e-ait-2e92a84c-87c49cccd-wx9fz" Oct 30 20:56:23.139530 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:23.132991 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94a3b297-c47d-4b2a-8e78-697a605c5481-plugins-config-volume\") pod \"payload-processing-e2e-ait-2e92a84c-87c49cccd-wx9fz\" (UID: \"94a3b297-c47d-4b2a-8e78-697a605c5481\") " pod="openshift-ingress/payload-processing-e2e-ait-2e92a84c-87c49cccd-wx9fz" Oct 30 20:56:23.143057 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:23.143013 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b305f151-d272-46e8-af32-5f1274ffce1c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-2e92a84c-748fd59c8c-hhx6n\" (UID: \"b305f151-d272-46e8-af32-5f1274ffce1c\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-2e92a84c-748fd59c8c-hhx6n" Oct 30 20:56:23.182640 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:23.165740 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c6cxp\" (UniqueName: \"kubernetes.io/projected/b305f151-d272-46e8-af32-5f1274ffce1c-kube-api-access-c6cxp\") pod \"payload-pre-processing-e2e-ait-2e92a84c-748fd59c8c-hhx6n\" (UID: \"b305f151-d272-46e8-af32-5f1274ffce1c\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-2e92a84c-748fd59c8c-hhx6n" Oct 30 20:56:23.194278 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:23.194181 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f7bzj\" (UniqueName: \"kubernetes.io/projected/94a3b297-c47d-4b2a-8e78-697a605c5481-kube-api-access-f7bzj\") pod \"payload-processing-e2e-ait-2e92a84c-87c49cccd-wx9fz\" (UID: \"94a3b297-c47d-4b2a-8e78-697a605c5481\") " pod="openshift-ingress/payload-processing-e2e-ait-2e92a84c-87c49cccd-wx9fz" Oct 30 20:56:23.212862 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:23.212831 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b8aee1/distinct-2-w1-b8aee1-kserve-6bcdd59797-cx6hr" Oct 30 20:56:23.218455 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:23.218419 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad7ff0/premium-w0-ad7ff0-kserve-5485d4b948-bpmgf" podUID="53cbd692-e9f8-46ba-8fde-3e5215a38904" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:56:23.223456 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:23.223425 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db534b/distinct-2-w2-db534b-kserve-8594c876-g92mh" podUID="955316d6-a9fd-4b75-a8ab-bbf382b776e2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:56:23.250651 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:23.250595 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b8aee1/premium-w1-b8aee1-kserve-786d575599-htc8d" podUID="77368afe-edc8-4fa5-8010-d321965fb38e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:56:23.307532 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:23.289906 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-cb60a2/distinct-w3-cb60a2-kserve-5bb76dc74-86fds" Oct 30 20:56:23.346641 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:23.346504 4611 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-2e92a84c-748fd59c8c-hhx6n" Oct 30 20:56:23.514649 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:23.491703 4611 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-2e92a84c-87c49cccd-wx9fz" Oct 30 20:56:23.527559 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:23.518196 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:56:23.527559 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:23.518305 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cb60a2/simulator-w3-cb60a2-kserve-6fd74b9d8b-vqfzf" podUID="711b167d-4e66-40d6-be75-33ede11f42c5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded" Oct 30 20:56:23.527559 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:23.525006 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b8aee1/premium-w1-b8aee1-kserve-786d575599-htc8d" Oct 30 20:56:23.687646 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:23.676045 4611 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-cb60a2/simulator-w3-cb60a2-kserve-6fd74b9d8b-vqfzf" podUID="711b167d-4e66-40d6-be75-33ede11f42c5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:56:23.687646 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:23.676795 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b8aee1/distinct-w1-b8aee1-kserve-5b857c55cd-x5b7b" Oct 30 20:56:23.701864 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:23.700847 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-14cca2/premium-w5-14cca2-kserve-6878777857-wpjtj" podUID="b99eb76b-d1c6-49e9-9809-c351c0f5a4de" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:56:23.767440 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:23.766946 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-f2ed1e/distinct-w4-f2ed1e-kserve-57cff9dcf8-8d5zl" Oct 30 20:56:23.790531 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:23.774785 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-3d909d/premium-w6-3d909d-kserve-765648946f-2qvd2" Oct 30 20:56:23.790531 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:23.774842 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-14cca2/distinct-w5-14cca2-kserve-587f6996b8-sgf6h" Oct 30 20:56:23.790531 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:23.774875 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ad7ff0/premium-w0-ad7ff0-kserve-5485d4b948-bpmgf" Oct 30 20:56:23.801566 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:23.801030 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-db534b/distinct-2-w2-db534b-kserve-8594c876-g92mh" Oct 30 20:56:23.815532 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:23.814872 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-cb60a2/simulator-w3-cb60a2-kserve-6fd74b9d8b-vqfzf" Oct 30 20:56:23.859418 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:23.859341 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-14cca2/premium-w5-14cca2-kserve-6878777857-wpjtj" Oct 30 20:56:24.150762 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:24.150729 4611 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-2e92a84c-748fd59c8c-hhx6n"] Oct 30 20:56:24.206829 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:24.206789 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-2e92a84c-748fd59c8c-hhx6n" event={"ID":"b305f151-d272-46e8-af32-5f1274ffce1c","Type":"ContainerStarted","Data":"2be37781b757420703cf7d1cf901af45f0c33e32c4cac7403e0b1aa5e1c98ffa"} Oct 30 20:56:24.271067 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:56:24.270711 4611 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94a3b297_c47d_4b2a_8e78_697a605c5481.slice/crio-b27015b0e23b378b4f2b37fcc5cbdea21824f200025f40327ce170446bd7d66e WatchSource:0}: Error finding container b27015b0e23b378b4f2b37fcc5cbdea21824f200025f40327ce170446bd7d66e: Status 404 returned error can't find the container with id b27015b0e23b378b4f2b37fcc5cbdea21824f200025f40327ce170446bd7d66e Oct 30 20:56:24.274766 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:24.274740 4611 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-2e92a84c-87c49cccd-wx9fz"] Oct 30 20:56:24.330163 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:24.330133 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:56:25.216260 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:25.216216 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-2e92a84c-748fd59c8c-hhx6n" event={"ID":"b305f151-d272-46e8-af32-5f1274ffce1c","Type":"ContainerStarted","Data":"2870530122e68a93897f4f92fe2dd8ebacdf3ad072d850a9d0120e4941b289f3"} Oct 30 20:56:25.216679 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:25.216319 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-2e92a84c-748fd59c8c-hhx6n" Oct 30 20:56:25.218060 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:25.218031 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-2e92a84c-87c49cccd-wx9fz" event={"ID":"94a3b297-c47d-4b2a-8e78-697a605c5481","Type":"ContainerStarted","Data":"0a5a9b2d3ac672fa37fb0d5d391255f4a2f335b17c0f67ffab2c58bda9e3e52b"} Oct 30 20:56:25.218199 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:25.218080 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-2e92a84c-87c49cccd-wx9fz" event={"ID":"94a3b297-c47d-4b2a-8e78-697a605c5481","Type":"ContainerStarted","Data":"b27015b0e23b378b4f2b37fcc5cbdea21824f200025f40327ce170446bd7d66e"} Oct 30 20:56:25.218199 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:25.218102 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-2e92a84c-87c49cccd-wx9fz" Oct 30 20:56:25.236008 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:25.235946 4611 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-2e92a84c-748fd59c8c-hhx6n" podStartSLOduration=8.235928705 podStartE2EDuration="8.235928705s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:56:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:56:25.23489247 +0000 UTC m=+1039.768919087" watchObservedRunningTime="2026-10-30 20:56:25.235928705 +0000 UTC m=+1039.769955324" Oct 30 20:56:25.257388 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:25.257330 4611 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-2e92a84c-87c49cccd-wx9fz" podStartSLOduration=8.257312069 podStartE2EDuration="8.257312069s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:56:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:56:25.253766443 +0000 UTC m=+1039.787793086" watchObservedRunningTime="2026-10-30 20:56:25.257312069 +0000 UTC m=+1039.791338686" Oct 30 20:56:25.259041 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:25.259016 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:56:26.302019 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:26.301985 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:56:27.171420 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:27.171382 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:56:27.882542 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:27.882485 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:56:28.570289 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:28.570249 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:56:28.713063 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:28.713025 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:56:29.409671 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:29.409635 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:56:30.530042 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:30.530003 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:56:31.251485 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:31.251441 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:56:31.647777 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:31.647741 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:56:32.209313 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:32.209283 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:56:32.889492 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:32.889459 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:56:33.582372 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:33.582324 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:56:33.918491 ip-10-0-133-192 kubenswrapper[4611]: E1030 20:56:33.918457 4611 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hwzc6" containerName="extractor" Oct 30 20:56:34.196688 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:34.196580 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:56:34.927158 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:34.927120 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:56:35.075733 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:35.075689 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:56:35.236903 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:35.236824 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:56:35.391295 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:35.391260 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:56:35.799265 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:35.799226 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:56:36.250652 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:36.250614 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-2e92a84c-748fd59c8c-hhx6n" Oct 30 20:56:36.251131 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:36.251037 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-2e92a84c-87c49cccd-wx9fz" Oct 30 20:56:36.680455 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:36.680419 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:56:37.411897 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:37.410620 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:56:38.198566 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:38.198270 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:56:38.823128 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:38.823046 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:56:39.519232 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:39.516074 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:56:40.060249 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:40.060215 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:56:40.210748 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:40.210713 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:56:40.362484 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:40.362448 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:56:40.957108 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:40.957067 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:56:41.410873 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:41.410837 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:56:41.905452 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:41.905401 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:56:42.220039 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:42.219960 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:56:43.166562 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:43.166498 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:56:44.040785 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:44.040722 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:56:44.684721 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:44.684686 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:56:45.483809 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:45.483772 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:56:46.317226 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:46.317190 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:56:47.229818 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:47.229774 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db534b/distinct-2-w2-db534b-kserve-8594c876-g92mh" podUID="955316d6-a9fd-4b75-a8ab-bbf382b776e2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:56:47.255477 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:47.230062 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b8aee1/distinct-2-w1-b8aee1-kserve-6bcdd59797-cx6hr" podUID="286eab7a-81a7-4768-a7f6-be6b3720028a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:56:47.256325 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:47.254287 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cb60a2/distinct-w3-cb60a2-kserve-5bb76dc74-86fds" podUID="52e301d9-f378-4e54-9515-01a6680d9fd7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded" Oct 30 20:56:47.295688 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:47.295534 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:56:47.569376 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:47.569279 4611 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-db534b/distinct-2-w2-db534b-kserve-8594c876-g92mh" podUID="955316d6-a9fd-4b75-a8ab-bbf382b776e2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/health\": context deadline exceeded" Oct 30 20:56:47.577281 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:47.577245 4611 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-ad7ff0/premium-w0-ad7ff0-kserve-5485d4b948-bpmgf" podUID="53cbd692-e9f8-46ba-8fde-3e5215a38904" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/health\": context deadline exceeded" Oct 30 20:56:47.651534 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:47.632785 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-14cca2/premium-w5-14cca2-kserve-6878777857-wpjtj" podUID="b99eb76b-d1c6-49e9-9809-c351c0f5a4de" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:56:47.652643 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:47.652606 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b8aee1/distinct-w1-b8aee1-kserve-5b857c55cd-x5b7b" podUID="1599e588-def8-42e1-b81d-ad0aaf119818" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:56:47.750805 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:47.750761 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f2ed1e/distinct-w4-f2ed1e-kserve-57cff9dcf8-8d5zl" podUID="653e3ec7-2147-4813-8946-1f25cfa425f6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:56:47.804532 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:47.778585 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-3d909d/premium-w6-3d909d-kserve-765648946f-2qvd2" podUID="fca5d697-109d-4e90-ab41-c40ad31af874" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded" Oct 30 20:56:47.804532 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:47.778641 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-14cca2/distinct-w5-14cca2-kserve-587f6996b8-sgf6h" podUID="02b652c3-0724-4a50-9e95-2a9890cc97c4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:56:47.826800 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:47.826716 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cb60a2/simulator-w3-cb60a2-kserve-6fd74b9d8b-vqfzf" podUID="711b167d-4e66-40d6-be75-33ede11f42c5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:56:47.828905 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:47.826990 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-3d909d/distinct-2-w6-3d909d-kserve-59c5d5c595-cjg2m" podUID="e2b4c221-ee94-4131-abd8-2313647c641e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:56:48.041534 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:48.003550 4611 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-6c6h6"] Oct 30 20:56:48.148541 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:48.130490 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:56:48.148871 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:48.135293 4611 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-6c6h6" Oct 30 20:56:48.169562 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:48.154853 4611 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-6c6h6"] Oct 30 20:56:48.169930 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:48.157688 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad7ff0/premium-w0-ad7ff0-kserve-5485d4b948-bpmgf" podUID="53cbd692-e9f8-46ba-8fde-3e5215a38904" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:56:48.171625 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:48.157728 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b8aee1/premium-w1-b8aee1-kserve-786d575599-htc8d" podUID="77368afe-edc8-4fa5-8010-d321965fb38e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded" Oct 30 20:56:48.223530 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:48.212621 4611 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-w44qs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:56:48.223530 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:48.212715 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-w44qs" podUID="6d6fd4b2-ad0a-4ce3-b5b3-966de34bd8b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:56:48.223530 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:48.215381 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2042268e-1c64-4fbe-9f2a-b826113d7f30-credential-socket\") pod \"e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-6c6h6\" (UID: \"2042268e-1c64-4fbe-9f2a-b826113d7f30\") " pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-6c6h6" Oct 30 20:56:48.223530 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:48.215424 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2042268e-1c64-4fbe-9f2a-b826113d7f30-istio-envoy\") pod \"e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-6c6h6\" (UID: \"2042268e-1c64-4fbe-9f2a-b826113d7f30\") " pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-6c6h6" Oct 30 20:56:48.223530 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:48.215480 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2042268e-1c64-4fbe-9f2a-b826113d7f30-istio-podinfo\") pod \"e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-6c6h6\" (UID: \"2042268e-1c64-4fbe-9f2a-b826113d7f30\") " pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-6c6h6" Oct 30 20:56:48.235631 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:48.235593 4611 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-8tvvl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:56:48.235873 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:48.235843 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8tvvl" podUID="46ae03da-7a80-4f78-8abd-391ebc697f2f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:56:48.245676 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:48.215523 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2042268e-1c64-4fbe-9f2a-b826113d7f30-istio-data\") pod \"e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-6c6h6\" (UID: \"2042268e-1c64-4fbe-9f2a-b826113d7f30\") " pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-6c6h6" Oct 30 20:56:48.245891 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:48.245870 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2042268e-1c64-4fbe-9f2a-b826113d7f30-istiod-ca-cert\") pod \"e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-6c6h6\" (UID: \"2042268e-1c64-4fbe-9f2a-b826113d7f30\") " pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-6c6h6" Oct 30 20:56:48.249529 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:48.246006 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2042268e-1c64-4fbe-9f2a-b826113d7f30-workload-socket\") pod \"e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-6c6h6\" (UID: \"2042268e-1c64-4fbe-9f2a-b826113d7f30\") " pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-6c6h6" Oct 30 20:56:48.261697 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:48.261669 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2042268e-1c64-4fbe-9f2a-b826113d7f30-workload-certs\") pod \"e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-6c6h6\" (UID: \"2042268e-1c64-4fbe-9f2a-b826113d7f30\") " pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-6c6h6" Oct 30 20:56:48.261885 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:48.261866 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sv277\" (UniqueName: \"kubernetes.io/projected/2042268e-1c64-4fbe-9f2a-b826113d7f30-kube-api-access-sv277\") pod \"e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-6c6h6\" (UID: \"2042268e-1c64-4fbe-9f2a-b826113d7f30\") " pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-6c6h6" Oct 30 20:56:48.262018 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:48.262002 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2042268e-1c64-4fbe-9f2a-b826113d7f30-istio-token\") pod \"e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-6c6h6\" (UID: \"2042268e-1c64-4fbe-9f2a-b826113d7f30\") " pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-6c6h6" Oct 30 20:56:48.272535 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:48.257173 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cb60a2/distinct-w3-cb60a2-kserve-5bb76dc74-86fds" podUID="52e301d9-f378-4e54-9515-01a6680d9fd7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded" Oct 30 20:56:48.272840 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:48.257208 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b8aee1/distinct-2-w1-b8aee1-kserve-6bcdd59797-cx6hr" podUID="286eab7a-81a7-4768-a7f6-be6b3720028a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:56:48.272840 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:48.257777 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db534b/distinct-2-w2-db534b-kserve-8594c876-g92mh" podUID="955316d6-a9fd-4b75-a8ab-bbf382b776e2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:56:48.299114 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:48.282241 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-cb60a2/distinct-w3-cb60a2-kserve-5bb76dc74-86fds" Oct 30 20:56:48.299114 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:48.293676 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b8aee1/distinct-2-w1-b8aee1-kserve-6bcdd59797-cx6hr" Oct 30 20:56:48.299114 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:48.293785 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-db534b/distinct-2-w2-db534b-kserve-8594c876-g92mh" Oct 30 20:56:48.366749 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:48.363651 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2042268e-1c64-4fbe-9f2a-b826113d7f30-workload-certs\") pod \"e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-6c6h6\" (UID: \"2042268e-1c64-4fbe-9f2a-b826113d7f30\") " pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-6c6h6" Oct 30 20:56:48.366749 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:48.363708 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sv277\" (UniqueName: \"kubernetes.io/projected/2042268e-1c64-4fbe-9f2a-b826113d7f30-kube-api-access-sv277\") pod \"e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-6c6h6\" (UID: \"2042268e-1c64-4fbe-9f2a-b826113d7f30\") " pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-6c6h6" Oct 30 20:56:48.366749 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:48.363751 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2042268e-1c64-4fbe-9f2a-b826113d7f30-istio-token\") pod \"e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-6c6h6\" (UID: \"2042268e-1c64-4fbe-9f2a-b826113d7f30\") " pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-6c6h6" Oct 30 20:56:48.366749 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:48.364090 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2042268e-1c64-4fbe-9f2a-b826113d7f30-credential-socket\") pod \"e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-6c6h6\" (UID: \"2042268e-1c64-4fbe-9f2a-b826113d7f30\") " pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-6c6h6" Oct 30 20:56:48.366749 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:48.364119 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2042268e-1c64-4fbe-9f2a-b826113d7f30-istio-envoy\") pod \"e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-6c6h6\" (UID: \"2042268e-1c64-4fbe-9f2a-b826113d7f30\") " pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-6c6h6" Oct 30 20:56:48.366749 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:48.364221 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2042268e-1c64-4fbe-9f2a-b826113d7f30-istio-podinfo\") pod \"e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-6c6h6\" (UID: \"2042268e-1c64-4fbe-9f2a-b826113d7f30\") " pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-6c6h6" Oct 30 20:56:48.366749 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:48.364249 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2042268e-1c64-4fbe-9f2a-b826113d7f30-istio-data\") pod \"e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-6c6h6\" (UID: \"2042268e-1c64-4fbe-9f2a-b826113d7f30\") " pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-6c6h6" Oct 30 20:56:48.366749 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:48.364284 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2042268e-1c64-4fbe-9f2a-b826113d7f30-istiod-ca-cert\") pod \"e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-6c6h6\" (UID: \"2042268e-1c64-4fbe-9f2a-b826113d7f30\") " pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-6c6h6" Oct 30 20:56:48.366749 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:48.364316 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2042268e-1c64-4fbe-9f2a-b826113d7f30-workload-socket\") pod \"e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-6c6h6\" (UID: \"2042268e-1c64-4fbe-9f2a-b826113d7f30\") " pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-6c6h6" Oct 30 20:56:48.366749 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:48.364835 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2042268e-1c64-4fbe-9f2a-b826113d7f30-workload-socket\") pod \"e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-6c6h6\" (UID: \"2042268e-1c64-4fbe-9f2a-b826113d7f30\") " pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-6c6h6" Oct 30 20:56:48.366749 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:48.365108 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2042268e-1c64-4fbe-9f2a-b826113d7f30-workload-certs\") pod \"e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-6c6h6\" (UID: \"2042268e-1c64-4fbe-9f2a-b826113d7f30\") " pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-6c6h6" Oct 30 20:56:48.366749 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:48.366417 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2042268e-1c64-4fbe-9f2a-b826113d7f30-credential-socket\") pod \"e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-6c6h6\" (UID: \"2042268e-1c64-4fbe-9f2a-b826113d7f30\") " pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-6c6h6" Oct 30 20:56:48.374032 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:48.374007 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2042268e-1c64-4fbe-9f2a-b826113d7f30-istio-data\") pod \"e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-6c6h6\" (UID: \"2042268e-1c64-4fbe-9f2a-b826113d7f30\") " pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-6c6h6" Oct 30 20:56:48.381322 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:48.381289 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2042268e-1c64-4fbe-9f2a-b826113d7f30-istiod-ca-cert\") pod \"e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-6c6h6\" (UID: \"2042268e-1c64-4fbe-9f2a-b826113d7f30\") " pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-6c6h6" Oct 30 20:56:48.645004 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:48.644952 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-14cca2/premium-w5-14cca2-kserve-6878777857-wpjtj" podUID="b99eb76b-d1c6-49e9-9809-c351c0f5a4de" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:56:48.816308 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:48.816273 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-14cca2/premium-w5-14cca2-kserve-6878777857-wpjtj" Oct 30 20:56:48.824737 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:48.824694 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f2ed1e/distinct-w4-f2ed1e-kserve-57cff9dcf8-8d5zl" podUID="653e3ec7-2147-4813-8946-1f25cfa425f6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded" Oct 30 20:56:48.859531 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:48.838479 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-3d909d/distinct-2-w6-3d909d-kserve-59c5d5c595-cjg2m" podUID="e2b4c221-ee94-4131-abd8-2313647c641e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": context deadline exceeded" Oct 30 20:56:49.053005 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:48.844177 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sv277\" (UniqueName: \"kubernetes.io/projected/2042268e-1c64-4fbe-9f2a-b826113d7f30-kube-api-access-sv277\") pod \"e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-6c6h6\" (UID: \"2042268e-1c64-4fbe-9f2a-b826113d7f30\") " pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-6c6h6" Oct 30 20:56:49.170025 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:48.859786 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cb60a2/simulator-w3-cb60a2-kserve-6fd74b9d8b-vqfzf" podUID="711b167d-4e66-40d6-be75-33ede11f42c5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded" Oct 30 20:56:49.170352 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:48.860348 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2042268e-1c64-4fbe-9f2a-b826113d7f30-istio-token\") pod \"e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-6c6h6\" (UID: \"2042268e-1c64-4fbe-9f2a-b826113d7f30\") " pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-6c6h6" Oct 30 20:56:49.170615 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:48.860607 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2042268e-1c64-4fbe-9f2a-b826113d7f30-istio-envoy\") pod \"e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-6c6h6\" (UID: \"2042268e-1c64-4fbe-9f2a-b826113d7f30\") " pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-6c6h6" Oct 30 20:56:49.170790 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:48.860890 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-f2ed1e/distinct-w4-f2ed1e-kserve-57cff9dcf8-8d5zl" Oct 30 20:56:49.178780 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:49.171051 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-3d909d/distinct-2-w6-3d909d-kserve-59c5d5c595-cjg2m" Oct 30 20:56:49.179266 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:49.179249 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-cb60a2/simulator-w3-cb60a2-kserve-6fd74b9d8b-vqfzf" Oct 30 20:56:49.252039 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:48.914917 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-3d909d/premium-w6-3d909d-kserve-765648946f-2qvd2" podUID="fca5d697-109d-4e90-ab41-c40ad31af874" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded" Oct 30 20:56:49.323590 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:49.278487 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-3d909d/premium-w6-3d909d-kserve-765648946f-2qvd2" Oct 30 20:56:49.396136 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:49.141611 4611 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-3d909d/premium-w6-3d909d-kserve-765648946f-2qvd2" podUID="fca5d697-109d-4e90-ab41-c40ad31af874" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:56:49.396633 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:49.202604 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad7ff0/premium-w0-ad7ff0-kserve-5485d4b948-bpmgf" podUID="53cbd692-e9f8-46ba-8fde-3e5215a38904" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:56:49.397108 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:48.882058 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2042268e-1c64-4fbe-9f2a-b826113d7f30-istio-podinfo\") pod \"e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-6c6h6\" (UID: \"2042268e-1c64-4fbe-9f2a-b826113d7f30\") " pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-6c6h6" Oct 30 20:56:49.397274 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:48.914558 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-14cca2/distinct-w5-14cca2-kserve-587f6996b8-sgf6h" podUID="02b652c3-0724-4a50-9e95-2a9890cc97c4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded" Oct 30 20:56:49.397656 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:49.203618 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b8aee1/premium-w1-b8aee1-kserve-786d575599-htc8d" podUID="77368afe-edc8-4fa5-8010-d321965fb38e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded" Oct 30 20:56:49.397989 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:49.397963 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ad7ff0/premium-w0-ad7ff0-kserve-5485d4b948-bpmgf" Oct 30 20:56:49.399313 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:49.399296 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-14cca2/distinct-w5-14cca2-kserve-587f6996b8-sgf6h" Oct 30 20:56:49.399478 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:49.399466 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b8aee1/premium-w1-b8aee1-kserve-786d575599-htc8d" Oct 30 20:56:49.399979 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:49.302813 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b8aee1/distinct-2-w1-b8aee1-kserve-6bcdd59797-cx6hr" podUID="286eab7a-81a7-4768-a7f6-be6b3720028a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:56:49.400664 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:49.302847 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cb60a2/distinct-w3-cb60a2-kserve-5bb76dc74-86fds" podUID="52e301d9-f378-4e54-9515-01a6680d9fd7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded" Oct 30 20:56:49.401324 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:49.012082 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b8aee1/distinct-w1-b8aee1-kserve-5b857c55cd-x5b7b" podUID="1599e588-def8-42e1-b81d-ad0aaf119818" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:56:49.401731 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:49.358421 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:56:49.401917 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:49.401902 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b8aee1/distinct-w1-b8aee1-kserve-5b857c55cd-x5b7b" Oct 30 20:56:49.402065 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:49.362654 4611 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-f2ed1e/distinct-w4-f2ed1e-kserve-57cff9dcf8-8d5zl" podUID="653e3ec7-2147-4813-8946-1f25cfa425f6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/health\": context deadline exceeded" Oct 30 20:56:49.402149 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:49.301723 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db534b/distinct-2-w2-db534b-kserve-8594c876-g92mh" podUID="955316d6-a9fd-4b75-a8ab-bbf382b776e2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:56:49.439830 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:49.439801 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-f2ed1e/distinct-w4-f2ed1e-kserve-57cff9dcf8-8d5zl" Oct 30 20:56:49.458048 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:49.457783 4611 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-6c6h6" Oct 30 20:56:49.622655 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:49.622355 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:56:49.631552 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:49.630395 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-14cca2/premium-w5-14cca2-kserve-6878777857-wpjtj" Oct 30 20:56:49.631552 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:49.630455 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-14cca2/distinct-w5-14cca2-kserve-587f6996b8-sgf6h" Oct 30 20:56:49.631552 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:49.630491 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b8aee1/distinct-2-w1-b8aee1-kserve-6bcdd59797-cx6hr" Oct 30 20:56:49.631552 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:49.630536 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-3d909d/distinct-2-w6-3d909d-kserve-59c5d5c595-cjg2m" Oct 30 20:56:49.631552 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:49.630566 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b8aee1/distinct-w1-b8aee1-kserve-5b857c55cd-x5b7b" Oct 30 20:56:49.631552 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:49.630650 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ad7ff0/premium-w0-ad7ff0-kserve-5485d4b948-bpmgf" Oct 30 20:56:49.631552 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:49.630715 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-cb60a2/distinct-w3-cb60a2-kserve-5bb76dc74-86fds" Oct 30 20:56:49.631552 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:49.630761 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-3d909d/premium-w6-3d909d-kserve-765648946f-2qvd2" Oct 30 20:56:49.631552 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:49.630792 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-cb60a2/simulator-w3-cb60a2-kserve-6fd74b9d8b-vqfzf" Oct 30 20:56:49.631552 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:49.630822 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-db534b/distinct-2-w2-db534b-kserve-8594c876-g92mh" Oct 30 20:56:49.632150 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:49.631940 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b8aee1/premium-w1-b8aee1-kserve-786d575599-htc8d" Oct 30 20:56:49.779950 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:49.779920 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:56:49.981370 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:56:49.981342 4611 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2042268e_1c64_4fbe_9f2a_b826113d7f30.slice/crio-21acae026f9bc9a30f1954c12f9e4d43846823b62def73ffb7936453893e3bc7 WatchSource:0}: Error finding container 21acae026f9bc9a30f1954c12f9e4d43846823b62def73ffb7936453893e3bc7: Status 404 returned error can't find the container with id 21acae026f9bc9a30f1954c12f9e4d43846823b62def73ffb7936453893e3bc7 Oct 30 20:56:49.985544 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:49.985486 4611 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:56:49.985649 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:49.985592 4611 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:56:49.985649 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:49.985636 4611 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:56:49.988446 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:49.988418 4611 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-6c6h6"] Oct 30 20:56:50.139586 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:50.136881 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-6c6h6" event={"ID":"2042268e-1c64-4fbe-9f2a-b826113d7f30","Type":"ContainerStarted","Data":"256030229a04a2e5065cc806c24ba748f6ae5c57276bd9ec8a9fef6b19d8abe2"} Oct 30 20:56:50.139586 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:50.136960 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-6c6h6" event={"ID":"2042268e-1c64-4fbe-9f2a-b826113d7f30","Type":"ContainerStarted","Data":"21acae026f9bc9a30f1954c12f9e4d43846823b62def73ffb7936453893e3bc7"} Oct 30 20:56:50.220884 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:50.220830 4611 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-6c6h6" podStartSLOduration=3.220811626 podStartE2EDuration="3.220811626s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:56:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:56:50.207733664 +0000 UTC m=+1064.741760281" watchObservedRunningTime="2026-10-30 20:56:50.220811626 +0000 UTC m=+1064.754838287" Oct 30 20:56:50.468532 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:50.466584 4611 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-6c6h6" Oct 30 20:56:50.696576 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:50.694815 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:56:50.969297 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:50.966818 4611 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-2e92a84c-87c49cccd-wx9fz"] Oct 30 20:56:51.011572 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:50.996565 4611 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-2e92a84c-748fd59c8c-hhx6n"] Oct 30 20:56:51.108281 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:51.108237 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:56:51.263883 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:51.263783 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:56:51.464259 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:51.464223 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:56:51.484584 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:51.475420 4611 patch_prober.go:29] interesting pod/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-6c6h6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body= Oct 30 20:56:51.503531 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:51.495728 4611 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-6c6h6" podUID="2042268e-1c64-4fbe-9f2a-b826113d7f30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" Oct 30 20:56:51.728182 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:51.728123 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:56:52.104668 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:52.102250 4611 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-a4317e67-openshift-default-5885d97bf-gx7wp"] Oct 30 20:56:52.113595 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:52.112276 4611 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-a4317e67-openshift-default-5885d97bf-gx7wp"] Oct 30 20:56:52.113595 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:52.112403 4611 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-a4317e67-openshift-default-5885d97bf-gx7wp" Oct 30 20:56:52.120809 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:52.120786 4611 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-a4317e67-openshift-default-dockercfg-nb9v6\"" Oct 30 20:56:52.277539 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:52.256731 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3bfb87f-8eb3-40eb-8c88-27495638d667-workload-socket\") pod \"e2e-derive-a4317e67-openshift-default-5885d97bf-gx7wp\" (UID: \"c3bfb87f-8eb3-40eb-8c88-27495638d667\") " pod="openshift-ingress/e2e-derive-a4317e67-openshift-default-5885d97bf-gx7wp" Oct 30 20:56:52.364920 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:52.364789 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3bfb87f-8eb3-40eb-8c88-27495638d667-workload-certs\") pod \"e2e-derive-a4317e67-openshift-default-5885d97bf-gx7wp\" (UID: \"c3bfb87f-8eb3-40eb-8c88-27495638d667\") " pod="openshift-ingress/e2e-derive-a4317e67-openshift-default-5885d97bf-gx7wp" Oct 30 20:56:52.389662 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:52.389622 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3bfb87f-8eb3-40eb-8c88-27495638d667-credential-socket\") pod \"e2e-derive-a4317e67-openshift-default-5885d97bf-gx7wp\" (UID: \"c3bfb87f-8eb3-40eb-8c88-27495638d667\") " pod="openshift-ingress/e2e-derive-a4317e67-openshift-default-5885d97bf-gx7wp" Oct 30 20:56:52.393529 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:52.389937 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5kcsj\" (UniqueName: \"kubernetes.io/projected/c3bfb87f-8eb3-40eb-8c88-27495638d667-kube-api-access-5kcsj\") pod \"e2e-derive-a4317e67-openshift-default-5885d97bf-gx7wp\" (UID: \"c3bfb87f-8eb3-40eb-8c88-27495638d667\") " pod="openshift-ingress/e2e-derive-a4317e67-openshift-default-5885d97bf-gx7wp" Oct 30 20:56:52.419644 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:52.419602 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3bfb87f-8eb3-40eb-8c88-27495638d667-istio-envoy\") pod \"e2e-derive-a4317e67-openshift-default-5885d97bf-gx7wp\" (UID: \"c3bfb87f-8eb3-40eb-8c88-27495638d667\") " pod="openshift-ingress/e2e-derive-a4317e67-openshift-default-5885d97bf-gx7wp" Oct 30 20:56:52.423530 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:52.419947 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3bfb87f-8eb3-40eb-8c88-27495638d667-istio-podinfo\") pod \"e2e-derive-a4317e67-openshift-default-5885d97bf-gx7wp\" (UID: \"c3bfb87f-8eb3-40eb-8c88-27495638d667\") " pod="openshift-ingress/e2e-derive-a4317e67-openshift-default-5885d97bf-gx7wp" Oct 30 20:56:52.462713 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:52.462672 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3bfb87f-8eb3-40eb-8c88-27495638d667-istio-token\") pod \"e2e-derive-a4317e67-openshift-default-5885d97bf-gx7wp\" (UID: \"c3bfb87f-8eb3-40eb-8c88-27495638d667\") " pod="openshift-ingress/e2e-derive-a4317e67-openshift-default-5885d97bf-gx7wp" Oct 30 20:56:52.466573 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:52.463012 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3bfb87f-8eb3-40eb-8c88-27495638d667-istiod-ca-cert\") pod \"e2e-derive-a4317e67-openshift-default-5885d97bf-gx7wp\" (UID: \"c3bfb87f-8eb3-40eb-8c88-27495638d667\") " pod="openshift-ingress/e2e-derive-a4317e67-openshift-default-5885d97bf-gx7wp" Oct 30 20:56:52.490708 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:52.490663 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3bfb87f-8eb3-40eb-8c88-27495638d667-istio-data\") pod \"e2e-derive-a4317e67-openshift-default-5885d97bf-gx7wp\" (UID: \"c3bfb87f-8eb3-40eb-8c88-27495638d667\") " pod="openshift-ingress/e2e-derive-a4317e67-openshift-default-5885d97bf-gx7wp" Oct 30 20:56:52.491311 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:52.461902 4611 patch_prober.go:29] interesting pod/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-6c6h6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body= Oct 30 20:56:52.491539 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:52.491491 4611 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-6c6h6" podUID="2042268e-1c64-4fbe-9f2a-b826113d7f30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" Oct 30 20:56:52.492068 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:52.256831 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:56:52.597566 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:52.592480 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3bfb87f-8eb3-40eb-8c88-27495638d667-workload-certs\") pod \"e2e-derive-a4317e67-openshift-default-5885d97bf-gx7wp\" (UID: \"c3bfb87f-8eb3-40eb-8c88-27495638d667\") " pod="openshift-ingress/e2e-derive-a4317e67-openshift-default-5885d97bf-gx7wp" Oct 30 20:56:52.602701 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:52.599468 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3bfb87f-8eb3-40eb-8c88-27495638d667-credential-socket\") pod \"e2e-derive-a4317e67-openshift-default-5885d97bf-gx7wp\" (UID: \"c3bfb87f-8eb3-40eb-8c88-27495638d667\") " pod="openshift-ingress/e2e-derive-a4317e67-openshift-default-5885d97bf-gx7wp" Oct 30 20:56:52.602701 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:52.599577 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5kcsj\" (UniqueName: \"kubernetes.io/projected/c3bfb87f-8eb3-40eb-8c88-27495638d667-kube-api-access-5kcsj\") pod \"e2e-derive-a4317e67-openshift-default-5885d97bf-gx7wp\" (UID: \"c3bfb87f-8eb3-40eb-8c88-27495638d667\") " pod="openshift-ingress/e2e-derive-a4317e67-openshift-default-5885d97bf-gx7wp" Oct 30 20:56:52.602701 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:52.599620 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3bfb87f-8eb3-40eb-8c88-27495638d667-istio-envoy\") pod \"e2e-derive-a4317e67-openshift-default-5885d97bf-gx7wp\" (UID: \"c3bfb87f-8eb3-40eb-8c88-27495638d667\") " pod="openshift-ingress/e2e-derive-a4317e67-openshift-default-5885d97bf-gx7wp" Oct 30 20:56:52.602701 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:52.599672 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3bfb87f-8eb3-40eb-8c88-27495638d667-istio-podinfo\") pod \"e2e-derive-a4317e67-openshift-default-5885d97bf-gx7wp\" (UID: \"c3bfb87f-8eb3-40eb-8c88-27495638d667\") " pod="openshift-ingress/e2e-derive-a4317e67-openshift-default-5885d97bf-gx7wp" Oct 30 20:56:52.602701 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:52.599762 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3bfb87f-8eb3-40eb-8c88-27495638d667-istio-token\") pod \"e2e-derive-a4317e67-openshift-default-5885d97bf-gx7wp\" (UID: \"c3bfb87f-8eb3-40eb-8c88-27495638d667\") " pod="openshift-ingress/e2e-derive-a4317e67-openshift-default-5885d97bf-gx7wp" Oct 30 20:56:52.602701 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:52.599853 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3bfb87f-8eb3-40eb-8c88-27495638d667-istiod-ca-cert\") pod \"e2e-derive-a4317e67-openshift-default-5885d97bf-gx7wp\" (UID: \"c3bfb87f-8eb3-40eb-8c88-27495638d667\") " pod="openshift-ingress/e2e-derive-a4317e67-openshift-default-5885d97bf-gx7wp" Oct 30 20:56:52.602701 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:52.599918 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3bfb87f-8eb3-40eb-8c88-27495638d667-istio-data\") pod \"e2e-derive-a4317e67-openshift-default-5885d97bf-gx7wp\" (UID: \"c3bfb87f-8eb3-40eb-8c88-27495638d667\") " pod="openshift-ingress/e2e-derive-a4317e67-openshift-default-5885d97bf-gx7wp" Oct 30 20:56:52.602701 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:52.600077 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3bfb87f-8eb3-40eb-8c88-27495638d667-workload-socket\") pod \"e2e-derive-a4317e67-openshift-default-5885d97bf-gx7wp\" (UID: \"c3bfb87f-8eb3-40eb-8c88-27495638d667\") " pod="openshift-ingress/e2e-derive-a4317e67-openshift-default-5885d97bf-gx7wp" Oct 30 20:56:52.602701 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:52.600668 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3bfb87f-8eb3-40eb-8c88-27495638d667-workload-socket\") pod \"e2e-derive-a4317e67-openshift-default-5885d97bf-gx7wp\" (UID: \"c3bfb87f-8eb3-40eb-8c88-27495638d667\") " pod="openshift-ingress/e2e-derive-a4317e67-openshift-default-5885d97bf-gx7wp" Oct 30 20:56:52.602701 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:52.598256 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3bfb87f-8eb3-40eb-8c88-27495638d667-workload-certs\") pod \"e2e-derive-a4317e67-openshift-default-5885d97bf-gx7wp\" (UID: \"c3bfb87f-8eb3-40eb-8c88-27495638d667\") " pod="openshift-ingress/e2e-derive-a4317e67-openshift-default-5885d97bf-gx7wp" Oct 30 20:56:52.603644 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:52.603607 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3bfb87f-8eb3-40eb-8c88-27495638d667-credential-socket\") pod \"e2e-derive-a4317e67-openshift-default-5885d97bf-gx7wp\" (UID: \"c3bfb87f-8eb3-40eb-8c88-27495638d667\") " pod="openshift-ingress/e2e-derive-a4317e67-openshift-default-5885d97bf-gx7wp" Oct 30 20:56:52.605130 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:52.604797 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3bfb87f-8eb3-40eb-8c88-27495638d667-istio-data\") pod \"e2e-derive-a4317e67-openshift-default-5885d97bf-gx7wp\" (UID: \"c3bfb87f-8eb3-40eb-8c88-27495638d667\") " pod="openshift-ingress/e2e-derive-a4317e67-openshift-default-5885d97bf-gx7wp" Oct 30 20:56:52.606785 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:52.606734 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3bfb87f-8eb3-40eb-8c88-27495638d667-istiod-ca-cert\") pod \"e2e-derive-a4317e67-openshift-default-5885d97bf-gx7wp\" (UID: \"c3bfb87f-8eb3-40eb-8c88-27495638d667\") " pod="openshift-ingress/e2e-derive-a4317e67-openshift-default-5885d97bf-gx7wp" Oct 30 20:56:52.610720 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:52.610671 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3bfb87f-8eb3-40eb-8c88-27495638d667-istio-envoy\") pod \"e2e-derive-a4317e67-openshift-default-5885d97bf-gx7wp\" (UID: \"c3bfb87f-8eb3-40eb-8c88-27495638d667\") " pod="openshift-ingress/e2e-derive-a4317e67-openshift-default-5885d97bf-gx7wp" Oct 30 20:56:52.611366 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:52.611336 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3bfb87f-8eb3-40eb-8c88-27495638d667-istio-podinfo\") pod \"e2e-derive-a4317e67-openshift-default-5885d97bf-gx7wp\" (UID: \"c3bfb87f-8eb3-40eb-8c88-27495638d667\") " pod="openshift-ingress/e2e-derive-a4317e67-openshift-default-5885d97bf-gx7wp" Oct 30 20:56:52.641577 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:52.621747 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3bfb87f-8eb3-40eb-8c88-27495638d667-istio-token\") pod \"e2e-derive-a4317e67-openshift-default-5885d97bf-gx7wp\" (UID: \"c3bfb87f-8eb3-40eb-8c88-27495638d667\") " pod="openshift-ingress/e2e-derive-a4317e67-openshift-default-5885d97bf-gx7wp" Oct 30 20:56:52.641577 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:52.622227 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5kcsj\" (UniqueName: \"kubernetes.io/projected/c3bfb87f-8eb3-40eb-8c88-27495638d667-kube-api-access-5kcsj\") pod \"e2e-derive-a4317e67-openshift-default-5885d97bf-gx7wp\" (UID: \"c3bfb87f-8eb3-40eb-8c88-27495638d667\") " pod="openshift-ingress/e2e-derive-a4317e67-openshift-default-5885d97bf-gx7wp" Oct 30 20:56:52.669798 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:52.669603 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:56:52.732561 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:52.728988 4611 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-a4317e67-openshift-default-5885d97bf-gx7wp" Oct 30 20:56:53.126236 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:53.111612 4611 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-de5833d3-v55zb"] Oct 30 20:56:53.126236 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:53.119413 4611 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-de5833d3-v55zb"] Oct 30 20:56:53.226180 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:53.225958 4611 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-a4317e67-openshift-default-5885d97bf-gx7wp"] Oct 30 20:56:53.233066 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:53.229692 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:56:53.237986 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:53.237892 4611 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:56:53.237986 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:53.237956 4611 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:56:53.238136 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:53.237991 4611 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:56:53.313545 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:53.313434 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-a4317e67-openshift-default-5885d97bf-gx7wp" event={"ID":"c3bfb87f-8eb3-40eb-8c88-27495638d667","Type":"ContainerStarted","Data":"a17cbba313da950d7559b4579c87182ad2fcecadb3250e2aa432f8c8780c4e52"} Oct 30 20:56:53.411387 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:53.411320 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:56:53.463072 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:53.462620 4611 patch_prober.go:29] interesting pod/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-6c6h6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body= Oct 30 20:56:53.463072 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:53.462686 4611 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-6c6h6" podUID="2042268e-1c64-4fbe-9f2a-b826113d7f30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" Oct 30 20:56:53.613764 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:53.613734 4611 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="93074e7e-2063-4c32-9c4c-36137d25d437" path="/var/lib/kubelet/pods/93074e7e-2063-4c32-9c4c-36137d25d437/volumes" Oct 30 20:56:53.865585 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:53.853176 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:56:54.027533 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:54.027273 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:56:54.194216 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:54.193932 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:56:54.328246 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:54.328210 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-a4317e67-openshift-default-5885d97bf-gx7wp" event={"ID":"c3bfb87f-8eb3-40eb-8c88-27495638d667","Type":"ContainerStarted","Data":"24778fe77e816b080bb543ffd31a35551fb493ca049b84398cecc026b61240df"} Oct 30 20:56:54.333664 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:54.333631 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:56:54.382530 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:54.382450 4611 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-a4317e67-openshift-default-5885d97bf-gx7wp" podStartSLOduration=2.382430442 podStartE2EDuration="2.382430442s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:56:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:56:54.351998803 +0000 UTC m=+1068.886025421" watchObservedRunningTime="2026-10-30 20:56:54.382430442 +0000 UTC m=+1068.916457059" Oct 30 20:56:54.445134 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:54.445047 4611 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-db534b/distinct-2-w2-db534b-kserve-8594c876-g92mh"] Oct 30 20:56:54.483534 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:54.483024 4611 patch_prober.go:29] interesting pod/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-6c6h6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body= Oct 30 20:56:54.483534 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:54.483086 4611 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-6c6h6" podUID="2042268e-1c64-4fbe-9f2a-b826113d7f30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" Oct 30 20:56:54.612254 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:54.612217 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:56:54.729654 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:54.729575 4611 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-a4317e67-openshift-default-5885d97bf-gx7wp" Oct 30 20:56:54.734631 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:54.734603 4611 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-a4317e67-openshift-default-5885d97bf-gx7wp" Oct 30 20:56:55.089811 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:55.088449 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:56:55.338266 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:55.338154 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-a4317e67-openshift-default-5885d97bf-gx7wp" Oct 30 20:56:55.339858 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:55.339795 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-a4317e67-openshift-default-5885d97bf-gx7wp" Oct 30 20:56:55.477546 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:55.475919 4611 patch_prober.go:29] interesting pod/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-6c6h6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body= Oct 30 20:56:55.477546 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:55.475984 4611 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-6c6h6" podUID="2042268e-1c64-4fbe-9f2a-b826113d7f30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" Oct 30 20:56:55.966526 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:55.966183 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:56:55.981559 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:55.973575 4611 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-2e92a84c-87c49cccd-wx9fz" podUID="94a3b297-c47d-4b2a-8e78-697a605c5481" containerName="payload-processing" containerID="cri-o://0a5a9b2d3ac672fa37fb0d5d391255f4a2f335b17c0f67ffab2c58bda9e3e52b" gracePeriod=25 Oct 30 20:56:55.997140 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:55.997094 4611 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-2e92a84c-748fd59c8c-hhx6n" podUID="b305f151-d272-46e8-af32-5f1274ffce1c" containerName="payload-pre-processing" containerID="cri-o://2870530122e68a93897f4f92fe2dd8ebacdf3ad072d850a9d0120e4941b289f3" gracePeriod=25 Oct 30 20:56:56.132037 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:56.131986 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:56:56.283738 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:56:56.283650 4611 logging.go:55] [core] [Channel #392 SubChannel #393]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.91:50052", ServerName: "10.133.0.91:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.91:50052: connect: connection refused" Oct 30 20:56:56.298475 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:56.298114 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:56:56.382537 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:56.366414 4611 generic.go:356] "Generic (PLEG): container finished" podID="94a3b297-c47d-4b2a-8e78-697a605c5481" containerID="0a5a9b2d3ac672fa37fb0d5d391255f4a2f335b17c0f67ffab2c58bda9e3e52b" exitCode=0 Oct 30 20:56:56.382537 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:56.366670 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-2e92a84c-87c49cccd-wx9fz" event={"ID":"94a3b297-c47d-4b2a-8e78-697a605c5481","Type":"ContainerDied","Data":"0a5a9b2d3ac672fa37fb0d5d391255f4a2f335b17c0f67ffab2c58bda9e3e52b"} Oct 30 20:56:56.382537 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:56.369134 4611 generic.go:356] "Generic (PLEG): container finished" podID="b305f151-d272-46e8-af32-5f1274ffce1c" containerID="2870530122e68a93897f4f92fe2dd8ebacdf3ad072d850a9d0120e4941b289f3" exitCode=0 Oct 30 20:56:56.382537 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:56.369576 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-2e92a84c-748fd59c8c-hhx6n" event={"ID":"b305f151-d272-46e8-af32-5f1274ffce1c","Type":"ContainerDied","Data":"2870530122e68a93897f4f92fe2dd8ebacdf3ad072d850a9d0120e4941b289f3"} Oct 30 20:56:56.486534 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:56.486023 4611 patch_prober.go:29] interesting pod/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-6c6h6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body= Oct 30 20:56:56.486534 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:56.486093 4611 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-6c6h6" podUID="2042268e-1c64-4fbe-9f2a-b826113d7f30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" Oct 30 20:56:56.639947 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:56.639910 4611 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-2e92a84c-87c49cccd-wx9fz" Oct 30 20:56:56.643684 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:56.643659 4611 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-2e92a84c-748fd59c8c-hhx6n" Oct 30 20:56:56.760162 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:56.759388 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:56:56.824728 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:56.824688 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/94a3b297-c47d-4b2a-8e78-697a605c5481-kube-api-access-f7bzj\" (UniqueName: \"kubernetes.io/projected/94a3b297-c47d-4b2a-8e78-697a605c5481-kube-api-access-f7bzj\") pod \"94a3b297-c47d-4b2a-8e78-697a605c5481\" (UID: \"94a3b297-c47d-4b2a-8e78-697a605c5481\") " Oct 30 20:56:56.824954 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:56.824754 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b305f151-d272-46e8-af32-5f1274ffce1c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b305f151-d272-46e8-af32-5f1274ffce1c-plugins-config-volume\") pod \"b305f151-d272-46e8-af32-5f1274ffce1c\" (UID: \"b305f151-d272-46e8-af32-5f1274ffce1c\") " Oct 30 20:56:56.825005 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:56.824960 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b305f151-d272-46e8-af32-5f1274ffce1c-kube-api-access-c6cxp\" (UniqueName: \"kubernetes.io/projected/b305f151-d272-46e8-af32-5f1274ffce1c-kube-api-access-c6cxp\") pod \"b305f151-d272-46e8-af32-5f1274ffce1c\" (UID: \"b305f151-d272-46e8-af32-5f1274ffce1c\") " Oct 30 20:56:56.825040 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:56.825016 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/94a3b297-c47d-4b2a-8e78-697a605c5481-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94a3b297-c47d-4b2a-8e78-697a605c5481-plugins-config-volume\") pod \"94a3b297-c47d-4b2a-8e78-697a605c5481\" (UID: \"94a3b297-c47d-4b2a-8e78-697a605c5481\") " Oct 30 20:56:56.826012 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:56.825984 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b305f151-d272-46e8-af32-5f1274ffce1c-plugins-config-volume" pod "b305f151-d272-46e8-af32-5f1274ffce1c" (UID: "b305f151-d272-46e8-af32-5f1274ffce1c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:56:56.826136 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:56.825986 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94a3b297-c47d-4b2a-8e78-697a605c5481-plugins-config-volume" pod "94a3b297-c47d-4b2a-8e78-697a605c5481" (UID: "94a3b297-c47d-4b2a-8e78-697a605c5481"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:56:56.827945 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:56.827922 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b305f151-d272-46e8-af32-5f1274ffce1c-kube-api-access-c6cxp" pod "b305f151-d272-46e8-af32-5f1274ffce1c" (UID: "b305f151-d272-46e8-af32-5f1274ffce1c"). InnerVolumeSpecName "kube-api-access-c6cxp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:56:56.828302 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:56.828273 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94a3b297-c47d-4b2a-8e78-697a605c5481-kube-api-access-f7bzj" pod "94a3b297-c47d-4b2a-8e78-697a605c5481" (UID: "94a3b297-c47d-4b2a-8e78-697a605c5481"). InnerVolumeSpecName "kube-api-access-f7bzj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:56:56.927848 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:56.927746 4611 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94a3b297-c47d-4b2a-8e78-697a605c5481-plugins-config-volume\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:56:56.928095 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:56.928067 4611 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f7bzj\" (UniqueName: \"kubernetes.io/projected/94a3b297-c47d-4b2a-8e78-697a605c5481-kube-api-access-f7bzj\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:56:56.928095 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:56.928092 4611 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b305f151-d272-46e8-af32-5f1274ffce1c-plugins-config-volume\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:56:56.928216 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:56.928106 4611 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c6cxp\" (UniqueName: \"kubernetes.io/projected/b305f151-d272-46e8-af32-5f1274ffce1c-kube-api-access-c6cxp\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:56:57.257488 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:57.254907 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-ait-2e92a84c-87c49cccd-wx9fz" podUID="94a3b297-c47d-4b2a-8e78-697a605c5481" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.92:50052\" within 1s: context deadline exceeded" Oct 30 20:56:57.257488 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:57.255280 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-ait-2e92a84c-748fd59c8c-hhx6n" podUID="b305f151-d272-46e8-af32-5f1274ffce1c" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.91:50052\" within 1s: context deadline exceeded" Oct 30 20:56:57.257488 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:56:57.255427 4611 logging.go:55] [core] [Channel #394 SubChannel #395]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.92:50052", ServerName: "10.133.0.92:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.92:50052: operation was canceled" Oct 30 20:56:57.392541 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:57.390367 4611 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-2e92a84c-748fd59c8c-hhx6n" Oct 30 20:56:57.396165 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:57.395713 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-2e92a84c-748fd59c8c-hhx6n" event={"ID":"b305f151-d272-46e8-af32-5f1274ffce1c","Type":"ContainerDied","Data":"2be37781b757420703cf7d1cf901af45f0c33e32c4cac7403e0b1aa5e1c98ffa"} Oct 30 20:56:57.396165 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:57.395790 4611 scope.go:122] "RemoveContainer" containerID="2870530122e68a93897f4f92fe2dd8ebacdf3ad072d850a9d0120e4941b289f3" Oct 30 20:56:57.400372 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:57.400345 4611 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-2e92a84c-87c49cccd-wx9fz" Oct 30 20:56:57.404126 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:57.403271 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-2e92a84c-87c49cccd-wx9fz" event={"ID":"94a3b297-c47d-4b2a-8e78-697a605c5481","Type":"ContainerDied","Data":"b27015b0e23b378b4f2b37fcc5cbdea21824f200025f40327ce170446bd7d66e"} Oct 30 20:56:57.440537 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:57.439549 4611 scope.go:122] "RemoveContainer" containerID="0a5a9b2d3ac672fa37fb0d5d391255f4a2f335b17c0f67ffab2c58bda9e3e52b" Oct 30 20:56:57.453882 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:57.453848 4611 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-2e92a84c-748fd59c8c-hhx6n"] Oct 30 20:56:57.461094 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:57.461066 4611 patch_prober.go:29] interesting pod/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-6c6h6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body= Oct 30 20:56:57.461209 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:57.461117 4611 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-6c6h6" podUID="2042268e-1c64-4fbe-9f2a-b826113d7f30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" Oct 30 20:56:57.466752 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:57.466719 4611 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-2e92a84c-748fd59c8c-hhx6n"] Oct 30 20:56:57.486769 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:57.486729 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:56:57.489591 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:57.489565 4611 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-2e92a84c-87c49cccd-wx9fz"] Oct 30 20:56:57.496935 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:57.496906 4611 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-2e92a84c-87c49cccd-wx9fz"] Oct 30 20:56:57.527858 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:57.527687 4611 patch_prober.go:29] interesting pod/e2e-ait-a8daf029-openshift-default-79cb595b95-qqft5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:56:57.527858 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:57.527794 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-a8daf029-openshift-default-79cb595b95-qqft5" podUID="8632241a-3073-4493-9fab-298607317a96" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:56:57.617943 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:57.617903 4611 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="94a3b297-c47d-4b2a-8e78-697a605c5481" path="/var/lib/kubelet/pods/94a3b297-c47d-4b2a-8e78-697a605c5481/volumes" Oct 30 20:56:57.618701 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:57.618682 4611 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b305f151-d272-46e8-af32-5f1274ffce1c" path="/var/lib/kubelet/pods/b305f151-d272-46e8-af32-5f1274ffce1c/volumes" Oct 30 20:56:57.647637 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:57.647606 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:56:57.819829 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:57.813390 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:56:58.286690 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:58.286648 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:56:58.494540 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:58.493339 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:56:58.657322 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:58.655025 4611 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-6c6h6" Oct 30 20:56:58.657322 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:58.656061 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-6c6h6" Oct 30 20:56:58.657322 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:58.656686 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:56:58.657715 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:58.657389 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-6c6h6" Oct 30 20:56:58.745947 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:58.745911 4611 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-a8daf029-openshift-default-79cb595b95-qqft5"] Oct 30 20:56:58.746589 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:58.746209 4611 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-a8daf029-openshift-default-79cb595b95-qqft5" podUID="8632241a-3073-4493-9fab-298607317a96" containerName="istio-proxy" containerID="cri-o://e00691c8cdcf7470c3b6bfae79ebf61a61c888085aad5613b0578654e7a980d0" gracePeriod=30 Oct 30 20:56:58.834538 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:58.823795 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:56:59.600530 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:59.598750 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:56:59.928733 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:59.928704 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:56:59.930253 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:56:59.930229 4611 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-a8daf029-openshift-default-79cb595b95-qqft5" Oct 30 20:57:00.044040 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:00.044003 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8632241a-3073-4493-9fab-298607317a96-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8632241a-3073-4493-9fab-298607317a96-credential-socket\") pod \"8632241a-3073-4493-9fab-298607317a96\" (UID: \"8632241a-3073-4493-9fab-298607317a96\") " Oct 30 20:57:00.044259 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:00.044055 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8632241a-3073-4493-9fab-298607317a96-istio-token\" (UniqueName: \"kubernetes.io/projected/8632241a-3073-4493-9fab-298607317a96-istio-token\") pod \"8632241a-3073-4493-9fab-298607317a96\" (UID: \"8632241a-3073-4493-9fab-298607317a96\") " Oct 30 20:57:00.044259 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:00.044116 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8632241a-3073-4493-9fab-298607317a96-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8632241a-3073-4493-9fab-298607317a96-istiod-ca-cert\") pod \"8632241a-3073-4493-9fab-298607317a96\" (UID: \"8632241a-3073-4493-9fab-298607317a96\") " Oct 30 20:57:00.044259 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:00.044161 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8632241a-3073-4493-9fab-298607317a96-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8632241a-3073-4493-9fab-298607317a96-istio-envoy\") pod \"8632241a-3073-4493-9fab-298607317a96\" (UID: \"8632241a-3073-4493-9fab-298607317a96\") " Oct 30 20:57:00.044451 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:00.044277 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8632241a-3073-4493-9fab-298607317a96-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8632241a-3073-4493-9fab-298607317a96-istio-data\") pod \"8632241a-3073-4493-9fab-298607317a96\" (UID: \"8632241a-3073-4493-9fab-298607317a96\") " Oct 30 20:57:00.044451 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:00.044336 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8632241a-3073-4493-9fab-298607317a96-credential-socket" pod "8632241a-3073-4493-9fab-298607317a96" (UID: "8632241a-3073-4493-9fab-298607317a96"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:57:00.044451 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:00.044355 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8632241a-3073-4493-9fab-298607317a96-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8632241a-3073-4493-9fab-298607317a96-istio-podinfo\") pod \"8632241a-3073-4493-9fab-298607317a96\" (UID: \"8632241a-3073-4493-9fab-298607317a96\") " Oct 30 20:57:00.044451 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:00.044397 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8632241a-3073-4493-9fab-298607317a96-kube-api-access-jjxzb\" (UniqueName: \"kubernetes.io/projected/8632241a-3073-4493-9fab-298607317a96-kube-api-access-jjxzb\") pod \"8632241a-3073-4493-9fab-298607317a96\" (UID: \"8632241a-3073-4493-9fab-298607317a96\") " Oct 30 20:57:00.044757 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:00.044685 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8632241a-3073-4493-9fab-298607317a96-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8632241a-3073-4493-9fab-298607317a96-workload-socket\") pod \"8632241a-3073-4493-9fab-298607317a96\" (UID: \"8632241a-3073-4493-9fab-298607317a96\") " Oct 30 20:57:00.044876 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:00.044839 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8632241a-3073-4493-9fab-298607317a96-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8632241a-3073-4493-9fab-298607317a96-workload-certs\") pod \"8632241a-3073-4493-9fab-298607317a96\" (UID: \"8632241a-3073-4493-9fab-298607317a96\") " Oct 30 20:57:00.045628 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:00.045593 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8632241a-3073-4493-9fab-298607317a96-workload-socket" pod "8632241a-3073-4493-9fab-298607317a96" (UID: "8632241a-3073-4493-9fab-298607317a96"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:57:00.045758 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:00.045668 4611 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8632241a-3073-4493-9fab-298607317a96-credential-socket\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:57:00.046052 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:00.045882 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8632241a-3073-4493-9fab-298607317a96-workload-certs" pod "8632241a-3073-4493-9fab-298607317a96" (UID: "8632241a-3073-4493-9fab-298607317a96"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:57:00.046052 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:00.045993 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8632241a-3073-4493-9fab-298607317a96-istiod-ca-cert" pod "8632241a-3073-4493-9fab-298607317a96" (UID: "8632241a-3073-4493-9fab-298607317a96"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:57:00.046528 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:00.046480 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8632241a-3073-4493-9fab-298607317a96-istio-data" pod "8632241a-3073-4493-9fab-298607317a96" (UID: "8632241a-3073-4493-9fab-298607317a96"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:57:00.053772 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:00.053719 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8632241a-3073-4493-9fab-298607317a96-istio-token" pod "8632241a-3073-4493-9fab-298607317a96" (UID: "8632241a-3073-4493-9fab-298607317a96"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:57:00.054639 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:00.054605 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8632241a-3073-4493-9fab-298607317a96-istio-podinfo" pod "8632241a-3073-4493-9fab-298607317a96" (UID: "8632241a-3073-4493-9fab-298607317a96"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:57:00.057974 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:00.057916 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8632241a-3073-4493-9fab-298607317a96-istio-envoy" pod "8632241a-3073-4493-9fab-298607317a96" (UID: "8632241a-3073-4493-9fab-298607317a96"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:57:00.058218 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:00.058195 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8632241a-3073-4493-9fab-298607317a96-kube-api-access-jjxzb" pod "8632241a-3073-4493-9fab-298607317a96" (UID: "8632241a-3073-4493-9fab-298607317a96"). InnerVolumeSpecName "kube-api-access-jjxzb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:57:00.147736 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:00.147632 4611 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8632241a-3073-4493-9fab-298607317a96-istio-envoy\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:57:00.147736 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:00.147671 4611 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8632241a-3073-4493-9fab-298607317a96-istio-data\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:57:00.147736 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:00.147684 4611 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8632241a-3073-4493-9fab-298607317a96-istio-podinfo\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:57:00.147736 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:00.147705 4611 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jjxzb\" (UniqueName: \"kubernetes.io/projected/8632241a-3073-4493-9fab-298607317a96-kube-api-access-jjxzb\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:57:00.147736 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:00.147719 4611 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8632241a-3073-4493-9fab-298607317a96-workload-socket\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:57:00.147736 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:00.147731 4611 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8632241a-3073-4493-9fab-298607317a96-workload-certs\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:57:00.148047 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:00.147743 4611 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8632241a-3073-4493-9fab-298607317a96-istio-token\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:57:00.148047 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:00.147756 4611 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8632241a-3073-4493-9fab-298607317a96-istiod-ca-cert\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:57:00.517836 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:00.517741 4611 generic.go:356] "Generic (PLEG): container finished" podID="8632241a-3073-4493-9fab-298607317a96" containerID="e00691c8cdcf7470c3b6bfae79ebf61a61c888085aad5613b0578654e7a980d0" exitCode=0 Oct 30 20:57:00.517836 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:00.517805 4611 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-a8daf029-openshift-default-79cb595b95-qqft5" Oct 30 20:57:00.518025 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:00.517829 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a8daf029-openshift-default-79cb595b95-qqft5" event={"ID":"8632241a-3073-4493-9fab-298607317a96","Type":"ContainerDied","Data":"e00691c8cdcf7470c3b6bfae79ebf61a61c888085aad5613b0578654e7a980d0"} Oct 30 20:57:00.518025 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:00.517895 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a8daf029-openshift-default-79cb595b95-qqft5" event={"ID":"8632241a-3073-4493-9fab-298607317a96","Type":"ContainerDied","Data":"de5babc5927a5f7c9e5f48ffd1bd60a7388060b6dba267fecd48589c0bf637fa"} Oct 30 20:57:00.518025 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:00.517924 4611 scope.go:122] "RemoveContainer" containerID="e00691c8cdcf7470c3b6bfae79ebf61a61c888085aad5613b0578654e7a980d0" Oct 30 20:57:00.532033 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:00.532014 4611 scope.go:122] "RemoveContainer" containerID="e00691c8cdcf7470c3b6bfae79ebf61a61c888085aad5613b0578654e7a980d0" Oct 30 20:57:00.532346 ip-10-0-133-192 kubenswrapper[4611]: E1030 20:57:00.532325 4611 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e00691c8cdcf7470c3b6bfae79ebf61a61c888085aad5613b0578654e7a980d0\": container with ID starting with e00691c8cdcf7470c3b6bfae79ebf61a61c888085aad5613b0578654e7a980d0 not found: ID does not exist" containerID="e00691c8cdcf7470c3b6bfae79ebf61a61c888085aad5613b0578654e7a980d0" Oct 30 20:57:00.532434 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:00.532352 4611 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e00691c8cdcf7470c3b6bfae79ebf61a61c888085aad5613b0578654e7a980d0"} err="failed to get container status \"e00691c8cdcf7470c3b6bfae79ebf61a61c888085aad5613b0578654e7a980d0\": rpc error: code = NotFound desc = could not find container \"e00691c8cdcf7470c3b6bfae79ebf61a61c888085aad5613b0578654e7a980d0\": container with ID starting with e00691c8cdcf7470c3b6bfae79ebf61a61c888085aad5613b0578654e7a980d0 not found: ID does not exist" Oct 30 20:57:00.549791 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:00.549765 4611 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-a8daf029-openshift-default-79cb595b95-qqft5"] Oct 30 20:57:00.554210 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:00.554181 4611 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-a8daf029-openshift-default-79cb595b95-qqft5"] Oct 30 20:57:00.789460 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:00.789369 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:57:01.613005 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:01.612965 4611 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8632241a-3073-4493-9fab-298607317a96" path="/var/lib/kubelet/pods/8632241a-3073-4493-9fab-298607317a96/volumes" Oct 30 20:57:01.614090 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:01.614057 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:57:01.812255 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:01.812223 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:57:02.281159 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:02.281124 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:57:02.422379 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:02.422323 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:57:03.023108 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:03.023075 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:57:03.043759 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:03.043719 4611 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d728d037-45fkc"] Oct 30 20:57:03.048734 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:03.048704 4611 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d728d037-45fkc"] Oct 30 20:57:03.607196 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:03.607160 4611 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1b15af64-44d3-4e30-8600-732e59af6877" path="/var/lib/kubelet/pods/1b15af64-44d3-4e30-8600-732e59af6877/volumes" Oct 30 20:57:03.676378 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:03.676345 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:57:04.061324 ip-10-0-133-192 kubenswrapper[4611]: E1030 20:57:04.061247 4611 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hwzc6" containerName="extractor" Oct 30 20:57:04.313341 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:04.313250 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:57:04.960148 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:04.960109 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:57:05.587766 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:05.587730 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:57:05.741611 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:05.741575 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:57:06.179668 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:06.175560 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:57:07.004732 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:07.004686 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:57:07.787915 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:07.787817 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:57:08.391398 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:08.391350 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:57:08.996454 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:08.995081 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:57:09.501914 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:09.501880 4611 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-db534b/distinct-2-w2-db534b-kserve-8594c876-g92mh" podUID="955316d6-a9fd-4b75-a8ab-bbf382b776e2" containerName="main" containerID="cri-o://697d51feb2e4144a1e3439ee261d8df6a1f9e97c03d30e623fedfa108d15479c" gracePeriod=45 Oct 30 20:57:09.616369 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:09.616320 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:57:09.632523 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:09.630217 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-db534b/distinct-2-w2-db534b-kserve-8594c876-g92mh" podUID="955316d6-a9fd-4b75-a8ab-bbf382b776e2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": dial tcp 10.133.0.72:8000: connect: connection refused" Oct 30 20:57:09.842168 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:09.842137 4611 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-db534b/distinct-2-w2-db534b-kserve-8594c876-g92mh" Oct 30 20:57:09.851003 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:09.850974 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:57:09.932022 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:09.931986 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/955316d6-a9fd-4b75-a8ab-bbf382b776e2-tls-certs\" (UniqueName: \"kubernetes.io/secret/955316d6-a9fd-4b75-a8ab-bbf382b776e2-tls-certs\") pod \"955316d6-a9fd-4b75-a8ab-bbf382b776e2\" (UID: \"955316d6-a9fd-4b75-a8ab-bbf382b776e2\") " Oct 30 20:57:09.932209 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:09.932079 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/955316d6-a9fd-4b75-a8ab-bbf382b776e2-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/955316d6-a9fd-4b75-a8ab-bbf382b776e2-model-cache\") pod \"955316d6-a9fd-4b75-a8ab-bbf382b776e2\" (UID: \"955316d6-a9fd-4b75-a8ab-bbf382b776e2\") " Oct 30 20:57:09.932279 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:09.932217 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/955316d6-a9fd-4b75-a8ab-bbf382b776e2-kube-api-access-pghmd\" (UniqueName: \"kubernetes.io/projected/955316d6-a9fd-4b75-a8ab-bbf382b776e2-kube-api-access-pghmd\") pod \"955316d6-a9fd-4b75-a8ab-bbf382b776e2\" (UID: \"955316d6-a9fd-4b75-a8ab-bbf382b776e2\") " Oct 30 20:57:09.932279 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:09.932264 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/955316d6-a9fd-4b75-a8ab-bbf382b776e2-home\" (UniqueName: \"kubernetes.io/empty-dir/955316d6-a9fd-4b75-a8ab-bbf382b776e2-home\") pod \"955316d6-a9fd-4b75-a8ab-bbf382b776e2\" (UID: \"955316d6-a9fd-4b75-a8ab-bbf382b776e2\") " Oct 30 20:57:09.932419 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:09.932358 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/955316d6-a9fd-4b75-a8ab-bbf382b776e2-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/955316d6-a9fd-4b75-a8ab-bbf382b776e2-tmp-dir\") pod \"955316d6-a9fd-4b75-a8ab-bbf382b776e2\" (UID: \"955316d6-a9fd-4b75-a8ab-bbf382b776e2\") " Oct 30 20:57:09.932419 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:09.932421 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/955316d6-a9fd-4b75-a8ab-bbf382b776e2-dshm\" (UniqueName: \"kubernetes.io/empty-dir/955316d6-a9fd-4b75-a8ab-bbf382b776e2-dshm\") pod \"955316d6-a9fd-4b75-a8ab-bbf382b776e2\" (UID: \"955316d6-a9fd-4b75-a8ab-bbf382b776e2\") " Oct 30 20:57:09.948554 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:09.943091 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/955316d6-a9fd-4b75-a8ab-bbf382b776e2-model-cache" pod "955316d6-a9fd-4b75-a8ab-bbf382b776e2" (UID: "955316d6-a9fd-4b75-a8ab-bbf382b776e2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:57:09.948554 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:09.943120 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/955316d6-a9fd-4b75-a8ab-bbf382b776e2-home" pod "955316d6-a9fd-4b75-a8ab-bbf382b776e2" (UID: "955316d6-a9fd-4b75-a8ab-bbf382b776e2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:57:09.948554 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:09.943322 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/955316d6-a9fd-4b75-a8ab-bbf382b776e2-tmp-dir" pod "955316d6-a9fd-4b75-a8ab-bbf382b776e2" (UID: "955316d6-a9fd-4b75-a8ab-bbf382b776e2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:57:09.948554 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:09.946086 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/955316d6-a9fd-4b75-a8ab-bbf382b776e2-dshm" pod "955316d6-a9fd-4b75-a8ab-bbf382b776e2" (UID: "955316d6-a9fd-4b75-a8ab-bbf382b776e2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:57:09.948554 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:09.948457 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/955316d6-a9fd-4b75-a8ab-bbf382b776e2-tls-certs" pod "955316d6-a9fd-4b75-a8ab-bbf382b776e2" (UID: "955316d6-a9fd-4b75-a8ab-bbf382b776e2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:57:09.954773 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:09.954749 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/955316d6-a9fd-4b75-a8ab-bbf382b776e2-kube-api-access-pghmd" pod "955316d6-a9fd-4b75-a8ab-bbf382b776e2" (UID: "955316d6-a9fd-4b75-a8ab-bbf382b776e2"). InnerVolumeSpecName "kube-api-access-pghmd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:57:10.024205 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:10.024170 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:57:10.035312 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:10.035282 4611 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/955316d6-a9fd-4b75-a8ab-bbf382b776e2-tmp-dir\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:57:10.035429 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:10.035316 4611 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/955316d6-a9fd-4b75-a8ab-bbf382b776e2-dshm\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:57:10.035429 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:10.035331 4611 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/955316d6-a9fd-4b75-a8ab-bbf382b776e2-tls-certs\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:57:10.035429 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:10.035345 4611 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/955316d6-a9fd-4b75-a8ab-bbf382b776e2-model-cache\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:57:10.035429 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:10.035362 4611 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pghmd\" (UniqueName: \"kubernetes.io/projected/955316d6-a9fd-4b75-a8ab-bbf382b776e2-kube-api-access-pghmd\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:57:10.035429 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:10.035379 4611 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/955316d6-a9fd-4b75-a8ab-bbf382b776e2-home\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:57:10.167912 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:10.167872 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:57:10.326713 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:10.326678 4611 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-8d0462d9-gw-openshift-default-6464c96d5c-vnmsn"] Oct 30 20:57:10.327408 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:10.327389 4611 cpu_manager.go:414] "containerMap: removing container" podUID="94a3b297-c47d-4b2a-8e78-697a605c5481" containerName="payload-processing" Oct 30 20:57:10.327459 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:10.327410 4611 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="94a3b297-c47d-4b2a-8e78-697a605c5481" containerName="payload-processing" Oct 30 20:57:10.327459 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:10.327442 4611 cpu_manager.go:414] "containerMap: removing container" podUID="8632241a-3073-4493-9fab-298607317a96" containerName="istio-proxy" Oct 30 20:57:10.327459 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:10.327448 4611 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8632241a-3073-4493-9fab-298607317a96" containerName="istio-proxy" Oct 30 20:57:10.327578 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:10.327461 4611 cpu_manager.go:414] "containerMap: removing container" podUID="b305f151-d272-46e8-af32-5f1274ffce1c" containerName="payload-pre-processing" Oct 30 20:57:10.327578 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:10.327467 4611 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b305f151-d272-46e8-af32-5f1274ffce1c" containerName="payload-pre-processing" Oct 30 20:57:10.327578 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:10.327483 4611 cpu_manager.go:414] "containerMap: removing container" podUID="955316d6-a9fd-4b75-a8ab-bbf382b776e2" containerName="main" Oct 30 20:57:10.327578 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:10.327488 4611 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="955316d6-a9fd-4b75-a8ab-bbf382b776e2" containerName="main" Oct 30 20:57:10.327722 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:10.327614 4611 memory_manager.go:365] "RemoveStaleState removing state" podUID="94a3b297-c47d-4b2a-8e78-697a605c5481" containerName="payload-processing" Oct 30 20:57:10.327722 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:10.327630 4611 memory_manager.go:365] "RemoveStaleState removing state" podUID="8632241a-3073-4493-9fab-298607317a96" containerName="istio-proxy" Oct 30 20:57:10.327722 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:10.327638 4611 memory_manager.go:365] "RemoveStaleState removing state" podUID="b305f151-d272-46e8-af32-5f1274ffce1c" containerName="payload-pre-processing" Oct 30 20:57:10.327722 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:10.327645 4611 memory_manager.go:365] "RemoveStaleState removing state" podUID="955316d6-a9fd-4b75-a8ab-bbf382b776e2" containerName="main" Oct 30 20:57:10.331819 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:10.331792 4611 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-8d0462d9-gw-openshift-default-6464c96d5c-vnmsn" Oct 30 20:57:10.386647 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:10.386391 4611 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-8d0462d9-gw-openshift-default-6464c96d5c-vnmsn"] Oct 30 20:57:10.390099 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:10.390072 4611 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-8d0462d9-gw-openshift-default-dockercfg-sj9bs\"" Oct 30 20:57:10.502589 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:10.501532 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22a8ecc8-ac35-4744-8c73-e196caf73f52-istio-envoy\") pod \"e2e-migrate-8d0462d9-gw-openshift-default-6464c96d5c-vnmsn\" (UID: \"22a8ecc8-ac35-4744-8c73-e196caf73f52\") " pod="openshift-ingress/e2e-migrate-8d0462d9-gw-openshift-default-6464c96d5c-vnmsn" Oct 30 20:57:10.502589 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:10.501687 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22a8ecc8-ac35-4744-8c73-e196caf73f52-istio-data\") pod \"e2e-migrate-8d0462d9-gw-openshift-default-6464c96d5c-vnmsn\" (UID: \"22a8ecc8-ac35-4744-8c73-e196caf73f52\") " pod="openshift-ingress/e2e-migrate-8d0462d9-gw-openshift-default-6464c96d5c-vnmsn" Oct 30 20:57:10.502589 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:10.501816 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22a8ecc8-ac35-4744-8c73-e196caf73f52-credential-socket\") pod \"e2e-migrate-8d0462d9-gw-openshift-default-6464c96d5c-vnmsn\" (UID: \"22a8ecc8-ac35-4744-8c73-e196caf73f52\") " pod="openshift-ingress/e2e-migrate-8d0462d9-gw-openshift-default-6464c96d5c-vnmsn" Oct 30 20:57:10.502589 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:10.501851 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22a8ecc8-ac35-4744-8c73-e196caf73f52-workload-certs\") pod \"e2e-migrate-8d0462d9-gw-openshift-default-6464c96d5c-vnmsn\" (UID: \"22a8ecc8-ac35-4744-8c73-e196caf73f52\") " pod="openshift-ingress/e2e-migrate-8d0462d9-gw-openshift-default-6464c96d5c-vnmsn" Oct 30 20:57:10.502589 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:10.501897 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22a8ecc8-ac35-4744-8c73-e196caf73f52-istiod-ca-cert\") pod \"e2e-migrate-8d0462d9-gw-openshift-default-6464c96d5c-vnmsn\" (UID: \"22a8ecc8-ac35-4744-8c73-e196caf73f52\") " pod="openshift-ingress/e2e-migrate-8d0462d9-gw-openshift-default-6464c96d5c-vnmsn" Oct 30 20:57:10.502589 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:10.501928 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l7jdc\" (UniqueName: \"kubernetes.io/projected/22a8ecc8-ac35-4744-8c73-e196caf73f52-kube-api-access-l7jdc\") pod \"e2e-migrate-8d0462d9-gw-openshift-default-6464c96d5c-vnmsn\" (UID: \"22a8ecc8-ac35-4744-8c73-e196caf73f52\") " pod="openshift-ingress/e2e-migrate-8d0462d9-gw-openshift-default-6464c96d5c-vnmsn" Oct 30 20:57:10.502589 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:10.501960 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22a8ecc8-ac35-4744-8c73-e196caf73f52-istio-token\") pod \"e2e-migrate-8d0462d9-gw-openshift-default-6464c96d5c-vnmsn\" (UID: \"22a8ecc8-ac35-4744-8c73-e196caf73f52\") " pod="openshift-ingress/e2e-migrate-8d0462d9-gw-openshift-default-6464c96d5c-vnmsn" Oct 30 20:57:10.502589 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:10.502164 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22a8ecc8-ac35-4744-8c73-e196caf73f52-workload-socket\") pod \"e2e-migrate-8d0462d9-gw-openshift-default-6464c96d5c-vnmsn\" (UID: \"22a8ecc8-ac35-4744-8c73-e196caf73f52\") " pod="openshift-ingress/e2e-migrate-8d0462d9-gw-openshift-default-6464c96d5c-vnmsn" Oct 30 20:57:10.502589 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:10.502215 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22a8ecc8-ac35-4744-8c73-e196caf73f52-istio-podinfo\") pod \"e2e-migrate-8d0462d9-gw-openshift-default-6464c96d5c-vnmsn\" (UID: \"22a8ecc8-ac35-4744-8c73-e196caf73f52\") " pod="openshift-ingress/e2e-migrate-8d0462d9-gw-openshift-default-6464c96d5c-vnmsn" Oct 30 20:57:10.517537 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:10.509439 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:57:10.604031 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:10.603997 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22a8ecc8-ac35-4744-8c73-e196caf73f52-istio-data\") pod \"e2e-migrate-8d0462d9-gw-openshift-default-6464c96d5c-vnmsn\" (UID: \"22a8ecc8-ac35-4744-8c73-e196caf73f52\") " pod="openshift-ingress/e2e-migrate-8d0462d9-gw-openshift-default-6464c96d5c-vnmsn" Oct 30 20:57:10.604657 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:10.604626 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22a8ecc8-ac35-4744-8c73-e196caf73f52-istio-data\") pod \"e2e-migrate-8d0462d9-gw-openshift-default-6464c96d5c-vnmsn\" (UID: \"22a8ecc8-ac35-4744-8c73-e196caf73f52\") " pod="openshift-ingress/e2e-migrate-8d0462d9-gw-openshift-default-6464c96d5c-vnmsn" Oct 30 20:57:10.604789 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:10.604679 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22a8ecc8-ac35-4744-8c73-e196caf73f52-credential-socket\") pod \"e2e-migrate-8d0462d9-gw-openshift-default-6464c96d5c-vnmsn\" (UID: \"22a8ecc8-ac35-4744-8c73-e196caf73f52\") " pod="openshift-ingress/e2e-migrate-8d0462d9-gw-openshift-default-6464c96d5c-vnmsn" Oct 30 20:57:10.604789 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:10.604714 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22a8ecc8-ac35-4744-8c73-e196caf73f52-workload-certs\") pod \"e2e-migrate-8d0462d9-gw-openshift-default-6464c96d5c-vnmsn\" (UID: \"22a8ecc8-ac35-4744-8c73-e196caf73f52\") " pod="openshift-ingress/e2e-migrate-8d0462d9-gw-openshift-default-6464c96d5c-vnmsn" Oct 30 20:57:10.604789 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:10.604744 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22a8ecc8-ac35-4744-8c73-e196caf73f52-istiod-ca-cert\") pod \"e2e-migrate-8d0462d9-gw-openshift-default-6464c96d5c-vnmsn\" (UID: \"22a8ecc8-ac35-4744-8c73-e196caf73f52\") " pod="openshift-ingress/e2e-migrate-8d0462d9-gw-openshift-default-6464c96d5c-vnmsn" Oct 30 20:57:10.604789 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:10.604773 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l7jdc\" (UniqueName: \"kubernetes.io/projected/22a8ecc8-ac35-4744-8c73-e196caf73f52-kube-api-access-l7jdc\") pod \"e2e-migrate-8d0462d9-gw-openshift-default-6464c96d5c-vnmsn\" (UID: \"22a8ecc8-ac35-4744-8c73-e196caf73f52\") " pod="openshift-ingress/e2e-migrate-8d0462d9-gw-openshift-default-6464c96d5c-vnmsn" Oct 30 20:57:10.605033 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:10.604806 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22a8ecc8-ac35-4744-8c73-e196caf73f52-istio-token\") pod \"e2e-migrate-8d0462d9-gw-openshift-default-6464c96d5c-vnmsn\" (UID: \"22a8ecc8-ac35-4744-8c73-e196caf73f52\") " pod="openshift-ingress/e2e-migrate-8d0462d9-gw-openshift-default-6464c96d5c-vnmsn" Oct 30 20:57:10.605033 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:10.604981 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22a8ecc8-ac35-4744-8c73-e196caf73f52-credential-socket\") pod \"e2e-migrate-8d0462d9-gw-openshift-default-6464c96d5c-vnmsn\" (UID: \"22a8ecc8-ac35-4744-8c73-e196caf73f52\") " pod="openshift-ingress/e2e-migrate-8d0462d9-gw-openshift-default-6464c96d5c-vnmsn" Oct 30 20:57:10.605033 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:10.604994 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22a8ecc8-ac35-4744-8c73-e196caf73f52-workload-certs\") pod \"e2e-migrate-8d0462d9-gw-openshift-default-6464c96d5c-vnmsn\" (UID: \"22a8ecc8-ac35-4744-8c73-e196caf73f52\") " pod="openshift-ingress/e2e-migrate-8d0462d9-gw-openshift-default-6464c96d5c-vnmsn" Oct 30 20:57:10.605033 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:10.604995 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22a8ecc8-ac35-4744-8c73-e196caf73f52-workload-socket\") pod \"e2e-migrate-8d0462d9-gw-openshift-default-6464c96d5c-vnmsn\" (UID: \"22a8ecc8-ac35-4744-8c73-e196caf73f52\") " pod="openshift-ingress/e2e-migrate-8d0462d9-gw-openshift-default-6464c96d5c-vnmsn" Oct 30 20:57:10.605249 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:10.605054 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22a8ecc8-ac35-4744-8c73-e196caf73f52-istio-podinfo\") pod \"e2e-migrate-8d0462d9-gw-openshift-default-6464c96d5c-vnmsn\" (UID: \"22a8ecc8-ac35-4744-8c73-e196caf73f52\") " pod="openshift-ingress/e2e-migrate-8d0462d9-gw-openshift-default-6464c96d5c-vnmsn" Oct 30 20:57:10.605249 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:10.605100 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22a8ecc8-ac35-4744-8c73-e196caf73f52-istio-envoy\") pod \"e2e-migrate-8d0462d9-gw-openshift-default-6464c96d5c-vnmsn\" (UID: \"22a8ecc8-ac35-4744-8c73-e196caf73f52\") " pod="openshift-ingress/e2e-migrate-8d0462d9-gw-openshift-default-6464c96d5c-vnmsn" Oct 30 20:57:10.605249 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:10.605237 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22a8ecc8-ac35-4744-8c73-e196caf73f52-workload-socket\") pod \"e2e-migrate-8d0462d9-gw-openshift-default-6464c96d5c-vnmsn\" (UID: \"22a8ecc8-ac35-4744-8c73-e196caf73f52\") " pod="openshift-ingress/e2e-migrate-8d0462d9-gw-openshift-default-6464c96d5c-vnmsn" Oct 30 20:57:10.606344 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:10.605763 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22a8ecc8-ac35-4744-8c73-e196caf73f52-istiod-ca-cert\") pod \"e2e-migrate-8d0462d9-gw-openshift-default-6464c96d5c-vnmsn\" (UID: \"22a8ecc8-ac35-4744-8c73-e196caf73f52\") " pod="openshift-ingress/e2e-migrate-8d0462d9-gw-openshift-default-6464c96d5c-vnmsn" Oct 30 20:57:10.610771 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:10.610744 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22a8ecc8-ac35-4744-8c73-e196caf73f52-istio-envoy\") pod \"e2e-migrate-8d0462d9-gw-openshift-default-6464c96d5c-vnmsn\" (UID: \"22a8ecc8-ac35-4744-8c73-e196caf73f52\") " pod="openshift-ingress/e2e-migrate-8d0462d9-gw-openshift-default-6464c96d5c-vnmsn" Oct 30 20:57:10.610878 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:10.610854 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22a8ecc8-ac35-4744-8c73-e196caf73f52-istio-podinfo\") pod \"e2e-migrate-8d0462d9-gw-openshift-default-6464c96d5c-vnmsn\" (UID: \"22a8ecc8-ac35-4744-8c73-e196caf73f52\") " pod="openshift-ingress/e2e-migrate-8d0462d9-gw-openshift-default-6464c96d5c-vnmsn" Oct 30 20:57:10.650158 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:10.638050 4611 generic.go:356] "Generic (PLEG): container finished" podID="955316d6-a9fd-4b75-a8ab-bbf382b776e2" containerID="697d51feb2e4144a1e3439ee261d8df6a1f9e97c03d30e623fedfa108d15479c" exitCode=0 Oct 30 20:57:10.650158 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:10.638183 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-db534b/distinct-2-w2-db534b-kserve-8594c876-g92mh" event={"ID":"955316d6-a9fd-4b75-a8ab-bbf382b776e2","Type":"ContainerDied","Data":"697d51feb2e4144a1e3439ee261d8df6a1f9e97c03d30e623fedfa108d15479c"} Oct 30 20:57:10.650158 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:10.638216 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-db534b/distinct-2-w2-db534b-kserve-8594c876-g92mh" event={"ID":"955316d6-a9fd-4b75-a8ab-bbf382b776e2","Type":"ContainerDied","Data":"577337b5f2ab13ba375a980ecd8e69f45abc26b876911385314607b4663f77a3"} Oct 30 20:57:10.650158 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:10.638238 4611 scope.go:122] "RemoveContainer" containerID="697d51feb2e4144a1e3439ee261d8df6a1f9e97c03d30e623fedfa108d15479c" Oct 30 20:57:10.650158 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:10.638384 4611 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-db534b/distinct-2-w2-db534b-kserve-8594c876-g92mh" Oct 30 20:57:10.656606 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:10.655254 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22a8ecc8-ac35-4744-8c73-e196caf73f52-istio-token\") pod \"e2e-migrate-8d0462d9-gw-openshift-default-6464c96d5c-vnmsn\" (UID: \"22a8ecc8-ac35-4744-8c73-e196caf73f52\") " pod="openshift-ingress/e2e-migrate-8d0462d9-gw-openshift-default-6464c96d5c-vnmsn" Oct 30 20:57:10.669784 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:10.669574 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:57:10.725561 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:10.720670 4611 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-db534b/distinct-2-w2-db534b-kserve-8594c876-g92mh"] Oct 30 20:57:10.725561 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:10.722020 4611 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-db534b/distinct-2-w2-db534b-kserve-8594c876-g92mh"] Oct 30 20:57:10.732086 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:10.727888 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l7jdc\" (UniqueName: \"kubernetes.io/projected/22a8ecc8-ac35-4744-8c73-e196caf73f52-kube-api-access-l7jdc\") pod \"e2e-migrate-8d0462d9-gw-openshift-default-6464c96d5c-vnmsn\" (UID: \"22a8ecc8-ac35-4744-8c73-e196caf73f52\") " pod="openshift-ingress/e2e-migrate-8d0462d9-gw-openshift-default-6464c96d5c-vnmsn" Oct 30 20:57:10.734080 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:10.734060 4611 scope.go:122] "RemoveContainer" containerID="697d51feb2e4144a1e3439ee261d8df6a1f9e97c03d30e623fedfa108d15479c" Oct 30 20:57:10.734449 ip-10-0-133-192 kubenswrapper[4611]: E1030 20:57:10.734425 4611 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"697d51feb2e4144a1e3439ee261d8df6a1f9e97c03d30e623fedfa108d15479c\": container with ID starting with 697d51feb2e4144a1e3439ee261d8df6a1f9e97c03d30e623fedfa108d15479c not found: ID does not exist" containerID="697d51feb2e4144a1e3439ee261d8df6a1f9e97c03d30e623fedfa108d15479c" Oct 30 20:57:10.734603 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:10.734574 4611 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"697d51feb2e4144a1e3439ee261d8df6a1f9e97c03d30e623fedfa108d15479c"} err="failed to get container status \"697d51feb2e4144a1e3439ee261d8df6a1f9e97c03d30e623fedfa108d15479c\": rpc error: code = NotFound desc = could not find container \"697d51feb2e4144a1e3439ee261d8df6a1f9e97c03d30e623fedfa108d15479c\": container with ID starting with 697d51feb2e4144a1e3439ee261d8df6a1f9e97c03d30e623fedfa108d15479c not found: ID does not exist" Oct 30 20:57:11.002474 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:11.002436 4611 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-8d0462d9-gw-openshift-default-6464c96d5c-vnmsn" Oct 30 20:57:11.355107 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:57:11.355070 4611 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22a8ecc8_ac35_4744_8c73_e196caf73f52.slice/crio-ed86896d45b50870e66fef8809f2e30aaccdb07b6eae3e5afd2c2e62108cb887 WatchSource:0}: Error finding container ed86896d45b50870e66fef8809f2e30aaccdb07b6eae3e5afd2c2e62108cb887: Status 404 returned error can't find the container with id ed86896d45b50870e66fef8809f2e30aaccdb07b6eae3e5afd2c2e62108cb887 Oct 30 20:57:11.355279 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:11.355117 4611 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-8d0462d9-gw-openshift-default-6464c96d5c-vnmsn"] Oct 30 20:57:11.357902 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:11.357869 4611 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:57:11.358002 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:11.357934 4611 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:57:11.358002 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:11.357961 4611 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:57:11.471454 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:11.471414 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:57:11.611886 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:11.611796 4611 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="955316d6-a9fd-4b75-a8ab-bbf382b776e2" path="/var/lib/kubelet/pods/955316d6-a9fd-4b75-a8ab-bbf382b776e2/volumes" Oct 30 20:57:11.629058 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:11.629016 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:57:11.666939 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:11.666899 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-8d0462d9-gw-openshift-default-6464c96d5c-vnmsn" event={"ID":"22a8ecc8-ac35-4744-8c73-e196caf73f52","Type":"ContainerStarted","Data":"c33c1bea362127a0b77552737a86f45c0efc7072471dbe76fee6bd528248b101"} Oct 30 20:57:11.667257 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:11.667234 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-8d0462d9-gw-openshift-default-6464c96d5c-vnmsn" event={"ID":"22a8ecc8-ac35-4744-8c73-e196caf73f52","Type":"ContainerStarted","Data":"ed86896d45b50870e66fef8809f2e30aaccdb07b6eae3e5afd2c2e62108cb887"} Oct 30 20:57:12.003735 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:12.003652 4611 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-8d0462d9-gw-openshift-default-6464c96d5c-vnmsn" Oct 30 20:57:12.010590 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:12.010551 4611 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-8d0462d9-gw-openshift-default-6464c96d5c-vnmsn" Oct 30 20:57:12.039344 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:12.039284 4611 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-8d0462d9-gw-openshift-default-6464c96d5c-vnmsn" podStartSLOduration=2.039265257 podStartE2EDuration="2.039265257s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:57:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:57:11.707351103 +0000 UTC m=+1086.241377732" watchObservedRunningTime="2026-10-30 20:57:12.039265257 +0000 UTC m=+1086.573291874" Oct 30 20:57:12.313984 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:12.313906 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:57:12.598418 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:12.598322 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:57:12.677174 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:12.677138 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-8d0462d9-gw-openshift-default-6464c96d5c-vnmsn" Oct 30 20:57:12.677973 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:12.677952 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-8d0462d9-gw-openshift-default-6464c96d5c-vnmsn" Oct 30 20:57:13.050229 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:13.050183 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:57:13.774605 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:13.774547 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:57:13.930024 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:13.929991 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:57:14.040777 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:14.040693 4611 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-87079fdf-mf2f5"] Oct 30 20:57:14.047571 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:14.047544 4611 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-87079fdf-mf2f5"] Oct 30 20:57:14.258659 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:14.258622 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:57:14.768086 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:14.768023 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:57:15.378037 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:15.377996 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:57:15.620411 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:15.620370 4611 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="58075892-0276-4403-aa80-69c374f8283d" path="/var/lib/kubelet/pods/58075892-0276-4403-aa80-69c374f8283d/volumes" Oct 30 20:57:15.753137 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:15.753045 4611 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-2e92a84c-openshift-default-8598669b69-kq4pw"] Oct 30 20:57:15.753316 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:15.753287 4611 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-2e92a84c-openshift-default-8598669b69-kq4pw" podUID="2e828845-f5af-44d1-8db6-5737ffb966f5" containerName="istio-proxy" containerID="cri-o://a95bd2542908f91dbbcbadca062ba8c97680b1d829ad6e7187c877efeff2b071" gracePeriod=30 Oct 30 20:57:15.829991 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:15.829939 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:57:16.205334 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:16.205294 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:57:16.924230 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:16.924193 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:57:17.711462 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:17.710421 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:57:18.409170 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:18.409130 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:57:19.017153 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:19.016964 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:57:19.719355 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:19.719321 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:57:19.880954 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:19.880916 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:57:20.576103 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:20.576067 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:57:21.032168 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:21.032138 4611 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-2e92a84c-openshift-default-8598669b69-kq4pw" Oct 30 20:57:21.047821 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:21.047780 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e828845-f5af-44d1-8db6-5737ffb966f5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e828845-f5af-44d1-8db6-5737ffb966f5-istio-data\") pod \"2e828845-f5af-44d1-8db6-5737ffb966f5\" (UID: \"2e828845-f5af-44d1-8db6-5737ffb966f5\") " Oct 30 20:57:21.048119 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:21.047885 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2e828845-f5af-44d1-8db6-5737ffb966f5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e828845-f5af-44d1-8db6-5737ffb966f5-istiod-ca-cert\") pod \"2e828845-f5af-44d1-8db6-5737ffb966f5\" (UID: \"2e828845-f5af-44d1-8db6-5737ffb966f5\") " Oct 30 20:57:21.048214 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:21.048187 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e828845-f5af-44d1-8db6-5737ffb966f5-istio-data" pod "2e828845-f5af-44d1-8db6-5737ffb966f5" (UID: "2e828845-f5af-44d1-8db6-5737ffb966f5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:57:21.048352 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:21.048292 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2e828845-f5af-44d1-8db6-5737ffb966f5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e828845-f5af-44d1-8db6-5737ffb966f5-istio-podinfo\") pod \"2e828845-f5af-44d1-8db6-5737ffb966f5\" (UID: \"2e828845-f5af-44d1-8db6-5737ffb966f5\") " Oct 30 20:57:21.048402 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:21.048365 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e828845-f5af-44d1-8db6-5737ffb966f5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e828845-f5af-44d1-8db6-5737ffb966f5-credential-socket\") pod \"2e828845-f5af-44d1-8db6-5737ffb966f5\" (UID: \"2e828845-f5af-44d1-8db6-5737ffb966f5\") " Oct 30 20:57:21.048402 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:21.048374 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e828845-f5af-44d1-8db6-5737ffb966f5-istiod-ca-cert" pod "2e828845-f5af-44d1-8db6-5737ffb966f5" (UID: "2e828845-f5af-44d1-8db6-5737ffb966f5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:57:21.048552 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:21.048478 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e828845-f5af-44d1-8db6-5737ffb966f5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e828845-f5af-44d1-8db6-5737ffb966f5-workload-socket\") pod \"2e828845-f5af-44d1-8db6-5737ffb966f5\" (UID: \"2e828845-f5af-44d1-8db6-5737ffb966f5\") " Oct 30 20:57:21.048641 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:21.048626 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e828845-f5af-44d1-8db6-5737ffb966f5-kube-api-access-wdwxc\" (UniqueName: \"kubernetes.io/projected/2e828845-f5af-44d1-8db6-5737ffb966f5-kube-api-access-wdwxc\") pod \"2e828845-f5af-44d1-8db6-5737ffb966f5\" (UID: \"2e828845-f5af-44d1-8db6-5737ffb966f5\") " Oct 30 20:57:21.048702 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:21.048687 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e828845-f5af-44d1-8db6-5737ffb966f5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e828845-f5af-44d1-8db6-5737ffb966f5-istio-envoy\") pod \"2e828845-f5af-44d1-8db6-5737ffb966f5\" (UID: \"2e828845-f5af-44d1-8db6-5737ffb966f5\") " Oct 30 20:57:21.048762 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:21.048724 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e828845-f5af-44d1-8db6-5737ffb966f5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e828845-f5af-44d1-8db6-5737ffb966f5-workload-certs\") pod \"2e828845-f5af-44d1-8db6-5737ffb966f5\" (UID: \"2e828845-f5af-44d1-8db6-5737ffb966f5\") " Oct 30 20:57:21.048762 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:21.048748 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e828845-f5af-44d1-8db6-5737ffb966f5-workload-socket" pod "2e828845-f5af-44d1-8db6-5737ffb966f5" (UID: "2e828845-f5af-44d1-8db6-5737ffb966f5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:57:21.048884 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:21.048771 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e828845-f5af-44d1-8db6-5737ffb966f5-istio-token\" (UniqueName: \"kubernetes.io/projected/2e828845-f5af-44d1-8db6-5737ffb966f5-istio-token\") pod \"2e828845-f5af-44d1-8db6-5737ffb966f5\" (UID: \"2e828845-f5af-44d1-8db6-5737ffb966f5\") " Oct 30 20:57:21.049705 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:21.049665 4611 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e828845-f5af-44d1-8db6-5737ffb966f5-istio-data\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:57:21.049705 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:21.049691 4611 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e828845-f5af-44d1-8db6-5737ffb966f5-istiod-ca-cert\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:57:21.049705 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:21.049706 4611 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e828845-f5af-44d1-8db6-5737ffb966f5-workload-socket\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:57:21.050701 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:21.050663 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e828845-f5af-44d1-8db6-5737ffb966f5-credential-socket" pod "2e828845-f5af-44d1-8db6-5737ffb966f5" (UID: "2e828845-f5af-44d1-8db6-5737ffb966f5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:57:21.051437 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:21.051410 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e828845-f5af-44d1-8db6-5737ffb966f5-workload-certs" pod "2e828845-f5af-44d1-8db6-5737ffb966f5" (UID: "2e828845-f5af-44d1-8db6-5737ffb966f5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:57:21.068725 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:21.068647 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e828845-f5af-44d1-8db6-5737ffb966f5-kube-api-access-wdwxc" pod "2e828845-f5af-44d1-8db6-5737ffb966f5" (UID: "2e828845-f5af-44d1-8db6-5737ffb966f5"). InnerVolumeSpecName "kube-api-access-wdwxc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:57:21.070603 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:21.070153 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e828845-f5af-44d1-8db6-5737ffb966f5-istio-envoy" pod "2e828845-f5af-44d1-8db6-5737ffb966f5" (UID: "2e828845-f5af-44d1-8db6-5737ffb966f5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:57:21.074137 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:21.072290 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2e828845-f5af-44d1-8db6-5737ffb966f5-istio-podinfo" pod "2e828845-f5af-44d1-8db6-5737ffb966f5" (UID: "2e828845-f5af-44d1-8db6-5737ffb966f5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:57:21.090321 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:21.090275 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e828845-f5af-44d1-8db6-5737ffb966f5-istio-token" pod "2e828845-f5af-44d1-8db6-5737ffb966f5" (UID: "2e828845-f5af-44d1-8db6-5737ffb966f5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:57:21.151247 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:21.151132 4611 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e828845-f5af-44d1-8db6-5737ffb966f5-istio-envoy\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:57:21.151247 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:21.151166 4611 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e828845-f5af-44d1-8db6-5737ffb966f5-workload-certs\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:57:21.151247 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:21.151178 4611 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e828845-f5af-44d1-8db6-5737ffb966f5-istio-token\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:57:21.151247 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:21.151186 4611 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e828845-f5af-44d1-8db6-5737ffb966f5-istio-podinfo\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:57:21.151247 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:21.151195 4611 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e828845-f5af-44d1-8db6-5737ffb966f5-credential-socket\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:57:21.151247 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:21.151203 4611 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wdwxc\" (UniqueName: \"kubernetes.io/projected/2e828845-f5af-44d1-8db6-5737ffb966f5-kube-api-access-wdwxc\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:57:21.388268 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:21.388162 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:57:21.817450 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:21.817414 4611 generic.go:356] "Generic (PLEG): container finished" podID="2e828845-f5af-44d1-8db6-5737ffb966f5" containerID="a95bd2542908f91dbbcbadca062ba8c97680b1d829ad6e7187c877efeff2b071" exitCode=0 Oct 30 20:57:21.817671 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:21.817462 4611 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-2e92a84c-openshift-default-8598669b69-kq4pw" Oct 30 20:57:21.817671 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:21.817494 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-2e92a84c-openshift-default-8598669b69-kq4pw" event={"ID":"2e828845-f5af-44d1-8db6-5737ffb966f5","Type":"ContainerDied","Data":"a95bd2542908f91dbbcbadca062ba8c97680b1d829ad6e7187c877efeff2b071"} Oct 30 20:57:21.817671 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:21.817569 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-2e92a84c-openshift-default-8598669b69-kq4pw" event={"ID":"2e828845-f5af-44d1-8db6-5737ffb966f5","Type":"ContainerDied","Data":"5085471df97d3ccd0319d629d6f2ed1ce61fc5c3cc1a24e4014e52feee40681d"} Oct 30 20:57:21.817671 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:21.817609 4611 scope.go:122] "RemoveContainer" containerID="a95bd2542908f91dbbcbadca062ba8c97680b1d829ad6e7187c877efeff2b071" Oct 30 20:57:21.829551 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:21.829529 4611 scope.go:122] "RemoveContainer" containerID="a95bd2542908f91dbbcbadca062ba8c97680b1d829ad6e7187c877efeff2b071" Oct 30 20:57:21.829856 ip-10-0-133-192 kubenswrapper[4611]: E1030 20:57:21.829830 4611 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a95bd2542908f91dbbcbadca062ba8c97680b1d829ad6e7187c877efeff2b071\": container with ID starting with a95bd2542908f91dbbcbadca062ba8c97680b1d829ad6e7187c877efeff2b071 not found: ID does not exist" containerID="a95bd2542908f91dbbcbadca062ba8c97680b1d829ad6e7187c877efeff2b071" Oct 30 20:57:21.829973 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:21.829860 4611 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a95bd2542908f91dbbcbadca062ba8c97680b1d829ad6e7187c877efeff2b071"} err="failed to get container status \"a95bd2542908f91dbbcbadca062ba8c97680b1d829ad6e7187c877efeff2b071\": rpc error: code = NotFound desc = could not find container \"a95bd2542908f91dbbcbadca062ba8c97680b1d829ad6e7187c877efeff2b071\": container with ID starting with a95bd2542908f91dbbcbadca062ba8c97680b1d829ad6e7187c877efeff2b071 not found: ID does not exist" Oct 30 20:57:21.842349 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:21.842327 4611 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-2e92a84c-openshift-default-8598669b69-kq4pw"] Oct 30 20:57:21.855385 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:21.855358 4611 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-2e92a84c-openshift-default-8598669b69-kq4pw"] Oct 30 20:57:22.214488 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:22.214451 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:57:23.040530 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:23.039834 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:57:23.677889 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:23.677854 4611 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2e828845-f5af-44d1-8db6-5737ffb966f5" path="/var/lib/kubelet/pods/2e828845-f5af-44d1-8db6-5737ffb966f5/volumes" Oct 30 20:57:23.829529 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:23.823577 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:57:24.309766 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:24.309719 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:57:24.627635 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:24.626629 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:57:25.307533 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:25.291450 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:57:25.536479 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:25.536413 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:57:26.411533 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:26.403805 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b8aee1/distinct-2-w1-b8aee1-kserve-6bcdd59797-cx6hr" podUID="286eab7a-81a7-4768-a7f6-be6b3720028a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:57:26.411533 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:26.404853 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cb60a2/distinct-w3-cb60a2-kserve-5bb76dc74-86fds" podUID="52e301d9-f378-4e54-9515-01a6680d9fd7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:57:26.490542 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:26.458608 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f2ed1e/distinct-w4-f2ed1e-kserve-57cff9dcf8-8d5zl" podUID="653e3ec7-2147-4813-8946-1f25cfa425f6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:57:26.556587 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:26.554162 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:57:26.646533 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:26.646130 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad7ff0/premium-w0-ad7ff0-kserve-5485d4b948-bpmgf" podUID="53cbd692-e9f8-46ba-8fde-3e5215a38904" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:57:26.669617 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:26.660698 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b8aee1/premium-w1-b8aee1-kserve-786d575599-htc8d" podUID="77368afe-edc8-4fa5-8010-d321965fb38e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded" Oct 30 20:57:26.669617 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:26.661204 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-3d909d/distinct-2-w6-3d909d-kserve-59c5d5c595-cjg2m" podUID="e2b4c221-ee94-4131-abd8-2313647c641e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": context deadline exceeded" Oct 30 20:57:26.669617 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:26.661564 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-3d909d/premium-w6-3d909d-kserve-765648946f-2qvd2" podUID="fca5d697-109d-4e90-ab41-c40ad31af874" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded" Oct 30 20:57:26.692962 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:26.692924 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-14cca2/distinct-w5-14cca2-kserve-587f6996b8-sgf6h" podUID="02b652c3-0724-4a50-9e95-2a9890cc97c4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:57:26.722642 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:26.722598 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-14cca2/premium-w5-14cca2-kserve-6878777857-wpjtj" podUID="b99eb76b-d1c6-49e9-9809-c351c0f5a4de" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:57:26.723268 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:26.723237 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b8aee1/distinct-w1-b8aee1-kserve-5b857c55cd-x5b7b" podUID="1599e588-def8-42e1-b81d-ad0aaf119818" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:57:26.780650 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:26.780608 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cb60a2/simulator-w3-cb60a2-kserve-6fd74b9d8b-vqfzf" podUID="711b167d-4e66-40d6-be75-33ede11f42c5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:57:26.852238 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:26.852185 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:57:27.158358 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:27.158320 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:57:27.322155 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:27.322118 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:57:27.408161 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:27.405260 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cb60a2/distinct-w3-cb60a2-kserve-5bb76dc74-86fds" podUID="52e301d9-f378-4e54-9515-01a6680d9fd7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:57:27.408161 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:27.406665 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-cb60a2/distinct-w3-cb60a2-kserve-5bb76dc74-86fds" Oct 30 20:57:27.484708 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:27.484601 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-f2ed1e/distinct-w4-f2ed1e-kserve-57cff9dcf8-8d5zl" podUID="653e3ec7-2147-4813-8946-1f25cfa425f6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:57:27.523640 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:27.506996 4611 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-ad7ff0/premium-w0-ad7ff0-kserve-5485d4b948-bpmgf" podUID="53cbd692-e9f8-46ba-8fde-3e5215a38904" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:57:27.523640 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:27.507296 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-f2ed1e/distinct-w4-f2ed1e-kserve-57cff9dcf8-8d5zl" Oct 30 20:57:27.670531 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:27.669618 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ad7ff0/premium-w0-ad7ff0-kserve-5485d4b948-bpmgf" podUID="53cbd692-e9f8-46ba-8fde-3e5215a38904" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded" Oct 30 20:57:27.690566 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:27.690081 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-3d909d/premium-w6-3d909d-kserve-765648946f-2qvd2" podUID="fca5d697-109d-4e90-ab41-c40ad31af874" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:57:27.754334 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:27.722079 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b8aee1/premium-w1-b8aee1-kserve-786d575599-htc8d" podUID="77368afe-edc8-4fa5-8010-d321965fb38e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:57:27.754334 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:27.723447 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-14cca2/premium-w5-14cca2-kserve-6878777857-wpjtj" podUID="b99eb76b-d1c6-49e9-9809-c351c0f5a4de" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:57:27.754334 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:27.728670 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b8aee1/distinct-w1-b8aee1-kserve-5b857c55cd-x5b7b" podUID="1599e588-def8-42e1-b81d-ad0aaf119818" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:57:27.811312 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:27.811137 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:57:27.841530 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:27.831367 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cb60a2/simulator-w3-cb60a2-kserve-6fd74b9d8b-vqfzf" podUID="711b167d-4e66-40d6-be75-33ede11f42c5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 20:57:27.880473 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:27.862640 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ad7ff0/premium-w0-ad7ff0-kserve-5485d4b948-bpmgf" Oct 30 20:57:27.885455 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:27.885428 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-3d909d/premium-w6-3d909d-kserve-765648946f-2qvd2" Oct 30 20:57:27.885707 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:27.885690 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b8aee1/premium-w1-b8aee1-kserve-786d575599-htc8d" Oct 30 20:57:27.885847 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:27.885835 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-14cca2/premium-w5-14cca2-kserve-6878777857-wpjtj" Oct 30 20:57:27.885965 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:27.885954 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b8aee1/distinct-w1-b8aee1-kserve-5b857c55cd-x5b7b" Oct 30 20:57:27.886064 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:27.886055 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-cb60a2/simulator-w3-cb60a2-kserve-6fd74b9d8b-vqfzf" Oct 30 20:57:27.886152 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:27.886142 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b8aee1/distinct-w1-b8aee1-kserve-5b857c55cd-x5b7b" Oct 30 20:57:27.914010 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:27.913983 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ad7ff0/premium-w0-ad7ff0-kserve-5485d4b948-bpmgf" Oct 30 20:57:27.934975 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:27.918999 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-f2ed1e/distinct-w4-f2ed1e-kserve-57cff9dcf8-8d5zl" Oct 30 20:57:27.957537 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:27.952687 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-14cca2/premium-w5-14cca2-kserve-6878777857-wpjtj" Oct 30 20:57:28.123092 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:28.123064 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-cb60a2/distinct-w3-cb60a2-kserve-5bb76dc74-86fds" Oct 30 20:57:28.139617 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:28.139592 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-3d909d/premium-w6-3d909d-kserve-765648946f-2qvd2" Oct 30 20:57:28.221568 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:28.185497 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:57:28.245536 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:28.244013 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b8aee1/premium-w1-b8aee1-kserve-786d575599-htc8d" Oct 30 20:57:28.245536 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:28.244102 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-cb60a2/simulator-w3-cb60a2-kserve-6fd74b9d8b-vqfzf" Oct 30 20:57:28.974417 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:28.974378 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:57:29.115163 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:29.115127 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:57:29.328175 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:29.328085 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:57:29.563533 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:29.557407 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:57:29.968313 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:29.968262 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:57:30.303794 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:30.303713 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:57:30.980832 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:30.980790 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:57:31.123587 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:31.123552 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:57:31.681375 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:31.681336 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:57:31.829681 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:31.829638 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:57:32.113169 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:32.113132 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:57:32.935527 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:32.930657 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:57:33.729275 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:33.729231 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:57:33.867490 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:33.867457 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:57:33.941779 ip-10-0-133-192 kubenswrapper[4611]: E1030 20:57:33.941745 4611 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hwzc6" containerName="extractor" Oct 30 20:57:34.404587 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:34.404503 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:57:35.002236 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:35.002198 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:57:35.380200 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:35.380158 4611 generic.go:356] "Generic (PLEG): container finished" podID="c3bfb87f-8eb3-40eb-8c88-27495638d667" containerID="24778fe77e816b080bb543ffd31a35551fb493ca049b84398cecc026b61240df" exitCode=0 Oct 30 20:57:35.380378 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:35.380281 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-a4317e67-openshift-default-5885d97bf-gx7wp" event={"ID":"c3bfb87f-8eb3-40eb-8c88-27495638d667","Type":"ContainerDied","Data":"24778fe77e816b080bb543ffd31a35551fb493ca049b84398cecc026b61240df"} Oct 30 20:57:35.380847 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:35.380817 4611 scope.go:122] "RemoveContainer" containerID="24778fe77e816b080bb543ffd31a35551fb493ca049b84398cecc026b61240df" Oct 30 20:57:35.382764 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:35.382730 4611 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:57:35.382858 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:35.382819 4611 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:57:35.382907 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:35.382868 4611 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:57:35.713295 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:35.713209 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:57:36.390318 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:36.390279 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-a4317e67-openshift-default-5885d97bf-gx7wp" event={"ID":"c3bfb87f-8eb3-40eb-8c88-27495638d667","Type":"ContainerStarted","Data":"d5501bea6093c8da3c826cadba3005e784d08887ba3f412b3a2fc6d00b351963"} Oct 30 20:57:36.468935 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:36.468888 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:57:36.729812 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:36.729736 4611 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-a4317e67-openshift-default-5885d97bf-gx7wp" Oct 30 20:57:36.730854 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:36.730829 4611 patch_prober.go:29] interesting pod/e2e-derive-a4317e67-openshift-default-5885d97bf-gx7wp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body= Oct 30 20:57:36.730945 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:36.730877 4611 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-a4317e67-openshift-default-5885d97bf-gx7wp" podUID="c3bfb87f-8eb3-40eb-8c88-27495638d667" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" Oct 30 20:57:37.010993 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:37.010897 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:57:37.170554 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:37.168374 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:57:37.372483 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:37.372450 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:57:37.495334 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:37.495297 4611 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-a4317e67-openshift-default-5885d97bf-gx7wp"] Oct 30 20:57:37.525187 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:37.524982 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:57:37.690621 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:37.689956 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:57:37.729907 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:37.729873 4611 patch_prober.go:29] interesting pod/e2e-derive-a4317e67-openshift-default-5885d97bf-gx7wp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body= Oct 30 20:57:37.730156 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:37.729933 4611 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-a4317e67-openshift-default-5885d97bf-gx7wp" podUID="c3bfb87f-8eb3-40eb-8c88-27495638d667" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" Oct 30 20:57:37.923669 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:37.923632 4611 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-kb6vb"] Oct 30 20:57:37.924154 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:37.924119 4611 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-kb6vb" podUID="41f0d1eb-188b-404a-8b31-cf9dc16b9fbc" containerName="istio-proxy" containerID="cri-o://57a88d136d133425ea062348fd5b75c500dc6d673847728ff7d889dc5e0082f2" gracePeriod=30 Oct 30 20:57:38.007687 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:38.007598 4611 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-5f78z"] Oct 30 20:57:38.007922 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:38.007889 4611 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-5f78z" podUID="06064c28-fd6d-4765-8635-8690665a9b83" containerName="istio-proxy" containerID="cri-o://aabb51bfc31c37d13fadd4cd218f1a931f8a4e4a27b04fa3e107400913404c18" gracePeriod=30 Oct 30 20:57:38.128560 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:38.125266 4611 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-lmpng"] Oct 30 20:57:38.128560 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:38.125538 4611 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-lmpng" podUID="61477918-6577-4957-807b-ebfa7a01f9d2" containerName="istio-proxy" containerID="cri-o://c2b4cfd5065653286408300dd2991b494221273438708cc131102f583d6f4b91" gracePeriod=30 Oct 30 20:57:38.154792 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:38.142604 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:57:38.154792 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:38.145006 4611 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-gl9z9"] Oct 30 20:57:38.154792 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:38.145282 4611 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-gl9z9" podUID="29cef334-f72f-471a-9793-f7907b62e622" containerName="istio-proxy" containerID="cri-o://e577f4543d3da626fa3715088ef07c55db24bb6b443dd39a8c4530fd7118b3cd" gracePeriod=30 Oct 30 20:57:38.443529 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:38.441982 4611 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-a4317e67-openshift-default-5885d97bf-gx7wp" podUID="c3bfb87f-8eb3-40eb-8c88-27495638d667" containerName="istio-proxy" containerID="cri-o://d5501bea6093c8da3c826cadba3005e784d08887ba3f412b3a2fc6d00b351963" gracePeriod=30 Oct 30 20:57:38.597744 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:38.597703 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:57:38.800440 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:38.800352 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:57:39.002848 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:39.002663 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:57:39.177314 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:39.177271 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:57:39.583537 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:39.583443 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:57:40.027477 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:40.027435 4611 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-020e7427-mrljk"] Oct 30 20:57:40.032891 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:40.032863 4611 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-020e7427-mrljk"] Oct 30 20:57:40.204070 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:40.204034 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:57:40.331322 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:40.331135 4611 patch_prober.go:29] interesting pod/e2e-worker-w2-db534b-openshift-default-85df4d79d9-kb6vb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 20:57:40.331322 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:40.331222 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-kb6vb" podUID="41f0d1eb-188b-404a-8b31-cf9dc16b9fbc" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 20:57:40.339071 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:40.339039 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-a4317e67-openshift-default-5885d97bf-gx7wp" Oct 30 20:57:40.397563 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:40.396167 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:57:40.826146 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:40.826107 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:57:41.338048 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:41.338004 4611 patch_prober.go:29] interesting pod/e2e-worker-w2-db534b-openshift-default-85df4d79d9-gl9z9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 20:57:41.339792 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:41.339756 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-gl9z9" podUID="29cef334-f72f-471a-9793-f7907b62e622" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 20:57:41.423327 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:41.423299 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:57:41.586531 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:41.586379 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:57:41.631537 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:41.631037 4611 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7a41f107-a2bb-45e9-a071-265238a96fc7" path="/var/lib/kubelet/pods/7a41f107-a2bb-45e9-a071-265238a96fc7/volumes" Oct 30 20:57:41.729957 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:41.729921 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:57:41.835944 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:41.835908 4611 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-8d0462d9-gw-openshift-default-6464c96d5c-vnmsn"] Oct 30 20:57:41.836414 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:41.836382 4611 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-8d0462d9-gw-openshift-default-6464c96d5c-vnmsn" podUID="22a8ecc8-ac35-4744-8c73-e196caf73f52" containerName="istio-proxy" containerID="cri-o://c33c1bea362127a0b77552737a86f45c0efc7072471dbe76fee6bd528248b101" gracePeriod=30 Oct 30 20:57:41.906687 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:41.906579 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:57:42.073533 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:42.073477 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:57:42.456895 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:42.456863 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:57:42.599620 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:42.599586 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:57:42.680049 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:42.680019 4611 patch_prober.go:29] interesting pod/e2e-migrate-8d0462d9-gw-openshift-default-6464c96d5c-vnmsn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 20:57:42.680212 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:42.680069 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-migrate-8d0462d9-gw-openshift-default-6464c96d5c-vnmsn" podUID="22a8ecc8-ac35-4744-8c73-e196caf73f52" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 20:57:43.179891 ip-10-0-133-192 kubenswrapper[4611]: E1030 20:57:43.179857 4611 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f0d1eb_188b_404a_8b31_cf9dc16b9fbc.slice/crio-conmon-57a88d136d133425ea062348fd5b75c500dc6d673847728ff7d889dc5e0082f2.scope\": RecentStats: unable to find data in memory cache]" Oct 30 20:57:43.410761 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:43.410410 4611 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-kb6vb" Oct 30 20:57:43.464951 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:43.464917 4611 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-5f78z" Oct 30 20:57:43.485384 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:43.485351 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06064c28-fd6d-4765-8635-8690665a9b83-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06064c28-fd6d-4765-8635-8690665a9b83-workload-socket\") pod \"06064c28-fd6d-4765-8635-8690665a9b83\" (UID: \"06064c28-fd6d-4765-8635-8690665a9b83\") " Oct 30 20:57:43.485568 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:43.485402 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/41f0d1eb-188b-404a-8b31-cf9dc16b9fbc-istio-token\" (UniqueName: \"kubernetes.io/projected/41f0d1eb-188b-404a-8b31-cf9dc16b9fbc-istio-token\") pod \"41f0d1eb-188b-404a-8b31-cf9dc16b9fbc\" (UID: \"41f0d1eb-188b-404a-8b31-cf9dc16b9fbc\") " Oct 30 20:57:43.485568 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:43.485431 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06064c28-fd6d-4765-8635-8690665a9b83-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06064c28-fd6d-4765-8635-8690665a9b83-credential-socket\") pod \"06064c28-fd6d-4765-8635-8690665a9b83\" (UID: \"06064c28-fd6d-4765-8635-8690665a9b83\") " Oct 30 20:57:43.485568 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:43.485469 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06064c28-fd6d-4765-8635-8690665a9b83-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06064c28-fd6d-4765-8635-8690665a9b83-workload-certs\") pod \"06064c28-fd6d-4765-8635-8690665a9b83\" (UID: \"06064c28-fd6d-4765-8635-8690665a9b83\") " Oct 30 20:57:43.485751 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:43.485602 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41f0d1eb-188b-404a-8b31-cf9dc16b9fbc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41f0d1eb-188b-404a-8b31-cf9dc16b9fbc-workload-socket\") pod \"41f0d1eb-188b-404a-8b31-cf9dc16b9fbc\" (UID: \"41f0d1eb-188b-404a-8b31-cf9dc16b9fbc\") " Oct 30 20:57:43.485751 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:43.485656 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/06064c28-fd6d-4765-8635-8690665a9b83-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06064c28-fd6d-4765-8635-8690665a9b83-istio-podinfo\") pod \"06064c28-fd6d-4765-8635-8690665a9b83\" (UID: \"06064c28-fd6d-4765-8635-8690665a9b83\") " Oct 30 20:57:43.485751 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:43.485735 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06064c28-fd6d-4765-8635-8690665a9b83-istio-token\" (UniqueName: \"kubernetes.io/projected/06064c28-fd6d-4765-8635-8690665a9b83-istio-token\") pod \"06064c28-fd6d-4765-8635-8690665a9b83\" (UID: \"06064c28-fd6d-4765-8635-8690665a9b83\") " Oct 30 20:57:43.485954 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:43.485772 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06064c28-fd6d-4765-8635-8690665a9b83-kube-api-access-zls8x\" (UniqueName: \"kubernetes.io/projected/06064c28-fd6d-4765-8635-8690665a9b83-kube-api-access-zls8x\") pod \"06064c28-fd6d-4765-8635-8690665a9b83\" (UID: \"06064c28-fd6d-4765-8635-8690665a9b83\") " Oct 30 20:57:43.485954 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:43.485847 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06064c28-fd6d-4765-8635-8690665a9b83-credential-socket" pod "06064c28-fd6d-4765-8635-8690665a9b83" (UID: "06064c28-fd6d-4765-8635-8690665a9b83"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:57:43.485954 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:43.485877 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06064c28-fd6d-4765-8635-8690665a9b83-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06064c28-fd6d-4765-8635-8690665a9b83-istio-envoy\") pod \"06064c28-fd6d-4765-8635-8690665a9b83\" (UID: \"06064c28-fd6d-4765-8635-8690665a9b83\") " Oct 30 20:57:43.486198 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:43.486173 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41f0d1eb-188b-404a-8b31-cf9dc16b9fbc-workload-socket" pod "41f0d1eb-188b-404a-8b31-cf9dc16b9fbc" (UID: "41f0d1eb-188b-404a-8b31-cf9dc16b9fbc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:57:43.486295 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:43.486229 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41f0d1eb-188b-404a-8b31-cf9dc16b9fbc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41f0d1eb-188b-404a-8b31-cf9dc16b9fbc-istio-envoy\") pod \"41f0d1eb-188b-404a-8b31-cf9dc16b9fbc\" (UID: \"41f0d1eb-188b-404a-8b31-cf9dc16b9fbc\") " Oct 30 20:57:43.486347 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:43.486323 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41f0d1eb-188b-404a-8b31-cf9dc16b9fbc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41f0d1eb-188b-404a-8b31-cf9dc16b9fbc-workload-certs\") pod \"41f0d1eb-188b-404a-8b31-cf9dc16b9fbc\" (UID: \"41f0d1eb-188b-404a-8b31-cf9dc16b9fbc\") " Oct 30 20:57:43.486402 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:43.486360 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/41f0d1eb-188b-404a-8b31-cf9dc16b9fbc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41f0d1eb-188b-404a-8b31-cf9dc16b9fbc-istiod-ca-cert\") pod \"41f0d1eb-188b-404a-8b31-cf9dc16b9fbc\" (UID: \"41f0d1eb-188b-404a-8b31-cf9dc16b9fbc\") " Oct 30 20:57:43.486456 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:43.486409 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/06064c28-fd6d-4765-8635-8690665a9b83-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06064c28-fd6d-4765-8635-8690665a9b83-istiod-ca-cert\") pod \"06064c28-fd6d-4765-8635-8690665a9b83\" (UID: \"06064c28-fd6d-4765-8635-8690665a9b83\") " Oct 30 20:57:43.486572 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:43.486466 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41f0d1eb-188b-404a-8b31-cf9dc16b9fbc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41f0d1eb-188b-404a-8b31-cf9dc16b9fbc-istio-data\") pod \"41f0d1eb-188b-404a-8b31-cf9dc16b9fbc\" (UID: \"41f0d1eb-188b-404a-8b31-cf9dc16b9fbc\") " Oct 30 20:57:43.486572 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:43.486537 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/41f0d1eb-188b-404a-8b31-cf9dc16b9fbc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41f0d1eb-188b-404a-8b31-cf9dc16b9fbc-istio-podinfo\") pod \"41f0d1eb-188b-404a-8b31-cf9dc16b9fbc\" (UID: \"41f0d1eb-188b-404a-8b31-cf9dc16b9fbc\") " Oct 30 20:57:43.486700 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:43.486603 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/41f0d1eb-188b-404a-8b31-cf9dc16b9fbc-kube-api-access-mdcc2\" (UniqueName: \"kubernetes.io/projected/41f0d1eb-188b-404a-8b31-cf9dc16b9fbc-kube-api-access-mdcc2\") pod \"41f0d1eb-188b-404a-8b31-cf9dc16b9fbc\" (UID: \"41f0d1eb-188b-404a-8b31-cf9dc16b9fbc\") " Oct 30 20:57:43.486919 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:43.486888 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41f0d1eb-188b-404a-8b31-cf9dc16b9fbc-istiod-ca-cert" pod "41f0d1eb-188b-404a-8b31-cf9dc16b9fbc" (UID: "41f0d1eb-188b-404a-8b31-cf9dc16b9fbc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:57:43.487006 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:43.486943 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41f0d1eb-188b-404a-8b31-cf9dc16b9fbc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41f0d1eb-188b-404a-8b31-cf9dc16b9fbc-credential-socket\") pod \"41f0d1eb-188b-404a-8b31-cf9dc16b9fbc\" (UID: \"41f0d1eb-188b-404a-8b31-cf9dc16b9fbc\") " Oct 30 20:57:43.487006 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:43.486987 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06064c28-fd6d-4765-8635-8690665a9b83-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06064c28-fd6d-4765-8635-8690665a9b83-istio-data\") pod \"06064c28-fd6d-4765-8635-8690665a9b83\" (UID: \"06064c28-fd6d-4765-8635-8690665a9b83\") " Oct 30 20:57:43.487988 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:43.487339 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06064c28-fd6d-4765-8635-8690665a9b83-workload-certs" pod "06064c28-fd6d-4765-8635-8690665a9b83" (UID: "06064c28-fd6d-4765-8635-8690665a9b83"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:57:43.487988 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:43.487633 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41f0d1eb-188b-404a-8b31-cf9dc16b9fbc-workload-certs" pod "41f0d1eb-188b-404a-8b31-cf9dc16b9fbc" (UID: "41f0d1eb-188b-404a-8b31-cf9dc16b9fbc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:57:43.487988 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:43.487957 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06064c28-fd6d-4765-8635-8690665a9b83-istiod-ca-cert" pod "06064c28-fd6d-4765-8635-8690665a9b83" (UID: "06064c28-fd6d-4765-8635-8690665a9b83"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:57:43.488269 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:43.488056 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41f0d1eb-188b-404a-8b31-cf9dc16b9fbc-istio-data" pod "41f0d1eb-188b-404a-8b31-cf9dc16b9fbc" (UID: "41f0d1eb-188b-404a-8b31-cf9dc16b9fbc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:57:43.488463 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:43.488438 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06064c28-fd6d-4765-8635-8690665a9b83-istio-data" pod "06064c28-fd6d-4765-8635-8690665a9b83" (UID: "06064c28-fd6d-4765-8635-8690665a9b83"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:57:43.488652 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:43.488633 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06064c28-fd6d-4765-8635-8690665a9b83-workload-socket" pod "06064c28-fd6d-4765-8635-8690665a9b83" (UID: "06064c28-fd6d-4765-8635-8690665a9b83"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:57:43.488742 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:43.488697 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41f0d1eb-188b-404a-8b31-cf9dc16b9fbc-credential-socket" pod "41f0d1eb-188b-404a-8b31-cf9dc16b9fbc" (UID: "41f0d1eb-188b-404a-8b31-cf9dc16b9fbc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:57:43.489033 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:43.489007 4611 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06064c28-fd6d-4765-8635-8690665a9b83-istiod-ca-cert\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:57:43.489033 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:43.489029 4611 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41f0d1eb-188b-404a-8b31-cf9dc16b9fbc-istio-data\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:57:43.489159 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:43.489044 4611 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41f0d1eb-188b-404a-8b31-cf9dc16b9fbc-credential-socket\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:57:43.489159 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:43.489057 4611 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06064c28-fd6d-4765-8635-8690665a9b83-istio-data\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:57:43.489159 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:43.489072 4611 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06064c28-fd6d-4765-8635-8690665a9b83-workload-socket\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:57:43.489159 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:43.489087 4611 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06064c28-fd6d-4765-8635-8690665a9b83-credential-socket\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:57:43.489159 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:43.489102 4611 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06064c28-fd6d-4765-8635-8690665a9b83-workload-certs\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:57:43.489159 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:43.489117 4611 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41f0d1eb-188b-404a-8b31-cf9dc16b9fbc-workload-socket\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:57:43.489159 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:43.489131 4611 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41f0d1eb-188b-404a-8b31-cf9dc16b9fbc-workload-certs\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:57:43.489159 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:43.489144 4611 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41f0d1eb-188b-404a-8b31-cf9dc16b9fbc-istiod-ca-cert\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:57:43.496732 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:43.496703 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/06064c28-fd6d-4765-8635-8690665a9b83-istio-podinfo" pod "06064c28-fd6d-4765-8635-8690665a9b83" (UID: "06064c28-fd6d-4765-8635-8690665a9b83"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:57:43.496817 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:43.496795 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/41f0d1eb-188b-404a-8b31-cf9dc16b9fbc-istio-podinfo" pod "41f0d1eb-188b-404a-8b31-cf9dc16b9fbc" (UID: "41f0d1eb-188b-404a-8b31-cf9dc16b9fbc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:57:43.496879 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:43.496821 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06064c28-fd6d-4765-8635-8690665a9b83-istio-token" pod "06064c28-fd6d-4765-8635-8690665a9b83" (UID: "06064c28-fd6d-4765-8635-8690665a9b83"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:57:43.496879 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:43.496867 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41f0d1eb-188b-404a-8b31-cf9dc16b9fbc-istio-token" pod "41f0d1eb-188b-404a-8b31-cf9dc16b9fbc" (UID: "41f0d1eb-188b-404a-8b31-cf9dc16b9fbc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:57:43.496974 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:43.496925 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41f0d1eb-188b-404a-8b31-cf9dc16b9fbc-kube-api-access-mdcc2" pod "41f0d1eb-188b-404a-8b31-cf9dc16b9fbc" (UID: "41f0d1eb-188b-404a-8b31-cf9dc16b9fbc"). InnerVolumeSpecName "kube-api-access-mdcc2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:57:43.496974 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:43.496966 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41f0d1eb-188b-404a-8b31-cf9dc16b9fbc-istio-envoy" pod "41f0d1eb-188b-404a-8b31-cf9dc16b9fbc" (UID: "41f0d1eb-188b-404a-8b31-cf9dc16b9fbc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:57:43.497685 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:43.497623 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06064c28-fd6d-4765-8635-8690665a9b83-istio-envoy" pod "06064c28-fd6d-4765-8635-8690665a9b83" (UID: "06064c28-fd6d-4765-8635-8690665a9b83"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:57:43.498668 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:43.498646 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06064c28-fd6d-4765-8635-8690665a9b83-kube-api-access-zls8x" pod "06064c28-fd6d-4765-8635-8690665a9b83" (UID: "06064c28-fd6d-4765-8635-8690665a9b83"). InnerVolumeSpecName "kube-api-access-zls8x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:57:43.506581 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:43.506560 4611 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-lmpng" Oct 30 20:57:43.551931 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:43.551906 4611 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-gl9z9" Oct 30 20:57:43.557191 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:43.557162 4611 generic.go:356] "Generic (PLEG): container finished" podID="06064c28-fd6d-4765-8635-8690665a9b83" containerID="aabb51bfc31c37d13fadd4cd218f1a931f8a4e4a27b04fa3e107400913404c18" exitCode=0 Oct 30 20:57:43.557341 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:43.557227 4611 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-5f78z" Oct 30 20:57:43.557341 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:43.557247 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-5f78z" event={"ID":"06064c28-fd6d-4765-8635-8690665a9b83","Type":"ContainerDied","Data":"aabb51bfc31c37d13fadd4cd218f1a931f8a4e4a27b04fa3e107400913404c18"} Oct 30 20:57:43.557341 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:43.557301 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-5f78z" event={"ID":"06064c28-fd6d-4765-8635-8690665a9b83","Type":"ContainerDied","Data":"494cd8b7fb015b421da4e68261471321425b163ce71c0c90b0ce2ab6664e04a1"} Oct 30 20:57:43.557341 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:43.557315 4611 scope.go:122] "RemoveContainer" containerID="aabb51bfc31c37d13fadd4cd218f1a931f8a4e4a27b04fa3e107400913404c18" Oct 30 20:57:43.567346 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:43.567324 4611 generic.go:356] "Generic (PLEG): container finished" podID="29cef334-f72f-471a-9793-f7907b62e622" containerID="e577f4543d3da626fa3715088ef07c55db24bb6b443dd39a8c4530fd7118b3cd" exitCode=0 Oct 30 20:57:43.567466 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:43.567425 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-gl9z9" event={"ID":"29cef334-f72f-471a-9793-f7907b62e622","Type":"ContainerDied","Data":"e577f4543d3da626fa3715088ef07c55db24bb6b443dd39a8c4530fd7118b3cd"} Oct 30 20:57:43.567466 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:43.567436 4611 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-gl9z9" Oct 30 20:57:43.567466 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:43.567471 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-gl9z9" event={"ID":"29cef334-f72f-471a-9793-f7907b62e622","Type":"ContainerDied","Data":"0f1409b2989e74d8f1ca2b08e2e5b6f93b49918b63e50c79675bdfc85ffeb599"} Oct 30 20:57:43.568866 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:43.568849 4611 scope.go:122] "RemoveContainer" containerID="bbf3a49efec4dd88896aa244e5313f8c00f1dad08d0e259457daea781577c113" Oct 30 20:57:43.569480 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:43.569302 4611 generic.go:356] "Generic (PLEG): container finished" podID="61477918-6577-4957-807b-ebfa7a01f9d2" containerID="c2b4cfd5065653286408300dd2991b494221273438708cc131102f583d6f4b91" exitCode=0 Oct 30 20:57:43.569480 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:43.569428 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-lmpng" event={"ID":"61477918-6577-4957-807b-ebfa7a01f9d2","Type":"ContainerDied","Data":"c2b4cfd5065653286408300dd2991b494221273438708cc131102f583d6f4b91"} Oct 30 20:57:43.569480 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:43.569469 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-lmpng" event={"ID":"61477918-6577-4957-807b-ebfa7a01f9d2","Type":"ContainerDied","Data":"4e57136caa11544abcd79158f21983af8deca0802beeae1b8a55216f17d9c412"} Oct 30 20:57:43.569626 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:43.569502 4611 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-lmpng" Oct 30 20:57:43.571267 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:43.571244 4611 generic.go:356] "Generic (PLEG): container finished" podID="41f0d1eb-188b-404a-8b31-cf9dc16b9fbc" containerID="57a88d136d133425ea062348fd5b75c500dc6d673847728ff7d889dc5e0082f2" exitCode=0 Oct 30 20:57:43.571362 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:43.571284 4611 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-kb6vb" Oct 30 20:57:43.571362 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:43.571322 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-kb6vb" event={"ID":"41f0d1eb-188b-404a-8b31-cf9dc16b9fbc","Type":"ContainerDied","Data":"57a88d136d133425ea062348fd5b75c500dc6d673847728ff7d889dc5e0082f2"} Oct 30 20:57:43.571459 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:43.571370 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-kb6vb" event={"ID":"41f0d1eb-188b-404a-8b31-cf9dc16b9fbc","Type":"ContainerDied","Data":"e0297d72bee10c4522c90c3fc28cbc94d68f4b484dfc3e6408808fd8cea9c66b"} Oct 30 20:57:43.589790 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:43.589765 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61477918-6577-4957-807b-ebfa7a01f9d2-istio-token\" (UniqueName: \"kubernetes.io/projected/61477918-6577-4957-807b-ebfa7a01f9d2-istio-token\") pod \"61477918-6577-4957-807b-ebfa7a01f9d2\" (UID: \"61477918-6577-4957-807b-ebfa7a01f9d2\") " Oct 30 20:57:43.589925 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:43.589862 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61477918-6577-4957-807b-ebfa7a01f9d2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61477918-6577-4957-807b-ebfa7a01f9d2-istio-envoy\") pod \"61477918-6577-4957-807b-ebfa7a01f9d2\" (UID: \"61477918-6577-4957-807b-ebfa7a01f9d2\") " Oct 30 20:57:43.589925 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:43.589906 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61477918-6577-4957-807b-ebfa7a01f9d2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61477918-6577-4957-807b-ebfa7a01f9d2-workload-certs\") pod \"61477918-6577-4957-807b-ebfa7a01f9d2\" (UID: \"61477918-6577-4957-807b-ebfa7a01f9d2\") " Oct 30 20:57:43.590009 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:43.589951 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29cef334-f72f-471a-9793-f7907b62e622-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29cef334-f72f-471a-9793-f7907b62e622-credential-socket\") pod \"29cef334-f72f-471a-9793-f7907b62e622\" (UID: \"29cef334-f72f-471a-9793-f7907b62e622\") " Oct 30 20:57:43.590063 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:43.590046 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/61477918-6577-4957-807b-ebfa7a01f9d2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61477918-6577-4957-807b-ebfa7a01f9d2-istio-podinfo\") pod \"61477918-6577-4957-807b-ebfa7a01f9d2\" (UID: \"61477918-6577-4957-807b-ebfa7a01f9d2\") " Oct 30 20:57:43.590098 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:43.590087 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/29cef334-f72f-471a-9793-f7907b62e622-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29cef334-f72f-471a-9793-f7907b62e622-istiod-ca-cert\") pod \"29cef334-f72f-471a-9793-f7907b62e622\" (UID: \"29cef334-f72f-471a-9793-f7907b62e622\") " Oct 30 20:57:43.590161 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:43.590122 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/29cef334-f72f-471a-9793-f7907b62e622-istio-token\" (UniqueName: \"kubernetes.io/projected/29cef334-f72f-471a-9793-f7907b62e622-istio-token\") pod \"29cef334-f72f-471a-9793-f7907b62e622\" (UID: \"29cef334-f72f-471a-9793-f7907b62e622\") " Oct 30 20:57:43.590218 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:43.590161 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29cef334-f72f-471a-9793-f7907b62e622-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29cef334-f72f-471a-9793-f7907b62e622-workload-socket\") pod \"29cef334-f72f-471a-9793-f7907b62e622\" (UID: \"29cef334-f72f-471a-9793-f7907b62e622\") " Oct 30 20:57:43.590290 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:43.590273 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/29cef334-f72f-471a-9793-f7907b62e622-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29cef334-f72f-471a-9793-f7907b62e622-istio-podinfo\") pod \"29cef334-f72f-471a-9793-f7907b62e622\" (UID: \"29cef334-f72f-471a-9793-f7907b62e622\") " Oct 30 20:57:43.590344 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:43.590304 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61477918-6577-4957-807b-ebfa7a01f9d2-workload-certs" pod "61477918-6577-4957-807b-ebfa7a01f9d2" (UID: "61477918-6577-4957-807b-ebfa7a01f9d2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:57:43.590403 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:43.590342 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61477918-6577-4957-807b-ebfa7a01f9d2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61477918-6577-4957-807b-ebfa7a01f9d2-credential-socket\") pod \"61477918-6577-4957-807b-ebfa7a01f9d2\" (UID: \"61477918-6577-4957-807b-ebfa7a01f9d2\") " Oct 30 20:57:43.590403 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:43.590397 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29cef334-f72f-471a-9793-f7907b62e622-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29cef334-f72f-471a-9793-f7907b62e622-istio-data\") pod \"29cef334-f72f-471a-9793-f7907b62e622\" (UID: \"29cef334-f72f-471a-9793-f7907b62e622\") " Oct 30 20:57:43.590530 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:43.590468 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/29cef334-f72f-471a-9793-f7907b62e622-kube-api-access-sdl6p\" (UniqueName: \"kubernetes.io/projected/29cef334-f72f-471a-9793-f7907b62e622-kube-api-access-sdl6p\") pod \"29cef334-f72f-471a-9793-f7907b62e622\" (UID: \"29cef334-f72f-471a-9793-f7907b62e622\") " Oct 30 20:57:43.590591 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:43.590503 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61477918-6577-4957-807b-ebfa7a01f9d2-kube-api-access-klg4q\" (UniqueName: \"kubernetes.io/projected/61477918-6577-4957-807b-ebfa7a01f9d2-kube-api-access-klg4q\") pod \"61477918-6577-4957-807b-ebfa7a01f9d2\" (UID: \"61477918-6577-4957-807b-ebfa7a01f9d2\") " Oct 30 20:57:43.590646 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:43.590588 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61477918-6577-4957-807b-ebfa7a01f9d2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61477918-6577-4957-807b-ebfa7a01f9d2-istio-data\") pod \"61477918-6577-4957-807b-ebfa7a01f9d2\" (UID: \"61477918-6577-4957-807b-ebfa7a01f9d2\") " Oct 30 20:57:43.590710 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:43.590645 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61477918-6577-4957-807b-ebfa7a01f9d2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61477918-6577-4957-807b-ebfa7a01f9d2-workload-socket\") pod \"61477918-6577-4957-807b-ebfa7a01f9d2\" (UID: \"61477918-6577-4957-807b-ebfa7a01f9d2\") " Oct 30 20:57:43.590768 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:43.590717 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/61477918-6577-4957-807b-ebfa7a01f9d2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61477918-6577-4957-807b-ebfa7a01f9d2-istiod-ca-cert\") pod \"61477918-6577-4957-807b-ebfa7a01f9d2\" (UID: \"61477918-6577-4957-807b-ebfa7a01f9d2\") " Oct 30 20:57:43.590880 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:43.590863 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29cef334-f72f-471a-9793-f7907b62e622-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29cef334-f72f-471a-9793-f7907b62e622-workload-certs\") pod \"29cef334-f72f-471a-9793-f7907b62e622\" (UID: \"29cef334-f72f-471a-9793-f7907b62e622\") " Oct 30 20:57:43.590950 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:43.590929 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29cef334-f72f-471a-9793-f7907b62e622-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29cef334-f72f-471a-9793-f7907b62e622-istio-envoy\") pod \"29cef334-f72f-471a-9793-f7907b62e622\" (UID: \"29cef334-f72f-471a-9793-f7907b62e622\") " Oct 30 20:57:43.591446 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:43.591150 4611 scope.go:122] "RemoveContainer" containerID="aabb51bfc31c37d13fadd4cd218f1a931f8a4e4a27b04fa3e107400913404c18" Oct 30 20:57:43.591917 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:43.591785 4611 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/06064c28-fd6d-4765-8635-8690665a9b83-istio-token\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:57:43.591917 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:43.591814 4611 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zls8x\" (UniqueName: \"kubernetes.io/projected/06064c28-fd6d-4765-8635-8690665a9b83-kube-api-access-zls8x\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:57:43.591917 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:43.591834 4611 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61477918-6577-4957-807b-ebfa7a01f9d2-workload-certs\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:57:43.591917 ip-10-0-133-192 kubenswrapper[4611]: E1030 20:57:43.591810 4611 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aabb51bfc31c37d13fadd4cd218f1a931f8a4e4a27b04fa3e107400913404c18\": container with ID starting with aabb51bfc31c37d13fadd4cd218f1a931f8a4e4a27b04fa3e107400913404c18 not found: ID does not exist" containerID="aabb51bfc31c37d13fadd4cd218f1a931f8a4e4a27b04fa3e107400913404c18" Oct 30 20:57:43.591917 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:43.591849 4611 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06064c28-fd6d-4765-8635-8690665a9b83-istio-envoy\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:57:43.591917 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:43.591862 4611 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41f0d1eb-188b-404a-8b31-cf9dc16b9fbc-istio-envoy\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:57:43.591917 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:43.591874 4611 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41f0d1eb-188b-404a-8b31-cf9dc16b9fbc-istio-podinfo\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:57:43.591917 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:43.591868 4611 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aabb51bfc31c37d13fadd4cd218f1a931f8a4e4a27b04fa3e107400913404c18"} err="failed to get container status \"aabb51bfc31c37d13fadd4cd218f1a931f8a4e4a27b04fa3e107400913404c18\": rpc error: code = NotFound desc = could not find container \"aabb51bfc31c37d13fadd4cd218f1a931f8a4e4a27b04fa3e107400913404c18\": container with ID starting with aabb51bfc31c37d13fadd4cd218f1a931f8a4e4a27b04fa3e107400913404c18 not found: ID does not exist" Oct 30 20:57:43.591917 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:43.590864 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29cef334-f72f-471a-9793-f7907b62e622-istiod-ca-cert" pod "29cef334-f72f-471a-9793-f7907b62e622" (UID: "29cef334-f72f-471a-9793-f7907b62e622"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:57:43.591917 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:43.591036 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61477918-6577-4957-807b-ebfa7a01f9d2-credential-socket" pod "61477918-6577-4957-807b-ebfa7a01f9d2" (UID: "61477918-6577-4957-807b-ebfa7a01f9d2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:57:43.591917 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:43.591893 4611 scope.go:122] "RemoveContainer" containerID="bbf3a49efec4dd88896aa244e5313f8c00f1dad08d0e259457daea781577c113" Oct 30 20:57:43.592498 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:43.591883 4611 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mdcc2\" (UniqueName: \"kubernetes.io/projected/41f0d1eb-188b-404a-8b31-cf9dc16b9fbc-kube-api-access-mdcc2\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:57:43.592498 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:43.591994 4611 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/41f0d1eb-188b-404a-8b31-cf9dc16b9fbc-istio-token\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:57:43.592498 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:43.592011 4611 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06064c28-fd6d-4765-8635-8690665a9b83-istio-podinfo\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:57:43.592498 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:43.591277 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29cef334-f72f-471a-9793-f7907b62e622-credential-socket" pod "29cef334-f72f-471a-9793-f7907b62e622" (UID: "29cef334-f72f-471a-9793-f7907b62e622"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:57:43.592498 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:43.591824 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29cef334-f72f-471a-9793-f7907b62e622-istio-data" pod "29cef334-f72f-471a-9793-f7907b62e622" (UID: "29cef334-f72f-471a-9793-f7907b62e622"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:57:43.592498 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:43.592127 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61477918-6577-4957-807b-ebfa7a01f9d2-workload-socket" pod "61477918-6577-4957-807b-ebfa7a01f9d2" (UID: "61477918-6577-4957-807b-ebfa7a01f9d2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:57:43.592498 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:43.592277 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61477918-6577-4957-807b-ebfa7a01f9d2-istiod-ca-cert" pod "61477918-6577-4957-807b-ebfa7a01f9d2" (UID: "61477918-6577-4957-807b-ebfa7a01f9d2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:57:43.592498 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:43.592330 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29cef334-f72f-471a-9793-f7907b62e622-workload-socket" pod "29cef334-f72f-471a-9793-f7907b62e622" (UID: "29cef334-f72f-471a-9793-f7907b62e622"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:57:43.592498 ip-10-0-133-192 kubenswrapper[4611]: E1030 20:57:43.592363 4611 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bbf3a49efec4dd88896aa244e5313f8c00f1dad08d0e259457daea781577c113\": container with ID starting with bbf3a49efec4dd88896aa244e5313f8c00f1dad08d0e259457daea781577c113 not found: ID does not exist" containerID="bbf3a49efec4dd88896aa244e5313f8c00f1dad08d0e259457daea781577c113" Oct 30 20:57:43.592498 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:43.592387 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61477918-6577-4957-807b-ebfa7a01f9d2-istio-data" pod "61477918-6577-4957-807b-ebfa7a01f9d2" (UID: "61477918-6577-4957-807b-ebfa7a01f9d2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:57:43.592498 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:43.592394 4611 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bbf3a49efec4dd88896aa244e5313f8c00f1dad08d0e259457daea781577c113"} err="failed to get container status \"bbf3a49efec4dd88896aa244e5313f8c00f1dad08d0e259457daea781577c113\": rpc error: code = NotFound desc = could not find container \"bbf3a49efec4dd88896aa244e5313f8c00f1dad08d0e259457daea781577c113\": container with ID starting with bbf3a49efec4dd88896aa244e5313f8c00f1dad08d0e259457daea781577c113 not found: ID does not exist" Oct 30 20:57:43.592498 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:43.592417 4611 scope.go:122] "RemoveContainer" containerID="e577f4543d3da626fa3715088ef07c55db24bb6b443dd39a8c4530fd7118b3cd" Oct 30 20:57:43.592498 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:43.592432 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29cef334-f72f-471a-9793-f7907b62e622-workload-certs" pod "29cef334-f72f-471a-9793-f7907b62e622" (UID: "29cef334-f72f-471a-9793-f7907b62e622"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:57:43.604224 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:43.604190 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29cef334-f72f-471a-9793-f7907b62e622-istio-envoy" pod "29cef334-f72f-471a-9793-f7907b62e622" (UID: "29cef334-f72f-471a-9793-f7907b62e622"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:57:43.604320 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:43.604202 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61477918-6577-4957-807b-ebfa7a01f9d2-istio-envoy" pod "61477918-6577-4957-807b-ebfa7a01f9d2" (UID: "61477918-6577-4957-807b-ebfa7a01f9d2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:57:43.604996 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:43.604966 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/61477918-6577-4957-807b-ebfa7a01f9d2-istio-podinfo" pod "61477918-6577-4957-807b-ebfa7a01f9d2" (UID: "61477918-6577-4957-807b-ebfa7a01f9d2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:57:43.605100 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:43.605001 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29cef334-f72f-471a-9793-f7907b62e622-kube-api-access-sdl6p" pod "29cef334-f72f-471a-9793-f7907b62e622" (UID: "29cef334-f72f-471a-9793-f7907b62e622"). InnerVolumeSpecName "kube-api-access-sdl6p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:57:43.605462 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:43.605438 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61477918-6577-4957-807b-ebfa7a01f9d2-istio-token" pod "61477918-6577-4957-807b-ebfa7a01f9d2" (UID: "61477918-6577-4957-807b-ebfa7a01f9d2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:57:43.619267 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:43.619239 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29cef334-f72f-471a-9793-f7907b62e622-istio-token" pod "29cef334-f72f-471a-9793-f7907b62e622" (UID: "29cef334-f72f-471a-9793-f7907b62e622"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:57:43.626458 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:43.626147 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/29cef334-f72f-471a-9793-f7907b62e622-istio-podinfo" pod "29cef334-f72f-471a-9793-f7907b62e622" (UID: "29cef334-f72f-471a-9793-f7907b62e622"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:57:43.635790 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:43.635755 4611 scope.go:122] "RemoveContainer" containerID="e577f4543d3da626fa3715088ef07c55db24bb6b443dd39a8c4530fd7118b3cd" Oct 30 20:57:43.636253 ip-10-0-133-192 kubenswrapper[4611]: E1030 20:57:43.636215 4611 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e577f4543d3da626fa3715088ef07c55db24bb6b443dd39a8c4530fd7118b3cd\": container with ID starting with e577f4543d3da626fa3715088ef07c55db24bb6b443dd39a8c4530fd7118b3cd not found: ID does not exist" containerID="e577f4543d3da626fa3715088ef07c55db24bb6b443dd39a8c4530fd7118b3cd" Oct 30 20:57:43.636320 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:43.636278 4611 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e577f4543d3da626fa3715088ef07c55db24bb6b443dd39a8c4530fd7118b3cd"} err="failed to get container status \"e577f4543d3da626fa3715088ef07c55db24bb6b443dd39a8c4530fd7118b3cd\": rpc error: code = NotFound desc = could not find container \"e577f4543d3da626fa3715088ef07c55db24bb6b443dd39a8c4530fd7118b3cd\": container with ID starting with e577f4543d3da626fa3715088ef07c55db24bb6b443dd39a8c4530fd7118b3cd not found: ID does not exist" Oct 30 20:57:43.636320 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:43.636303 4611 scope.go:122] "RemoveContainer" containerID="c2b4cfd5065653286408300dd2991b494221273438708cc131102f583d6f4b91" Oct 30 20:57:43.639916 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:43.639892 4611 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-5f78z"] Oct 30 20:57:43.640029 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:43.639939 4611 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-5f78z"] Oct 30 20:57:43.640029 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:43.639951 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61477918-6577-4957-807b-ebfa7a01f9d2-kube-api-access-klg4q" pod "61477918-6577-4957-807b-ebfa7a01f9d2" (UID: "61477918-6577-4957-807b-ebfa7a01f9d2"). InnerVolumeSpecName "kube-api-access-klg4q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:57:43.640029 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:43.639964 4611 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-kb6vb"] Oct 30 20:57:43.640298 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:43.640276 4611 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-kb6vb"] Oct 30 20:57:43.657614 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:43.657592 4611 scope.go:122] "RemoveContainer" containerID="c2b4cfd5065653286408300dd2991b494221273438708cc131102f583d6f4b91" Oct 30 20:57:43.658188 ip-10-0-133-192 kubenswrapper[4611]: E1030 20:57:43.658167 4611 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c2b4cfd5065653286408300dd2991b494221273438708cc131102f583d6f4b91\": container with ID starting with c2b4cfd5065653286408300dd2991b494221273438708cc131102f583d6f4b91 not found: ID does not exist" containerID="c2b4cfd5065653286408300dd2991b494221273438708cc131102f583d6f4b91" Oct 30 20:57:43.658293 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:43.658195 4611 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2b4cfd5065653286408300dd2991b494221273438708cc131102f583d6f4b91"} err="failed to get container status \"c2b4cfd5065653286408300dd2991b494221273438708cc131102f583d6f4b91\": rpc error: code = NotFound desc = could not find container \"c2b4cfd5065653286408300dd2991b494221273438708cc131102f583d6f4b91\": container with ID starting with c2b4cfd5065653286408300dd2991b494221273438708cc131102f583d6f4b91 not found: ID does not exist" Oct 30 20:57:43.658293 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:43.658211 4611 scope.go:122] "RemoveContainer" containerID="57a88d136d133425ea062348fd5b75c500dc6d673847728ff7d889dc5e0082f2" Oct 30 20:57:43.668174 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:43.668157 4611 scope.go:122] "RemoveContainer" containerID="57a88d136d133425ea062348fd5b75c500dc6d673847728ff7d889dc5e0082f2" Oct 30 20:57:43.668449 ip-10-0-133-192 kubenswrapper[4611]: E1030 20:57:43.668432 4611 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"57a88d136d133425ea062348fd5b75c500dc6d673847728ff7d889dc5e0082f2\": container with ID starting with 57a88d136d133425ea062348fd5b75c500dc6d673847728ff7d889dc5e0082f2 not found: ID does not exist" containerID="57a88d136d133425ea062348fd5b75c500dc6d673847728ff7d889dc5e0082f2" Oct 30 20:57:43.668526 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:43.668456 4611 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"57a88d136d133425ea062348fd5b75c500dc6d673847728ff7d889dc5e0082f2"} err="failed to get container status \"57a88d136d133425ea062348fd5b75c500dc6d673847728ff7d889dc5e0082f2\": rpc error: code = NotFound desc = could not find container \"57a88d136d133425ea062348fd5b75c500dc6d673847728ff7d889dc5e0082f2\": container with ID starting with 57a88d136d133425ea062348fd5b75c500dc6d673847728ff7d889dc5e0082f2 not found: ID does not exist" Oct 30 20:57:43.694092 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:43.694058 4611 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29cef334-f72f-471a-9793-f7907b62e622-istio-envoy\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:57:43.694092 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:43.694091 4611 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61477918-6577-4957-807b-ebfa7a01f9d2-istio-token\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:57:43.694320 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:43.694107 4611 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61477918-6577-4957-807b-ebfa7a01f9d2-istio-envoy\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:57:43.694320 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:43.694122 4611 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29cef334-f72f-471a-9793-f7907b62e622-credential-socket\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:57:43.694320 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:43.694136 4611 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61477918-6577-4957-807b-ebfa7a01f9d2-istio-podinfo\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:57:43.694320 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:43.694150 4611 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29cef334-f72f-471a-9793-f7907b62e622-istiod-ca-cert\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:57:43.694320 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:43.694163 4611 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/29cef334-f72f-471a-9793-f7907b62e622-istio-token\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:57:43.694320 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:43.694176 4611 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29cef334-f72f-471a-9793-f7907b62e622-workload-socket\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:57:43.694320 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:43.694190 4611 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29cef334-f72f-471a-9793-f7907b62e622-istio-podinfo\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:57:43.694320 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:43.694203 4611 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61477918-6577-4957-807b-ebfa7a01f9d2-credential-socket\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:57:43.694320 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:43.694216 4611 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29cef334-f72f-471a-9793-f7907b62e622-istio-data\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:57:43.694320 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:43.694229 4611 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sdl6p\" (UniqueName: \"kubernetes.io/projected/29cef334-f72f-471a-9793-f7907b62e622-kube-api-access-sdl6p\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:57:43.694320 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:43.694244 4611 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-klg4q\" (UniqueName: \"kubernetes.io/projected/61477918-6577-4957-807b-ebfa7a01f9d2-kube-api-access-klg4q\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:57:43.694320 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:43.694257 4611 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61477918-6577-4957-807b-ebfa7a01f9d2-istio-data\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:57:43.694320 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:43.694270 4611 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61477918-6577-4957-807b-ebfa7a01f9d2-workload-socket\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:57:43.694320 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:43.694284 4611 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61477918-6577-4957-807b-ebfa7a01f9d2-istiod-ca-cert\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:57:43.694320 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:43.694297 4611 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29cef334-f72f-471a-9793-f7907b62e622-workload-certs\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:57:43.810146 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:43.810101 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:57:43.887101 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:43.887029 4611 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-gl9z9"] Oct 30 20:57:43.891311 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:43.891282 4611 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-gl9z9"] Oct 30 20:57:43.905618 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:43.905583 4611 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-lmpng"] Oct 30 20:57:43.911436 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:43.911409 4611 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-db534b-openshift-default-85df4d79d9-lmpng"] Oct 30 20:57:44.590217 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:44.590181 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:57:45.291890 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:45.291806 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:57:45.607566 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:45.607537 4611 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="06064c28-fd6d-4765-8635-8690665a9b83" path="/var/lib/kubelet/pods/06064c28-fd6d-4765-8635-8690665a9b83/volumes" Oct 30 20:57:45.608074 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:45.608058 4611 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="29cef334-f72f-471a-9793-f7907b62e622" path="/var/lib/kubelet/pods/29cef334-f72f-471a-9793-f7907b62e622/volumes" Oct 30 20:57:45.608519 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:45.608493 4611 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="41f0d1eb-188b-404a-8b31-cf9dc16b9fbc" path="/var/lib/kubelet/pods/41f0d1eb-188b-404a-8b31-cf9dc16b9fbc/volumes" Oct 30 20:57:45.609273 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:45.609256 4611 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="61477918-6577-4957-807b-ebfa7a01f9d2" path="/var/lib/kubelet/pods/61477918-6577-4957-807b-ebfa7a01f9d2/volumes" Oct 30 20:57:45.843324 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:45.843287 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:57:45.993789 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:45.993703 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:57:46.196644 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:46.196603 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:57:46.784045 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:46.782370 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:57:47.190980 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:47.190953 4611 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-8d0462d9-gw-openshift-default-6464c96d5c-vnmsn" Oct 30 20:57:47.252996 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:47.252963 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22a8ecc8-ac35-4744-8c73-e196caf73f52-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22a8ecc8-ac35-4744-8c73-e196caf73f52-credential-socket\") pod \"22a8ecc8-ac35-4744-8c73-e196caf73f52\" (UID: \"22a8ecc8-ac35-4744-8c73-e196caf73f52\") " Oct 30 20:57:47.253187 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:47.253120 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22a8ecc8-ac35-4744-8c73-e196caf73f52-kube-api-access-l7jdc\" (UniqueName: \"kubernetes.io/projected/22a8ecc8-ac35-4744-8c73-e196caf73f52-kube-api-access-l7jdc\") pod \"22a8ecc8-ac35-4744-8c73-e196caf73f52\" (UID: \"22a8ecc8-ac35-4744-8c73-e196caf73f52\") " Oct 30 20:57:47.253283 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:47.253224 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22a8ecc8-ac35-4744-8c73-e196caf73f52-credential-socket" pod "22a8ecc8-ac35-4744-8c73-e196caf73f52" (UID: "22a8ecc8-ac35-4744-8c73-e196caf73f52"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:57:47.253704 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:47.253493 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/22a8ecc8-ac35-4744-8c73-e196caf73f52-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22a8ecc8-ac35-4744-8c73-e196caf73f52-istio-podinfo\") pod \"22a8ecc8-ac35-4744-8c73-e196caf73f52\" (UID: \"22a8ecc8-ac35-4744-8c73-e196caf73f52\") " Oct 30 20:57:47.253704 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:47.253560 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22a8ecc8-ac35-4744-8c73-e196caf73f52-istio-token\" (UniqueName: \"kubernetes.io/projected/22a8ecc8-ac35-4744-8c73-e196caf73f52-istio-token\") pod \"22a8ecc8-ac35-4744-8c73-e196caf73f52\" (UID: \"22a8ecc8-ac35-4744-8c73-e196caf73f52\") " Oct 30 20:57:47.253704 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:47.253622 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22a8ecc8-ac35-4744-8c73-e196caf73f52-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22a8ecc8-ac35-4744-8c73-e196caf73f52-workload-certs\") pod \"22a8ecc8-ac35-4744-8c73-e196caf73f52\" (UID: \"22a8ecc8-ac35-4744-8c73-e196caf73f52\") " Oct 30 20:57:47.253974 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:47.253741 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22a8ecc8-ac35-4744-8c73-e196caf73f52-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22a8ecc8-ac35-4744-8c73-e196caf73f52-istio-envoy\") pod \"22a8ecc8-ac35-4744-8c73-e196caf73f52\" (UID: \"22a8ecc8-ac35-4744-8c73-e196caf73f52\") " Oct 30 20:57:47.253974 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:47.253921 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22a8ecc8-ac35-4744-8c73-e196caf73f52-workload-certs" pod "22a8ecc8-ac35-4744-8c73-e196caf73f52" (UID: "22a8ecc8-ac35-4744-8c73-e196caf73f52"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:57:47.254213 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:47.254193 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22a8ecc8-ac35-4744-8c73-e196caf73f52-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22a8ecc8-ac35-4744-8c73-e196caf73f52-istio-data\") pod \"22a8ecc8-ac35-4744-8c73-e196caf73f52\" (UID: \"22a8ecc8-ac35-4744-8c73-e196caf73f52\") " Oct 30 20:57:47.254567 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:47.254530 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22a8ecc8-ac35-4744-8c73-e196caf73f52-istio-data" pod "22a8ecc8-ac35-4744-8c73-e196caf73f52" (UID: "22a8ecc8-ac35-4744-8c73-e196caf73f52"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:57:47.255064 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:47.254962 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22a8ecc8-ac35-4744-8c73-e196caf73f52-workload-socket" pod "22a8ecc8-ac35-4744-8c73-e196caf73f52" (UID: "22a8ecc8-ac35-4744-8c73-e196caf73f52"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:57:47.255198 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:47.254762 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22a8ecc8-ac35-4744-8c73-e196caf73f52-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22a8ecc8-ac35-4744-8c73-e196caf73f52-workload-socket\") pod \"22a8ecc8-ac35-4744-8c73-e196caf73f52\" (UID: \"22a8ecc8-ac35-4744-8c73-e196caf73f52\") " Oct 30 20:57:47.255456 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:47.255437 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/22a8ecc8-ac35-4744-8c73-e196caf73f52-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22a8ecc8-ac35-4744-8c73-e196caf73f52-istiod-ca-cert\") pod \"22a8ecc8-ac35-4744-8c73-e196caf73f52\" (UID: \"22a8ecc8-ac35-4744-8c73-e196caf73f52\") " Oct 30 20:57:47.256265 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:47.256149 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22a8ecc8-ac35-4744-8c73-e196caf73f52-istiod-ca-cert" pod "22a8ecc8-ac35-4744-8c73-e196caf73f52" (UID: "22a8ecc8-ac35-4744-8c73-e196caf73f52"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:57:47.256419 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:47.256399 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22a8ecc8-ac35-4744-8c73-e196caf73f52-istio-envoy" pod "22a8ecc8-ac35-4744-8c73-e196caf73f52" (UID: "22a8ecc8-ac35-4744-8c73-e196caf73f52"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:57:47.256868 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:47.256842 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/22a8ecc8-ac35-4744-8c73-e196caf73f52-istio-podinfo" pod "22a8ecc8-ac35-4744-8c73-e196caf73f52" (UID: "22a8ecc8-ac35-4744-8c73-e196caf73f52"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:57:47.256949 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:47.256875 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22a8ecc8-ac35-4744-8c73-e196caf73f52-kube-api-access-l7jdc" pod "22a8ecc8-ac35-4744-8c73-e196caf73f52" (UID: "22a8ecc8-ac35-4744-8c73-e196caf73f52"). InnerVolumeSpecName "kube-api-access-l7jdc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:57:47.257940 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:47.257906 4611 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22a8ecc8-ac35-4744-8c73-e196caf73f52-credential-socket\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:57:47.258063 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:47.257939 4611 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l7jdc\" (UniqueName: \"kubernetes.io/projected/22a8ecc8-ac35-4744-8c73-e196caf73f52-kube-api-access-l7jdc\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:57:47.258063 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:47.257956 4611 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22a8ecc8-ac35-4744-8c73-e196caf73f52-istio-podinfo\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:57:47.258063 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:47.257971 4611 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22a8ecc8-ac35-4744-8c73-e196caf73f52-workload-certs\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:57:47.258063 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:47.257986 4611 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22a8ecc8-ac35-4744-8c73-e196caf73f52-istio-envoy\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:57:47.258063 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:47.258000 4611 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22a8ecc8-ac35-4744-8c73-e196caf73f52-istio-data\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:57:47.258063 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:47.258015 4611 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22a8ecc8-ac35-4744-8c73-e196caf73f52-workload-socket\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:57:47.258063 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:47.258029 4611 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22a8ecc8-ac35-4744-8c73-e196caf73f52-istiod-ca-cert\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:57:47.259824 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:47.259798 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22a8ecc8-ac35-4744-8c73-e196caf73f52-istio-token" pod "22a8ecc8-ac35-4744-8c73-e196caf73f52" (UID: "22a8ecc8-ac35-4744-8c73-e196caf73f52"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:57:47.360198 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:47.360161 4611 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22a8ecc8-ac35-4744-8c73-e196caf73f52-istio-token\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:57:47.373341 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:47.373319 4611 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-a4317e67-openshift-default-5885d97bf-gx7wp" Oct 30 20:57:47.380089 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:47.380061 4611 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-ad7ff0/premium-w0-ad7ff0-kserve-5485d4b948-bpmgf"] Oct 30 20:57:47.428143 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:47.428110 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:57:47.461726 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:47.461039 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c3bfb87f-8eb3-40eb-8c88-27495638d667-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3bfb87f-8eb3-40eb-8c88-27495638d667-istiod-ca-cert\") pod \"c3bfb87f-8eb3-40eb-8c88-27495638d667\" (UID: \"c3bfb87f-8eb3-40eb-8c88-27495638d667\") " Oct 30 20:57:47.461726 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:47.461088 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3bfb87f-8eb3-40eb-8c88-27495638d667-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3bfb87f-8eb3-40eb-8c88-27495638d667-workload-socket\") pod \"c3bfb87f-8eb3-40eb-8c88-27495638d667\" (UID: \"c3bfb87f-8eb3-40eb-8c88-27495638d667\") " Oct 30 20:57:47.461726 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:47.461169 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3bfb87f-8eb3-40eb-8c88-27495638d667-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3bfb87f-8eb3-40eb-8c88-27495638d667-credential-socket\") pod \"c3bfb87f-8eb3-40eb-8c88-27495638d667\" (UID: \"c3bfb87f-8eb3-40eb-8c88-27495638d667\") " Oct 30 20:57:47.461726 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:47.461227 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3bfb87f-8eb3-40eb-8c88-27495638d667-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3bfb87f-8eb3-40eb-8c88-27495638d667-istio-envoy\") pod \"c3bfb87f-8eb3-40eb-8c88-27495638d667\" (UID: \"c3bfb87f-8eb3-40eb-8c88-27495638d667\") " Oct 30 20:57:47.461726 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:47.461371 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c3bfb87f-8eb3-40eb-8c88-27495638d667-kube-api-access-5kcsj\" (UniqueName: \"kubernetes.io/projected/c3bfb87f-8eb3-40eb-8c88-27495638d667-kube-api-access-5kcsj\") pod \"c3bfb87f-8eb3-40eb-8c88-27495638d667\" (UID: \"c3bfb87f-8eb3-40eb-8c88-27495638d667\") " Oct 30 20:57:47.461726 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:47.461405 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c3bfb87f-8eb3-40eb-8c88-27495638d667-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3bfb87f-8eb3-40eb-8c88-27495638d667-istio-podinfo\") pod \"c3bfb87f-8eb3-40eb-8c88-27495638d667\" (UID: \"c3bfb87f-8eb3-40eb-8c88-27495638d667\") " Oct 30 20:57:47.461726 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:47.461447 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3bfb87f-8eb3-40eb-8c88-27495638d667-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3bfb87f-8eb3-40eb-8c88-27495638d667-workload-certs\") pod \"c3bfb87f-8eb3-40eb-8c88-27495638d667\" (UID: \"c3bfb87f-8eb3-40eb-8c88-27495638d667\") " Oct 30 20:57:47.461726 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:47.461404 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3bfb87f-8eb3-40eb-8c88-27495638d667-istiod-ca-cert" pod "c3bfb87f-8eb3-40eb-8c88-27495638d667" (UID: "c3bfb87f-8eb3-40eb-8c88-27495638d667"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:57:47.461726 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:47.461497 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3bfb87f-8eb3-40eb-8c88-27495638d667-credential-socket" pod "c3bfb87f-8eb3-40eb-8c88-27495638d667" (UID: "c3bfb87f-8eb3-40eb-8c88-27495638d667"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:57:47.462322 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:47.461751 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c3bfb87f-8eb3-40eb-8c88-27495638d667-istio-token\" (UniqueName: \"kubernetes.io/projected/c3bfb87f-8eb3-40eb-8c88-27495638d667-istio-token\") pod \"c3bfb87f-8eb3-40eb-8c88-27495638d667\" (UID: \"c3bfb87f-8eb3-40eb-8c88-27495638d667\") " Oct 30 20:57:47.462322 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:47.461798 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3bfb87f-8eb3-40eb-8c88-27495638d667-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3bfb87f-8eb3-40eb-8c88-27495638d667-istio-data\") pod \"c3bfb87f-8eb3-40eb-8c88-27495638d667\" (UID: \"c3bfb87f-8eb3-40eb-8c88-27495638d667\") " Oct 30 20:57:47.462322 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:47.462039 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3bfb87f-8eb3-40eb-8c88-27495638d667-workload-socket" pod "c3bfb87f-8eb3-40eb-8c88-27495638d667" (UID: "c3bfb87f-8eb3-40eb-8c88-27495638d667"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:57:47.462756 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:47.462564 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3bfb87f-8eb3-40eb-8c88-27495638d667-workload-certs" pod "c3bfb87f-8eb3-40eb-8c88-27495638d667" (UID: "c3bfb87f-8eb3-40eb-8c88-27495638d667"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:57:47.462902 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:47.462783 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3bfb87f-8eb3-40eb-8c88-27495638d667-istio-data" pod "c3bfb87f-8eb3-40eb-8c88-27495638d667" (UID: "c3bfb87f-8eb3-40eb-8c88-27495638d667"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:57:47.463097 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:47.463024 4611 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3bfb87f-8eb3-40eb-8c88-27495638d667-istiod-ca-cert\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:57:47.463097 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:47.463050 4611 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3bfb87f-8eb3-40eb-8c88-27495638d667-workload-socket\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:57:47.463097 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:47.463063 4611 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3bfb87f-8eb3-40eb-8c88-27495638d667-credential-socket\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:57:47.463097 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:47.463076 4611 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3bfb87f-8eb3-40eb-8c88-27495638d667-workload-certs\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:57:47.464847 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:47.464714 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c3bfb87f-8eb3-40eb-8c88-27495638d667-istio-podinfo" pod "c3bfb87f-8eb3-40eb-8c88-27495638d667" (UID: "c3bfb87f-8eb3-40eb-8c88-27495638d667"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:57:47.464847 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:47.464813 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3bfb87f-8eb3-40eb-8c88-27495638d667-kube-api-access-5kcsj" pod "c3bfb87f-8eb3-40eb-8c88-27495638d667" (UID: "c3bfb87f-8eb3-40eb-8c88-27495638d667"). InnerVolumeSpecName "kube-api-access-5kcsj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:57:47.465767 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:47.465737 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3bfb87f-8eb3-40eb-8c88-27495638d667-istio-envoy" pod "c3bfb87f-8eb3-40eb-8c88-27495638d667" (UID: "c3bfb87f-8eb3-40eb-8c88-27495638d667"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:57:47.466564 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:47.466542 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3bfb87f-8eb3-40eb-8c88-27495638d667-istio-token" pod "c3bfb87f-8eb3-40eb-8c88-27495638d667" (UID: "c3bfb87f-8eb3-40eb-8c88-27495638d667"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:57:47.565581 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:47.565534 4611 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5kcsj\" (UniqueName: \"kubernetes.io/projected/c3bfb87f-8eb3-40eb-8c88-27495638d667-kube-api-access-5kcsj\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:57:47.565581 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:47.565574 4611 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3bfb87f-8eb3-40eb-8c88-27495638d667-istio-podinfo\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:57:47.565581 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:47.565585 4611 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3bfb87f-8eb3-40eb-8c88-27495638d667-istio-token\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:57:47.565838 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:47.565595 4611 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3bfb87f-8eb3-40eb-8c88-27495638d667-istio-data\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:57:47.565838 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:47.565604 4611 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3bfb87f-8eb3-40eb-8c88-27495638d667-istio-envoy\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:57:47.572469 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:47.572434 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:57:47.620380 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:47.620281 4611 generic.go:356] "Generic (PLEG): container finished" podID="c3bfb87f-8eb3-40eb-8c88-27495638d667" containerID="d5501bea6093c8da3c826cadba3005e784d08887ba3f412b3a2fc6d00b351963" exitCode=0 Oct 30 20:57:47.620659 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:47.620434 4611 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-a4317e67-openshift-default-5885d97bf-gx7wp" Oct 30 20:57:47.620659 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:47.620545 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-a4317e67-openshift-default-5885d97bf-gx7wp" event={"ID":"c3bfb87f-8eb3-40eb-8c88-27495638d667","Type":"ContainerDied","Data":"d5501bea6093c8da3c826cadba3005e784d08887ba3f412b3a2fc6d00b351963"} Oct 30 20:57:47.620659 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:47.620593 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-a4317e67-openshift-default-5885d97bf-gx7wp" event={"ID":"c3bfb87f-8eb3-40eb-8c88-27495638d667","Type":"ContainerDied","Data":"a17cbba313da950d7559b4579c87182ad2fcecadb3250e2aa432f8c8780c4e52"} Oct 30 20:57:47.620659 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:47.620624 4611 scope.go:122] "RemoveContainer" containerID="d5501bea6093c8da3c826cadba3005e784d08887ba3f412b3a2fc6d00b351963" Oct 30 20:57:47.623373 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:47.623271 4611 generic.go:356] "Generic (PLEG): container finished" podID="22a8ecc8-ac35-4744-8c73-e196caf73f52" containerID="c33c1bea362127a0b77552737a86f45c0efc7072471dbe76fee6bd528248b101" exitCode=0 Oct 30 20:57:47.623373 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:47.623320 4611 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-8d0462d9-gw-openshift-default-6464c96d5c-vnmsn" Oct 30 20:57:47.623587 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:47.623380 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-8d0462d9-gw-openshift-default-6464c96d5c-vnmsn" event={"ID":"22a8ecc8-ac35-4744-8c73-e196caf73f52","Type":"ContainerDied","Data":"c33c1bea362127a0b77552737a86f45c0efc7072471dbe76fee6bd528248b101"} Oct 30 20:57:47.623587 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:47.623416 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-8d0462d9-gw-openshift-default-6464c96d5c-vnmsn" event={"ID":"22a8ecc8-ac35-4744-8c73-e196caf73f52","Type":"ContainerDied","Data":"ed86896d45b50870e66fef8809f2e30aaccdb07b6eae3e5afd2c2e62108cb887"} Oct 30 20:57:47.650162 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:47.650133 4611 scope.go:122] "RemoveContainer" containerID="24778fe77e816b080bb543ffd31a35551fb493ca049b84398cecc026b61240df" Oct 30 20:57:47.659613 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:47.659585 4611 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-8d0462d9-gw-openshift-default-6464c96d5c-vnmsn"] Oct 30 20:57:47.667567 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:47.667539 4611 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-8d0462d9-gw-openshift-default-6464c96d5c-vnmsn"] Oct 30 20:57:47.668740 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:47.668674 4611 scope.go:122] "RemoveContainer" containerID="d5501bea6093c8da3c826cadba3005e784d08887ba3f412b3a2fc6d00b351963" Oct 30 20:57:47.669152 ip-10-0-133-192 kubenswrapper[4611]: E1030 20:57:47.669129 4611 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d5501bea6093c8da3c826cadba3005e784d08887ba3f412b3a2fc6d00b351963\": container with ID starting with d5501bea6093c8da3c826cadba3005e784d08887ba3f412b3a2fc6d00b351963 not found: ID does not exist" containerID="d5501bea6093c8da3c826cadba3005e784d08887ba3f412b3a2fc6d00b351963" Oct 30 20:57:47.669274 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:47.669162 4611 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5501bea6093c8da3c826cadba3005e784d08887ba3f412b3a2fc6d00b351963"} err="failed to get container status \"d5501bea6093c8da3c826cadba3005e784d08887ba3f412b3a2fc6d00b351963\": rpc error: code = NotFound desc = could not find container \"d5501bea6093c8da3c826cadba3005e784d08887ba3f412b3a2fc6d00b351963\": container with ID starting with d5501bea6093c8da3c826cadba3005e784d08887ba3f412b3a2fc6d00b351963 not found: ID does not exist" Oct 30 20:57:47.669274 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:47.669185 4611 scope.go:122] "RemoveContainer" containerID="24778fe77e816b080bb543ffd31a35551fb493ca049b84398cecc026b61240df" Oct 30 20:57:47.669473 ip-10-0-133-192 kubenswrapper[4611]: E1030 20:57:47.669456 4611 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"24778fe77e816b080bb543ffd31a35551fb493ca049b84398cecc026b61240df\": container with ID starting with 24778fe77e816b080bb543ffd31a35551fb493ca049b84398cecc026b61240df not found: ID does not exist" containerID="24778fe77e816b080bb543ffd31a35551fb493ca049b84398cecc026b61240df" Oct 30 20:57:47.669577 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:47.669476 4611 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"24778fe77e816b080bb543ffd31a35551fb493ca049b84398cecc026b61240df"} err="failed to get container status \"24778fe77e816b080bb543ffd31a35551fb493ca049b84398cecc026b61240df\": rpc error: code = NotFound desc = could not find container \"24778fe77e816b080bb543ffd31a35551fb493ca049b84398cecc026b61240df\": container with ID starting with 24778fe77e816b080bb543ffd31a35551fb493ca049b84398cecc026b61240df not found: ID does not exist" Oct 30 20:57:47.669577 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:47.669494 4611 scope.go:122] "RemoveContainer" containerID="c33c1bea362127a0b77552737a86f45c0efc7072471dbe76fee6bd528248b101" Oct 30 20:57:47.672769 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:47.672742 4611 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-a4317e67-openshift-default-5885d97bf-gx7wp"] Oct 30 20:57:47.678286 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:47.678265 4611 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-a4317e67-openshift-default-5885d97bf-gx7wp"] Oct 30 20:57:47.681560 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:47.681541 4611 scope.go:122] "RemoveContainer" containerID="c33c1bea362127a0b77552737a86f45c0efc7072471dbe76fee6bd528248b101" Oct 30 20:57:47.681838 ip-10-0-133-192 kubenswrapper[4611]: E1030 20:57:47.681821 4611 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c33c1bea362127a0b77552737a86f45c0efc7072471dbe76fee6bd528248b101\": container with ID starting with c33c1bea362127a0b77552737a86f45c0efc7072471dbe76fee6bd528248b101 not found: ID does not exist" containerID="c33c1bea362127a0b77552737a86f45c0efc7072471dbe76fee6bd528248b101" Oct 30 20:57:47.681906 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:47.681847 4611 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c33c1bea362127a0b77552737a86f45c0efc7072471dbe76fee6bd528248b101"} err="failed to get container status \"c33c1bea362127a0b77552737a86f45c0efc7072471dbe76fee6bd528248b101\": rpc error: code = NotFound desc = could not find container \"c33c1bea362127a0b77552737a86f45c0efc7072471dbe76fee6bd528248b101\": container with ID starting with c33c1bea362127a0b77552737a86f45c0efc7072471dbe76fee6bd528248b101 not found: ID does not exist" Oct 30 20:57:48.192791 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:48.192752 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:57:48.734059 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:48.734027 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:57:48.992026 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:48.991941 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:57:49.609133 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:49.609078 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:57:49.615895 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:49.615863 4611 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="22a8ecc8-ac35-4744-8c73-e196caf73f52" path="/var/lib/kubelet/pods/22a8ecc8-ac35-4744-8c73-e196caf73f52/volumes" Oct 30 20:57:49.616734 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:49.616659 4611 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c3bfb87f-8eb3-40eb-8c88-27495638d667" path="/var/lib/kubelet/pods/c3bfb87f-8eb3-40eb-8c88-27495638d667/volumes" Oct 30 20:57:50.237527 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:50.237478 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:57:50.895577 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:50.893069 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:57:51.001157 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:51.001121 4611 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-cb60a2/distinct-w3-cb60a2-kserve-5bb76dc74-86fds"] Oct 30 20:57:51.023695 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:51.023664 4611 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-cb60a2/simulator-w3-cb60a2-kserve-6fd74b9d8b-vqfzf"] Oct 30 20:57:51.146382 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:51.146272 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:57:51.696602 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:51.696537 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:57:52.417317 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:52.417282 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:57:52.773294 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:52.773203 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:57:52.952277 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:52.950868 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:57:53.166044 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:53.162265 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:57:53.876311 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:53.876272 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:57:54.489870 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:54.489835 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:57:55.370118 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:55.370083 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:57:56.006254 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:56.004841 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:57:56.368763 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:56.368727 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:57:56.517982 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:56.514052 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:57:56.761151 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:56.761058 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:57:57.209204 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:57.209173 4611 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ee99e329-9w8fj"] Oct 30 20:57:57.209785 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:57.209769 4611 cpu_manager.go:414] "containerMap: removing container" podUID="06064c28-fd6d-4765-8635-8690665a9b83" containerName="istio-proxy" Oct 30 20:57:57.209785 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:57.209785 4611 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06064c28-fd6d-4765-8635-8690665a9b83" containerName="istio-proxy" Oct 30 20:57:57.209892 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:57.209814 4611 cpu_manager.go:414] "containerMap: removing container" podUID="2e828845-f5af-44d1-8db6-5737ffb966f5" containerName="istio-proxy" Oct 30 20:57:57.209892 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:57.209820 4611 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e828845-f5af-44d1-8db6-5737ffb966f5" containerName="istio-proxy" Oct 30 20:57:57.209892 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:57.209828 4611 cpu_manager.go:414] "containerMap: removing container" podUID="61477918-6577-4957-807b-ebfa7a01f9d2" containerName="istio-proxy" Oct 30 20:57:57.209892 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:57.209832 4611 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61477918-6577-4957-807b-ebfa7a01f9d2" containerName="istio-proxy" Oct 30 20:57:57.209892 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:57.209841 4611 cpu_manager.go:414] "containerMap: removing container" podUID="22a8ecc8-ac35-4744-8c73-e196caf73f52" containerName="istio-proxy" Oct 30 20:57:57.209892 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:57.209846 4611 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="22a8ecc8-ac35-4744-8c73-e196caf73f52" containerName="istio-proxy" Oct 30 20:57:57.209892 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:57.209853 4611 cpu_manager.go:414] "containerMap: removing container" podUID="29cef334-f72f-471a-9793-f7907b62e622" containerName="istio-proxy" Oct 30 20:57:57.209892 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:57.209858 4611 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="29cef334-f72f-471a-9793-f7907b62e622" containerName="istio-proxy" Oct 30 20:57:57.209892 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:57.209867 4611 cpu_manager.go:414] "containerMap: removing container" podUID="41f0d1eb-188b-404a-8b31-cf9dc16b9fbc" containerName="istio-proxy" Oct 30 20:57:57.209892 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:57.209872 4611 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="41f0d1eb-188b-404a-8b31-cf9dc16b9fbc" containerName="istio-proxy" Oct 30 20:57:57.209892 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:57.209890 4611 cpu_manager.go:414] "containerMap: removing container" podUID="c3bfb87f-8eb3-40eb-8c88-27495638d667" containerName="istio-proxy" Oct 30 20:57:57.209892 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:57.209895 4611 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3bfb87f-8eb3-40eb-8c88-27495638d667" containerName="istio-proxy" Oct 30 20:57:57.210249 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:57.209912 4611 cpu_manager.go:414] "containerMap: removing container" podUID="c3bfb87f-8eb3-40eb-8c88-27495638d667" containerName="istio-proxy" Oct 30 20:57:57.210249 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:57.209917 4611 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3bfb87f-8eb3-40eb-8c88-27495638d667" containerName="istio-proxy" Oct 30 20:57:57.210249 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:57.210008 4611 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e828845-f5af-44d1-8db6-5737ffb966f5" containerName="istio-proxy" Oct 30 20:57:57.210249 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:57.210018 4611 memory_manager.go:365] "RemoveStaleState removing state" podUID="61477918-6577-4957-807b-ebfa7a01f9d2" containerName="istio-proxy" Oct 30 20:57:57.210249 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:57.210025 4611 memory_manager.go:365] "RemoveStaleState removing state" podUID="22a8ecc8-ac35-4744-8c73-e196caf73f52" containerName="istio-proxy" Oct 30 20:57:57.210249 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:57.210030 4611 memory_manager.go:365] "RemoveStaleState removing state" podUID="29cef334-f72f-471a-9793-f7907b62e622" containerName="istio-proxy" Oct 30 20:57:57.210249 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:57.210036 4611 memory_manager.go:365] "RemoveStaleState removing state" podUID="41f0d1eb-188b-404a-8b31-cf9dc16b9fbc" containerName="istio-proxy" Oct 30 20:57:57.210249 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:57.210045 4611 memory_manager.go:365] "RemoveStaleState removing state" podUID="c3bfb87f-8eb3-40eb-8c88-27495638d667" containerName="istio-proxy" Oct 30 20:57:57.210249 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:57.210053 4611 memory_manager.go:365] "RemoveStaleState removing state" podUID="c3bfb87f-8eb3-40eb-8c88-27495638d667" containerName="istio-proxy" Oct 30 20:57:57.210249 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:57.210059 4611 memory_manager.go:365] "RemoveStaleState removing state" podUID="06064c28-fd6d-4765-8635-8690665a9b83" containerName="istio-proxy" Oct 30 20:57:57.210249 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:57.210065 4611 memory_manager.go:365] "RemoveStaleState removing state" podUID="06064c28-fd6d-4765-8635-8690665a9b83" containerName="istio-proxy" Oct 30 20:57:57.213739 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:57.213712 4611 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee99e329-9w8fj" Oct 30 20:57:57.219359 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:57.219336 4611 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7hf8r\"" Oct 30 20:57:57.219692 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:57.219642 4611 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:57:57.222402 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:57.222382 4611 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ee99e329-9w8fj"] Oct 30 20:57:57.345458 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:57.345418 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/763cbacf-a524-4397-bf3a-2878d0e15a6d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ee99e329-9w8fj\" (UID: \"763cbacf-a524-4397-bf3a-2878d0e15a6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee99e329-9w8fj" Oct 30 20:57:57.345679 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:57.345578 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cwcnv\" (UniqueName: \"kubernetes.io/projected/763cbacf-a524-4397-bf3a-2878d0e15a6d-kube-api-access-cwcnv\") pod \"maas-api-revoke-sub-ee99e329-9w8fj\" (UID: \"763cbacf-a524-4397-bf3a-2878d0e15a6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee99e329-9w8fj" Oct 30 20:57:57.345748 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:57.345705 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/763cbacf-a524-4397-bf3a-2878d0e15a6d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ee99e329-9w8fj\" (UID: \"763cbacf-a524-4397-bf3a-2878d0e15a6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee99e329-9w8fj" Oct 30 20:57:57.447281 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:57.447246 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/763cbacf-a524-4397-bf3a-2878d0e15a6d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ee99e329-9w8fj\" (UID: \"763cbacf-a524-4397-bf3a-2878d0e15a6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee99e329-9w8fj" Oct 30 20:57:57.447434 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:57.447296 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cwcnv\" (UniqueName: \"kubernetes.io/projected/763cbacf-a524-4397-bf3a-2878d0e15a6d-kube-api-access-cwcnv\") pod \"maas-api-revoke-sub-ee99e329-9w8fj\" (UID: \"763cbacf-a524-4397-bf3a-2878d0e15a6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee99e329-9w8fj" Oct 30 20:57:57.447434 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:57.447389 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/763cbacf-a524-4397-bf3a-2878d0e15a6d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ee99e329-9w8fj\" (UID: \"763cbacf-a524-4397-bf3a-2878d0e15a6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee99e329-9w8fj" Oct 30 20:57:57.448096 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:57.448068 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/763cbacf-a524-4397-bf3a-2878d0e15a6d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ee99e329-9w8fj\" (UID: \"763cbacf-a524-4397-bf3a-2878d0e15a6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee99e329-9w8fj" Oct 30 20:57:57.449783 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:57.449762 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/763cbacf-a524-4397-bf3a-2878d0e15a6d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ee99e329-9w8fj\" (UID: \"763cbacf-a524-4397-bf3a-2878d0e15a6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee99e329-9w8fj" Oct 30 20:57:57.458804 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:57.458779 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cwcnv\" (UniqueName: \"kubernetes.io/projected/763cbacf-a524-4397-bf3a-2878d0e15a6d-kube-api-access-cwcnv\") pod \"maas-api-revoke-sub-ee99e329-9w8fj\" (UID: \"763cbacf-a524-4397-bf3a-2878d0e15a6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee99e329-9w8fj" Oct 30 20:57:57.546500 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:57.544749 4611 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee99e329-9w8fj" Oct 30 20:57:57.600461 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:57.599799 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:57:57.884606 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:57.884575 4611 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ee99e329-9w8fj"] Oct 30 20:57:57.885308 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:57:57.885280 4611 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod763cbacf_a524_4397_bf3a_2878d0e15a6d.slice/crio-c15570d54ffca4e8f145cbb477c0b1863e8877e8cdfdf0cb683449ac7a455b25 WatchSource:0}: Error finding container c15570d54ffca4e8f145cbb477c0b1863e8877e8cdfdf0cb683449ac7a455b25: Status 404 returned error can't find the container with id c15570d54ffca4e8f145cbb477c0b1863e8877e8cdfdf0cb683449ac7a455b25 Oct 30 20:57:58.359999 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:58.359961 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:57:58.724601 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:58.724562 4611 generic.go:356] "Generic (PLEG): container finished" podID="763cbacf-a524-4397-bf3a-2878d0e15a6d" containerID="be94e0c3a5311bf034bb48181a805f887425335488aa69afe5ab33cc0cceb696" exitCode=0 Oct 30 20:57:58.725005 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:58.724659 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee99e329-9w8fj" event={"ID":"763cbacf-a524-4397-bf3a-2878d0e15a6d","Type":"ContainerDied","Data":"be94e0c3a5311bf034bb48181a805f887425335488aa69afe5ab33cc0cceb696"} Oct 30 20:57:58.725005 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:58.724719 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee99e329-9w8fj" event={"ID":"763cbacf-a524-4397-bf3a-2878d0e15a6d","Type":"ContainerStarted","Data":"c15570d54ffca4e8f145cbb477c0b1863e8877e8cdfdf0cb683449ac7a455b25"} Oct 30 20:57:58.998177 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:58.998076 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:57:59.791438 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:59.791400 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:57:59.945883 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:57:59.945860 4611 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee99e329-9w8fj" Oct 30 20:58:00.095560 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:00.095488 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/763cbacf-a524-4397-bf3a-2878d0e15a6d-kube-api-access-cwcnv\" (UniqueName: \"kubernetes.io/projected/763cbacf-a524-4397-bf3a-2878d0e15a6d-kube-api-access-cwcnv\") pod \"763cbacf-a524-4397-bf3a-2878d0e15a6d\" (UID: \"763cbacf-a524-4397-bf3a-2878d0e15a6d\") " Oct 30 20:58:00.095710 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:00.095616 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/763cbacf-a524-4397-bf3a-2878d0e15a6d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/763cbacf-a524-4397-bf3a-2878d0e15a6d-maas-api-openshift-ca\") pod \"763cbacf-a524-4397-bf3a-2878d0e15a6d\" (UID: \"763cbacf-a524-4397-bf3a-2878d0e15a6d\") " Oct 30 20:58:00.095762 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:00.095749 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/763cbacf-a524-4397-bf3a-2878d0e15a6d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/763cbacf-a524-4397-bf3a-2878d0e15a6d-maas-api-xks-ca\") pod \"763cbacf-a524-4397-bf3a-2878d0e15a6d\" (UID: \"763cbacf-a524-4397-bf3a-2878d0e15a6d\") " Oct 30 20:58:00.096015 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:00.095989 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/763cbacf-a524-4397-bf3a-2878d0e15a6d-maas-api-openshift-ca" pod "763cbacf-a524-4397-bf3a-2878d0e15a6d" (UID: "763cbacf-a524-4397-bf3a-2878d0e15a6d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:58:00.096284 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:00.096267 4611 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/763cbacf-a524-4397-bf3a-2878d0e15a6d-maas-api-openshift-ca\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:00.097566 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:00.097543 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/763cbacf-a524-4397-bf3a-2878d0e15a6d-kube-api-access-cwcnv" pod "763cbacf-a524-4397-bf3a-2878d0e15a6d" (UID: "763cbacf-a524-4397-bf3a-2878d0e15a6d"). InnerVolumeSpecName "kube-api-access-cwcnv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:58:00.097632 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:00.097598 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/763cbacf-a524-4397-bf3a-2878d0e15a6d-maas-api-xks-ca" pod "763cbacf-a524-4397-bf3a-2878d0e15a6d" (UID: "763cbacf-a524-4397-bf3a-2878d0e15a6d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:58:00.197216 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:00.197186 4611 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cwcnv\" (UniqueName: \"kubernetes.io/projected/763cbacf-a524-4397-bf3a-2878d0e15a6d-kube-api-access-cwcnv\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:00.197216 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:00.197216 4611 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/763cbacf-a524-4397-bf3a-2878d0e15a6d-maas-api-xks-ca\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:00.388359 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:00.388324 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:58:00.741171 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:00.741093 4611 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee99e329-9w8fj" Oct 30 20:58:00.741171 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:00.741131 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee99e329-9w8fj" event={"ID":"763cbacf-a524-4397-bf3a-2878d0e15a6d","Type":"ContainerDied","Data":"c15570d54ffca4e8f145cbb477c0b1863e8877e8cdfdf0cb683449ac7a455b25"} Oct 30 20:58:00.741352 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:00.741187 4611 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c15570d54ffca4e8f145cbb477c0b1863e8877e8cdfdf0cb683449ac7a455b25" Oct 30 20:58:01.075883 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:01.075794 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:58:01.692636 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:01.692583 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:58:02.305911 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:02.305868 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:58:02.403792 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:02.403733 4611 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-ad7ff0/premium-w0-ad7ff0-kserve-5485d4b948-bpmgf" podUID="53cbd692-e9f8-46ba-8fde-3e5215a38904" containerName="main" containerID="cri-o://01be21bd89a61b89518c9b2365f7320e7b6b1dcc5dfe6c2da5d92a14bd255f70" gracePeriod=45 Oct 30 20:58:02.720154 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:02.720125 4611 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-ad7ff0/premium-w0-ad7ff0-kserve-5485d4b948-bpmgf" Oct 30 20:58:02.763501 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:02.763462 4611 generic.go:356] "Generic (PLEG): container finished" podID="53cbd692-e9f8-46ba-8fde-3e5215a38904" containerID="01be21bd89a61b89518c9b2365f7320e7b6b1dcc5dfe6c2da5d92a14bd255f70" exitCode=0 Oct 30 20:58:02.763501 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:02.763546 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-ad7ff0/premium-w0-ad7ff0-kserve-5485d4b948-bpmgf" event={"ID":"53cbd692-e9f8-46ba-8fde-3e5215a38904","Type":"ContainerDied","Data":"01be21bd89a61b89518c9b2365f7320e7b6b1dcc5dfe6c2da5d92a14bd255f70"} Oct 30 20:58:02.763838 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:02.763585 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-ad7ff0/premium-w0-ad7ff0-kserve-5485d4b948-bpmgf" event={"ID":"53cbd692-e9f8-46ba-8fde-3e5215a38904","Type":"ContainerDied","Data":"5a05ccbe8e50a91ebabe3bb62c97d9d7c8063ea376f57f9a7d0e7d04b48750a3"} Oct 30 20:58:02.763838 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:02.763610 4611 scope.go:122] "RemoveContainer" containerID="01be21bd89a61b89518c9b2365f7320e7b6b1dcc5dfe6c2da5d92a14bd255f70" Oct 30 20:58:02.763838 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:02.763756 4611 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-ad7ff0/premium-w0-ad7ff0-kserve-5485d4b948-bpmgf" Oct 30 20:58:02.777995 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:02.777969 4611 scope.go:122] "RemoveContainer" containerID="01be21bd89a61b89518c9b2365f7320e7b6b1dcc5dfe6c2da5d92a14bd255f70" Oct 30 20:58:02.778269 ip-10-0-133-192 kubenswrapper[4611]: E1030 20:58:02.778248 4611 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"01be21bd89a61b89518c9b2365f7320e7b6b1dcc5dfe6c2da5d92a14bd255f70\": container with ID starting with 01be21bd89a61b89518c9b2365f7320e7b6b1dcc5dfe6c2da5d92a14bd255f70 not found: ID does not exist" containerID="01be21bd89a61b89518c9b2365f7320e7b6b1dcc5dfe6c2da5d92a14bd255f70" Oct 30 20:58:02.778341 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:02.778281 4611 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"01be21bd89a61b89518c9b2365f7320e7b6b1dcc5dfe6c2da5d92a14bd255f70"} err="failed to get container status \"01be21bd89a61b89518c9b2365f7320e7b6b1dcc5dfe6c2da5d92a14bd255f70\": rpc error: code = NotFound desc = could not find container \"01be21bd89a61b89518c9b2365f7320e7b6b1dcc5dfe6c2da5d92a14bd255f70\": container with ID starting with 01be21bd89a61b89518c9b2365f7320e7b6b1dcc5dfe6c2da5d92a14bd255f70 not found: ID does not exist" Oct 30 20:58:02.844406 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:02.844372 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53cbd692-e9f8-46ba-8fde-3e5215a38904-dshm\" (UniqueName: \"kubernetes.io/empty-dir/53cbd692-e9f8-46ba-8fde-3e5215a38904-dshm\") pod \"53cbd692-e9f8-46ba-8fde-3e5215a38904\" (UID: \"53cbd692-e9f8-46ba-8fde-3e5215a38904\") " Oct 30 20:58:02.844568 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:02.844555 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/53cbd692-e9f8-46ba-8fde-3e5215a38904-tls-certs\" (UniqueName: \"kubernetes.io/secret/53cbd692-e9f8-46ba-8fde-3e5215a38904-tls-certs\") pod \"53cbd692-e9f8-46ba-8fde-3e5215a38904\" (UID: \"53cbd692-e9f8-46ba-8fde-3e5215a38904\") " Oct 30 20:58:02.844619 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:02.844597 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/53cbd692-e9f8-46ba-8fde-3e5215a38904-kube-api-access-m6mt5\" (UniqueName: \"kubernetes.io/projected/53cbd692-e9f8-46ba-8fde-3e5215a38904-kube-api-access-m6mt5\") pod \"53cbd692-e9f8-46ba-8fde-3e5215a38904\" (UID: \"53cbd692-e9f8-46ba-8fde-3e5215a38904\") " Oct 30 20:58:02.844744 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:02.844720 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53cbd692-e9f8-46ba-8fde-3e5215a38904-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/53cbd692-e9f8-46ba-8fde-3e5215a38904-model-cache\") pod \"53cbd692-e9f8-46ba-8fde-3e5215a38904\" (UID: \"53cbd692-e9f8-46ba-8fde-3e5215a38904\") " Oct 30 20:58:02.845260 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:02.844905 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53cbd692-e9f8-46ba-8fde-3e5215a38904-home\" (UniqueName: \"kubernetes.io/empty-dir/53cbd692-e9f8-46ba-8fde-3e5215a38904-home\") pod \"53cbd692-e9f8-46ba-8fde-3e5215a38904\" (UID: \"53cbd692-e9f8-46ba-8fde-3e5215a38904\") " Oct 30 20:58:02.845260 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:02.844943 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53cbd692-e9f8-46ba-8fde-3e5215a38904-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/53cbd692-e9f8-46ba-8fde-3e5215a38904-tmp-dir\") pod \"53cbd692-e9f8-46ba-8fde-3e5215a38904\" (UID: \"53cbd692-e9f8-46ba-8fde-3e5215a38904\") " Oct 30 20:58:02.845260 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:02.844987 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53cbd692-e9f8-46ba-8fde-3e5215a38904-model-cache" pod "53cbd692-e9f8-46ba-8fde-3e5215a38904" (UID: "53cbd692-e9f8-46ba-8fde-3e5215a38904"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:58:02.845260 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:02.845166 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53cbd692-e9f8-46ba-8fde-3e5215a38904-home" pod "53cbd692-e9f8-46ba-8fde-3e5215a38904" (UID: "53cbd692-e9f8-46ba-8fde-3e5215a38904"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:58:02.845260 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:02.845231 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53cbd692-e9f8-46ba-8fde-3e5215a38904-tmp-dir" pod "53cbd692-e9f8-46ba-8fde-3e5215a38904" (UID: "53cbd692-e9f8-46ba-8fde-3e5215a38904"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:58:02.845629 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:02.845550 4611 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/53cbd692-e9f8-46ba-8fde-3e5215a38904-home\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:02.845629 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:02.845566 4611 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/53cbd692-e9f8-46ba-8fde-3e5215a38904-tmp-dir\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:02.845629 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:02.845574 4611 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/53cbd692-e9f8-46ba-8fde-3e5215a38904-model-cache\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:02.846820 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:02.846801 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53cbd692-e9f8-46ba-8fde-3e5215a38904-kube-api-access-m6mt5" pod "53cbd692-e9f8-46ba-8fde-3e5215a38904" (UID: "53cbd692-e9f8-46ba-8fde-3e5215a38904"). InnerVolumeSpecName "kube-api-access-m6mt5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:58:02.846964 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:02.846945 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53cbd692-e9f8-46ba-8fde-3e5215a38904-dshm" pod "53cbd692-e9f8-46ba-8fde-3e5215a38904" (UID: "53cbd692-e9f8-46ba-8fde-3e5215a38904"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:58:02.847021 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:02.846962 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/53cbd692-e9f8-46ba-8fde-3e5215a38904-tls-certs" pod "53cbd692-e9f8-46ba-8fde-3e5215a38904" (UID: "53cbd692-e9f8-46ba-8fde-3e5215a38904"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:58:02.930412 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:02.929659 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:58:02.946709 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:02.946680 4611 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/53cbd692-e9f8-46ba-8fde-3e5215a38904-dshm\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:02.946709 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:02.946705 4611 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/53cbd692-e9f8-46ba-8fde-3e5215a38904-tls-certs\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:02.946709 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:02.946715 4611 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m6mt5\" (UniqueName: \"kubernetes.io/projected/53cbd692-e9f8-46ba-8fde-3e5215a38904-kube-api-access-m6mt5\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:03.095741 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:03.095707 4611 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-ad7ff0/premium-w0-ad7ff0-kserve-5485d4b948-bpmgf"] Oct 30 20:58:03.102099 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:03.102066 4611 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-ad7ff0/premium-w0-ad7ff0-kserve-5485d4b948-bpmgf"] Oct 30 20:58:03.610285 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:03.609604 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:58:03.624538 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:03.624469 4611 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="53cbd692-e9f8-46ba-8fde-3e5215a38904" path="/var/lib/kubelet/pods/53cbd692-e9f8-46ba-8fde-3e5215a38904/volumes" Oct 30 20:58:03.918139 ip-10-0-133-192 kubenswrapper[4611]: E1030 20:58:03.918048 4611 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hwzc6" containerName="extractor" Oct 30 20:58:04.040839 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:04.040798 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:58:04.338980 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:04.338888 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:58:05.082964 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:05.082569 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:58:05.946194 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:05.946155 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:58:06.023783 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:06.023738 4611 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-cb60a2/distinct-w3-cb60a2-kserve-5bb76dc74-86fds" podUID="52e301d9-f378-4e54-9515-01a6680d9fd7" containerName="main" containerID="cri-o://36cf0baf044dd3dc38d449c968df0fbd734cf76c3a1acfd9d63f3083987ced58" gracePeriod=45 Oct 30 20:58:06.045801 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:06.045771 4611 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-cb60a2/simulator-w3-cb60a2-kserve-6fd74b9d8b-vqfzf" podUID="711b167d-4e66-40d6-be75-33ede11f42c5" containerName="main" containerID="cri-o://9f55ee67ad822c60030083e1f0ed09869e45b9eb2d5d2eeef4659752733052d8" gracePeriod=45 Oct 30 20:58:06.119364 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:06.119334 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cb60a2/distinct-w3-cb60a2-kserve-5bb76dc74-86fds" podUID="52e301d9-f378-4e54-9515-01a6680d9fd7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": dial tcp 10.133.0.60:8000: connect: connection refused" Oct 30 20:58:06.229530 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:06.229458 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cb60a2/simulator-w3-cb60a2-kserve-6fd74b9d8b-vqfzf" podUID="711b167d-4e66-40d6-be75-33ede11f42c5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": dial tcp 10.133.0.50:8000: connect: connection refused" Oct 30 20:58:06.373759 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:06.373723 4611 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-cb60a2/distinct-w3-cb60a2-kserve-5bb76dc74-86fds" Oct 30 20:58:06.377663 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:06.377641 4611 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-cb60a2/simulator-w3-cb60a2-kserve-6fd74b9d8b-vqfzf" Oct 30 20:58:06.404735 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:06.404709 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/52e301d9-f378-4e54-9515-01a6680d9fd7-tls-certs\" (UniqueName: \"kubernetes.io/secret/52e301d9-f378-4e54-9515-01a6680d9fd7-tls-certs\") pod \"52e301d9-f378-4e54-9515-01a6680d9fd7\" (UID: \"52e301d9-f378-4e54-9515-01a6680d9fd7\") " Oct 30 20:58:06.416218 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:06.405209 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52e301d9-f378-4e54-9515-01a6680d9fd7-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/52e301d9-f378-4e54-9515-01a6680d9fd7-tmp-dir\") pod \"52e301d9-f378-4e54-9515-01a6680d9fd7\" (UID: \"52e301d9-f378-4e54-9515-01a6680d9fd7\") " Oct 30 20:58:06.416218 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:06.405246 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/711b167d-4e66-40d6-be75-33ede11f42c5-dshm\" (UniqueName: \"kubernetes.io/empty-dir/711b167d-4e66-40d6-be75-33ede11f42c5-dshm\") pod \"711b167d-4e66-40d6-be75-33ede11f42c5\" (UID: \"711b167d-4e66-40d6-be75-33ede11f42c5\") " Oct 30 20:58:06.416218 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:06.405280 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52e301d9-f378-4e54-9515-01a6680d9fd7-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/52e301d9-f378-4e54-9515-01a6680d9fd7-model-cache\") pod \"52e301d9-f378-4e54-9515-01a6680d9fd7\" (UID: \"52e301d9-f378-4e54-9515-01a6680d9fd7\") " Oct 30 20:58:06.416218 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:06.405334 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/711b167d-4e66-40d6-be75-33ede11f42c5-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/711b167d-4e66-40d6-be75-33ede11f42c5-tmp-dir\") pod \"711b167d-4e66-40d6-be75-33ede11f42c5\" (UID: \"711b167d-4e66-40d6-be75-33ede11f42c5\") " Oct 30 20:58:06.416218 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:06.405378 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52e301d9-f378-4e54-9515-01a6680d9fd7-kube-api-access-tl6c9\" (UniqueName: \"kubernetes.io/projected/52e301d9-f378-4e54-9515-01a6680d9fd7-kube-api-access-tl6c9\") pod \"52e301d9-f378-4e54-9515-01a6680d9fd7\" (UID: \"52e301d9-f378-4e54-9515-01a6680d9fd7\") " Oct 30 20:58:06.416218 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:06.405445 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/711b167d-4e66-40d6-be75-33ede11f42c5-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/711b167d-4e66-40d6-be75-33ede11f42c5-model-cache\") pod \"711b167d-4e66-40d6-be75-33ede11f42c5\" (UID: \"711b167d-4e66-40d6-be75-33ede11f42c5\") " Oct 30 20:58:06.416218 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:06.407742 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52e301d9-f378-4e54-9515-01a6680d9fd7-model-cache" pod "52e301d9-f378-4e54-9515-01a6680d9fd7" (UID: "52e301d9-f378-4e54-9515-01a6680d9fd7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:58:06.416218 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:06.407778 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52e301d9-f378-4e54-9515-01a6680d9fd7-tmp-dir" pod "52e301d9-f378-4e54-9515-01a6680d9fd7" (UID: "52e301d9-f378-4e54-9515-01a6680d9fd7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:58:06.416218 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:06.407999 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/711b167d-4e66-40d6-be75-33ede11f42c5-tmp-dir" pod "711b167d-4e66-40d6-be75-33ede11f42c5" (UID: "711b167d-4e66-40d6-be75-33ede11f42c5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:58:06.416218 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:06.408261 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/711b167d-4e66-40d6-be75-33ede11f42c5-model-cache" pod "711b167d-4e66-40d6-be75-33ede11f42c5" (UID: "711b167d-4e66-40d6-be75-33ede11f42c5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:58:06.420863 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:06.416894 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/711b167d-4e66-40d6-be75-33ede11f42c5-home\" (UniqueName: \"kubernetes.io/empty-dir/711b167d-4e66-40d6-be75-33ede11f42c5-home\") pod \"711b167d-4e66-40d6-be75-33ede11f42c5\" (UID: \"711b167d-4e66-40d6-be75-33ede11f42c5\") " Oct 30 20:58:06.420863 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:06.416954 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/711b167d-4e66-40d6-be75-33ede11f42c5-tls-certs\" (UniqueName: \"kubernetes.io/secret/711b167d-4e66-40d6-be75-33ede11f42c5-tls-certs\") pod \"711b167d-4e66-40d6-be75-33ede11f42c5\" (UID: \"711b167d-4e66-40d6-be75-33ede11f42c5\") " Oct 30 20:58:06.420863 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:06.417039 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/711b167d-4e66-40d6-be75-33ede11f42c5-kube-api-access-xvpzz\" (UniqueName: \"kubernetes.io/projected/711b167d-4e66-40d6-be75-33ede11f42c5-kube-api-access-xvpzz\") pod \"711b167d-4e66-40d6-be75-33ede11f42c5\" (UID: \"711b167d-4e66-40d6-be75-33ede11f42c5\") " Oct 30 20:58:06.420863 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:06.417110 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52e301d9-f378-4e54-9515-01a6680d9fd7-home\" (UniqueName: \"kubernetes.io/empty-dir/52e301d9-f378-4e54-9515-01a6680d9fd7-home\") pod \"52e301d9-f378-4e54-9515-01a6680d9fd7\" (UID: \"52e301d9-f378-4e54-9515-01a6680d9fd7\") " Oct 30 20:58:06.420863 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:06.417148 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52e301d9-f378-4e54-9515-01a6680d9fd7-dshm\" (UniqueName: \"kubernetes.io/empty-dir/52e301d9-f378-4e54-9515-01a6680d9fd7-dshm\") pod \"52e301d9-f378-4e54-9515-01a6680d9fd7\" (UID: \"52e301d9-f378-4e54-9515-01a6680d9fd7\") " Oct 30 20:58:06.420863 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:06.420557 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52e301d9-f378-4e54-9515-01a6680d9fd7-home" pod "52e301d9-f378-4e54-9515-01a6680d9fd7" (UID: "52e301d9-f378-4e54-9515-01a6680d9fd7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:58:06.421248 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:06.420998 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/711b167d-4e66-40d6-be75-33ede11f42c5-home" pod "711b167d-4e66-40d6-be75-33ede11f42c5" (UID: "711b167d-4e66-40d6-be75-33ede11f42c5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:58:06.421646 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:06.421625 4611 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/52e301d9-f378-4e54-9515-01a6680d9fd7-tmp-dir\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:06.421765 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:06.421754 4611 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/52e301d9-f378-4e54-9515-01a6680d9fd7-model-cache\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:06.421844 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:06.421833 4611 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/711b167d-4e66-40d6-be75-33ede11f42c5-tmp-dir\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:06.421922 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:06.421911 4611 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/711b167d-4e66-40d6-be75-33ede11f42c5-model-cache\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:06.421997 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:06.421987 4611 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/711b167d-4e66-40d6-be75-33ede11f42c5-home\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:06.422067 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:06.422058 4611 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/52e301d9-f378-4e54-9515-01a6680d9fd7-home\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:06.429990 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:06.429918 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/711b167d-4e66-40d6-be75-33ede11f42c5-kube-api-access-xvpzz" pod "711b167d-4e66-40d6-be75-33ede11f42c5" (UID: "711b167d-4e66-40d6-be75-33ede11f42c5"). InnerVolumeSpecName "kube-api-access-xvpzz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:58:06.429990 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:06.429957 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52e301d9-f378-4e54-9515-01a6680d9fd7-kube-api-access-tl6c9" pod "52e301d9-f378-4e54-9515-01a6680d9fd7" (UID: "52e301d9-f378-4e54-9515-01a6680d9fd7"). InnerVolumeSpecName "kube-api-access-tl6c9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:58:06.432957 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:06.432281 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/711b167d-4e66-40d6-be75-33ede11f42c5-dshm" pod "711b167d-4e66-40d6-be75-33ede11f42c5" (UID: "711b167d-4e66-40d6-be75-33ede11f42c5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:58:06.438056 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:06.437918 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/711b167d-4e66-40d6-be75-33ede11f42c5-tls-certs" pod "711b167d-4e66-40d6-be75-33ede11f42c5" (UID: "711b167d-4e66-40d6-be75-33ede11f42c5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:58:06.438056 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:06.438030 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/52e301d9-f378-4e54-9515-01a6680d9fd7-tls-certs" pod "52e301d9-f378-4e54-9515-01a6680d9fd7" (UID: "52e301d9-f378-4e54-9515-01a6680d9fd7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:58:06.439060 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:06.439035 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52e301d9-f378-4e54-9515-01a6680d9fd7-dshm" pod "52e301d9-f378-4e54-9515-01a6680d9fd7" (UID: "52e301d9-f378-4e54-9515-01a6680d9fd7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:58:06.486447 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:06.486338 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:58:06.523132 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:06.523101 4611 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/52e301d9-f378-4e54-9515-01a6680d9fd7-tls-certs\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:06.523132 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:06.523129 4611 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/711b167d-4e66-40d6-be75-33ede11f42c5-dshm\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:06.523299 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:06.523143 4611 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tl6c9\" (UniqueName: \"kubernetes.io/projected/52e301d9-f378-4e54-9515-01a6680d9fd7-kube-api-access-tl6c9\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:06.523299 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:06.523157 4611 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/711b167d-4e66-40d6-be75-33ede11f42c5-tls-certs\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:06.523299 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:06.523170 4611 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xvpzz\" (UniqueName: \"kubernetes.io/projected/711b167d-4e66-40d6-be75-33ede11f42c5-kube-api-access-xvpzz\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:06.523299 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:06.523183 4611 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/52e301d9-f378-4e54-9515-01a6680d9fd7-dshm\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:06.803362 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:06.803271 4611 generic.go:356] "Generic (PLEG): container finished" podID="711b167d-4e66-40d6-be75-33ede11f42c5" containerID="9f55ee67ad822c60030083e1f0ed09869e45b9eb2d5d2eeef4659752733052d8" exitCode=0 Oct 30 20:58:06.803362 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:06.803328 4611 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-cb60a2/simulator-w3-cb60a2-kserve-6fd74b9d8b-vqfzf" Oct 30 20:58:06.803640 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:06.803359 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-cb60a2/simulator-w3-cb60a2-kserve-6fd74b9d8b-vqfzf" event={"ID":"711b167d-4e66-40d6-be75-33ede11f42c5","Type":"ContainerDied","Data":"9f55ee67ad822c60030083e1f0ed09869e45b9eb2d5d2eeef4659752733052d8"} Oct 30 20:58:06.803640 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:06.803403 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-cb60a2/simulator-w3-cb60a2-kserve-6fd74b9d8b-vqfzf" event={"ID":"711b167d-4e66-40d6-be75-33ede11f42c5","Type":"ContainerDied","Data":"4c228916f6d3f64e0cdd9605efff747a55b580bace8ffa02febff8f046316694"} Oct 30 20:58:06.803640 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:06.803420 4611 scope.go:122] "RemoveContainer" containerID="9f55ee67ad822c60030083e1f0ed09869e45b9eb2d5d2eeef4659752733052d8" Oct 30 20:58:06.805111 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:06.805090 4611 generic.go:356] "Generic (PLEG): container finished" podID="52e301d9-f378-4e54-9515-01a6680d9fd7" containerID="36cf0baf044dd3dc38d449c968df0fbd734cf76c3a1acfd9d63f3083987ced58" exitCode=0 Oct 30 20:58:06.805232 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:06.805189 4611 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-cb60a2/distinct-w3-cb60a2-kserve-5bb76dc74-86fds" Oct 30 20:58:06.805232 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:06.805196 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-cb60a2/distinct-w3-cb60a2-kserve-5bb76dc74-86fds" event={"ID":"52e301d9-f378-4e54-9515-01a6680d9fd7","Type":"ContainerDied","Data":"36cf0baf044dd3dc38d449c968df0fbd734cf76c3a1acfd9d63f3083987ced58"} Oct 30 20:58:06.805232 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:06.805221 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-cb60a2/distinct-w3-cb60a2-kserve-5bb76dc74-86fds" event={"ID":"52e301d9-f378-4e54-9515-01a6680d9fd7","Type":"ContainerDied","Data":"5573e26a002b83469b1768d80b0d819d38c1c8aea0f4663b2c212c6f0bc1c2b7"} Oct 30 20:58:06.815816 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:06.815801 4611 scope.go:122] "RemoveContainer" containerID="9f55ee67ad822c60030083e1f0ed09869e45b9eb2d5d2eeef4659752733052d8" Oct 30 20:58:06.816084 ip-10-0-133-192 kubenswrapper[4611]: E1030 20:58:06.816063 4611 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f55ee67ad822c60030083e1f0ed09869e45b9eb2d5d2eeef4659752733052d8\": container with ID starting with 9f55ee67ad822c60030083e1f0ed09869e45b9eb2d5d2eeef4659752733052d8 not found: ID does not exist" containerID="9f55ee67ad822c60030083e1f0ed09869e45b9eb2d5d2eeef4659752733052d8" Oct 30 20:58:06.816133 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:06.816092 4611 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f55ee67ad822c60030083e1f0ed09869e45b9eb2d5d2eeef4659752733052d8"} err="failed to get container status \"9f55ee67ad822c60030083e1f0ed09869e45b9eb2d5d2eeef4659752733052d8\": rpc error: code = NotFound desc = could not find container \"9f55ee67ad822c60030083e1f0ed09869e45b9eb2d5d2eeef4659752733052d8\": container with ID starting with 9f55ee67ad822c60030083e1f0ed09869e45b9eb2d5d2eeef4659752733052d8 not found: ID does not exist" Oct 30 20:58:06.816133 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:06.816110 4611 scope.go:122] "RemoveContainer" containerID="36cf0baf044dd3dc38d449c968df0fbd734cf76c3a1acfd9d63f3083987ced58" Oct 30 20:58:06.824872 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:06.824857 4611 scope.go:122] "RemoveContainer" containerID="36cf0baf044dd3dc38d449c968df0fbd734cf76c3a1acfd9d63f3083987ced58" Oct 30 20:58:06.825117 ip-10-0-133-192 kubenswrapper[4611]: E1030 20:58:06.825098 4611 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"36cf0baf044dd3dc38d449c968df0fbd734cf76c3a1acfd9d63f3083987ced58\": container with ID starting with 36cf0baf044dd3dc38d449c968df0fbd734cf76c3a1acfd9d63f3083987ced58 not found: ID does not exist" containerID="36cf0baf044dd3dc38d449c968df0fbd734cf76c3a1acfd9d63f3083987ced58" Oct 30 20:58:06.825166 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:06.825124 4611 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"36cf0baf044dd3dc38d449c968df0fbd734cf76c3a1acfd9d63f3083987ced58"} err="failed to get container status \"36cf0baf044dd3dc38d449c968df0fbd734cf76c3a1acfd9d63f3083987ced58\": rpc error: code = NotFound desc = could not find container \"36cf0baf044dd3dc38d449c968df0fbd734cf76c3a1acfd9d63f3083987ced58\": container with ID starting with 36cf0baf044dd3dc38d449c968df0fbd734cf76c3a1acfd9d63f3083987ced58 not found: ID does not exist" Oct 30 20:58:06.830703 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:06.830683 4611 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-cb60a2/simulator-w3-cb60a2-kserve-6fd74b9d8b-vqfzf"] Oct 30 20:58:06.838558 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:06.838539 4611 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-cb60a2/simulator-w3-cb60a2-kserve-6fd74b9d8b-vqfzf"] Oct 30 20:58:06.847323 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:06.847302 4611 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-cb60a2/distinct-w3-cb60a2-kserve-5bb76dc74-86fds"] Oct 30 20:58:06.852829 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:06.852810 4611 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-cb60a2/distinct-w3-cb60a2-kserve-5bb76dc74-86fds"] Oct 30 20:58:07.282762 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:07.282726 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:58:07.474367 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:07.474337 4611 scope.go:122] "RemoveContainer" containerID="a9c16efe2304ca324ef84b191e5dbeaa4bf4950620f7877cae1f835e3e4c29df" Oct 30 20:58:07.488921 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:07.488886 4611 scope.go:122] "RemoveContainer" containerID="3ae27b1e50bb852c8549eef0ab5d09d2a49dd6f9440a828e529f497524e3ece1" Oct 30 20:58:07.609006 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:07.608972 4611 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="52e301d9-f378-4e54-9515-01a6680d9fd7" path="/var/lib/kubelet/pods/52e301d9-f378-4e54-9515-01a6680d9fd7/volumes" Oct 30 20:58:07.609765 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:07.609739 4611 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="711b167d-4e66-40d6-be75-33ede11f42c5" path="/var/lib/kubelet/pods/711b167d-4e66-40d6-be75-33ede11f42c5/volumes" Oct 30 20:58:07.909431 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:07.909366 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:58:08.607448 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:08.607401 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:58:09.295548 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:09.295478 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:58:09.994034 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:09.993990 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:58:10.918485 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:10.918449 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:58:11.598274 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:11.598224 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:58:11.770225 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:11.770188 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:58:12.147548 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:12.147494 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:58:12.301092 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:12.301056 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:58:12.451172 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:12.451072 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:58:12.608107 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:12.608022 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:58:12.939177 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:12.939138 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:58:13.526567 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:13.526497 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:58:14.515242 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:14.515182 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:58:14.946637 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:14.946604 4611 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-cb60a2-76455477c8-6w5qv"] Oct 30 20:58:14.946912 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:14.946883 4611 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-cb60a2-76455477c8-6w5qv" podUID="a74a618b-03cd-4949-9b6d-f91301dc19d1" containerName="maas-api" containerID="cri-o://7a190eb18771a8d343ae56f5ead972d1a702fda1f89c99959b8ae9ba0a723c73" gracePeriod=30 Oct 30 20:58:15.024085 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:15.024042 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:58:15.422722 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:15.422695 4611 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-cb60a2-76455477c8-6w5qv" Oct 30 20:58:15.449174 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:15.449137 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a74a618b-03cd-4949-9b6d-f91301dc19d1-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a74a618b-03cd-4949-9b6d-f91301dc19d1-maas-api-tls\") pod \"a74a618b-03cd-4949-9b6d-f91301dc19d1\" (UID: \"a74a618b-03cd-4949-9b6d-f91301dc19d1\") " Oct 30 20:58:15.449330 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:15.449251 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a74a618b-03cd-4949-9b6d-f91301dc19d1-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a74a618b-03cd-4949-9b6d-f91301dc19d1-metrics-certs\") pod \"a74a618b-03cd-4949-9b6d-f91301dc19d1\" (UID: \"a74a618b-03cd-4949-9b6d-f91301dc19d1\") " Oct 30 20:58:15.449330 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:15.449303 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a74a618b-03cd-4949-9b6d-f91301dc19d1-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a74a618b-03cd-4949-9b6d-f91301dc19d1-openshift-service-ca\") pod \"a74a618b-03cd-4949-9b6d-f91301dc19d1\" (UID: \"a74a618b-03cd-4949-9b6d-f91301dc19d1\") " Oct 30 20:58:15.449583 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:15.449496 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a74a618b-03cd-4949-9b6d-f91301dc19d1-kube-api-access-p5gfb\" (UniqueName: \"kubernetes.io/projected/a74a618b-03cd-4949-9b6d-f91301dc19d1-kube-api-access-p5gfb\") pod \"a74a618b-03cd-4949-9b6d-f91301dc19d1\" (UID: \"a74a618b-03cd-4949-9b6d-f91301dc19d1\") " Oct 30 20:58:15.450072 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:15.450040 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a74a618b-03cd-4949-9b6d-f91301dc19d1-openshift-service-ca" pod "a74a618b-03cd-4949-9b6d-f91301dc19d1" (UID: "a74a618b-03cd-4949-9b6d-f91301dc19d1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:58:15.450205 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:15.450186 4611 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a74a618b-03cd-4949-9b6d-f91301dc19d1-openshift-service-ca\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:15.452499 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:15.452266 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a74a618b-03cd-4949-9b6d-f91301dc19d1-metrics-certs" pod "a74a618b-03cd-4949-9b6d-f91301dc19d1" (UID: "a74a618b-03cd-4949-9b6d-f91301dc19d1"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:58:15.452978 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:15.452955 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a74a618b-03cd-4949-9b6d-f91301dc19d1-maas-api-tls" pod "a74a618b-03cd-4949-9b6d-f91301dc19d1" (UID: "a74a618b-03cd-4949-9b6d-f91301dc19d1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:58:15.453484 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:15.453430 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a74a618b-03cd-4949-9b6d-f91301dc19d1-kube-api-access-p5gfb" pod "a74a618b-03cd-4949-9b6d-f91301dc19d1" (UID: "a74a618b-03cd-4949-9b6d-f91301dc19d1"). InnerVolumeSpecName "kube-api-access-p5gfb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:58:15.552110 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:15.552071 4611 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p5gfb\" (UniqueName: \"kubernetes.io/projected/a74a618b-03cd-4949-9b6d-f91301dc19d1-kube-api-access-p5gfb\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:15.552110 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:15.552109 4611 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a74a618b-03cd-4949-9b6d-f91301dc19d1-maas-api-tls\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:15.552590 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:15.552124 4611 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a74a618b-03cd-4949-9b6d-f91301dc19d1-metrics-certs\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:15.874665 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:15.874626 4611 generic.go:356] "Generic (PLEG): container finished" podID="a74a618b-03cd-4949-9b6d-f91301dc19d1" containerID="7a190eb18771a8d343ae56f5ead972d1a702fda1f89c99959b8ae9ba0a723c73" exitCode=0 Oct 30 20:58:15.874852 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:15.874675 4611 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-cb60a2-76455477c8-6w5qv" Oct 30 20:58:15.874852 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:15.874706 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-cb60a2-76455477c8-6w5qv" event={"ID":"a74a618b-03cd-4949-9b6d-f91301dc19d1","Type":"ContainerDied","Data":"7a190eb18771a8d343ae56f5ead972d1a702fda1f89c99959b8ae9ba0a723c73"} Oct 30 20:58:15.874852 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:15.874758 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-cb60a2-76455477c8-6w5qv" event={"ID":"a74a618b-03cd-4949-9b6d-f91301dc19d1","Type":"ContainerDied","Data":"60126af043f2d4f7b92812b7b8a0d3178da3fc4ff26933c4a12557de7e3144e6"} Oct 30 20:58:15.874852 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:15.874781 4611 scope.go:122] "RemoveContainer" containerID="7a190eb18771a8d343ae56f5ead972d1a702fda1f89c99959b8ae9ba0a723c73" Oct 30 20:58:15.884951 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:15.884925 4611 scope.go:122] "RemoveContainer" containerID="7a190eb18771a8d343ae56f5ead972d1a702fda1f89c99959b8ae9ba0a723c73" Oct 30 20:58:15.885186 ip-10-0-133-192 kubenswrapper[4611]: E1030 20:58:15.885166 4611 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a190eb18771a8d343ae56f5ead972d1a702fda1f89c99959b8ae9ba0a723c73\": container with ID starting with 7a190eb18771a8d343ae56f5ead972d1a702fda1f89c99959b8ae9ba0a723c73 not found: ID does not exist" containerID="7a190eb18771a8d343ae56f5ead972d1a702fda1f89c99959b8ae9ba0a723c73" Oct 30 20:58:15.885254 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:15.885198 4611 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a190eb18771a8d343ae56f5ead972d1a702fda1f89c99959b8ae9ba0a723c73"} err="failed to get container status \"7a190eb18771a8d343ae56f5ead972d1a702fda1f89c99959b8ae9ba0a723c73\": rpc error: code = NotFound desc = could not find container \"7a190eb18771a8d343ae56f5ead972d1a702fda1f89c99959b8ae9ba0a723c73\": container with ID starting with 7a190eb18771a8d343ae56f5ead972d1a702fda1f89c99959b8ae9ba0a723c73 not found: ID does not exist" Oct 30 20:58:15.894769 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:15.894746 4611 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-cb60a2-76455477c8-6w5qv"] Oct 30 20:58:15.909078 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:15.909058 4611 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-cb60a2-76455477c8-6w5qv"] Oct 30 20:58:16.295470 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:16.294272 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:58:16.991109 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:16.989493 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:58:17.595082 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:17.595036 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:58:17.624731 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:17.624694 4611 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a74a618b-03cd-4949-9b6d-f91301dc19d1" path="/var/lib/kubelet/pods/a74a618b-03cd-4949-9b6d-f91301dc19d1/volumes" Oct 30 20:58:18.244808 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:18.244775 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:58:18.811391 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:18.810579 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:58:19.417879 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:19.417840 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:58:20.069224 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:20.069186 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:58:20.216348 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:20.216312 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:58:20.384131 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:20.384070 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:58:20.406056 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:20.406017 4611 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-6xtqj"] Oct 30 20:58:20.406341 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:20.406303 4611 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-6xtqj" podUID="63d294fd-5293-4752-9116-0aafa81d5e4f" containerName="istio-proxy" containerID="cri-o://a215e69034ca57fda6e5dbb3630c604d2a87905bf54b35fa8f5e49e148efb3ad" gracePeriod=30 Oct 30 20:58:20.416081 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:20.416048 4611 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-mwkjq"] Oct 30 20:58:20.416803 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:20.416771 4611 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-mwkjq" podUID="59bd8e32-17bd-4753-ab30-1e32a1a2085f" containerName="istio-proxy" containerID="cri-o://b2f26a42242237c9d55b800f03e076c751ae5f0f865aa72e2a225bdc99efc463" gracePeriod=30 Oct 30 20:58:20.444543 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:20.444472 4611 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-kpn5r"] Oct 30 20:58:20.444947 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:20.444755 4611 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-kpn5r" podUID="25d72035-53b1-4de2-b594-34c64b9b1560" containerName="istio-proxy" containerID="cri-o://702ebef0ccce8010474bb2a51a9b783cde0b5a29f31f3643c02904f15a075355" gracePeriod=30 Oct 30 20:58:20.588871 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:20.588834 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:58:21.155224 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:21.155172 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:58:22.035802 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:22.035764 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:58:22.709252 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:22.709218 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:58:23.268484 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:23.268451 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:58:23.422057 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:23.422022 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:58:24.014241 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:24.014197 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:58:24.138023 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:24.137982 4611 patch_prober.go:29] interesting pod/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-mwkjq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 20:58:24.138190 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:24.138053 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-mwkjq" podUID="59bd8e32-17bd-4753-ab30-1e32a1a2085f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 20:58:24.192556 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:24.192425 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:58:24.375925 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:24.375890 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:58:24.490530 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:24.490475 4611 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-9d6hc"] Oct 30 20:58:24.490804 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:24.490774 4611 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-9d6hc" podUID="f1f5526b-53f1-4c2b-a801-49ee75e16f9c" containerName="istio-proxy" containerID="cri-o://f1034ebd6de0aa7d70504195eb408bf9c78e55bc0caeffefdc776f925bdba0c2" gracePeriod=30 Oct 30 20:58:24.496522 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:24.496485 4611 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-sxmcf"] Oct 30 20:58:24.496757 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:24.496719 4611 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-sxmcf" podUID="cfbb7630-0e90-41df-9d4e-ade03ff1f522" containerName="istio-proxy" containerID="cri-o://27e240f53d2b9886252ca3e879894bca4d4306d4b9d49918f76959e495672fa2" gracePeriod=30 Oct 30 20:58:24.500155 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:24.500130 4611 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-2pw94"] Oct 30 20:58:24.500303 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:24.500285 4611 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-2pw94" podUID="8603ecfb-6faf-4f18-bb0b-2aee0341b0e3" containerName="istio-proxy" containerID="cri-o://02bf56353d665042d4709c90beafe218f03c0517f1183ba82c0dfba1a7f1a493" gracePeriod=30 Oct 30 20:58:24.553287 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:24.553256 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:58:24.934217 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:24.934180 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:58:25.610392 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:25.610092 4611 patch_prober.go:29] interesting pod/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-2pw94 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 20:58:25.610392 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:25.610159 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-2pw94" podUID="8603ecfb-6faf-4f18-bb0b-2aee0341b0e3" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 20:58:25.651294 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:25.651242 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:58:25.726734 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:25.726702 4611 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-6xtqj" Oct 30 20:58:25.823913 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:25.823880 4611 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-kpn5r" Oct 30 20:58:25.834251 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:25.834224 4611 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-mwkjq" Oct 30 20:58:25.834826 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:25.834803 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63d294fd-5293-4752-9116-0aafa81d5e4f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63d294fd-5293-4752-9116-0aafa81d5e4f-workload-socket\") pod \"63d294fd-5293-4752-9116-0aafa81d5e4f\" (UID: \"63d294fd-5293-4752-9116-0aafa81d5e4f\") " Oct 30 20:58:25.834933 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:25.834859 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63d294fd-5293-4752-9116-0aafa81d5e4f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63d294fd-5293-4752-9116-0aafa81d5e4f-credential-socket\") pod \"63d294fd-5293-4752-9116-0aafa81d5e4f\" (UID: \"63d294fd-5293-4752-9116-0aafa81d5e4f\") " Oct 30 20:58:25.834933 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:25.834913 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63d294fd-5293-4752-9116-0aafa81d5e4f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63d294fd-5293-4752-9116-0aafa81d5e4f-istio-data\") pod \"63d294fd-5293-4752-9116-0aafa81d5e4f\" (UID: \"63d294fd-5293-4752-9116-0aafa81d5e4f\") " Oct 30 20:58:25.835050 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:25.835012 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63d294fd-5293-4752-9116-0aafa81d5e4f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63d294fd-5293-4752-9116-0aafa81d5e4f-workload-certs\") pod \"63d294fd-5293-4752-9116-0aafa81d5e4f\" (UID: \"63d294fd-5293-4752-9116-0aafa81d5e4f\") " Oct 30 20:58:25.835128 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:25.835102 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63d294fd-5293-4752-9116-0aafa81d5e4f-workload-socket" pod "63d294fd-5293-4752-9116-0aafa81d5e4f" (UID: "63d294fd-5293-4752-9116-0aafa81d5e4f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:58:25.835224 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:25.835199 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/63d294fd-5293-4752-9116-0aafa81d5e4f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63d294fd-5293-4752-9116-0aafa81d5e4f-istiod-ca-cert\") pod \"63d294fd-5293-4752-9116-0aafa81d5e4f\" (UID: \"63d294fd-5293-4752-9116-0aafa81d5e4f\") " Oct 30 20:58:25.835329 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:25.835240 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63d294fd-5293-4752-9116-0aafa81d5e4f-credential-socket" pod "63d294fd-5293-4752-9116-0aafa81d5e4f" (UID: "63d294fd-5293-4752-9116-0aafa81d5e4f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:58:25.835329 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:25.835288 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/63d294fd-5293-4752-9116-0aafa81d5e4f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63d294fd-5293-4752-9116-0aafa81d5e4f-istio-podinfo\") pod \"63d294fd-5293-4752-9116-0aafa81d5e4f\" (UID: \"63d294fd-5293-4752-9116-0aafa81d5e4f\") " Oct 30 20:58:25.835406 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:25.835321 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63d294fd-5293-4752-9116-0aafa81d5e4f-istio-data" pod "63d294fd-5293-4752-9116-0aafa81d5e4f" (UID: "63d294fd-5293-4752-9116-0aafa81d5e4f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:58:25.835406 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:25.835392 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/63d294fd-5293-4752-9116-0aafa81d5e4f-kube-api-access-wgzm5\" (UniqueName: \"kubernetes.io/projected/63d294fd-5293-4752-9116-0aafa81d5e4f-kube-api-access-wgzm5\") pod \"63d294fd-5293-4752-9116-0aafa81d5e4f\" (UID: \"63d294fd-5293-4752-9116-0aafa81d5e4f\") " Oct 30 20:58:25.835474 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:25.835442 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63d294fd-5293-4752-9116-0aafa81d5e4f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63d294fd-5293-4752-9116-0aafa81d5e4f-istio-envoy\") pod \"63d294fd-5293-4752-9116-0aafa81d5e4f\" (UID: \"63d294fd-5293-4752-9116-0aafa81d5e4f\") " Oct 30 20:58:25.835474 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:25.835452 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63d294fd-5293-4752-9116-0aafa81d5e4f-workload-certs" pod "63d294fd-5293-4752-9116-0aafa81d5e4f" (UID: "63d294fd-5293-4752-9116-0aafa81d5e4f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:58:25.835474 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:25.835469 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/63d294fd-5293-4752-9116-0aafa81d5e4f-istio-token\" (UniqueName: \"kubernetes.io/projected/63d294fd-5293-4752-9116-0aafa81d5e4f-istio-token\") pod \"63d294fd-5293-4752-9116-0aafa81d5e4f\" (UID: \"63d294fd-5293-4752-9116-0aafa81d5e4f\") " Oct 30 20:58:25.835725 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:25.835701 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63d294fd-5293-4752-9116-0aafa81d5e4f-istiod-ca-cert" pod "63d294fd-5293-4752-9116-0aafa81d5e4f" (UID: "63d294fd-5293-4752-9116-0aafa81d5e4f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:58:25.836211 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:25.836184 4611 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63d294fd-5293-4752-9116-0aafa81d5e4f-workload-certs\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:25.836382 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:25.836214 4611 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63d294fd-5293-4752-9116-0aafa81d5e4f-istiod-ca-cert\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:25.836382 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:25.836229 4611 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63d294fd-5293-4752-9116-0aafa81d5e4f-workload-socket\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:25.836382 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:25.836242 4611 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63d294fd-5293-4752-9116-0aafa81d5e4f-credential-socket\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:25.836382 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:25.836255 4611 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63d294fd-5293-4752-9116-0aafa81d5e4f-istio-data\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:25.838211 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:25.838186 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63d294fd-5293-4752-9116-0aafa81d5e4f-kube-api-access-wgzm5" pod "63d294fd-5293-4752-9116-0aafa81d5e4f" (UID: "63d294fd-5293-4752-9116-0aafa81d5e4f"). InnerVolumeSpecName "kube-api-access-wgzm5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:58:25.839002 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:25.838985 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63d294fd-5293-4752-9116-0aafa81d5e4f-istio-token" pod "63d294fd-5293-4752-9116-0aafa81d5e4f" (UID: "63d294fd-5293-4752-9116-0aafa81d5e4f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:58:25.839964 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:25.839940 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63d294fd-5293-4752-9116-0aafa81d5e4f-istio-envoy" pod "63d294fd-5293-4752-9116-0aafa81d5e4f" (UID: "63d294fd-5293-4752-9116-0aafa81d5e4f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:58:25.844046 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:25.844026 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/63d294fd-5293-4752-9116-0aafa81d5e4f-istio-podinfo" pod "63d294fd-5293-4752-9116-0aafa81d5e4f" (UID: "63d294fd-5293-4752-9116-0aafa81d5e4f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:58:25.937834 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:25.937796 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/25d72035-53b1-4de2-b594-34c64b9b1560-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25d72035-53b1-4de2-b594-34c64b9b1560-istiod-ca-cert\") pod \"25d72035-53b1-4de2-b594-34c64b9b1560\" (UID: \"25d72035-53b1-4de2-b594-34c64b9b1560\") " Oct 30 20:58:25.938026 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:25.937851 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/25d72035-53b1-4de2-b594-34c64b9b1560-kube-api-access-v9222\" (UniqueName: \"kubernetes.io/projected/25d72035-53b1-4de2-b594-34c64b9b1560-kube-api-access-v9222\") pod \"25d72035-53b1-4de2-b594-34c64b9b1560\" (UID: \"25d72035-53b1-4de2-b594-34c64b9b1560\") " Oct 30 20:58:25.938026 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:25.937926 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/59bd8e32-17bd-4753-ab30-1e32a1a2085f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59bd8e32-17bd-4753-ab30-1e32a1a2085f-istio-podinfo\") pod \"59bd8e32-17bd-4753-ab30-1e32a1a2085f\" (UID: \"59bd8e32-17bd-4753-ab30-1e32a1a2085f\") " Oct 30 20:58:25.938026 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:25.937981 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/59bd8e32-17bd-4753-ab30-1e32a1a2085f-istio-token\" (UniqueName: \"kubernetes.io/projected/59bd8e32-17bd-4753-ab30-1e32a1a2085f-istio-token\") pod \"59bd8e32-17bd-4753-ab30-1e32a1a2085f\" (UID: \"59bd8e32-17bd-4753-ab30-1e32a1a2085f\") " Oct 30 20:58:25.938188 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:25.938025 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/25d72035-53b1-4de2-b594-34c64b9b1560-istio-token\" (UniqueName: \"kubernetes.io/projected/25d72035-53b1-4de2-b594-34c64b9b1560-istio-token\") pod \"25d72035-53b1-4de2-b594-34c64b9b1560\" (UID: \"25d72035-53b1-4de2-b594-34c64b9b1560\") " Oct 30 20:58:25.938188 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:25.938054 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/25d72035-53b1-4de2-b594-34c64b9b1560-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25d72035-53b1-4de2-b594-34c64b9b1560-istio-podinfo\") pod \"25d72035-53b1-4de2-b594-34c64b9b1560\" (UID: \"25d72035-53b1-4de2-b594-34c64b9b1560\") " Oct 30 20:58:25.938188 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:25.938087 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/59bd8e32-17bd-4753-ab30-1e32a1a2085f-kube-api-access-9vr9n\" (UniqueName: \"kubernetes.io/projected/59bd8e32-17bd-4753-ab30-1e32a1a2085f-kube-api-access-9vr9n\") pod \"59bd8e32-17bd-4753-ab30-1e32a1a2085f\" (UID: \"59bd8e32-17bd-4753-ab30-1e32a1a2085f\") " Oct 30 20:58:25.938188 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:25.938173 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25d72035-53b1-4de2-b594-34c64b9b1560-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25d72035-53b1-4de2-b594-34c64b9b1560-credential-socket\") pod \"25d72035-53b1-4de2-b594-34c64b9b1560\" (UID: \"25d72035-53b1-4de2-b594-34c64b9b1560\") " Oct 30 20:58:25.938391 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:25.938241 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25d72035-53b1-4de2-b594-34c64b9b1560-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25d72035-53b1-4de2-b594-34c64b9b1560-workload-certs\") pod \"25d72035-53b1-4de2-b594-34c64b9b1560\" (UID: \"25d72035-53b1-4de2-b594-34c64b9b1560\") " Oct 30 20:58:25.938391 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:25.938280 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59bd8e32-17bd-4753-ab30-1e32a1a2085f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59bd8e32-17bd-4753-ab30-1e32a1a2085f-workload-certs\") pod \"59bd8e32-17bd-4753-ab30-1e32a1a2085f\" (UID: \"59bd8e32-17bd-4753-ab30-1e32a1a2085f\") " Oct 30 20:58:25.938498 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:25.938386 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59bd8e32-17bd-4753-ab30-1e32a1a2085f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59bd8e32-17bd-4753-ab30-1e32a1a2085f-istio-envoy\") pod \"59bd8e32-17bd-4753-ab30-1e32a1a2085f\" (UID: \"59bd8e32-17bd-4753-ab30-1e32a1a2085f\") " Oct 30 20:58:25.938498 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:25.938431 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59bd8e32-17bd-4753-ab30-1e32a1a2085f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59bd8e32-17bd-4753-ab30-1e32a1a2085f-istio-data\") pod \"59bd8e32-17bd-4753-ab30-1e32a1a2085f\" (UID: \"59bd8e32-17bd-4753-ab30-1e32a1a2085f\") " Oct 30 20:58:25.938498 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:25.938479 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25d72035-53b1-4de2-b594-34c64b9b1560-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25d72035-53b1-4de2-b594-34c64b9b1560-istio-data\") pod \"25d72035-53b1-4de2-b594-34c64b9b1560\" (UID: \"25d72035-53b1-4de2-b594-34c64b9b1560\") " Oct 30 20:58:25.938679 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:25.938568 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/59bd8e32-17bd-4753-ab30-1e32a1a2085f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59bd8e32-17bd-4753-ab30-1e32a1a2085f-istiod-ca-cert\") pod \"59bd8e32-17bd-4753-ab30-1e32a1a2085f\" (UID: \"59bd8e32-17bd-4753-ab30-1e32a1a2085f\") " Oct 30 20:58:25.938679 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:25.938621 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25d72035-53b1-4de2-b594-34c64b9b1560-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25d72035-53b1-4de2-b594-34c64b9b1560-workload-socket\") pod \"25d72035-53b1-4de2-b594-34c64b9b1560\" (UID: \"25d72035-53b1-4de2-b594-34c64b9b1560\") " Oct 30 20:58:25.938679 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:25.938670 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59bd8e32-17bd-4753-ab30-1e32a1a2085f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59bd8e32-17bd-4753-ab30-1e32a1a2085f-credential-socket\") pod \"59bd8e32-17bd-4753-ab30-1e32a1a2085f\" (UID: \"59bd8e32-17bd-4753-ab30-1e32a1a2085f\") " Oct 30 20:58:25.938848 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:25.938725 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25d72035-53b1-4de2-b594-34c64b9b1560-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25d72035-53b1-4de2-b594-34c64b9b1560-istio-envoy\") pod \"25d72035-53b1-4de2-b594-34c64b9b1560\" (UID: \"25d72035-53b1-4de2-b594-34c64b9b1560\") " Oct 30 20:58:25.938848 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:25.938772 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59bd8e32-17bd-4753-ab30-1e32a1a2085f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59bd8e32-17bd-4753-ab30-1e32a1a2085f-workload-socket\") pod \"59bd8e32-17bd-4753-ab30-1e32a1a2085f\" (UID: \"59bd8e32-17bd-4753-ab30-1e32a1a2085f\") " Oct 30 20:58:25.939652 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:25.939585 4611 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63d294fd-5293-4752-9116-0aafa81d5e4f-istio-podinfo\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:25.939652 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:25.939616 4611 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wgzm5\" (UniqueName: \"kubernetes.io/projected/63d294fd-5293-4752-9116-0aafa81d5e4f-kube-api-access-wgzm5\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:25.939652 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:25.939630 4611 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63d294fd-5293-4752-9116-0aafa81d5e4f-istio-envoy\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:25.939652 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:25.939645 4611 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63d294fd-5293-4752-9116-0aafa81d5e4f-istio-token\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:25.939927 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:25.939904 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59bd8e32-17bd-4753-ab30-1e32a1a2085f-workload-socket" pod "59bd8e32-17bd-4753-ab30-1e32a1a2085f" (UID: "59bd8e32-17bd-4753-ab30-1e32a1a2085f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:58:25.940244 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:25.940222 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25d72035-53b1-4de2-b594-34c64b9b1560-istiod-ca-cert" pod "25d72035-53b1-4de2-b594-34c64b9b1560" (UID: "25d72035-53b1-4de2-b594-34c64b9b1560"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:58:25.940850 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:25.940822 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59bd8e32-17bd-4753-ab30-1e32a1a2085f-workload-certs" pod "59bd8e32-17bd-4753-ab30-1e32a1a2085f" (UID: "59bd8e32-17bd-4753-ab30-1e32a1a2085f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:58:25.941043 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:25.941015 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25d72035-53b1-4de2-b594-34c64b9b1560-workload-certs" pod "25d72035-53b1-4de2-b594-34c64b9b1560" (UID: "25d72035-53b1-4de2-b594-34c64b9b1560"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:58:25.941261 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:25.941234 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59bd8e32-17bd-4753-ab30-1e32a1a2085f-credential-socket" pod "59bd8e32-17bd-4753-ab30-1e32a1a2085f" (UID: "59bd8e32-17bd-4753-ab30-1e32a1a2085f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:58:25.941478 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:25.941453 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25d72035-53b1-4de2-b594-34c64b9b1560-workload-socket" pod "25d72035-53b1-4de2-b594-34c64b9b1560" (UID: "25d72035-53b1-4de2-b594-34c64b9b1560"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:58:25.941618 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:25.941582 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59bd8e32-17bd-4753-ab30-1e32a1a2085f-istiod-ca-cert" pod "59bd8e32-17bd-4753-ab30-1e32a1a2085f" (UID: "59bd8e32-17bd-4753-ab30-1e32a1a2085f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:58:25.941948 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:25.941921 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25d72035-53b1-4de2-b594-34c64b9b1560-credential-socket" pod "25d72035-53b1-4de2-b594-34c64b9b1560" (UID: "25d72035-53b1-4de2-b594-34c64b9b1560"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:58:25.944198 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:25.944176 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/25d72035-53b1-4de2-b594-34c64b9b1560-istio-podinfo" pod "25d72035-53b1-4de2-b594-34c64b9b1560" (UID: "25d72035-53b1-4de2-b594-34c64b9b1560"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:58:25.944414 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:25.944346 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59bd8e32-17bd-4753-ab30-1e32a1a2085f-istio-envoy" pod "59bd8e32-17bd-4753-ab30-1e32a1a2085f" (UID: "59bd8e32-17bd-4753-ab30-1e32a1a2085f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:58:25.944414 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:25.944354 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59bd8e32-17bd-4753-ab30-1e32a1a2085f-istio-data" pod "59bd8e32-17bd-4753-ab30-1e32a1a2085f" (UID: "59bd8e32-17bd-4753-ab30-1e32a1a2085f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:58:25.944665 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:25.944463 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25d72035-53b1-4de2-b594-34c64b9b1560-istio-envoy" pod "25d72035-53b1-4de2-b594-34c64b9b1560" (UID: "25d72035-53b1-4de2-b594-34c64b9b1560"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:58:25.944665 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:25.944571 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25d72035-53b1-4de2-b594-34c64b9b1560-istio-data" pod "25d72035-53b1-4de2-b594-34c64b9b1560" (UID: "25d72035-53b1-4de2-b594-34c64b9b1560"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:58:25.944958 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:25.944938 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59bd8e32-17bd-4753-ab30-1e32a1a2085f-kube-api-access-9vr9n" pod "59bd8e32-17bd-4753-ab30-1e32a1a2085f" (UID: "59bd8e32-17bd-4753-ab30-1e32a1a2085f"). InnerVolumeSpecName "kube-api-access-9vr9n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:58:25.945198 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:25.945175 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59bd8e32-17bd-4753-ab30-1e32a1a2085f-istio-token" pod "59bd8e32-17bd-4753-ab30-1e32a1a2085f" (UID: "59bd8e32-17bd-4753-ab30-1e32a1a2085f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:58:25.947795 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:25.947770 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25d72035-53b1-4de2-b594-34c64b9b1560-kube-api-access-v9222" pod "25d72035-53b1-4de2-b594-34c64b9b1560" (UID: "25d72035-53b1-4de2-b594-34c64b9b1560"). InnerVolumeSpecName "kube-api-access-v9222". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:58:25.947795 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:25.947787 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25d72035-53b1-4de2-b594-34c64b9b1560-istio-token" pod "25d72035-53b1-4de2-b594-34c64b9b1560" (UID: "25d72035-53b1-4de2-b594-34c64b9b1560"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:58:25.949293 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:25.949274 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/59bd8e32-17bd-4753-ab30-1e32a1a2085f-istio-podinfo" pod "59bd8e32-17bd-4753-ab30-1e32a1a2085f" (UID: "59bd8e32-17bd-4753-ab30-1e32a1a2085f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:58:25.960586 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:25.960561 4611 generic.go:356] "Generic (PLEG): container finished" podID="59bd8e32-17bd-4753-ab30-1e32a1a2085f" containerID="b2f26a42242237c9d55b800f03e076c751ae5f0f865aa72e2a225bdc99efc463" exitCode=0 Oct 30 20:58:25.960666 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:25.960610 4611 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-mwkjq" Oct 30 20:58:25.960666 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:25.960631 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-mwkjq" event={"ID":"59bd8e32-17bd-4753-ab30-1e32a1a2085f","Type":"ContainerDied","Data":"b2f26a42242237c9d55b800f03e076c751ae5f0f865aa72e2a225bdc99efc463"} Oct 30 20:58:25.960754 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:25.960681 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-mwkjq" event={"ID":"59bd8e32-17bd-4753-ab30-1e32a1a2085f","Type":"ContainerDied","Data":"e089fb9ae62e8db71d7725f4dddb373e78880da2c4552048c2f57da16e09e185"} Oct 30 20:58:25.960754 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:25.960706 4611 scope.go:122] "RemoveContainer" containerID="b2f26a42242237c9d55b800f03e076c751ae5f0f865aa72e2a225bdc99efc463" Oct 30 20:58:25.962642 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:25.962619 4611 generic.go:356] "Generic (PLEG): container finished" podID="63d294fd-5293-4752-9116-0aafa81d5e4f" containerID="a215e69034ca57fda6e5dbb3630c604d2a87905bf54b35fa8f5e49e148efb3ad" exitCode=0 Oct 30 20:58:25.962745 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:25.962682 4611 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-6xtqj" Oct 30 20:58:25.962745 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:25.962697 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-6xtqj" event={"ID":"63d294fd-5293-4752-9116-0aafa81d5e4f","Type":"ContainerDied","Data":"a215e69034ca57fda6e5dbb3630c604d2a87905bf54b35fa8f5e49e148efb3ad"} Oct 30 20:58:25.962851 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:25.962747 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-6xtqj" event={"ID":"63d294fd-5293-4752-9116-0aafa81d5e4f","Type":"ContainerDied","Data":"6ebdf9bc66c8d06ec5e5175d22164b5a4a0daa22f36a73ca3d6874bf5ef0ebf2"} Oct 30 20:58:25.969521 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:25.965193 4611 generic.go:356] "Generic (PLEG): container finished" podID="25d72035-53b1-4de2-b594-34c64b9b1560" containerID="702ebef0ccce8010474bb2a51a9b783cde0b5a29f31f3643c02904f15a075355" exitCode=0 Oct 30 20:58:25.969521 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:25.965233 4611 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-kpn5r" Oct 30 20:58:25.969521 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:25.965274 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-kpn5r" event={"ID":"25d72035-53b1-4de2-b594-34c64b9b1560","Type":"ContainerDied","Data":"702ebef0ccce8010474bb2a51a9b783cde0b5a29f31f3643c02904f15a075355"} Oct 30 20:58:25.969521 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:25.965312 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-kpn5r" event={"ID":"25d72035-53b1-4de2-b594-34c64b9b1560","Type":"ContainerDied","Data":"9e26e9839e37016426ef8bb1778a5245222a9ea492ad2b23182087f6a869867b"} Oct 30 20:58:25.972269 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:25.972178 4611 scope.go:122] "RemoveContainer" containerID="b2f26a42242237c9d55b800f03e076c751ae5f0f865aa72e2a225bdc99efc463" Oct 30 20:58:25.972504 ip-10-0-133-192 kubenswrapper[4611]: E1030 20:58:25.972478 4611 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b2f26a42242237c9d55b800f03e076c751ae5f0f865aa72e2a225bdc99efc463\": container with ID starting with b2f26a42242237c9d55b800f03e076c751ae5f0f865aa72e2a225bdc99efc463 not found: ID does not exist" containerID="b2f26a42242237c9d55b800f03e076c751ae5f0f865aa72e2a225bdc99efc463" Oct 30 20:58:25.972621 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:25.972522 4611 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2f26a42242237c9d55b800f03e076c751ae5f0f865aa72e2a225bdc99efc463"} err="failed to get container status \"b2f26a42242237c9d55b800f03e076c751ae5f0f865aa72e2a225bdc99efc463\": rpc error: code = NotFound desc = could not find container \"b2f26a42242237c9d55b800f03e076c751ae5f0f865aa72e2a225bdc99efc463\": container with ID starting with b2f26a42242237c9d55b800f03e076c751ae5f0f865aa72e2a225bdc99efc463 not found: ID does not exist" Oct 30 20:58:25.972621 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:25.972542 4611 scope.go:122] "RemoveContainer" containerID="a215e69034ca57fda6e5dbb3630c604d2a87905bf54b35fa8f5e49e148efb3ad" Oct 30 20:58:25.982752 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:25.982738 4611 scope.go:122] "RemoveContainer" containerID="9616177cca0955ef3462c13286375e84733d645de3ecd8ef6b9861e7317f425c" Oct 30 20:58:25.991454 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:25.991430 4611 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-mwkjq"] Oct 30 20:58:25.993110 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:25.993088 4611 scope.go:122] "RemoveContainer" containerID="a215e69034ca57fda6e5dbb3630c604d2a87905bf54b35fa8f5e49e148efb3ad" Oct 30 20:58:25.993427 ip-10-0-133-192 kubenswrapper[4611]: E1030 20:58:25.993384 4611 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a215e69034ca57fda6e5dbb3630c604d2a87905bf54b35fa8f5e49e148efb3ad\": container with ID starting with a215e69034ca57fda6e5dbb3630c604d2a87905bf54b35fa8f5e49e148efb3ad not found: ID does not exist" containerID="a215e69034ca57fda6e5dbb3630c604d2a87905bf54b35fa8f5e49e148efb3ad" Oct 30 20:58:25.993538 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:25.993423 4611 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a215e69034ca57fda6e5dbb3630c604d2a87905bf54b35fa8f5e49e148efb3ad"} err="failed to get container status \"a215e69034ca57fda6e5dbb3630c604d2a87905bf54b35fa8f5e49e148efb3ad\": rpc error: code = NotFound desc = could not find container \"a215e69034ca57fda6e5dbb3630c604d2a87905bf54b35fa8f5e49e148efb3ad\": container with ID starting with a215e69034ca57fda6e5dbb3630c604d2a87905bf54b35fa8f5e49e148efb3ad not found: ID does not exist" Oct 30 20:58:25.993538 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:25.993438 4611 scope.go:122] "RemoveContainer" containerID="9616177cca0955ef3462c13286375e84733d645de3ecd8ef6b9861e7317f425c" Oct 30 20:58:25.993776 ip-10-0-133-192 kubenswrapper[4611]: E1030 20:58:25.993757 4611 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9616177cca0955ef3462c13286375e84733d645de3ecd8ef6b9861e7317f425c\": container with ID starting with 9616177cca0955ef3462c13286375e84733d645de3ecd8ef6b9861e7317f425c not found: ID does not exist" containerID="9616177cca0955ef3462c13286375e84733d645de3ecd8ef6b9861e7317f425c" Oct 30 20:58:25.993915 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:25.993780 4611 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9616177cca0955ef3462c13286375e84733d645de3ecd8ef6b9861e7317f425c"} err="failed to get container status \"9616177cca0955ef3462c13286375e84733d645de3ecd8ef6b9861e7317f425c\": rpc error: code = NotFound desc = could not find container \"9616177cca0955ef3462c13286375e84733d645de3ecd8ef6b9861e7317f425c\": container with ID starting with 9616177cca0955ef3462c13286375e84733d645de3ecd8ef6b9861e7317f425c not found: ID does not exist" Oct 30 20:58:25.993915 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:25.993794 4611 scope.go:122] "RemoveContainer" containerID="702ebef0ccce8010474bb2a51a9b783cde0b5a29f31f3643c02904f15a075355" Oct 30 20:58:25.996802 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:25.996782 4611 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-mwkjq"] Oct 30 20:58:26.003674 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:26.003644 4611 scope.go:122] "RemoveContainer" containerID="702ebef0ccce8010474bb2a51a9b783cde0b5a29f31f3643c02904f15a075355" Oct 30 20:58:26.004040 ip-10-0-133-192 kubenswrapper[4611]: E1030 20:58:26.003990 4611 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"702ebef0ccce8010474bb2a51a9b783cde0b5a29f31f3643c02904f15a075355\": container with ID starting with 702ebef0ccce8010474bb2a51a9b783cde0b5a29f31f3643c02904f15a075355 not found: ID does not exist" containerID="702ebef0ccce8010474bb2a51a9b783cde0b5a29f31f3643c02904f15a075355" Oct 30 20:58:26.004322 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:26.004291 4611 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"702ebef0ccce8010474bb2a51a9b783cde0b5a29f31f3643c02904f15a075355"} err="failed to get container status \"702ebef0ccce8010474bb2a51a9b783cde0b5a29f31f3643c02904f15a075355\": rpc error: code = NotFound desc = could not find container \"702ebef0ccce8010474bb2a51a9b783cde0b5a29f31f3643c02904f15a075355\": container with ID starting with 702ebef0ccce8010474bb2a51a9b783cde0b5a29f31f3643c02904f15a075355 not found: ID does not exist" Oct 30 20:58:26.006982 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:26.006964 4611 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-kpn5r"] Oct 30 20:58:26.015915 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:26.015886 4611 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-kpn5r"] Oct 30 20:58:26.027965 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:26.027938 4611 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-6xtqj"] Oct 30 20:58:26.032645 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:26.032625 4611 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-ad7ff0-openshift-default-6f59db585c-6xtqj"] Oct 30 20:58:26.040461 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:26.040435 4611 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25d72035-53b1-4de2-b594-34c64b9b1560-istiod-ca-cert\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:26.040568 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:26.040464 4611 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v9222\" (UniqueName: \"kubernetes.io/projected/25d72035-53b1-4de2-b594-34c64b9b1560-kube-api-access-v9222\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:26.040568 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:26.040481 4611 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59bd8e32-17bd-4753-ab30-1e32a1a2085f-istio-podinfo\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:26.040568 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:26.040495 4611 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59bd8e32-17bd-4753-ab30-1e32a1a2085f-istio-token\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:26.040568 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:26.040553 4611 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/25d72035-53b1-4de2-b594-34c64b9b1560-istio-token\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:26.040568 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:26.040567 4611 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25d72035-53b1-4de2-b594-34c64b9b1560-istio-podinfo\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:26.040806 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:26.040581 4611 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9vr9n\" (UniqueName: \"kubernetes.io/projected/59bd8e32-17bd-4753-ab30-1e32a1a2085f-kube-api-access-9vr9n\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:26.040806 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:26.040595 4611 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25d72035-53b1-4de2-b594-34c64b9b1560-credential-socket\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:26.040806 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:26.040608 4611 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25d72035-53b1-4de2-b594-34c64b9b1560-workload-certs\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:26.040806 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:26.040623 4611 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59bd8e32-17bd-4753-ab30-1e32a1a2085f-workload-certs\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:26.040806 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:26.040637 4611 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59bd8e32-17bd-4753-ab30-1e32a1a2085f-istio-envoy\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:26.040806 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:26.040651 4611 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59bd8e32-17bd-4753-ab30-1e32a1a2085f-istio-data\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:26.040806 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:26.040664 4611 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25d72035-53b1-4de2-b594-34c64b9b1560-istio-data\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:26.040806 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:26.040678 4611 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59bd8e32-17bd-4753-ab30-1e32a1a2085f-istiod-ca-cert\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:26.040806 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:26.040691 4611 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25d72035-53b1-4de2-b594-34c64b9b1560-workload-socket\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:26.040806 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:26.040705 4611 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59bd8e32-17bd-4753-ab30-1e32a1a2085f-credential-socket\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:26.040806 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:26.040719 4611 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25d72035-53b1-4de2-b594-34c64b9b1560-istio-envoy\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:26.040806 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:26.040734 4611 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59bd8e32-17bd-4753-ab30-1e32a1a2085f-workload-socket\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:26.391063 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:26.391010 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:58:27.096790 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:27.096752 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:58:27.462841 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:27.462801 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:58:27.607146 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:27.607112 4611 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="25d72035-53b1-4de2-b594-34c64b9b1560" path="/var/lib/kubelet/pods/25d72035-53b1-4de2-b594-34c64b9b1560/volumes" Oct 30 20:58:27.607643 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:27.607627 4611 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="59bd8e32-17bd-4753-ab30-1e32a1a2085f" path="/var/lib/kubelet/pods/59bd8e32-17bd-4753-ab30-1e32a1a2085f/volumes" Oct 30 20:58:27.608114 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:27.608098 4611 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="63d294fd-5293-4752-9116-0aafa81d5e4f" path="/var/lib/kubelet/pods/63d294fd-5293-4752-9116-0aafa81d5e4f/volumes" Oct 30 20:58:27.609678 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:27.609642 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:58:27.785749 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:27.785640 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:58:28.625597 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:28.617288 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:58:29.842583 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:29.838887 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:58:30.061533 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:30.061035 4611 generic.go:356] "Generic (PLEG): container finished" podID="cfbb7630-0e90-41df-9d4e-ade03ff1f522" containerID="27e240f53d2b9886252ca3e879894bca4d4306d4b9d49918f76959e495672fa2" exitCode=0 Oct 30 20:58:30.061533 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:30.061123 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-sxmcf" event={"ID":"cfbb7630-0e90-41df-9d4e-ade03ff1f522","Type":"ContainerDied","Data":"27e240f53d2b9886252ca3e879894bca4d4306d4b9d49918f76959e495672fa2"} Oct 30 20:58:30.074556 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:30.063350 4611 generic.go:356] "Generic (PLEG): container finished" podID="8603ecfb-6faf-4f18-bb0b-2aee0341b0e3" containerID="02bf56353d665042d4709c90beafe218f03c0517f1183ba82c0dfba1a7f1a493" exitCode=0 Oct 30 20:58:30.074556 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:30.063487 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-2pw94" event={"ID":"8603ecfb-6faf-4f18-bb0b-2aee0341b0e3","Type":"ContainerDied","Data":"02bf56353d665042d4709c90beafe218f03c0517f1183ba82c0dfba1a7f1a493"} Oct 30 20:58:30.074556 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:30.069782 4611 generic.go:356] "Generic (PLEG): container finished" podID="f1f5526b-53f1-4c2b-a801-49ee75e16f9c" containerID="f1034ebd6de0aa7d70504195eb408bf9c78e55bc0caeffefdc776f925bdba0c2" exitCode=0 Oct 30 20:58:30.074556 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:30.069921 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-9d6hc" event={"ID":"f1f5526b-53f1-4c2b-a801-49ee75e16f9c","Type":"ContainerDied","Data":"f1034ebd6de0aa7d70504195eb408bf9c78e55bc0caeffefdc776f925bdba0c2"} Oct 30 20:58:30.895807 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:30.878726 4611 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-sxmcf" Oct 30 20:58:30.931546 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:30.930391 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:58:31.029825 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:31.029785 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cfbb7630-0e90-41df-9d4e-ade03ff1f522-istio-token\" (UniqueName: \"kubernetes.io/projected/cfbb7630-0e90-41df-9d4e-ade03ff1f522-istio-token\") pod \"cfbb7630-0e90-41df-9d4e-ade03ff1f522\" (UID: \"cfbb7630-0e90-41df-9d4e-ade03ff1f522\") " Oct 30 20:58:31.030070 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:31.030049 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cfbb7630-0e90-41df-9d4e-ade03ff1f522-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cfbb7630-0e90-41df-9d4e-ade03ff1f522-istio-envoy\") pod \"cfbb7630-0e90-41df-9d4e-ade03ff1f522\" (UID: \"cfbb7630-0e90-41df-9d4e-ade03ff1f522\") " Oct 30 20:58:31.030216 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:31.030199 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cfbb7630-0e90-41df-9d4e-ade03ff1f522-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cfbb7630-0e90-41df-9d4e-ade03ff1f522-istiod-ca-cert\") pod \"cfbb7630-0e90-41df-9d4e-ade03ff1f522\" (UID: \"cfbb7630-0e90-41df-9d4e-ade03ff1f522\") " Oct 30 20:58:31.030372 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:31.030354 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cfbb7630-0e90-41df-9d4e-ade03ff1f522-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cfbb7630-0e90-41df-9d4e-ade03ff1f522-workload-certs\") pod \"cfbb7630-0e90-41df-9d4e-ade03ff1f522\" (UID: \"cfbb7630-0e90-41df-9d4e-ade03ff1f522\") " Oct 30 20:58:31.030521 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:31.030490 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cfbb7630-0e90-41df-9d4e-ade03ff1f522-kube-api-access-gg4dc\" (UniqueName: \"kubernetes.io/projected/cfbb7630-0e90-41df-9d4e-ade03ff1f522-kube-api-access-gg4dc\") pod \"cfbb7630-0e90-41df-9d4e-ade03ff1f522\" (UID: \"cfbb7630-0e90-41df-9d4e-ade03ff1f522\") " Oct 30 20:58:31.030695 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:31.030677 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cfbb7630-0e90-41df-9d4e-ade03ff1f522-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfbb7630-0e90-41df-9d4e-ade03ff1f522-workload-socket\") pod \"cfbb7630-0e90-41df-9d4e-ade03ff1f522\" (UID: \"cfbb7630-0e90-41df-9d4e-ade03ff1f522\") " Oct 30 20:58:31.030949 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:31.030929 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cfbb7630-0e90-41df-9d4e-ade03ff1f522-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cfbb7630-0e90-41df-9d4e-ade03ff1f522-istio-data\") pod \"cfbb7630-0e90-41df-9d4e-ade03ff1f522\" (UID: \"cfbb7630-0e90-41df-9d4e-ade03ff1f522\") " Oct 30 20:58:31.031145 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:31.031128 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cfbb7630-0e90-41df-9d4e-ade03ff1f522-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cfbb7630-0e90-41df-9d4e-ade03ff1f522-istio-podinfo\") pod \"cfbb7630-0e90-41df-9d4e-ade03ff1f522\" (UID: \"cfbb7630-0e90-41df-9d4e-ade03ff1f522\") " Oct 30 20:58:31.031286 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:31.031267 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cfbb7630-0e90-41df-9d4e-ade03ff1f522-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfbb7630-0e90-41df-9d4e-ade03ff1f522-credential-socket\") pod \"cfbb7630-0e90-41df-9d4e-ade03ff1f522\" (UID: \"cfbb7630-0e90-41df-9d4e-ade03ff1f522\") " Oct 30 20:58:31.050529 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:31.038895 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfbb7630-0e90-41df-9d4e-ade03ff1f522-workload-certs" pod "cfbb7630-0e90-41df-9d4e-ade03ff1f522" (UID: "cfbb7630-0e90-41df-9d4e-ade03ff1f522"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:58:31.050529 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:31.043993 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfbb7630-0e90-41df-9d4e-ade03ff1f522-istio-data" pod "cfbb7630-0e90-41df-9d4e-ade03ff1f522" (UID: "cfbb7630-0e90-41df-9d4e-ade03ff1f522"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:58:31.050529 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:31.044692 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfbb7630-0e90-41df-9d4e-ade03ff1f522-workload-socket" pod "cfbb7630-0e90-41df-9d4e-ade03ff1f522" (UID: "cfbb7630-0e90-41df-9d4e-ade03ff1f522"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:58:31.055651 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:31.055621 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfbb7630-0e90-41df-9d4e-ade03ff1f522-credential-socket" pod "cfbb7630-0e90-41df-9d4e-ade03ff1f522" (UID: "cfbb7630-0e90-41df-9d4e-ade03ff1f522"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:58:31.058917 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:31.058893 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cfbb7630-0e90-41df-9d4e-ade03ff1f522-istiod-ca-cert" pod "cfbb7630-0e90-41df-9d4e-ade03ff1f522" (UID: "cfbb7630-0e90-41df-9d4e-ade03ff1f522"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:58:31.134672 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:31.131070 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cfbb7630-0e90-41df-9d4e-ade03ff1f522-istio-podinfo" pod "cfbb7630-0e90-41df-9d4e-ade03ff1f522" (UID: "cfbb7630-0e90-41df-9d4e-ade03ff1f522"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:58:31.134672 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:31.133407 4611 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cfbb7630-0e90-41df-9d4e-ade03ff1f522-istio-data\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:31.134672 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:31.133435 4611 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cfbb7630-0e90-41df-9d4e-ade03ff1f522-istio-podinfo\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:31.134672 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:31.133448 4611 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfbb7630-0e90-41df-9d4e-ade03ff1f522-credential-socket\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:31.134672 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:31.133460 4611 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cfbb7630-0e90-41df-9d4e-ade03ff1f522-istiod-ca-cert\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:31.134672 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:31.133471 4611 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cfbb7630-0e90-41df-9d4e-ade03ff1f522-workload-certs\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:31.134672 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:31.133483 4611 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfbb7630-0e90-41df-9d4e-ade03ff1f522-workload-socket\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:31.135149 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:31.134680 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cfbb7630-0e90-41df-9d4e-ade03ff1f522-kube-api-access-gg4dc" pod "cfbb7630-0e90-41df-9d4e-ade03ff1f522" (UID: "cfbb7630-0e90-41df-9d4e-ade03ff1f522"). InnerVolumeSpecName "kube-api-access-gg4dc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:58:31.136322 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:31.136294 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfbb7630-0e90-41df-9d4e-ade03ff1f522-istio-envoy" pod "cfbb7630-0e90-41df-9d4e-ade03ff1f522" (UID: "cfbb7630-0e90-41df-9d4e-ade03ff1f522"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:58:31.136983 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:31.136939 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cfbb7630-0e90-41df-9d4e-ade03ff1f522-istio-token" pod "cfbb7630-0e90-41df-9d4e-ade03ff1f522" (UID: "cfbb7630-0e90-41df-9d4e-ade03ff1f522"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:58:31.137770 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:31.137738 4611 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-sxmcf" Oct 30 20:58:31.137862 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:31.137770 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-sxmcf" event={"ID":"cfbb7630-0e90-41df-9d4e-ade03ff1f522","Type":"ContainerDied","Data":"e223e1927a99adac226fd76e9ba1b52086516f9760b25e1c9ee4698db6a76bd3"} Oct 30 20:58:31.137922 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:31.137877 4611 scope.go:122] "RemoveContainer" containerID="27e240f53d2b9886252ca3e879894bca4d4306d4b9d49918f76959e495672fa2" Oct 30 20:58:31.139316 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:31.138671 4611 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-9d6hc" Oct 30 20:58:31.141474 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:31.141449 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-9d6hc" event={"ID":"f1f5526b-53f1-4c2b-a801-49ee75e16f9c","Type":"ContainerDied","Data":"ac2c235cc3b9199171c242828d7bc92bae609e69629c035b155ed64c47c177ae"} Oct 30 20:58:31.207804 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:31.204952 4611 scope.go:122] "RemoveContainer" containerID="f1034ebd6de0aa7d70504195eb408bf9c78e55bc0caeffefdc776f925bdba0c2" Oct 30 20:58:31.208789 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:31.208712 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:58:31.212637 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:31.212251 4611 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-sxmcf"] Oct 30 20:58:31.220583 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:31.219725 4611 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-sxmcf"] Oct 30 20:58:31.238358 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:31.236303 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f1f5526b-53f1-4c2b-a801-49ee75e16f9c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1f5526b-53f1-4c2b-a801-49ee75e16f9c-istio-podinfo\") pod \"f1f5526b-53f1-4c2b-a801-49ee75e16f9c\" (UID: \"f1f5526b-53f1-4c2b-a801-49ee75e16f9c\") " Oct 30 20:58:31.238358 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:31.236365 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f1f5526b-53f1-4c2b-a801-49ee75e16f9c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1f5526b-53f1-4c2b-a801-49ee75e16f9c-istiod-ca-cert\") pod \"f1f5526b-53f1-4c2b-a801-49ee75e16f9c\" (UID: \"f1f5526b-53f1-4c2b-a801-49ee75e16f9c\") " Oct 30 20:58:31.238358 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:31.236436 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1f5526b-53f1-4c2b-a801-49ee75e16f9c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1f5526b-53f1-4c2b-a801-49ee75e16f9c-istio-envoy\") pod \"f1f5526b-53f1-4c2b-a801-49ee75e16f9c\" (UID: \"f1f5526b-53f1-4c2b-a801-49ee75e16f9c\") " Oct 30 20:58:31.238358 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:31.236723 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1f5526b-53f1-4c2b-a801-49ee75e16f9c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1f5526b-53f1-4c2b-a801-49ee75e16f9c-workload-certs\") pod \"f1f5526b-53f1-4c2b-a801-49ee75e16f9c\" (UID: \"f1f5526b-53f1-4c2b-a801-49ee75e16f9c\") " Oct 30 20:58:31.238358 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:31.236780 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1f5526b-53f1-4c2b-a801-49ee75e16f9c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1f5526b-53f1-4c2b-a801-49ee75e16f9c-credential-socket\") pod \"f1f5526b-53f1-4c2b-a801-49ee75e16f9c\" (UID: \"f1f5526b-53f1-4c2b-a801-49ee75e16f9c\") " Oct 30 20:58:31.238358 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:31.236885 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1f5526b-53f1-4c2b-a801-49ee75e16f9c-kube-api-access-hd265\" (UniqueName: \"kubernetes.io/projected/f1f5526b-53f1-4c2b-a801-49ee75e16f9c-kube-api-access-hd265\") pod \"f1f5526b-53f1-4c2b-a801-49ee75e16f9c\" (UID: \"f1f5526b-53f1-4c2b-a801-49ee75e16f9c\") " Oct 30 20:58:31.238358 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:31.236958 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1f5526b-53f1-4c2b-a801-49ee75e16f9c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1f5526b-53f1-4c2b-a801-49ee75e16f9c-workload-socket\") pod \"f1f5526b-53f1-4c2b-a801-49ee75e16f9c\" (UID: \"f1f5526b-53f1-4c2b-a801-49ee75e16f9c\") " Oct 30 20:58:31.238358 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:31.237014 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1f5526b-53f1-4c2b-a801-49ee75e16f9c-istio-token\" (UniqueName: \"kubernetes.io/projected/f1f5526b-53f1-4c2b-a801-49ee75e16f9c-istio-token\") pod \"f1f5526b-53f1-4c2b-a801-49ee75e16f9c\" (UID: \"f1f5526b-53f1-4c2b-a801-49ee75e16f9c\") " Oct 30 20:58:31.238358 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:31.237076 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1f5526b-53f1-4c2b-a801-49ee75e16f9c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1f5526b-53f1-4c2b-a801-49ee75e16f9c-istio-data\") pod \"f1f5526b-53f1-4c2b-a801-49ee75e16f9c\" (UID: \"f1f5526b-53f1-4c2b-a801-49ee75e16f9c\") " Oct 30 20:58:31.238358 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:31.237234 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1f5526b-53f1-4c2b-a801-49ee75e16f9c-workload-certs" pod "f1f5526b-53f1-4c2b-a801-49ee75e16f9c" (UID: "f1f5526b-53f1-4c2b-a801-49ee75e16f9c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:58:31.238358 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:31.237565 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1f5526b-53f1-4c2b-a801-49ee75e16f9c-istiod-ca-cert" pod "f1f5526b-53f1-4c2b-a801-49ee75e16f9c" (UID: "f1f5526b-53f1-4c2b-a801-49ee75e16f9c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:58:31.238358 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:31.237787 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1f5526b-53f1-4c2b-a801-49ee75e16f9c-istio-data" pod "f1f5526b-53f1-4c2b-a801-49ee75e16f9c" (UID: "f1f5526b-53f1-4c2b-a801-49ee75e16f9c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:58:31.238358 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:31.237949 4611 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1f5526b-53f1-4c2b-a801-49ee75e16f9c-istiod-ca-cert\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:31.238358 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:31.237971 4611 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cfbb7630-0e90-41df-9d4e-ade03ff1f522-istio-token\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:31.238358 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:31.237986 4611 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cfbb7630-0e90-41df-9d4e-ade03ff1f522-istio-envoy\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:31.238358 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:31.238000 4611 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1f5526b-53f1-4c2b-a801-49ee75e16f9c-workload-certs\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:31.238358 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:31.238015 4611 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gg4dc\" (UniqueName: \"kubernetes.io/projected/cfbb7630-0e90-41df-9d4e-ade03ff1f522-kube-api-access-gg4dc\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:31.238358 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:31.238030 4611 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1f5526b-53f1-4c2b-a801-49ee75e16f9c-istio-data\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:31.238358 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:31.237967 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1f5526b-53f1-4c2b-a801-49ee75e16f9c-credential-socket" pod "f1f5526b-53f1-4c2b-a801-49ee75e16f9c" (UID: "f1f5526b-53f1-4c2b-a801-49ee75e16f9c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:58:31.238358 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:31.238028 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1f5526b-53f1-4c2b-a801-49ee75e16f9c-workload-socket" pod "f1f5526b-53f1-4c2b-a801-49ee75e16f9c" (UID: "f1f5526b-53f1-4c2b-a801-49ee75e16f9c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:58:31.240043 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:31.239959 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f1f5526b-53f1-4c2b-a801-49ee75e16f9c-istio-podinfo" pod "f1f5526b-53f1-4c2b-a801-49ee75e16f9c" (UID: "f1f5526b-53f1-4c2b-a801-49ee75e16f9c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:58:31.240483 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:31.240437 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1f5526b-53f1-4c2b-a801-49ee75e16f9c-kube-api-access-hd265" pod "f1f5526b-53f1-4c2b-a801-49ee75e16f9c" (UID: "f1f5526b-53f1-4c2b-a801-49ee75e16f9c"). InnerVolumeSpecName "kube-api-access-hd265". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:58:31.241294 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:31.241267 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1f5526b-53f1-4c2b-a801-49ee75e16f9c-istio-token" pod "f1f5526b-53f1-4c2b-a801-49ee75e16f9c" (UID: "f1f5526b-53f1-4c2b-a801-49ee75e16f9c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:58:31.241756 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:31.241668 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1f5526b-53f1-4c2b-a801-49ee75e16f9c-istio-envoy" pod "f1f5526b-53f1-4c2b-a801-49ee75e16f9c" (UID: "f1f5526b-53f1-4c2b-a801-49ee75e16f9c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:58:31.243939 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:31.243917 4611 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-2pw94" Oct 30 20:58:31.339477 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:31.339441 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8603ecfb-6faf-4f18-bb0b-2aee0341b0e3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8603ecfb-6faf-4f18-bb0b-2aee0341b0e3-istiod-ca-cert\") pod \"8603ecfb-6faf-4f18-bb0b-2aee0341b0e3\" (UID: \"8603ecfb-6faf-4f18-bb0b-2aee0341b0e3\") " Oct 30 20:58:31.339641 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:31.339590 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8603ecfb-6faf-4f18-bb0b-2aee0341b0e3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8603ecfb-6faf-4f18-bb0b-2aee0341b0e3-workload-certs\") pod \"8603ecfb-6faf-4f18-bb0b-2aee0341b0e3\" (UID: \"8603ecfb-6faf-4f18-bb0b-2aee0341b0e3\") " Oct 30 20:58:31.339704 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:31.339672 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8603ecfb-6faf-4f18-bb0b-2aee0341b0e3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8603ecfb-6faf-4f18-bb0b-2aee0341b0e3-istio-podinfo\") pod \"8603ecfb-6faf-4f18-bb0b-2aee0341b0e3\" (UID: \"8603ecfb-6faf-4f18-bb0b-2aee0341b0e3\") " Oct 30 20:58:31.339759 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:31.339716 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8603ecfb-6faf-4f18-bb0b-2aee0341b0e3-kube-api-access-gsjg7\" (UniqueName: \"kubernetes.io/projected/8603ecfb-6faf-4f18-bb0b-2aee0341b0e3-kube-api-access-gsjg7\") pod \"8603ecfb-6faf-4f18-bb0b-2aee0341b0e3\" (UID: \"8603ecfb-6faf-4f18-bb0b-2aee0341b0e3\") " Oct 30 20:58:31.339827 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:31.339809 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8603ecfb-6faf-4f18-bb0b-2aee0341b0e3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8603ecfb-6faf-4f18-bb0b-2aee0341b0e3-workload-socket\") pod \"8603ecfb-6faf-4f18-bb0b-2aee0341b0e3\" (UID: \"8603ecfb-6faf-4f18-bb0b-2aee0341b0e3\") " Oct 30 20:58:31.339879 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:31.339832 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8603ecfb-6faf-4f18-bb0b-2aee0341b0e3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8603ecfb-6faf-4f18-bb0b-2aee0341b0e3-istio-data\") pod \"8603ecfb-6faf-4f18-bb0b-2aee0341b0e3\" (UID: \"8603ecfb-6faf-4f18-bb0b-2aee0341b0e3\") " Oct 30 20:58:31.339938 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:31.339881 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8603ecfb-6faf-4f18-bb0b-2aee0341b0e3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8603ecfb-6faf-4f18-bb0b-2aee0341b0e3-credential-socket\") pod \"8603ecfb-6faf-4f18-bb0b-2aee0341b0e3\" (UID: \"8603ecfb-6faf-4f18-bb0b-2aee0341b0e3\") " Oct 30 20:58:31.339938 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:31.339919 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8603ecfb-6faf-4f18-bb0b-2aee0341b0e3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8603ecfb-6faf-4f18-bb0b-2aee0341b0e3-istio-envoy\") pod \"8603ecfb-6faf-4f18-bb0b-2aee0341b0e3\" (UID: \"8603ecfb-6faf-4f18-bb0b-2aee0341b0e3\") " Oct 30 20:58:31.340043 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:31.339933 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8603ecfb-6faf-4f18-bb0b-2aee0341b0e3-workload-certs" pod "8603ecfb-6faf-4f18-bb0b-2aee0341b0e3" (UID: "8603ecfb-6faf-4f18-bb0b-2aee0341b0e3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:58:31.340043 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:31.339966 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8603ecfb-6faf-4f18-bb0b-2aee0341b0e3-istio-token\" (UniqueName: \"kubernetes.io/projected/8603ecfb-6faf-4f18-bb0b-2aee0341b0e3-istio-token\") pod \"8603ecfb-6faf-4f18-bb0b-2aee0341b0e3\" (UID: \"8603ecfb-6faf-4f18-bb0b-2aee0341b0e3\") " Oct 30 20:58:31.340160 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:31.340132 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8603ecfb-6faf-4f18-bb0b-2aee0341b0e3-workload-socket" pod "8603ecfb-6faf-4f18-bb0b-2aee0341b0e3" (UID: "8603ecfb-6faf-4f18-bb0b-2aee0341b0e3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:58:31.340553 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:31.340502 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8603ecfb-6faf-4f18-bb0b-2aee0341b0e3-istio-data" pod "8603ecfb-6faf-4f18-bb0b-2aee0341b0e3" (UID: "8603ecfb-6faf-4f18-bb0b-2aee0341b0e3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:58:31.340684 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:31.340575 4611 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f1f5526b-53f1-4c2b-a801-49ee75e16f9c-istio-token\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:31.340684 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:31.340598 4611 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8603ecfb-6faf-4f18-bb0b-2aee0341b0e3-workload-certs\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:31.340684 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:31.340613 4611 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1f5526b-53f1-4c2b-a801-49ee75e16f9c-istio-podinfo\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:31.340684 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:31.340614 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8603ecfb-6faf-4f18-bb0b-2aee0341b0e3-istiod-ca-cert" pod "8603ecfb-6faf-4f18-bb0b-2aee0341b0e3" (UID: "8603ecfb-6faf-4f18-bb0b-2aee0341b0e3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:58:31.340684 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:31.340626 4611 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1f5526b-53f1-4c2b-a801-49ee75e16f9c-istio-envoy\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:31.340684 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:31.340640 4611 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8603ecfb-6faf-4f18-bb0b-2aee0341b0e3-workload-socket\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:31.340684 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:31.340654 4611 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1f5526b-53f1-4c2b-a801-49ee75e16f9c-credential-socket\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:31.340684 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:31.340667 4611 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hd265\" (UniqueName: \"kubernetes.io/projected/f1f5526b-53f1-4c2b-a801-49ee75e16f9c-kube-api-access-hd265\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:31.340684 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:31.340681 4611 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1f5526b-53f1-4c2b-a801-49ee75e16f9c-workload-socket\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:31.341095 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:31.340708 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8603ecfb-6faf-4f18-bb0b-2aee0341b0e3-credential-socket" pod "8603ecfb-6faf-4f18-bb0b-2aee0341b0e3" (UID: "8603ecfb-6faf-4f18-bb0b-2aee0341b0e3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:58:31.342832 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:31.342801 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8603ecfb-6faf-4f18-bb0b-2aee0341b0e3-istio-envoy" pod "8603ecfb-6faf-4f18-bb0b-2aee0341b0e3" (UID: "8603ecfb-6faf-4f18-bb0b-2aee0341b0e3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:58:31.343161 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:31.343105 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8603ecfb-6faf-4f18-bb0b-2aee0341b0e3-kube-api-access-gsjg7" pod "8603ecfb-6faf-4f18-bb0b-2aee0341b0e3" (UID: "8603ecfb-6faf-4f18-bb0b-2aee0341b0e3"). InnerVolumeSpecName "kube-api-access-gsjg7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:58:31.343161 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:31.343131 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8603ecfb-6faf-4f18-bb0b-2aee0341b0e3-istio-token" pod "8603ecfb-6faf-4f18-bb0b-2aee0341b0e3" (UID: "8603ecfb-6faf-4f18-bb0b-2aee0341b0e3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:58:31.343495 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:31.343476 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8603ecfb-6faf-4f18-bb0b-2aee0341b0e3-istio-podinfo" pod "8603ecfb-6faf-4f18-bb0b-2aee0341b0e3" (UID: "8603ecfb-6faf-4f18-bb0b-2aee0341b0e3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:58:31.441720 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:31.441645 4611 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8603ecfb-6faf-4f18-bb0b-2aee0341b0e3-istiod-ca-cert\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:31.441720 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:31.441672 4611 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8603ecfb-6faf-4f18-bb0b-2aee0341b0e3-istio-podinfo\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:31.441720 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:31.441682 4611 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gsjg7\" (UniqueName: \"kubernetes.io/projected/8603ecfb-6faf-4f18-bb0b-2aee0341b0e3-kube-api-access-gsjg7\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:31.441720 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:31.441692 4611 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8603ecfb-6faf-4f18-bb0b-2aee0341b0e3-istio-data\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:31.441720 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:31.441701 4611 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8603ecfb-6faf-4f18-bb0b-2aee0341b0e3-credential-socket\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:31.441720 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:31.441709 4611 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8603ecfb-6faf-4f18-bb0b-2aee0341b0e3-istio-envoy\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:31.441720 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:31.441719 4611 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8603ecfb-6faf-4f18-bb0b-2aee0341b0e3-istio-token\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:31.609428 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:31.609388 4611 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cfbb7630-0e90-41df-9d4e-ade03ff1f522" path="/var/lib/kubelet/pods/cfbb7630-0e90-41df-9d4e-ade03ff1f522/volumes" Oct 30 20:58:31.820019 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:31.819919 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:58:32.022326 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:32.022291 4611 patch_prober.go:29] interesting pod/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-9d6hc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:58:32.022811 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:32.022353 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-9d6hc" podUID="f1f5526b-53f1-4c2b-a801-49ee75e16f9c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:58:32.152028 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:32.151996 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-2pw94" event={"ID":"8603ecfb-6faf-4f18-bb0b-2aee0341b0e3","Type":"ContainerDied","Data":"4f8f9bb17315ab4f1215f5f2b7163ab03036f09fd72c75918b3bb1b63009eefa"} Oct 30 20:58:32.152192 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:32.152055 4611 scope.go:122] "RemoveContainer" containerID="02bf56353d665042d4709c90beafe218f03c0517f1183ba82c0dfba1a7f1a493" Oct 30 20:58:32.152192 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:32.152010 4611 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-2pw94" Oct 30 20:58:32.153359 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:32.153213 4611 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-9d6hc" Oct 30 20:58:32.180448 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:32.180415 4611 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-2pw94"] Oct 30 20:58:32.184722 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:32.184703 4611 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-2pw94"] Oct 30 20:58:32.194665 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:32.194646 4611 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-9d6hc"] Oct 30 20:58:32.198857 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:32.198838 4611 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-cb60a2-openshift-default-57db98bd79-9d6hc"] Oct 30 20:58:32.539079 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:32.538997 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:58:33.203286 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:33.201921 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:58:33.567492 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:33.567400 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:58:33.606095 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:33.606061 4611 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8603ecfb-6faf-4f18-bb0b-2aee0341b0e3" path="/var/lib/kubelet/pods/8603ecfb-6faf-4f18-bb0b-2aee0341b0e3/volumes" Oct 30 20:58:33.606574 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:33.606559 4611 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f1f5526b-53f1-4c2b-a801-49ee75e16f9c" path="/var/lib/kubelet/pods/f1f5526b-53f1-4c2b-a801-49ee75e16f9c/volumes" Oct 30 20:58:33.824714 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:33.824622 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:58:33.918556 ip-10-0-133-192 kubenswrapper[4611]: E1030 20:58:33.918490 4611 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hwzc6" containerName="extractor" Oct 30 20:58:33.974401 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:33.974367 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:58:34.819288 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:34.819252 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:58:35.597401 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:35.597223 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:58:35.916658 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:35.916624 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:58:36.065347 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:36.065310 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:58:36.503165 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:36.503132 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:58:37.446009 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:37.445974 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:58:38.087031 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:38.086993 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:58:38.589169 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:38.589136 4611 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-14cca2/distinct-w5-14cca2-kserve-587f6996b8-sgf6h"] Oct 30 20:58:38.597913 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:38.597884 4611 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-14cca2/premium-w5-14cca2-kserve-6878777857-wpjtj"] Oct 30 20:58:38.664810 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:38.664767 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:58:38.833699 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:38.833620 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:58:39.491754 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:39.491712 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:58:40.206961 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:40.206902 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:58:41.016120 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:41.016085 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:58:41.611056 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:41.611017 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:58:42.288694 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:42.288090 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:58:42.894391 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:42.894332 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:58:43.683815 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:43.683773 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:58:43.950996 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:43.950925 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:58:44.118152 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:44.118079 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:58:44.616652 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:44.616604 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:58:45.087743 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:45.087666 4611 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d1eb66ad-jjwfx"] Oct 30 20:58:45.088238 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:45.088223 4611 cpu_manager.go:414] "containerMap: removing container" podUID="63d294fd-5293-4752-9116-0aafa81d5e4f" containerName="istio-proxy" Oct 30 20:58:45.088288 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:45.088239 4611 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63d294fd-5293-4752-9116-0aafa81d5e4f" containerName="istio-proxy" Oct 30 20:58:45.088288 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:45.088253 4611 cpu_manager.go:414] "containerMap: removing container" podUID="cfbb7630-0e90-41df-9d4e-ade03ff1f522" containerName="istio-proxy" Oct 30 20:58:45.088288 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:45.088258 4611 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cfbb7630-0e90-41df-9d4e-ade03ff1f522" containerName="istio-proxy" Oct 30 20:58:45.088288 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:45.088269 4611 cpu_manager.go:414] "containerMap: removing container" podUID="a74a618b-03cd-4949-9b6d-f91301dc19d1" containerName="maas-api" Oct 30 20:58:45.088288 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:45.088274 4611 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a74a618b-03cd-4949-9b6d-f91301dc19d1" containerName="maas-api" Oct 30 20:58:45.088432 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:45.088291 4611 cpu_manager.go:414] "containerMap: removing container" podUID="8603ecfb-6faf-4f18-bb0b-2aee0341b0e3" containerName="istio-proxy" Oct 30 20:58:45.088432 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:45.088296 4611 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8603ecfb-6faf-4f18-bb0b-2aee0341b0e3" containerName="istio-proxy" Oct 30 20:58:45.088432 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:45.088306 4611 cpu_manager.go:414] "containerMap: removing container" podUID="711b167d-4e66-40d6-be75-33ede11f42c5" containerName="main" Oct 30 20:58:45.088432 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:45.088311 4611 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="711b167d-4e66-40d6-be75-33ede11f42c5" containerName="main" Oct 30 20:58:45.088432 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:45.088320 4611 cpu_manager.go:414] "containerMap: removing container" podUID="59bd8e32-17bd-4753-ab30-1e32a1a2085f" containerName="istio-proxy" Oct 30 20:58:45.088432 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:45.088324 4611 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="59bd8e32-17bd-4753-ab30-1e32a1a2085f" containerName="istio-proxy" Oct 30 20:58:45.088432 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:45.088336 4611 cpu_manager.go:414] "containerMap: removing container" podUID="06064c28-fd6d-4765-8635-8690665a9b83" containerName="istio-proxy" Oct 30 20:58:45.088432 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:45.088340 4611 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06064c28-fd6d-4765-8635-8690665a9b83" containerName="istio-proxy" Oct 30 20:58:45.088432 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:45.088357 4611 cpu_manager.go:414] "containerMap: removing container" podUID="52e301d9-f378-4e54-9515-01a6680d9fd7" containerName="main" Oct 30 20:58:45.088432 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:45.088361 4611 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="52e301d9-f378-4e54-9515-01a6680d9fd7" containerName="main" Oct 30 20:58:45.088432 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:45.088368 4611 cpu_manager.go:414] "containerMap: removing container" podUID="53cbd692-e9f8-46ba-8fde-3e5215a38904" containerName="main" Oct 30 20:58:45.088432 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:45.088372 4611 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="53cbd692-e9f8-46ba-8fde-3e5215a38904" containerName="main" Oct 30 20:58:45.088432 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:45.088379 4611 cpu_manager.go:414] "containerMap: removing container" podUID="25d72035-53b1-4de2-b594-34c64b9b1560" containerName="istio-proxy" Oct 30 20:58:45.088432 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:45.088384 4611 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="25d72035-53b1-4de2-b594-34c64b9b1560" containerName="istio-proxy" Oct 30 20:58:45.088432 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:45.088393 4611 cpu_manager.go:414] "containerMap: removing container" podUID="f1f5526b-53f1-4c2b-a801-49ee75e16f9c" containerName="istio-proxy" Oct 30 20:58:45.088432 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:45.088398 4611 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1f5526b-53f1-4c2b-a801-49ee75e16f9c" containerName="istio-proxy" Oct 30 20:58:45.088432 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:45.088405 4611 cpu_manager.go:414] "containerMap: removing container" podUID="763cbacf-a524-4397-bf3a-2878d0e15a6d" containerName="revoke-keys" Oct 30 20:58:45.088432 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:45.088409 4611 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="763cbacf-a524-4397-bf3a-2878d0e15a6d" containerName="revoke-keys" Oct 30 20:58:45.088936 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:45.088485 4611 memory_manager.go:365] "RemoveStaleState removing state" podUID="52e301d9-f378-4e54-9515-01a6680d9fd7" containerName="main" Oct 30 20:58:45.088936 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:45.088494 4611 memory_manager.go:365] "RemoveStaleState removing state" podUID="53cbd692-e9f8-46ba-8fde-3e5215a38904" containerName="main" Oct 30 20:58:45.088936 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:45.088502 4611 memory_manager.go:365] "RemoveStaleState removing state" podUID="25d72035-53b1-4de2-b594-34c64b9b1560" containerName="istio-proxy" Oct 30 20:58:45.088936 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:45.088522 4611 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1f5526b-53f1-4c2b-a801-49ee75e16f9c" containerName="istio-proxy" Oct 30 20:58:45.088936 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:45.088528 4611 memory_manager.go:365] "RemoveStaleState removing state" podUID="763cbacf-a524-4397-bf3a-2878d0e15a6d" containerName="revoke-keys" Oct 30 20:58:45.088936 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:45.088536 4611 memory_manager.go:365] "RemoveStaleState removing state" podUID="63d294fd-5293-4752-9116-0aafa81d5e4f" containerName="istio-proxy" Oct 30 20:58:45.088936 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:45.088542 4611 memory_manager.go:365] "RemoveStaleState removing state" podUID="cfbb7630-0e90-41df-9d4e-ade03ff1f522" containerName="istio-proxy" Oct 30 20:58:45.088936 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:45.088548 4611 memory_manager.go:365] "RemoveStaleState removing state" podUID="a74a618b-03cd-4949-9b6d-f91301dc19d1" containerName="maas-api" Oct 30 20:58:45.088936 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:45.088555 4611 memory_manager.go:365] "RemoveStaleState removing state" podUID="8603ecfb-6faf-4f18-bb0b-2aee0341b0e3" containerName="istio-proxy" Oct 30 20:58:45.088936 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:45.088561 4611 memory_manager.go:365] "RemoveStaleState removing state" podUID="711b167d-4e66-40d6-be75-33ede11f42c5" containerName="main" Oct 30 20:58:45.088936 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:45.088566 4611 memory_manager.go:365] "RemoveStaleState removing state" podUID="59bd8e32-17bd-4753-ab30-1e32a1a2085f" containerName="istio-proxy" Oct 30 20:58:45.088936 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:45.088573 4611 memory_manager.go:365] "RemoveStaleState removing state" podUID="63d294fd-5293-4752-9116-0aafa81d5e4f" containerName="istio-proxy" Oct 30 20:58:45.093992 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:45.093972 4611 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1eb66ad-jjwfx" Oct 30 20:58:45.102090 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:45.102059 4611 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d1eb66ad-jjwfx"] Oct 30 20:58:45.168017 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:45.167967 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/78c5ded8-a0ed-4176-bc86-f6e3e233b455-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d1eb66ad-jjwfx\" (UID: \"78c5ded8-a0ed-4176-bc86-f6e3e233b455\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1eb66ad-jjwfx" Oct 30 20:58:45.168164 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:45.168025 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5hdpw\" (UniqueName: \"kubernetes.io/projected/78c5ded8-a0ed-4176-bc86-f6e3e233b455-kube-api-access-5hdpw\") pod \"maas-api-revoke-sub-d1eb66ad-jjwfx\" (UID: \"78c5ded8-a0ed-4176-bc86-f6e3e233b455\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1eb66ad-jjwfx" Oct 30 20:58:45.168164 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:45.168109 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/78c5ded8-a0ed-4176-bc86-f6e3e233b455-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d1eb66ad-jjwfx\" (UID: \"78c5ded8-a0ed-4176-bc86-f6e3e233b455\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1eb66ad-jjwfx" Oct 30 20:58:45.198068 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:45.198040 4611 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:58:45.198240 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:45.198221 4611 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7hf8r\"" Oct 30 20:58:45.269640 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:45.269610 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/78c5ded8-a0ed-4176-bc86-f6e3e233b455-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d1eb66ad-jjwfx\" (UID: \"78c5ded8-a0ed-4176-bc86-f6e3e233b455\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1eb66ad-jjwfx" Oct 30 20:58:45.269813 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:45.269663 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5hdpw\" (UniqueName: \"kubernetes.io/projected/78c5ded8-a0ed-4176-bc86-f6e3e233b455-kube-api-access-5hdpw\") pod \"maas-api-revoke-sub-d1eb66ad-jjwfx\" (UID: \"78c5ded8-a0ed-4176-bc86-f6e3e233b455\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1eb66ad-jjwfx" Oct 30 20:58:45.269879 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:45.269828 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/78c5ded8-a0ed-4176-bc86-f6e3e233b455-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d1eb66ad-jjwfx\" (UID: \"78c5ded8-a0ed-4176-bc86-f6e3e233b455\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1eb66ad-jjwfx" Oct 30 20:58:45.270231 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:45.270213 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/78c5ded8-a0ed-4176-bc86-f6e3e233b455-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d1eb66ad-jjwfx\" (UID: \"78c5ded8-a0ed-4176-bc86-f6e3e233b455\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1eb66ad-jjwfx" Oct 30 20:58:45.272062 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:45.272044 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/78c5ded8-a0ed-4176-bc86-f6e3e233b455-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d1eb66ad-jjwfx\" (UID: \"78c5ded8-a0ed-4176-bc86-f6e3e233b455\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1eb66ad-jjwfx" Oct 30 20:58:45.285191 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:45.285169 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5hdpw\" (UniqueName: \"kubernetes.io/projected/78c5ded8-a0ed-4176-bc86-f6e3e233b455-kube-api-access-5hdpw\") pod \"maas-api-revoke-sub-d1eb66ad-jjwfx\" (UID: \"78c5ded8-a0ed-4176-bc86-f6e3e233b455\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1eb66ad-jjwfx" Oct 30 20:58:45.389495 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:45.389456 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:58:45.412683 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:45.412654 4611 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1eb66ad-jjwfx" Oct 30 20:58:45.701838 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:45.701796 4611 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d1eb66ad-jjwfx"] Oct 30 20:58:45.703237 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:58:45.703207 4611 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78c5ded8_a0ed_4176_bc86_f6e3e233b455.slice/crio-11f0da933fbfd8c5de5d9f37c1f93dd9830015abe48a73a06f67e4c419ce3b89 WatchSource:0}: Error finding container 11f0da933fbfd8c5de5d9f37c1f93dd9830015abe48a73a06f67e4c419ce3b89: Status 404 returned error can't find the container with id 11f0da933fbfd8c5de5d9f37c1f93dd9830015abe48a73a06f67e4c419ce3b89 Oct 30 20:58:46.092442 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:46.092403 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:58:46.254490 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:46.254397 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1eb66ad-jjwfx" event={"ID":"78c5ded8-a0ed-4176-bc86-f6e3e233b455","Type":"ContainerStarted","Data":"af320cd17fbeb1e6606e834c84593d4e5f8b21dbfbf265ac2b860e75e69b8c76"} Oct 30 20:58:46.254490 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:46.254452 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1eb66ad-jjwfx" event={"ID":"78c5ded8-a0ed-4176-bc86-f6e3e233b455","Type":"ContainerStarted","Data":"11f0da933fbfd8c5de5d9f37c1f93dd9830015abe48a73a06f67e4c419ce3b89"} Oct 30 20:58:46.270601 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:46.270551 4611 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1eb66ad-jjwfx" podStartSLOduration=1.270538476 podStartE2EDuration="1.270538476s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:58:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:58:46.269846161 +0000 UTC m=+1180.803872776" watchObservedRunningTime="2026-10-30 20:58:46.270538476 +0000 UTC m=+1180.804565092" Oct 30 20:58:46.702976 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:46.702935 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:58:47.262960 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:47.262924 4611 generic.go:356] "Generic (PLEG): container finished" podID="78c5ded8-a0ed-4176-bc86-f6e3e233b455" containerID="af320cd17fbeb1e6606e834c84593d4e5f8b21dbfbf265ac2b860e75e69b8c76" exitCode=0 Oct 30 20:58:47.263329 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:47.262999 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1eb66ad-jjwfx" event={"ID":"78c5ded8-a0ed-4176-bc86-f6e3e233b455","Type":"ContainerDied","Data":"af320cd17fbeb1e6606e834c84593d4e5f8b21dbfbf265ac2b860e75e69b8c76"} Oct 30 20:58:47.424054 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:47.417209 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:58:48.082455 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:48.081561 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:58:48.425698 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:48.425676 4611 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1eb66ad-jjwfx" Oct 30 20:58:48.530973 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:48.530943 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/78c5ded8-a0ed-4176-bc86-f6e3e233b455-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/78c5ded8-a0ed-4176-bc86-f6e3e233b455-maas-api-xks-ca\") pod \"78c5ded8-a0ed-4176-bc86-f6e3e233b455\" (UID: \"78c5ded8-a0ed-4176-bc86-f6e3e233b455\") " Oct 30 20:58:48.531105 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:48.531055 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/78c5ded8-a0ed-4176-bc86-f6e3e233b455-kube-api-access-5hdpw\" (UniqueName: \"kubernetes.io/projected/78c5ded8-a0ed-4176-bc86-f6e3e233b455-kube-api-access-5hdpw\") pod \"78c5ded8-a0ed-4176-bc86-f6e3e233b455\" (UID: \"78c5ded8-a0ed-4176-bc86-f6e3e233b455\") " Oct 30 20:58:48.531157 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:48.531130 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/78c5ded8-a0ed-4176-bc86-f6e3e233b455-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/78c5ded8-a0ed-4176-bc86-f6e3e233b455-maas-api-openshift-ca\") pod \"78c5ded8-a0ed-4176-bc86-f6e3e233b455\" (UID: \"78c5ded8-a0ed-4176-bc86-f6e3e233b455\") " Oct 30 20:58:48.531488 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:48.531463 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78c5ded8-a0ed-4176-bc86-f6e3e233b455-maas-api-openshift-ca" pod "78c5ded8-a0ed-4176-bc86-f6e3e233b455" (UID: "78c5ded8-a0ed-4176-bc86-f6e3e233b455"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:58:48.531650 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:48.531628 4611 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/78c5ded8-a0ed-4176-bc86-f6e3e233b455-maas-api-openshift-ca\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:48.532996 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:48.532956 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/78c5ded8-a0ed-4176-bc86-f6e3e233b455-maas-api-xks-ca" pod "78c5ded8-a0ed-4176-bc86-f6e3e233b455" (UID: "78c5ded8-a0ed-4176-bc86-f6e3e233b455"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:58:48.533098 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:48.533059 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78c5ded8-a0ed-4176-bc86-f6e3e233b455-kube-api-access-5hdpw" pod "78c5ded8-a0ed-4176-bc86-f6e3e233b455" (UID: "78c5ded8-a0ed-4176-bc86-f6e3e233b455"). InnerVolumeSpecName "kube-api-access-5hdpw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:58:48.632643 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:48.632616 4611 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/78c5ded8-a0ed-4176-bc86-f6e3e233b455-maas-api-xks-ca\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:48.632643 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:48.632640 4611 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5hdpw\" (UniqueName: \"kubernetes.io/projected/78c5ded8-a0ed-4176-bc86-f6e3e233b455-kube-api-access-5hdpw\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:48.767041 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:48.766961 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:58:48.925304 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:48.925266 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:58:49.279066 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:49.279041 4611 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1eb66ad-jjwfx" Oct 30 20:58:49.279303 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:49.279039 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1eb66ad-jjwfx" event={"ID":"78c5ded8-a0ed-4176-bc86-f6e3e233b455","Type":"ContainerDied","Data":"11f0da933fbfd8c5de5d9f37c1f93dd9830015abe48a73a06f67e4c419ce3b89"} Oct 30 20:58:49.279303 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:49.279159 4611 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="11f0da933fbfd8c5de5d9f37c1f93dd9830015abe48a73a06f67e4c419ce3b89" Oct 30 20:58:49.525394 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:49.525358 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:58:50.335522 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:50.332852 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:58:51.004490 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:51.004446 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:58:51.708142 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:51.708099 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:58:52.415600 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:52.415564 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:58:52.583261 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:52.583226 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:58:53.265374 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:53.265334 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:58:53.610658 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:53.610611 4611 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-14cca2/distinct-w5-14cca2-kserve-587f6996b8-sgf6h" podUID="02b652c3-0724-4a50-9e95-2a9890cc97c4" containerName="main" containerID="cri-o://d1b4cba96b77d59a8b7e17e5ea7041b95d5bac7a658002706fdc139ef29a806f" gracePeriod=45 Oct 30 20:58:53.619367 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:53.619333 4611 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-14cca2/premium-w5-14cca2-kserve-6878777857-wpjtj" podUID="b99eb76b-d1c6-49e9-9809-c351c0f5a4de" containerName="main" containerID="cri-o://d406ff442d0c9a437cc2ff961f6cb0177e6a528cd2212db4510a648c6f8e2446" gracePeriod=45 Oct 30 20:58:53.622181 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:53.622151 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-14cca2/distinct-w5-14cca2-kserve-587f6996b8-sgf6h" podUID="02b652c3-0724-4a50-9e95-2a9890cc97c4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": dial tcp 10.133.0.59:8000: connect: connection refused" Oct 30 20:58:53.966371 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:53.965576 4611 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-14cca2/distinct-w5-14cca2-kserve-587f6996b8-sgf6h" Oct 30 20:58:53.974356 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:53.969336 4611 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-14cca2/premium-w5-14cca2-kserve-6878777857-wpjtj" Oct 30 20:58:54.111176 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:54.111136 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02b652c3-0724-4a50-9e95-2a9890cc97c4-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/02b652c3-0724-4a50-9e95-2a9890cc97c4-model-cache\") pod \"02b652c3-0724-4a50-9e95-2a9890cc97c4\" (UID: \"02b652c3-0724-4a50-9e95-2a9890cc97c4\") " Oct 30 20:58:54.111176 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:54.111178 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b99eb76b-d1c6-49e9-9809-c351c0f5a4de-kube-api-access-wbl24\" (UniqueName: \"kubernetes.io/projected/b99eb76b-d1c6-49e9-9809-c351c0f5a4de-kube-api-access-wbl24\") pod \"b99eb76b-d1c6-49e9-9809-c351c0f5a4de\" (UID: \"b99eb76b-d1c6-49e9-9809-c351c0f5a4de\") " Oct 30 20:58:54.111435 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:54.111308 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/02b652c3-0724-4a50-9e95-2a9890cc97c4-kube-api-access-5fx85\" (UniqueName: \"kubernetes.io/projected/02b652c3-0724-4a50-9e95-2a9890cc97c4-kube-api-access-5fx85\") pod \"02b652c3-0724-4a50-9e95-2a9890cc97c4\" (UID: \"02b652c3-0724-4a50-9e95-2a9890cc97c4\") " Oct 30 20:58:54.111435 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:54.111407 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b99eb76b-d1c6-49e9-9809-c351c0f5a4de-tls-certs\" (UniqueName: \"kubernetes.io/secret/b99eb76b-d1c6-49e9-9809-c351c0f5a4de-tls-certs\") pod \"b99eb76b-d1c6-49e9-9809-c351c0f5a4de\" (UID: \"b99eb76b-d1c6-49e9-9809-c351c0f5a4de\") " Oct 30 20:58:54.111601 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:54.111446 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02b652c3-0724-4a50-9e95-2a9890cc97c4-model-cache" pod "02b652c3-0724-4a50-9e95-2a9890cc97c4" (UID: "02b652c3-0724-4a50-9e95-2a9890cc97c4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:58:54.111601 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:54.111472 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02b652c3-0724-4a50-9e95-2a9890cc97c4-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/02b652c3-0724-4a50-9e95-2a9890cc97c4-tmp-dir\") pod \"02b652c3-0724-4a50-9e95-2a9890cc97c4\" (UID: \"02b652c3-0724-4a50-9e95-2a9890cc97c4\") " Oct 30 20:58:54.111601 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:54.111541 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b99eb76b-d1c6-49e9-9809-c351c0f5a4de-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b99eb76b-d1c6-49e9-9809-c351c0f5a4de-model-cache\") pod \"b99eb76b-d1c6-49e9-9809-c351c0f5a4de\" (UID: \"b99eb76b-d1c6-49e9-9809-c351c0f5a4de\") " Oct 30 20:58:54.111821 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:54.111618 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b99eb76b-d1c6-49e9-9809-c351c0f5a4de-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b99eb76b-d1c6-49e9-9809-c351c0f5a4de-tmp-dir\") pod \"b99eb76b-d1c6-49e9-9809-c351c0f5a4de\" (UID: \"b99eb76b-d1c6-49e9-9809-c351c0f5a4de\") " Oct 30 20:58:54.111821 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:54.111719 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02b652c3-0724-4a50-9e95-2a9890cc97c4-tmp-dir" pod "02b652c3-0724-4a50-9e95-2a9890cc97c4" (UID: "02b652c3-0724-4a50-9e95-2a9890cc97c4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:58:54.112015 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:54.111996 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b99eb76b-d1c6-49e9-9809-c351c0f5a4de-tmp-dir" pod "b99eb76b-d1c6-49e9-9809-c351c0f5a4de" (UID: "b99eb76b-d1c6-49e9-9809-c351c0f5a4de"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:58:54.112080 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:54.111939 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b99eb76b-d1c6-49e9-9809-c351c0f5a4de-model-cache" pod "b99eb76b-d1c6-49e9-9809-c351c0f5a4de" (UID: "b99eb76b-d1c6-49e9-9809-c351c0f5a4de"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:58:54.112080 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:54.112057 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02b652c3-0724-4a50-9e95-2a9890cc97c4-dshm\" (UniqueName: \"kubernetes.io/empty-dir/02b652c3-0724-4a50-9e95-2a9890cc97c4-dshm\") pod \"02b652c3-0724-4a50-9e95-2a9890cc97c4\" (UID: \"02b652c3-0724-4a50-9e95-2a9890cc97c4\") " Oct 30 20:58:54.113216 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:54.112255 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b99eb76b-d1c6-49e9-9809-c351c0f5a4de-home\" (UniqueName: \"kubernetes.io/empty-dir/b99eb76b-d1c6-49e9-9809-c351c0f5a4de-home\") pod \"b99eb76b-d1c6-49e9-9809-c351c0f5a4de\" (UID: \"b99eb76b-d1c6-49e9-9809-c351c0f5a4de\") " Oct 30 20:58:54.113216 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:54.112291 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b99eb76b-d1c6-49e9-9809-c351c0f5a4de-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b99eb76b-d1c6-49e9-9809-c351c0f5a4de-dshm\") pod \"b99eb76b-d1c6-49e9-9809-c351c0f5a4de\" (UID: \"b99eb76b-d1c6-49e9-9809-c351c0f5a4de\") " Oct 30 20:58:54.113216 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:54.112336 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/02b652c3-0724-4a50-9e95-2a9890cc97c4-tls-certs\" (UniqueName: \"kubernetes.io/secret/02b652c3-0724-4a50-9e95-2a9890cc97c4-tls-certs\") pod \"02b652c3-0724-4a50-9e95-2a9890cc97c4\" (UID: \"02b652c3-0724-4a50-9e95-2a9890cc97c4\") " Oct 30 20:58:54.113216 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:54.112383 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02b652c3-0724-4a50-9e95-2a9890cc97c4-home\" (UniqueName: \"kubernetes.io/empty-dir/02b652c3-0724-4a50-9e95-2a9890cc97c4-home\") pod \"02b652c3-0724-4a50-9e95-2a9890cc97c4\" (UID: \"02b652c3-0724-4a50-9e95-2a9890cc97c4\") " Oct 30 20:58:54.113216 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:54.112475 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b99eb76b-d1c6-49e9-9809-c351c0f5a4de-home" pod "b99eb76b-d1c6-49e9-9809-c351c0f5a4de" (UID: "b99eb76b-d1c6-49e9-9809-c351c0f5a4de"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:58:54.113216 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:54.112728 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02b652c3-0724-4a50-9e95-2a9890cc97c4-home" pod "02b652c3-0724-4a50-9e95-2a9890cc97c4" (UID: "02b652c3-0724-4a50-9e95-2a9890cc97c4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:58:54.113216 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:54.113110 4611 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b99eb76b-d1c6-49e9-9809-c351c0f5a4de-model-cache\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:54.113216 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:54.113134 4611 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b99eb76b-d1c6-49e9-9809-c351c0f5a4de-tmp-dir\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:54.113216 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:54.113149 4611 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b99eb76b-d1c6-49e9-9809-c351c0f5a4de-home\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:54.113216 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:54.113164 4611 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/02b652c3-0724-4a50-9e95-2a9890cc97c4-home\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:54.113216 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:54.113177 4611 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/02b652c3-0724-4a50-9e95-2a9890cc97c4-model-cache\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:54.113216 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:54.113190 4611 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/02b652c3-0724-4a50-9e95-2a9890cc97c4-tmp-dir\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:54.113957 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:54.113864 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b99eb76b-d1c6-49e9-9809-c351c0f5a4de-kube-api-access-wbl24" pod "b99eb76b-d1c6-49e9-9809-c351c0f5a4de" (UID: "b99eb76b-d1c6-49e9-9809-c351c0f5a4de"). InnerVolumeSpecName "kube-api-access-wbl24". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:58:54.114014 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:54.113972 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02b652c3-0724-4a50-9e95-2a9890cc97c4-kube-api-access-5fx85" pod "02b652c3-0724-4a50-9e95-2a9890cc97c4" (UID: "02b652c3-0724-4a50-9e95-2a9890cc97c4"). InnerVolumeSpecName "kube-api-access-5fx85". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:58:54.114135 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:54.114112 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b99eb76b-d1c6-49e9-9809-c351c0f5a4de-tls-certs" pod "b99eb76b-d1c6-49e9-9809-c351c0f5a4de" (UID: "b99eb76b-d1c6-49e9-9809-c351c0f5a4de"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:58:54.114818 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:54.114796 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b99eb76b-d1c6-49e9-9809-c351c0f5a4de-dshm" pod "b99eb76b-d1c6-49e9-9809-c351c0f5a4de" (UID: "b99eb76b-d1c6-49e9-9809-c351c0f5a4de"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:58:54.115969 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:54.115945 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/02b652c3-0724-4a50-9e95-2a9890cc97c4-tls-certs" pod "02b652c3-0724-4a50-9e95-2a9890cc97c4" (UID: "02b652c3-0724-4a50-9e95-2a9890cc97c4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:58:54.116086 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:54.115969 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02b652c3-0724-4a50-9e95-2a9890cc97c4-dshm" pod "02b652c3-0724-4a50-9e95-2a9890cc97c4" (UID: "02b652c3-0724-4a50-9e95-2a9890cc97c4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:58:54.198773 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:54.194978 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:58:54.215066 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:54.215030 4611 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/02b652c3-0724-4a50-9e95-2a9890cc97c4-dshm\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:54.215066 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:54.215062 4611 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b99eb76b-d1c6-49e9-9809-c351c0f5a4de-dshm\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:54.215288 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:54.215076 4611 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/02b652c3-0724-4a50-9e95-2a9890cc97c4-tls-certs\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:54.215288 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:54.215089 4611 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wbl24\" (UniqueName: \"kubernetes.io/projected/b99eb76b-d1c6-49e9-9809-c351c0f5a4de-kube-api-access-wbl24\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:54.215288 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:54.215104 4611 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5fx85\" (UniqueName: \"kubernetes.io/projected/02b652c3-0724-4a50-9e95-2a9890cc97c4-kube-api-access-5fx85\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:54.215288 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:54.215116 4611 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b99eb76b-d1c6-49e9-9809-c351c0f5a4de-tls-certs\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:54.320660 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:54.320618 4611 generic.go:356] "Generic (PLEG): container finished" podID="b99eb76b-d1c6-49e9-9809-c351c0f5a4de" containerID="d406ff442d0c9a437cc2ff961f6cb0177e6a528cd2212db4510a648c6f8e2446" exitCode=0 Oct 30 20:58:54.320866 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:54.320762 4611 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-14cca2/premium-w5-14cca2-kserve-6878777857-wpjtj" Oct 30 20:58:54.320866 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:54.320782 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-14cca2/premium-w5-14cca2-kserve-6878777857-wpjtj" event={"ID":"b99eb76b-d1c6-49e9-9809-c351c0f5a4de","Type":"ContainerDied","Data":"d406ff442d0c9a437cc2ff961f6cb0177e6a528cd2212db4510a648c6f8e2446"} Oct 30 20:58:54.320866 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:54.320822 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-14cca2/premium-w5-14cca2-kserve-6878777857-wpjtj" event={"ID":"b99eb76b-d1c6-49e9-9809-c351c0f5a4de","Type":"ContainerDied","Data":"76a0c637a5b71d45e9c6b3862658c946946c7fb05aa6e54316e8326be92fd82e"} Oct 30 20:58:54.320866 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:54.320853 4611 scope.go:122] "RemoveContainer" containerID="d406ff442d0c9a437cc2ff961f6cb0177e6a528cd2212db4510a648c6f8e2446" Oct 30 20:58:54.323165 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:54.323102 4611 generic.go:356] "Generic (PLEG): container finished" podID="02b652c3-0724-4a50-9e95-2a9890cc97c4" containerID="d1b4cba96b77d59a8b7e17e5ea7041b95d5bac7a658002706fdc139ef29a806f" exitCode=0 Oct 30 20:58:54.323292 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:54.323193 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-14cca2/distinct-w5-14cca2-kserve-587f6996b8-sgf6h" event={"ID":"02b652c3-0724-4a50-9e95-2a9890cc97c4","Type":"ContainerDied","Data":"d1b4cba96b77d59a8b7e17e5ea7041b95d5bac7a658002706fdc139ef29a806f"} Oct 30 20:58:54.323292 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:54.323243 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-14cca2/distinct-w5-14cca2-kserve-587f6996b8-sgf6h" event={"ID":"02b652c3-0724-4a50-9e95-2a9890cc97c4","Type":"ContainerDied","Data":"669182bc8bed8cd3666e25e5f0c0d558a1f950b836b942e983b0c59d6445d118"} Oct 30 20:58:54.323406 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:54.323354 4611 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-14cca2/distinct-w5-14cca2-kserve-587f6996b8-sgf6h" Oct 30 20:58:54.336279 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:54.336260 4611 scope.go:122] "RemoveContainer" containerID="d406ff442d0c9a437cc2ff961f6cb0177e6a528cd2212db4510a648c6f8e2446" Oct 30 20:58:54.336640 ip-10-0-133-192 kubenswrapper[4611]: E1030 20:58:54.336612 4611 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d406ff442d0c9a437cc2ff961f6cb0177e6a528cd2212db4510a648c6f8e2446\": container with ID starting with d406ff442d0c9a437cc2ff961f6cb0177e6a528cd2212db4510a648c6f8e2446 not found: ID does not exist" containerID="d406ff442d0c9a437cc2ff961f6cb0177e6a528cd2212db4510a648c6f8e2446" Oct 30 20:58:54.336735 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:54.336649 4611 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d406ff442d0c9a437cc2ff961f6cb0177e6a528cd2212db4510a648c6f8e2446"} err="failed to get container status \"d406ff442d0c9a437cc2ff961f6cb0177e6a528cd2212db4510a648c6f8e2446\": rpc error: code = NotFound desc = could not find container \"d406ff442d0c9a437cc2ff961f6cb0177e6a528cd2212db4510a648c6f8e2446\": container with ID starting with d406ff442d0c9a437cc2ff961f6cb0177e6a528cd2212db4510a648c6f8e2446 not found: ID does not exist" Oct 30 20:58:54.336735 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:54.336671 4611 scope.go:122] "RemoveContainer" containerID="d1b4cba96b77d59a8b7e17e5ea7041b95d5bac7a658002706fdc139ef29a806f" Oct 30 20:58:54.347902 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:54.347881 4611 scope.go:122] "RemoveContainer" containerID="d1b4cba96b77d59a8b7e17e5ea7041b95d5bac7a658002706fdc139ef29a806f" Oct 30 20:58:54.348158 ip-10-0-133-192 kubenswrapper[4611]: E1030 20:58:54.348141 4611 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d1b4cba96b77d59a8b7e17e5ea7041b95d5bac7a658002706fdc139ef29a806f\": container with ID starting with d1b4cba96b77d59a8b7e17e5ea7041b95d5bac7a658002706fdc139ef29a806f not found: ID does not exist" containerID="d1b4cba96b77d59a8b7e17e5ea7041b95d5bac7a658002706fdc139ef29a806f" Oct 30 20:58:54.348208 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:54.348165 4611 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d1b4cba96b77d59a8b7e17e5ea7041b95d5bac7a658002706fdc139ef29a806f"} err="failed to get container status \"d1b4cba96b77d59a8b7e17e5ea7041b95d5bac7a658002706fdc139ef29a806f\": rpc error: code = NotFound desc = could not find container \"d1b4cba96b77d59a8b7e17e5ea7041b95d5bac7a658002706fdc139ef29a806f\": container with ID starting with d1b4cba96b77d59a8b7e17e5ea7041b95d5bac7a658002706fdc139ef29a806f not found: ID does not exist" Oct 30 20:58:54.357586 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:54.357561 4611 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-14cca2/distinct-w5-14cca2-kserve-587f6996b8-sgf6h"] Oct 30 20:58:54.361774 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:54.361750 4611 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-14cca2/distinct-w5-14cca2-kserve-587f6996b8-sgf6h"] Oct 30 20:58:54.373263 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:54.373237 4611 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-14cca2/premium-w5-14cca2-kserve-6878777857-wpjtj"] Oct 30 20:58:54.376552 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:54.376503 4611 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-14cca2/premium-w5-14cca2-kserve-6878777857-wpjtj"] Oct 30 20:58:54.909195 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:54.909102 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:58:54.955617 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:54.955576 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-14cca2/premium-w5-14cca2-kserve-6878777857-wpjtj" podUID="b99eb76b-d1c6-49e9-9809-c351c0f5a4de" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:58:55.582410 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:55.582364 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:58:55.607530 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:55.607481 4611 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="02b652c3-0724-4a50-9e95-2a9890cc97c4" path="/var/lib/kubelet/pods/02b652c3-0724-4a50-9e95-2a9890cc97c4/volumes" Oct 30 20:58:55.608125 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:55.608102 4611 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b99eb76b-d1c6-49e9-9809-c351c0f5a4de" path="/var/lib/kubelet/pods/b99eb76b-d1c6-49e9-9809-c351c0f5a4de/volumes" Oct 30 20:58:56.203528 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:56.203477 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:58:56.793802 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:56.793765 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:58:57.482839 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:57.482770 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:58:58.228538 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:58.224576 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:58:58.825321 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:58.825235 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:58:59.371069 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:59.371034 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:58:59.536720 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:58:59.536680 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:59:00.272237 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:00.272195 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:59:00.941541 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:00.941490 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:59:01.131625 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:01.131588 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:59:01.279892 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:01.279791 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:59:01.434967 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:01.434932 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:59:01.828082 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:01.828046 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:59:02.355454 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:02.355416 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:59:03.215358 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:03.215303 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:59:03.916707 ip-10-0-133-192 kubenswrapper[4611]: E1030 20:59:03.916673 4611 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hwzc6" containerName="extractor" Oct 30 20:59:03.978331 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:03.978273 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:59:04.685548 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:04.685498 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:59:05.309322 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:05.309283 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:59:05.776794 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:05.776764 4611 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7w5wz_7e2574c1-b59c-453c-9501-25ec13a1ccad/ovn-acl-logging/0.log" Oct 30 20:59:05.778215 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:05.778192 4611 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7w5wz_7e2574c1-b59c-453c-9501-25ec13a1ccad/ovn-acl-logging/0.log" Oct 30 20:59:05.973160 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:05.973116 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:59:06.607754 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:06.607719 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:59:07.211080 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:07.211042 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:59:07.568559 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:07.568472 4611 scope.go:122] "RemoveContainer" containerID="5a112870a7bd0d7ec22f2e6ac21b8fec865daabbcbd37215a94c46c0242807a7" Oct 30 20:59:07.578032 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:07.578015 4611 scope.go:122] "RemoveContainer" containerID="1c479d7a9e1eac8587c32d7ceca486804b3fd524eece15219dd5eb4cfed6c0ab" Oct 30 20:59:07.898945 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:07.898908 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:59:08.494560 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:08.493239 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:59:08.807569 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:08.807460 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:59:08.963031 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:08.962994 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:59:09.111676 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:09.111623 4611 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-6pfl5"] Oct 30 20:59:09.111873 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:09.111846 4611 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-6pfl5" podUID="06695006-f33e-474a-bfa5-40eb62bd42f8" containerName="istio-proxy" containerID="cri-o://087ddc58167f65b47f6e684ebfdfb7d6fc341e0c4e55750c35fe52dce7ad0473" gracePeriod=30 Oct 30 20:59:09.126747 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:09.126722 4611 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-ccszm"] Oct 30 20:59:09.126954 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:09.126931 4611 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-ccszm" podUID="69e69ba0-b2fd-4201-8c22-5ecac952d4bb" containerName="istio-proxy" containerID="cri-o://9aab1ebf1ca04f57868eba0f42a0d2ede0cdd21b941a5105b9671a0dbc642a80" gracePeriod=30 Oct 30 20:59:09.155943 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:09.155912 4611 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-k5p2c"] Oct 30 20:59:09.156120 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:09.156099 4611 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-k5p2c" podUID="85895a05-5edd-47b7-bd4e-8112526dd4d8" containerName="istio-proxy" containerID="cri-o://5bd45afa60cd729f1061937bf05738a9c2230e255ecedb8bea35a1b842e0da08" gracePeriod=30 Oct 30 20:59:09.219140 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:09.219007 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:59:09.576901 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:09.576817 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:59:10.208621 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:10.208580 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:59:10.612974 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:10.612939 4611 patch_prober.go:29] interesting pod/e2e-worker-w5-14cca2-openshift-default-d7f87db59-ccszm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 20:59:10.613127 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:10.613004 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-ccszm" podUID="69e69ba0-b2fd-4201-8c22-5ecac952d4bb" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 20:59:10.984044 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:10.983742 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:59:11.688147 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:11.688107 4611 patch_prober.go:29] interesting pod/e2e-worker-w5-14cca2-openshift-default-d7f87db59-6pfl5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 20:59:11.688688 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:11.688173 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-6pfl5" podUID="06695006-f33e-474a-bfa5-40eb62bd42f8" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 20:59:11.691022 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:11.690990 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:59:12.313721 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:12.313677 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:59:13.095329 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:13.095291 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:59:13.791310 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:13.791271 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:59:14.417856 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:14.417809 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:59:14.467435 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:14.467402 4611 generic.go:356] "Generic (PLEG): container finished" podID="06695006-f33e-474a-bfa5-40eb62bd42f8" containerID="087ddc58167f65b47f6e684ebfdfb7d6fc341e0c4e55750c35fe52dce7ad0473" exitCode=0 Oct 30 20:59:14.467653 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:14.467476 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-6pfl5" event={"ID":"06695006-f33e-474a-bfa5-40eb62bd42f8","Type":"ContainerDied","Data":"087ddc58167f65b47f6e684ebfdfb7d6fc341e0c4e55750c35fe52dce7ad0473"} Oct 30 20:59:14.467653 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:14.467554 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-6pfl5" event={"ID":"06695006-f33e-474a-bfa5-40eb62bd42f8","Type":"ContainerDied","Data":"f1d317ed8399fd402fd0848fc383a41e38b436aa21d27912592be2aa68f9bdac"} Oct 30 20:59:14.467653 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:14.467580 4611 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f1d317ed8399fd402fd0848fc383a41e38b436aa21d27912592be2aa68f9bdac" Oct 30 20:59:14.470863 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:14.470837 4611 generic.go:356] "Generic (PLEG): container finished" podID="85895a05-5edd-47b7-bd4e-8112526dd4d8" containerID="5bd45afa60cd729f1061937bf05738a9c2230e255ecedb8bea35a1b842e0da08" exitCode=0 Oct 30 20:59:14.471222 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:14.470906 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-k5p2c" event={"ID":"85895a05-5edd-47b7-bd4e-8112526dd4d8","Type":"ContainerDied","Data":"5bd45afa60cd729f1061937bf05738a9c2230e255ecedb8bea35a1b842e0da08"} Oct 30 20:59:14.471222 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:14.470964 4611 scope.go:122] "RemoveContainer" containerID="a177d3a5bbd6f448456bee62c37268c18299145eed1615119d4d3979cd4e5e79" Oct 30 20:59:14.472869 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:14.472822 4611 generic.go:356] "Generic (PLEG): container finished" podID="69e69ba0-b2fd-4201-8c22-5ecac952d4bb" containerID="9aab1ebf1ca04f57868eba0f42a0d2ede0cdd21b941a5105b9671a0dbc642a80" exitCode=0 Oct 30 20:59:14.473020 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:14.472946 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-ccszm" event={"ID":"69e69ba0-b2fd-4201-8c22-5ecac952d4bb","Type":"ContainerDied","Data":"9aab1ebf1ca04f57868eba0f42a0d2ede0cdd21b941a5105b9671a0dbc642a80"} Oct 30 20:59:14.473020 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:14.472993 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-ccszm" event={"ID":"69e69ba0-b2fd-4201-8c22-5ecac952d4bb","Type":"ContainerDied","Data":"49f62b300d3c8db8bd1a4e53cf63735443b25131368d849b608f6628dc8c83e2"} Oct 30 20:59:14.473020 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:14.473017 4611 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="49f62b300d3c8db8bd1a4e53cf63735443b25131368d849b608f6628dc8c83e2" Oct 30 20:59:14.494528 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:14.493706 4611 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-6pfl5" Oct 30 20:59:14.498268 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:14.498236 4611 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-ccszm" Oct 30 20:59:14.542207 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:14.542183 4611 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-k5p2c" Oct 30 20:59:14.544458 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:14.544433 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69e69ba0-b2fd-4201-8c22-5ecac952d4bb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69e69ba0-b2fd-4201-8c22-5ecac952d4bb-istio-envoy\") pod \"69e69ba0-b2fd-4201-8c22-5ecac952d4bb\" (UID: \"69e69ba0-b2fd-4201-8c22-5ecac952d4bb\") " Oct 30 20:59:14.544562 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:14.544502 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/69e69ba0-b2fd-4201-8c22-5ecac952d4bb-kube-api-access-j965r\" (UniqueName: \"kubernetes.io/projected/69e69ba0-b2fd-4201-8c22-5ecac952d4bb-kube-api-access-j965r\") pod \"69e69ba0-b2fd-4201-8c22-5ecac952d4bb\" (UID: \"69e69ba0-b2fd-4201-8c22-5ecac952d4bb\") " Oct 30 20:59:14.544622 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:14.544606 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06695006-f33e-474a-bfa5-40eb62bd42f8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06695006-f33e-474a-bfa5-40eb62bd42f8-istio-envoy\") pod \"06695006-f33e-474a-bfa5-40eb62bd42f8\" (UID: \"06695006-f33e-474a-bfa5-40eb62bd42f8\") " Oct 30 20:59:14.544664 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:14.544648 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/06695006-f33e-474a-bfa5-40eb62bd42f8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06695006-f33e-474a-bfa5-40eb62bd42f8-istiod-ca-cert\") pod \"06695006-f33e-474a-bfa5-40eb62bd42f8\" (UID: \"06695006-f33e-474a-bfa5-40eb62bd42f8\") " Oct 30 20:59:14.544732 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:14.544713 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06695006-f33e-474a-bfa5-40eb62bd42f8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06695006-f33e-474a-bfa5-40eb62bd42f8-workload-socket\") pod \"06695006-f33e-474a-bfa5-40eb62bd42f8\" (UID: \"06695006-f33e-474a-bfa5-40eb62bd42f8\") " Oct 30 20:59:14.544790 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:14.544758 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06695006-f33e-474a-bfa5-40eb62bd42f8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06695006-f33e-474a-bfa5-40eb62bd42f8-credential-socket\") pod \"06695006-f33e-474a-bfa5-40eb62bd42f8\" (UID: \"06695006-f33e-474a-bfa5-40eb62bd42f8\") " Oct 30 20:59:14.544842 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:14.544787 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06695006-f33e-474a-bfa5-40eb62bd42f8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06695006-f33e-474a-bfa5-40eb62bd42f8-istio-data\") pod \"06695006-f33e-474a-bfa5-40eb62bd42f8\" (UID: \"06695006-f33e-474a-bfa5-40eb62bd42f8\") " Oct 30 20:59:14.544842 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:14.544830 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06695006-f33e-474a-bfa5-40eb62bd42f8-istio-token\" (UniqueName: \"kubernetes.io/projected/06695006-f33e-474a-bfa5-40eb62bd42f8-istio-token\") pod \"06695006-f33e-474a-bfa5-40eb62bd42f8\" (UID: \"06695006-f33e-474a-bfa5-40eb62bd42f8\") " Oct 30 20:59:14.544954 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:14.544860 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69e69ba0-b2fd-4201-8c22-5ecac952d4bb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69e69ba0-b2fd-4201-8c22-5ecac952d4bb-workload-certs\") pod \"69e69ba0-b2fd-4201-8c22-5ecac952d4bb\" (UID: \"69e69ba0-b2fd-4201-8c22-5ecac952d4bb\") " Oct 30 20:59:14.544954 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:14.544891 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69e69ba0-b2fd-4201-8c22-5ecac952d4bb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69e69ba0-b2fd-4201-8c22-5ecac952d4bb-istio-data\") pod \"69e69ba0-b2fd-4201-8c22-5ecac952d4bb\" (UID: \"69e69ba0-b2fd-4201-8c22-5ecac952d4bb\") " Oct 30 20:59:14.544954 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:14.544943 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/69e69ba0-b2fd-4201-8c22-5ecac952d4bb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69e69ba0-b2fd-4201-8c22-5ecac952d4bb-istio-podinfo\") pod \"69e69ba0-b2fd-4201-8c22-5ecac952d4bb\" (UID: \"69e69ba0-b2fd-4201-8c22-5ecac952d4bb\") " Oct 30 20:59:14.545108 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:14.544985 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/06695006-f33e-474a-bfa5-40eb62bd42f8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06695006-f33e-474a-bfa5-40eb62bd42f8-istio-podinfo\") pod \"06695006-f33e-474a-bfa5-40eb62bd42f8\" (UID: \"06695006-f33e-474a-bfa5-40eb62bd42f8\") " Oct 30 20:59:14.545108 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:14.545020 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69e69ba0-b2fd-4201-8c22-5ecac952d4bb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69e69ba0-b2fd-4201-8c22-5ecac952d4bb-workload-socket\") pod \"69e69ba0-b2fd-4201-8c22-5ecac952d4bb\" (UID: \"69e69ba0-b2fd-4201-8c22-5ecac952d4bb\") " Oct 30 20:59:14.545108 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:14.545055 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06695006-f33e-474a-bfa5-40eb62bd42f8-workload-socket" pod "06695006-f33e-474a-bfa5-40eb62bd42f8" (UID: "06695006-f33e-474a-bfa5-40eb62bd42f8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:59:14.545275 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:14.545117 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06695006-f33e-474a-bfa5-40eb62bd42f8-kube-api-access-76brf\" (UniqueName: \"kubernetes.io/projected/06695006-f33e-474a-bfa5-40eb62bd42f8-kube-api-access-76brf\") pod \"06695006-f33e-474a-bfa5-40eb62bd42f8\" (UID: \"06695006-f33e-474a-bfa5-40eb62bd42f8\") " Oct 30 20:59:14.545275 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:14.545145 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/69e69ba0-b2fd-4201-8c22-5ecac952d4bb-istio-token\" (UniqueName: \"kubernetes.io/projected/69e69ba0-b2fd-4201-8c22-5ecac952d4bb-istio-token\") pod \"69e69ba0-b2fd-4201-8c22-5ecac952d4bb\" (UID: \"69e69ba0-b2fd-4201-8c22-5ecac952d4bb\") " Oct 30 20:59:14.545275 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:14.545192 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69e69ba0-b2fd-4201-8c22-5ecac952d4bb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69e69ba0-b2fd-4201-8c22-5ecac952d4bb-credential-socket\") pod \"69e69ba0-b2fd-4201-8c22-5ecac952d4bb\" (UID: \"69e69ba0-b2fd-4201-8c22-5ecac952d4bb\") " Oct 30 20:59:14.545275 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:14.545259 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06695006-f33e-474a-bfa5-40eb62bd42f8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06695006-f33e-474a-bfa5-40eb62bd42f8-workload-certs\") pod \"06695006-f33e-474a-bfa5-40eb62bd42f8\" (UID: \"06695006-f33e-474a-bfa5-40eb62bd42f8\") " Oct 30 20:59:14.545406 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:14.545307 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/69e69ba0-b2fd-4201-8c22-5ecac952d4bb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69e69ba0-b2fd-4201-8c22-5ecac952d4bb-istiod-ca-cert\") pod \"69e69ba0-b2fd-4201-8c22-5ecac952d4bb\" (UID: \"69e69ba0-b2fd-4201-8c22-5ecac952d4bb\") " Oct 30 20:59:14.545406 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:14.545339 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06695006-f33e-474a-bfa5-40eb62bd42f8-credential-socket" pod "06695006-f33e-474a-bfa5-40eb62bd42f8" (UID: "06695006-f33e-474a-bfa5-40eb62bd42f8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:59:14.545467 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:14.545428 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06695006-f33e-474a-bfa5-40eb62bd42f8-istio-data" pod "06695006-f33e-474a-bfa5-40eb62bd42f8" (UID: "06695006-f33e-474a-bfa5-40eb62bd42f8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:59:14.545583 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:14.545561 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69e69ba0-b2fd-4201-8c22-5ecac952d4bb-workload-socket" pod "69e69ba0-b2fd-4201-8c22-5ecac952d4bb" (UID: "69e69ba0-b2fd-4201-8c22-5ecac952d4bb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:59:14.546033 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:14.546008 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69e69ba0-b2fd-4201-8c22-5ecac952d4bb-istio-data" pod "69e69ba0-b2fd-4201-8c22-5ecac952d4bb" (UID: "69e69ba0-b2fd-4201-8c22-5ecac952d4bb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:59:14.546033 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:14.546022 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69e69ba0-b2fd-4201-8c22-5ecac952d4bb-istiod-ca-cert" pod "69e69ba0-b2fd-4201-8c22-5ecac952d4bb" (UID: "69e69ba0-b2fd-4201-8c22-5ecac952d4bb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:59:14.546225 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:14.546203 4611 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06695006-f33e-474a-bfa5-40eb62bd42f8-workload-socket\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:59:14.546375 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:14.546318 4611 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06695006-f33e-474a-bfa5-40eb62bd42f8-credential-socket\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:59:14.546375 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:14.546347 4611 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06695006-f33e-474a-bfa5-40eb62bd42f8-istio-data\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:59:14.546375 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:14.546362 4611 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69e69ba0-b2fd-4201-8c22-5ecac952d4bb-workload-socket\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:59:14.547633 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:14.547593 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69e69ba0-b2fd-4201-8c22-5ecac952d4bb-workload-certs" pod "69e69ba0-b2fd-4201-8c22-5ecac952d4bb" (UID: "69e69ba0-b2fd-4201-8c22-5ecac952d4bb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:59:14.549228 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:14.548754 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06695006-f33e-474a-bfa5-40eb62bd42f8-istio-envoy" pod "06695006-f33e-474a-bfa5-40eb62bd42f8" (UID: "06695006-f33e-474a-bfa5-40eb62bd42f8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:59:14.549228 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:14.545643 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06695006-f33e-474a-bfa5-40eb62bd42f8-istiod-ca-cert" pod "06695006-f33e-474a-bfa5-40eb62bd42f8" (UID: "06695006-f33e-474a-bfa5-40eb62bd42f8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:59:14.549228 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:14.548997 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69e69ba0-b2fd-4201-8c22-5ecac952d4bb-credential-socket" pod "69e69ba0-b2fd-4201-8c22-5ecac952d4bb" (UID: "69e69ba0-b2fd-4201-8c22-5ecac952d4bb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:59:14.551009 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:14.550820 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69e69ba0-b2fd-4201-8c22-5ecac952d4bb-kube-api-access-j965r" pod "69e69ba0-b2fd-4201-8c22-5ecac952d4bb" (UID: "69e69ba0-b2fd-4201-8c22-5ecac952d4bb"). InnerVolumeSpecName "kube-api-access-j965r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:59:14.551853 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:14.551439 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06695006-f33e-474a-bfa5-40eb62bd42f8-workload-certs" pod "06695006-f33e-474a-bfa5-40eb62bd42f8" (UID: "06695006-f33e-474a-bfa5-40eb62bd42f8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:59:14.552280 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:14.552249 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06695006-f33e-474a-bfa5-40eb62bd42f8-istio-token" pod "06695006-f33e-474a-bfa5-40eb62bd42f8" (UID: "06695006-f33e-474a-bfa5-40eb62bd42f8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:59:14.552905 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:14.552873 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69e69ba0-b2fd-4201-8c22-5ecac952d4bb-istio-envoy" pod "69e69ba0-b2fd-4201-8c22-5ecac952d4bb" (UID: "69e69ba0-b2fd-4201-8c22-5ecac952d4bb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:59:14.553102 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:14.553083 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06695006-f33e-474a-bfa5-40eb62bd42f8-kube-api-access-76brf" pod "06695006-f33e-474a-bfa5-40eb62bd42f8" (UID: "06695006-f33e-474a-bfa5-40eb62bd42f8"). InnerVolumeSpecName "kube-api-access-76brf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:59:14.553459 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:14.553440 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/06695006-f33e-474a-bfa5-40eb62bd42f8-istio-podinfo" pod "06695006-f33e-474a-bfa5-40eb62bd42f8" (UID: "06695006-f33e-474a-bfa5-40eb62bd42f8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:59:14.553817 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:14.553787 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/69e69ba0-b2fd-4201-8c22-5ecac952d4bb-istio-podinfo" pod "69e69ba0-b2fd-4201-8c22-5ecac952d4bb" (UID: "69e69ba0-b2fd-4201-8c22-5ecac952d4bb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:59:14.554659 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:14.554633 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69e69ba0-b2fd-4201-8c22-5ecac952d4bb-istio-token" pod "69e69ba0-b2fd-4201-8c22-5ecac952d4bb" (UID: "69e69ba0-b2fd-4201-8c22-5ecac952d4bb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:59:14.647247 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:14.647157 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85895a05-5edd-47b7-bd4e-8112526dd4d8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85895a05-5edd-47b7-bd4e-8112526dd4d8-istio-envoy\") pod \"85895a05-5edd-47b7-bd4e-8112526dd4d8\" (UID: \"85895a05-5edd-47b7-bd4e-8112526dd4d8\") " Oct 30 20:59:14.647483 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:14.647266 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85895a05-5edd-47b7-bd4e-8112526dd4d8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85895a05-5edd-47b7-bd4e-8112526dd4d8-workload-socket\") pod \"85895a05-5edd-47b7-bd4e-8112526dd4d8\" (UID: \"85895a05-5edd-47b7-bd4e-8112526dd4d8\") " Oct 30 20:59:14.647483 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:14.647307 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85895a05-5edd-47b7-bd4e-8112526dd4d8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85895a05-5edd-47b7-bd4e-8112526dd4d8-workload-certs\") pod \"85895a05-5edd-47b7-bd4e-8112526dd4d8\" (UID: \"85895a05-5edd-47b7-bd4e-8112526dd4d8\") " Oct 30 20:59:14.647483 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:14.647371 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/85895a05-5edd-47b7-bd4e-8112526dd4d8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85895a05-5edd-47b7-bd4e-8112526dd4d8-istiod-ca-cert\") pod \"85895a05-5edd-47b7-bd4e-8112526dd4d8\" (UID: \"85895a05-5edd-47b7-bd4e-8112526dd4d8\") " Oct 30 20:59:14.647483 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:14.647403 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/85895a05-5edd-47b7-bd4e-8112526dd4d8-kube-api-access-cxx86\" (UniqueName: \"kubernetes.io/projected/85895a05-5edd-47b7-bd4e-8112526dd4d8-kube-api-access-cxx86\") pod \"85895a05-5edd-47b7-bd4e-8112526dd4d8\" (UID: \"85895a05-5edd-47b7-bd4e-8112526dd4d8\") " Oct 30 20:59:14.647483 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:14.647437 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/85895a05-5edd-47b7-bd4e-8112526dd4d8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85895a05-5edd-47b7-bd4e-8112526dd4d8-istio-podinfo\") pod \"85895a05-5edd-47b7-bd4e-8112526dd4d8\" (UID: \"85895a05-5edd-47b7-bd4e-8112526dd4d8\") " Oct 30 20:59:14.647483 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:14.647472 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/85895a05-5edd-47b7-bd4e-8112526dd4d8-istio-token\" (UniqueName: \"kubernetes.io/projected/85895a05-5edd-47b7-bd4e-8112526dd4d8-istio-token\") pod \"85895a05-5edd-47b7-bd4e-8112526dd4d8\" (UID: \"85895a05-5edd-47b7-bd4e-8112526dd4d8\") " Oct 30 20:59:14.648323 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:14.647518 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85895a05-5edd-47b7-bd4e-8112526dd4d8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85895a05-5edd-47b7-bd4e-8112526dd4d8-credential-socket\") pod \"85895a05-5edd-47b7-bd4e-8112526dd4d8\" (UID: \"85895a05-5edd-47b7-bd4e-8112526dd4d8\") " Oct 30 20:59:14.648323 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:14.647704 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85895a05-5edd-47b7-bd4e-8112526dd4d8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85895a05-5edd-47b7-bd4e-8112526dd4d8-istio-data\") pod \"85895a05-5edd-47b7-bd4e-8112526dd4d8\" (UID: \"85895a05-5edd-47b7-bd4e-8112526dd4d8\") " Oct 30 20:59:14.648323 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:14.648077 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85895a05-5edd-47b7-bd4e-8112526dd4d8-workload-certs" pod "85895a05-5edd-47b7-bd4e-8112526dd4d8" (UID: "85895a05-5edd-47b7-bd4e-8112526dd4d8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:59:14.648880 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:14.648416 4611 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/06695006-f33e-474a-bfa5-40eb62bd42f8-istio-token\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:59:14.648880 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:14.648442 4611 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69e69ba0-b2fd-4201-8c22-5ecac952d4bb-workload-certs\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:59:14.648880 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:14.648457 4611 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69e69ba0-b2fd-4201-8c22-5ecac952d4bb-istio-data\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:59:14.648880 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:14.648471 4611 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69e69ba0-b2fd-4201-8c22-5ecac952d4bb-istio-podinfo\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:59:14.648880 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:14.648485 4611 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06695006-f33e-474a-bfa5-40eb62bd42f8-istio-podinfo\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:59:14.648880 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:14.648552 4611 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-76brf\" (UniqueName: \"kubernetes.io/projected/06695006-f33e-474a-bfa5-40eb62bd42f8-kube-api-access-76brf\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:59:14.648880 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:14.648570 4611 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69e69ba0-b2fd-4201-8c22-5ecac952d4bb-istio-token\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:59:14.648880 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:14.648583 4611 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69e69ba0-b2fd-4201-8c22-5ecac952d4bb-credential-socket\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:59:14.648880 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:14.648597 4611 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06695006-f33e-474a-bfa5-40eb62bd42f8-workload-certs\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:59:14.648880 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:14.648613 4611 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69e69ba0-b2fd-4201-8c22-5ecac952d4bb-istiod-ca-cert\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:59:14.648880 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:14.648628 4611 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69e69ba0-b2fd-4201-8c22-5ecac952d4bb-istio-envoy\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:59:14.648880 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:14.648643 4611 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j965r\" (UniqueName: \"kubernetes.io/projected/69e69ba0-b2fd-4201-8c22-5ecac952d4bb-kube-api-access-j965r\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:59:14.648880 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:14.648656 4611 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06695006-f33e-474a-bfa5-40eb62bd42f8-istio-envoy\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:59:14.648880 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:14.648670 4611 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06695006-f33e-474a-bfa5-40eb62bd42f8-istiod-ca-cert\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:59:14.648880 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:14.648684 4611 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85895a05-5edd-47b7-bd4e-8112526dd4d8-workload-certs\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:59:14.650103 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:14.649041 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85895a05-5edd-47b7-bd4e-8112526dd4d8-istiod-ca-cert" pod "85895a05-5edd-47b7-bd4e-8112526dd4d8" (UID: "85895a05-5edd-47b7-bd4e-8112526dd4d8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:59:14.650103 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:14.649358 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85895a05-5edd-47b7-bd4e-8112526dd4d8-credential-socket" pod "85895a05-5edd-47b7-bd4e-8112526dd4d8" (UID: "85895a05-5edd-47b7-bd4e-8112526dd4d8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:59:14.650103 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:14.649434 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85895a05-5edd-47b7-bd4e-8112526dd4d8-istio-data" pod "85895a05-5edd-47b7-bd4e-8112526dd4d8" (UID: "85895a05-5edd-47b7-bd4e-8112526dd4d8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:59:14.650103 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:14.649635 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85895a05-5edd-47b7-bd4e-8112526dd4d8-workload-socket" pod "85895a05-5edd-47b7-bd4e-8112526dd4d8" (UID: "85895a05-5edd-47b7-bd4e-8112526dd4d8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:59:14.651019 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:14.650996 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85895a05-5edd-47b7-bd4e-8112526dd4d8-istio-envoy" pod "85895a05-5edd-47b7-bd4e-8112526dd4d8" (UID: "85895a05-5edd-47b7-bd4e-8112526dd4d8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:59:14.651345 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:14.651300 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85895a05-5edd-47b7-bd4e-8112526dd4d8-istio-token" pod "85895a05-5edd-47b7-bd4e-8112526dd4d8" (UID: "85895a05-5edd-47b7-bd4e-8112526dd4d8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:59:14.653282 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:14.653249 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/85895a05-5edd-47b7-bd4e-8112526dd4d8-istio-podinfo" pod "85895a05-5edd-47b7-bd4e-8112526dd4d8" (UID: "85895a05-5edd-47b7-bd4e-8112526dd4d8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:59:14.653683 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:14.653657 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85895a05-5edd-47b7-bd4e-8112526dd4d8-kube-api-access-cxx86" pod "85895a05-5edd-47b7-bd4e-8112526dd4d8" (UID: "85895a05-5edd-47b7-bd4e-8112526dd4d8"). InnerVolumeSpecName "kube-api-access-cxx86". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:59:14.749441 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:14.749411 4611 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85895a05-5edd-47b7-bd4e-8112526dd4d8-istio-podinfo\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:59:14.749441 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:14.749438 4611 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/85895a05-5edd-47b7-bd4e-8112526dd4d8-istio-token\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:59:14.749621 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:14.749448 4611 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85895a05-5edd-47b7-bd4e-8112526dd4d8-credential-socket\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:59:14.749621 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:14.749457 4611 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85895a05-5edd-47b7-bd4e-8112526dd4d8-istio-data\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:59:14.749621 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:14.749465 4611 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85895a05-5edd-47b7-bd4e-8112526dd4d8-istio-envoy\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:59:14.749621 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:14.749473 4611 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85895a05-5edd-47b7-bd4e-8112526dd4d8-workload-socket\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:59:14.749621 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:14.749481 4611 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85895a05-5edd-47b7-bd4e-8112526dd4d8-istiod-ca-cert\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:59:14.749621 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:14.749489 4611 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cxx86\" (UniqueName: \"kubernetes.io/projected/85895a05-5edd-47b7-bd4e-8112526dd4d8-kube-api-access-cxx86\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:59:15.179074 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:15.179038 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:59:15.343643 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:15.343610 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:59:15.491866 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:15.491773 4611 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-k5p2c" Oct 30 20:59:15.491866 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:15.491806 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-k5p2c" event={"ID":"85895a05-5edd-47b7-bd4e-8112526dd4d8","Type":"ContainerDied","Data":"31393a264ed7e656a1a5cbf87637d3ee5d63f4f888c6a3603e5ee1d3e629b931"} Oct 30 20:59:15.492390 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:15.491867 4611 scope.go:122] "RemoveContainer" containerID="5bd45afa60cd729f1061937bf05738a9c2230e255ecedb8bea35a1b842e0da08" Oct 30 20:59:15.492390 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:15.492098 4611 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-ccszm" Oct 30 20:59:15.492390 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:15.492250 4611 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-6pfl5" Oct 30 20:59:15.527386 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:15.527354 4611 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-k5p2c"] Oct 30 20:59:15.530627 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:15.530594 4611 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-k5p2c"] Oct 30 20:59:15.547454 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:15.547427 4611 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-ccszm"] Oct 30 20:59:15.553406 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:15.553383 4611 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-ccszm"] Oct 30 20:59:15.563281 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:15.563256 4611 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-6pfl5"] Oct 30 20:59:15.568738 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:15.568717 4611 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-14cca2-openshift-default-d7f87db59-6pfl5"] Oct 30 20:59:15.605913 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:15.605887 4611 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="06695006-f33e-474a-bfa5-40eb62bd42f8" path="/var/lib/kubelet/pods/06695006-f33e-474a-bfa5-40eb62bd42f8/volumes" Oct 30 20:59:15.606357 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:15.606344 4611 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="69e69ba0-b2fd-4201-8c22-5ecac952d4bb" path="/var/lib/kubelet/pods/69e69ba0-b2fd-4201-8c22-5ecac952d4bb/volumes" Oct 30 20:59:15.606797 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:15.606783 4611 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="85895a05-5edd-47b7-bd4e-8112526dd4d8" path="/var/lib/kubelet/pods/85895a05-5edd-47b7-bd4e-8112526dd4d8/volumes" Oct 30 20:59:15.996281 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:15.996241 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:59:16.842150 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:16.842118 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:59:17.745695 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:17.745654 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:59:17.897703 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:17.897667 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:59:18.392577 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:18.385747 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:59:19.036324 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:19.036287 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:59:19.718955 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:19.718892 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:59:20.382252 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:20.382199 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:59:21.069268 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:21.065264 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:59:21.804475 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:21.804438 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:59:22.499552 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:22.493598 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:59:23.089197 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:23.089162 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:59:23.721112 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:23.721074 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:59:24.405746 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:24.405699 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:59:24.667048 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:24.666965 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:59:25.289645 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:25.289609 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:59:25.922261 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:25.922223 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:59:26.601581 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:26.601545 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:59:27.288658 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:27.288603 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:59:27.905493 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:27.905461 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:59:28.593685 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:28.593651 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:59:29.211942 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:29.211725 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:59:29.884617 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:29.884579 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:59:30.491624 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:30.491586 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:59:31.182473 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:31.182425 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:59:31.803558 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:31.803486 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:59:32.397617 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:32.397557 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:59:33.013065 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:33.013014 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:59:33.680700 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:33.680656 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:59:33.916849 ip-10-0-133-192 kubenswrapper[4611]: E1030 20:59:33.916817 4611 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hwzc6" containerName="extractor" Oct 30 20:59:34.213381 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:34.213347 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:59:34.376326 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:34.376277 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:59:35.133355 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:35.133315 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:59:35.790769 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:35.790663 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:59:36.463669 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:36.463577 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:59:37.118384 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:37.118345 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:59:37.721622 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:37.721581 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:59:38.393050 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:38.393017 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:59:39.125948 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:39.125909 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:59:39.132519 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:39.132465 4611 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-3d909d/distinct-2-w6-3d909d-kserve-59c5d5c595-cjg2m"] Oct 30 20:59:39.149227 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:39.149195 4611 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-3d909d/premium-w6-3d909d-kserve-765648946f-2qvd2"] Oct 30 20:59:39.335379 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:39.335345 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:59:39.995360 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:39.995325 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:59:40.705350 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:40.705307 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:59:41.420316 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:41.420275 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:59:42.102530 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:42.101182 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:59:42.795449 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:42.795415 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:59:43.491574 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:43.491401 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:59:44.088724 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:44.088520 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:59:44.467037 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:44.467003 4611 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-3d909d-74bfb44f9b-5f2x8"] Oct 30 20:59:44.484015 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:44.483981 4611 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-3d909d-57b9b6dcf9-l9pwk"] Oct 30 20:59:44.582211 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:44.582177 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:59:44.643273 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:44.643236 4611 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-be73052e-plwg2"] Oct 30 20:59:44.643801 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:44.643786 4611 cpu_manager.go:414] "containerMap: removing container" podUID="02b652c3-0724-4a50-9e95-2a9890cc97c4" containerName="main" Oct 30 20:59:44.643850 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:44.643802 4611 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="02b652c3-0724-4a50-9e95-2a9890cc97c4" containerName="main" Oct 30 20:59:44.643850 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:44.643812 4611 cpu_manager.go:414] "containerMap: removing container" podUID="85895a05-5edd-47b7-bd4e-8112526dd4d8" containerName="istio-proxy" Oct 30 20:59:44.643850 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:44.643817 4611 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="85895a05-5edd-47b7-bd4e-8112526dd4d8" containerName="istio-proxy" Oct 30 20:59:44.643850 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:44.643825 4611 cpu_manager.go:414] "containerMap: removing container" podUID="06695006-f33e-474a-bfa5-40eb62bd42f8" containerName="istio-proxy" Oct 30 20:59:44.643850 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:44.643833 4611 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06695006-f33e-474a-bfa5-40eb62bd42f8" containerName="istio-proxy" Oct 30 20:59:44.644007 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:44.643863 4611 cpu_manager.go:414] "containerMap: removing container" podUID="69e69ba0-b2fd-4201-8c22-5ecac952d4bb" containerName="istio-proxy" Oct 30 20:59:44.644007 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:44.643868 4611 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="69e69ba0-b2fd-4201-8c22-5ecac952d4bb" containerName="istio-proxy" Oct 30 20:59:44.644007 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:44.643876 4611 cpu_manager.go:414] "containerMap: removing container" podUID="78c5ded8-a0ed-4176-bc86-f6e3e233b455" containerName="revoke-keys" Oct 30 20:59:44.644007 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:44.643882 4611 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="78c5ded8-a0ed-4176-bc86-f6e3e233b455" containerName="revoke-keys" Oct 30 20:59:44.644007 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:44.643889 4611 cpu_manager.go:414] "containerMap: removing container" podUID="85895a05-5edd-47b7-bd4e-8112526dd4d8" containerName="istio-proxy" Oct 30 20:59:44.644007 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:44.643894 4611 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="85895a05-5edd-47b7-bd4e-8112526dd4d8" containerName="istio-proxy" Oct 30 20:59:44.644007 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:44.643903 4611 cpu_manager.go:414] "containerMap: removing container" podUID="63d294fd-5293-4752-9116-0aafa81d5e4f" containerName="istio-proxy" Oct 30 20:59:44.644007 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:44.643908 4611 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63d294fd-5293-4752-9116-0aafa81d5e4f" containerName="istio-proxy" Oct 30 20:59:44.644007 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:44.643931 4611 cpu_manager.go:414] "containerMap: removing container" podUID="b99eb76b-d1c6-49e9-9809-c351c0f5a4de" containerName="main" Oct 30 20:59:44.644007 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:44.643935 4611 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b99eb76b-d1c6-49e9-9809-c351c0f5a4de" containerName="main" Oct 30 20:59:44.644394 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:44.644020 4611 memory_manager.go:365] "RemoveStaleState removing state" podUID="b99eb76b-d1c6-49e9-9809-c351c0f5a4de" containerName="main" Oct 30 20:59:44.644394 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:44.644029 4611 memory_manager.go:365] "RemoveStaleState removing state" podUID="02b652c3-0724-4a50-9e95-2a9890cc97c4" containerName="main" Oct 30 20:59:44.644394 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:44.644034 4611 memory_manager.go:365] "RemoveStaleState removing state" podUID="85895a05-5edd-47b7-bd4e-8112526dd4d8" containerName="istio-proxy" Oct 30 20:59:44.644394 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:44.644040 4611 memory_manager.go:365] "RemoveStaleState removing state" podUID="06695006-f33e-474a-bfa5-40eb62bd42f8" containerName="istio-proxy" Oct 30 20:59:44.644394 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:44.644048 4611 memory_manager.go:365] "RemoveStaleState removing state" podUID="69e69ba0-b2fd-4201-8c22-5ecac952d4bb" containerName="istio-proxy" Oct 30 20:59:44.644394 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:44.644054 4611 memory_manager.go:365] "RemoveStaleState removing state" podUID="78c5ded8-a0ed-4176-bc86-f6e3e233b455" containerName="revoke-keys" Oct 30 20:59:44.644394 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:44.644060 4611 memory_manager.go:365] "RemoveStaleState removing state" podUID="85895a05-5edd-47b7-bd4e-8112526dd4d8" containerName="istio-proxy" Oct 30 20:59:44.647858 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:44.647834 4611 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be73052e-plwg2" Oct 30 20:59:44.658640 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:44.658616 4611 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-be73052e-plwg2"] Oct 30 20:59:44.737196 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:44.737011 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/44c74378-cef2-413a-a344-81d8093a89e8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-be73052e-plwg2\" (UID: \"44c74378-cef2-413a-a344-81d8093a89e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be73052e-plwg2" Oct 30 20:59:44.737196 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:44.737140 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2q2jh\" (UniqueName: \"kubernetes.io/projected/44c74378-cef2-413a-a344-81d8093a89e8-kube-api-access-2q2jh\") pod \"maas-api-revoke-sub-be73052e-plwg2\" (UID: \"44c74378-cef2-413a-a344-81d8093a89e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be73052e-plwg2" Oct 30 20:59:44.750726 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:44.737277 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/44c74378-cef2-413a-a344-81d8093a89e8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-be73052e-plwg2\" (UID: \"44c74378-cef2-413a-a344-81d8093a89e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be73052e-plwg2" Oct 30 20:59:44.759197 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:44.759154 4611 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:59:44.759687 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:44.759642 4611 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7hf8r\"" Oct 30 20:59:44.760851 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:44.760829 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:59:44.838625 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:44.838576 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/44c74378-cef2-413a-a344-81d8093a89e8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-be73052e-plwg2\" (UID: \"44c74378-cef2-413a-a344-81d8093a89e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be73052e-plwg2" Oct 30 20:59:44.838791 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:44.838726 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/44c74378-cef2-413a-a344-81d8093a89e8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-be73052e-plwg2\" (UID: \"44c74378-cef2-413a-a344-81d8093a89e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be73052e-plwg2" Oct 30 20:59:44.838913 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:44.838895 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2q2jh\" (UniqueName: \"kubernetes.io/projected/44c74378-cef2-413a-a344-81d8093a89e8-kube-api-access-2q2jh\") pod \"maas-api-revoke-sub-be73052e-plwg2\" (UID: \"44c74378-cef2-413a-a344-81d8093a89e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be73052e-plwg2" Oct 30 20:59:44.839825 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:44.839796 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/44c74378-cef2-413a-a344-81d8093a89e8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-be73052e-plwg2\" (UID: \"44c74378-cef2-413a-a344-81d8093a89e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be73052e-plwg2" Oct 30 20:59:44.842135 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:44.842110 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/44c74378-cef2-413a-a344-81d8093a89e8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-be73052e-plwg2\" (UID: \"44c74378-cef2-413a-a344-81d8093a89e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be73052e-plwg2" Oct 30 20:59:44.850225 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:44.850196 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2q2jh\" (UniqueName: \"kubernetes.io/projected/44c74378-cef2-413a-a344-81d8093a89e8-kube-api-access-2q2jh\") pod \"maas-api-revoke-sub-be73052e-plwg2\" (UID: \"44c74378-cef2-413a-a344-81d8093a89e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be73052e-plwg2" Oct 30 20:59:44.968704 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:44.968666 4611 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be73052e-plwg2" Oct 30 20:59:45.048857 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:45.048808 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:59:45.260975 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:45.260944 4611 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-be73052e-plwg2"] Oct 30 20:59:45.271115 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:59:45.263076 4611 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44c74378_cef2_413a_a344_81d8093a89e8.slice/crio-579edc9fd5ba437f36de05308bbaa7dee264e9b928c6c8e5e819f185a2ec05ec WatchSource:0}: Error finding container 579edc9fd5ba437f36de05308bbaa7dee264e9b928c6c8e5e819f185a2ec05ec: Status 404 returned error can't find the container with id 579edc9fd5ba437f36de05308bbaa7dee264e9b928c6c8e5e819f185a2ec05ec Oct 30 20:59:45.690918 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:45.690886 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be73052e-plwg2" event={"ID":"44c74378-cef2-413a-a344-81d8093a89e8","Type":"ContainerStarted","Data":"78e0e5c6a863eb41854621bf79532a02aa17184d3e242af256f658a0b93aa5d1"} Oct 30 20:59:45.691338 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:45.690952 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be73052e-plwg2" event={"ID":"44c74378-cef2-413a-a344-81d8093a89e8","Type":"ContainerStarted","Data":"579edc9fd5ba437f36de05308bbaa7dee264e9b928c6c8e5e819f185a2ec05ec"} Oct 30 20:59:45.709969 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:45.709910 4611 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be73052e-plwg2" podStartSLOduration=1.709891616 podStartE2EDuration="1.709891616s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:59:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:59:45.709133065 +0000 UTC m=+1240.243159693" watchObservedRunningTime="2026-10-30 20:59:45.709891616 +0000 UTC m=+1240.243918246" Oct 30 20:59:45.716293 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:45.716252 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:59:46.191299 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:46.191261 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:59:46.421263 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:46.421228 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:59:46.621849 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:46.621814 4611 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-812bfe65-openshift-default-74647fb5f7-f85fj"] Oct 30 20:59:46.626195 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:46.626171 4611 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-812bfe65-openshift-default-74647fb5f7-f85fj" Oct 30 20:59:46.631474 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:46.631451 4611 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-812bfe65-openshift-default-dockercfg-fxppw\"" Oct 30 20:59:46.641798 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:46.641771 4611 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-812bfe65-openshift-default-74647fb5f7-f85fj"] Oct 30 20:59:46.698078 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:46.698008 4611 generic.go:356] "Generic (PLEG): container finished" podID="44c74378-cef2-413a-a344-81d8093a89e8" containerID="78e0e5c6a863eb41854621bf79532a02aa17184d3e242af256f658a0b93aa5d1" exitCode=0 Oct 30 20:59:46.698422 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:46.698084 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be73052e-plwg2" event={"ID":"44c74378-cef2-413a-a344-81d8093a89e8","Type":"ContainerDied","Data":"78e0e5c6a863eb41854621bf79532a02aa17184d3e242af256f658a0b93aa5d1"} Oct 30 20:59:46.745217 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:46.745177 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:59:46.789537 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:46.783179 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/918ff269-fb66-4479-9e8d-d0184421bc7c-istio-envoy\") pod \"e2e-mt-812bfe65-openshift-default-74647fb5f7-f85fj\" (UID: \"918ff269-fb66-4479-9e8d-d0184421bc7c\") " pod="openshift-ingress/e2e-mt-812bfe65-openshift-default-74647fb5f7-f85fj" Oct 30 20:59:46.789537 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:46.783328 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/918ff269-fb66-4479-9e8d-d0184421bc7c-istiod-ca-cert\") pod \"e2e-mt-812bfe65-openshift-default-74647fb5f7-f85fj\" (UID: \"918ff269-fb66-4479-9e8d-d0184421bc7c\") " pod="openshift-ingress/e2e-mt-812bfe65-openshift-default-74647fb5f7-f85fj" Oct 30 20:59:46.789537 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:46.783443 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/918ff269-fb66-4479-9e8d-d0184421bc7c-istio-podinfo\") pod \"e2e-mt-812bfe65-openshift-default-74647fb5f7-f85fj\" (UID: \"918ff269-fb66-4479-9e8d-d0184421bc7c\") " pod="openshift-ingress/e2e-mt-812bfe65-openshift-default-74647fb5f7-f85fj" Oct 30 20:59:46.789537 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:46.783476 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/918ff269-fb66-4479-9e8d-d0184421bc7c-credential-socket\") pod \"e2e-mt-812bfe65-openshift-default-74647fb5f7-f85fj\" (UID: \"918ff269-fb66-4479-9e8d-d0184421bc7c\") " pod="openshift-ingress/e2e-mt-812bfe65-openshift-default-74647fb5f7-f85fj" Oct 30 20:59:46.789537 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:46.783504 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/918ff269-fb66-4479-9e8d-d0184421bc7c-workload-certs\") pod \"e2e-mt-812bfe65-openshift-default-74647fb5f7-f85fj\" (UID: \"918ff269-fb66-4479-9e8d-d0184421bc7c\") " pod="openshift-ingress/e2e-mt-812bfe65-openshift-default-74647fb5f7-f85fj" Oct 30 20:59:46.789537 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:46.783625 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/918ff269-fb66-4479-9e8d-d0184421bc7c-workload-socket\") pod \"e2e-mt-812bfe65-openshift-default-74647fb5f7-f85fj\" (UID: \"918ff269-fb66-4479-9e8d-d0184421bc7c\") " pod="openshift-ingress/e2e-mt-812bfe65-openshift-default-74647fb5f7-f85fj" Oct 30 20:59:46.789537 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:46.783696 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/918ff269-fb66-4479-9e8d-d0184421bc7c-istio-data\") pod \"e2e-mt-812bfe65-openshift-default-74647fb5f7-f85fj\" (UID: \"918ff269-fb66-4479-9e8d-d0184421bc7c\") " pod="openshift-ingress/e2e-mt-812bfe65-openshift-default-74647fb5f7-f85fj" Oct 30 20:59:46.789537 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:46.783763 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/918ff269-fb66-4479-9e8d-d0184421bc7c-istio-token\") pod \"e2e-mt-812bfe65-openshift-default-74647fb5f7-f85fj\" (UID: \"918ff269-fb66-4479-9e8d-d0184421bc7c\") " pod="openshift-ingress/e2e-mt-812bfe65-openshift-default-74647fb5f7-f85fj" Oct 30 20:59:46.789537 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:46.783802 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hkblh\" (UniqueName: \"kubernetes.io/projected/918ff269-fb66-4479-9e8d-d0184421bc7c-kube-api-access-hkblh\") pod \"e2e-mt-812bfe65-openshift-default-74647fb5f7-f85fj\" (UID: \"918ff269-fb66-4479-9e8d-d0184421bc7c\") " pod="openshift-ingress/e2e-mt-812bfe65-openshift-default-74647fb5f7-f85fj" Oct 30 20:59:46.884952 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:46.884921 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/918ff269-fb66-4479-9e8d-d0184421bc7c-istio-podinfo\") pod \"e2e-mt-812bfe65-openshift-default-74647fb5f7-f85fj\" (UID: \"918ff269-fb66-4479-9e8d-d0184421bc7c\") " pod="openshift-ingress/e2e-mt-812bfe65-openshift-default-74647fb5f7-f85fj" Oct 30 20:59:46.885129 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:46.884958 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/918ff269-fb66-4479-9e8d-d0184421bc7c-credential-socket\") pod \"e2e-mt-812bfe65-openshift-default-74647fb5f7-f85fj\" (UID: \"918ff269-fb66-4479-9e8d-d0184421bc7c\") " pod="openshift-ingress/e2e-mt-812bfe65-openshift-default-74647fb5f7-f85fj" Oct 30 20:59:46.885129 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:46.884978 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/918ff269-fb66-4479-9e8d-d0184421bc7c-workload-certs\") pod \"e2e-mt-812bfe65-openshift-default-74647fb5f7-f85fj\" (UID: \"918ff269-fb66-4479-9e8d-d0184421bc7c\") " pod="openshift-ingress/e2e-mt-812bfe65-openshift-default-74647fb5f7-f85fj" Oct 30 20:59:46.885129 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:46.885038 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/918ff269-fb66-4479-9e8d-d0184421bc7c-workload-socket\") pod \"e2e-mt-812bfe65-openshift-default-74647fb5f7-f85fj\" (UID: \"918ff269-fb66-4479-9e8d-d0184421bc7c\") " pod="openshift-ingress/e2e-mt-812bfe65-openshift-default-74647fb5f7-f85fj" Oct 30 20:59:46.885129 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:46.885068 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/918ff269-fb66-4479-9e8d-d0184421bc7c-istio-data\") pod \"e2e-mt-812bfe65-openshift-default-74647fb5f7-f85fj\" (UID: \"918ff269-fb66-4479-9e8d-d0184421bc7c\") " pod="openshift-ingress/e2e-mt-812bfe65-openshift-default-74647fb5f7-f85fj" Oct 30 20:59:46.885129 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:46.885112 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/918ff269-fb66-4479-9e8d-d0184421bc7c-istio-token\") pod \"e2e-mt-812bfe65-openshift-default-74647fb5f7-f85fj\" (UID: \"918ff269-fb66-4479-9e8d-d0184421bc7c\") " pod="openshift-ingress/e2e-mt-812bfe65-openshift-default-74647fb5f7-f85fj" Oct 30 20:59:46.885392 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:46.885151 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hkblh\" (UniqueName: \"kubernetes.io/projected/918ff269-fb66-4479-9e8d-d0184421bc7c-kube-api-access-hkblh\") pod \"e2e-mt-812bfe65-openshift-default-74647fb5f7-f85fj\" (UID: \"918ff269-fb66-4479-9e8d-d0184421bc7c\") " pod="openshift-ingress/e2e-mt-812bfe65-openshift-default-74647fb5f7-f85fj" Oct 30 20:59:46.885392 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:46.885280 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/918ff269-fb66-4479-9e8d-d0184421bc7c-istio-envoy\") pod \"e2e-mt-812bfe65-openshift-default-74647fb5f7-f85fj\" (UID: \"918ff269-fb66-4479-9e8d-d0184421bc7c\") " pod="openshift-ingress/e2e-mt-812bfe65-openshift-default-74647fb5f7-f85fj" Oct 30 20:59:46.885392 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:46.885353 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/918ff269-fb66-4479-9e8d-d0184421bc7c-istiod-ca-cert\") pod \"e2e-mt-812bfe65-openshift-default-74647fb5f7-f85fj\" (UID: \"918ff269-fb66-4479-9e8d-d0184421bc7c\") " pod="openshift-ingress/e2e-mt-812bfe65-openshift-default-74647fb5f7-f85fj" Oct 30 20:59:46.885598 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:46.885404 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/918ff269-fb66-4479-9e8d-d0184421bc7c-credential-socket\") pod \"e2e-mt-812bfe65-openshift-default-74647fb5f7-f85fj\" (UID: \"918ff269-fb66-4479-9e8d-d0184421bc7c\") " pod="openshift-ingress/e2e-mt-812bfe65-openshift-default-74647fb5f7-f85fj" Oct 30 20:59:46.885598 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:46.885449 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/918ff269-fb66-4479-9e8d-d0184421bc7c-workload-certs\") pod \"e2e-mt-812bfe65-openshift-default-74647fb5f7-f85fj\" (UID: \"918ff269-fb66-4479-9e8d-d0184421bc7c\") " pod="openshift-ingress/e2e-mt-812bfe65-openshift-default-74647fb5f7-f85fj" Oct 30 20:59:46.885598 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:46.885529 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/918ff269-fb66-4479-9e8d-d0184421bc7c-workload-socket\") pod \"e2e-mt-812bfe65-openshift-default-74647fb5f7-f85fj\" (UID: \"918ff269-fb66-4479-9e8d-d0184421bc7c\") " pod="openshift-ingress/e2e-mt-812bfe65-openshift-default-74647fb5f7-f85fj" Oct 30 20:59:46.885598 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:46.885586 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/918ff269-fb66-4479-9e8d-d0184421bc7c-istio-data\") pod \"e2e-mt-812bfe65-openshift-default-74647fb5f7-f85fj\" (UID: \"918ff269-fb66-4479-9e8d-d0184421bc7c\") " pod="openshift-ingress/e2e-mt-812bfe65-openshift-default-74647fb5f7-f85fj" Oct 30 20:59:46.886022 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:46.886001 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/918ff269-fb66-4479-9e8d-d0184421bc7c-istiod-ca-cert\") pod \"e2e-mt-812bfe65-openshift-default-74647fb5f7-f85fj\" (UID: \"918ff269-fb66-4479-9e8d-d0184421bc7c\") " pod="openshift-ingress/e2e-mt-812bfe65-openshift-default-74647fb5f7-f85fj" Oct 30 20:59:46.887639 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:46.887620 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/918ff269-fb66-4479-9e8d-d0184421bc7c-istio-envoy\") pod \"e2e-mt-812bfe65-openshift-default-74647fb5f7-f85fj\" (UID: \"918ff269-fb66-4479-9e8d-d0184421bc7c\") " pod="openshift-ingress/e2e-mt-812bfe65-openshift-default-74647fb5f7-f85fj" Oct 30 20:59:46.887765 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:46.887747 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/918ff269-fb66-4479-9e8d-d0184421bc7c-istio-podinfo\") pod \"e2e-mt-812bfe65-openshift-default-74647fb5f7-f85fj\" (UID: \"918ff269-fb66-4479-9e8d-d0184421bc7c\") " pod="openshift-ingress/e2e-mt-812bfe65-openshift-default-74647fb5f7-f85fj" Oct 30 20:59:46.894550 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:46.894209 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:59:46.896844 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:46.896822 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/918ff269-fb66-4479-9e8d-d0184421bc7c-istio-token\") pod \"e2e-mt-812bfe65-openshift-default-74647fb5f7-f85fj\" (UID: \"918ff269-fb66-4479-9e8d-d0184421bc7c\") " pod="openshift-ingress/e2e-mt-812bfe65-openshift-default-74647fb5f7-f85fj" Oct 30 20:59:46.897699 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:46.897676 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hkblh\" (UniqueName: \"kubernetes.io/projected/918ff269-fb66-4479-9e8d-d0184421bc7c-kube-api-access-hkblh\") pod \"e2e-mt-812bfe65-openshift-default-74647fb5f7-f85fj\" (UID: \"918ff269-fb66-4479-9e8d-d0184421bc7c\") " pod="openshift-ingress/e2e-mt-812bfe65-openshift-default-74647fb5f7-f85fj" Oct 30 20:59:46.943322 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:46.943288 4611 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-812bfe65-openshift-default-74647fb5f7-f85fj" Oct 30 20:59:47.036141 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:47.036104 4611 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7e8fa042-8xjrj"] Oct 30 20:59:47.039523 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:47.039483 4611 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7e8fa042-8xjrj"] Oct 30 20:59:47.238308 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:47.238281 4611 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-812bfe65-openshift-default-74647fb5f7-f85fj"] Oct 30 20:59:47.239318 ip-10-0-133-192 kubenswrapper[4611]: W1030 20:59:47.239292 4611 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod918ff269_fb66_4479_9e8d_d0184421bc7c.slice/crio-951a442d5d9efae0eba8c3718c65806b6d63e25eb78670878ad9db1661c76aa4 WatchSource:0}: Error finding container 951a442d5d9efae0eba8c3718c65806b6d63e25eb78670878ad9db1661c76aa4: Status 404 returned error can't find the container with id 951a442d5d9efae0eba8c3718c65806b6d63e25eb78670878ad9db1661c76aa4 Oct 30 20:59:47.241972 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:47.241934 4611 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:59:47.242067 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:47.242016 4611 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:59:47.242067 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:47.242057 4611 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:59:47.298976 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:47.298172 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:59:47.607358 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:47.607321 4611 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="23c4eefd-503e-4988-8777-cd4f6f91247b" path="/var/lib/kubelet/pods/23c4eefd-503e-4988-8777-cd4f6f91247b/volumes" Oct 30 20:59:47.706401 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:47.706356 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-812bfe65-openshift-default-74647fb5f7-f85fj" event={"ID":"918ff269-fb66-4479-9e8d-d0184421bc7c","Type":"ContainerStarted","Data":"374d672a1b8de629149cd6f13a4700bc2904de58b732cc0733a2facef6ba11c4"} Oct 30 20:59:47.706864 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:47.706423 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-812bfe65-openshift-default-74647fb5f7-f85fj" event={"ID":"918ff269-fb66-4479-9e8d-d0184421bc7c","Type":"ContainerStarted","Data":"951a442d5d9efae0eba8c3718c65806b6d63e25eb78670878ad9db1661c76aa4"} Oct 30 20:59:47.731002 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:47.730944 4611 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-812bfe65-openshift-default-74647fb5f7-f85fj" podStartSLOduration=1.730925805 podStartE2EDuration="1.730925805s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:59:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:59:47.726791944 +0000 UTC m=+1242.260818573" watchObservedRunningTime="2026-10-30 20:59:47.730925805 +0000 UTC m=+1242.264952422" Oct 30 20:59:47.884055 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:47.884026 4611 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be73052e-plwg2" Oct 30 20:59:47.944243 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:47.944214 4611 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-812bfe65-openshift-default-74647fb5f7-f85fj" Oct 30 20:59:47.991817 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:47.991779 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:59:48.000365 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:48.000339 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/44c74378-cef2-413a-a344-81d8093a89e8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/44c74378-cef2-413a-a344-81d8093a89e8-maas-api-openshift-ca\") pod \"44c74378-cef2-413a-a344-81d8093a89e8\" (UID: \"44c74378-cef2-413a-a344-81d8093a89e8\") " Oct 30 20:59:48.000526 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:48.000378 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44c74378-cef2-413a-a344-81d8093a89e8-kube-api-access-2q2jh\" (UniqueName: \"kubernetes.io/projected/44c74378-cef2-413a-a344-81d8093a89e8-kube-api-access-2q2jh\") pod \"44c74378-cef2-413a-a344-81d8093a89e8\" (UID: \"44c74378-cef2-413a-a344-81d8093a89e8\") " Oct 30 20:59:48.000526 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:48.000425 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/44c74378-cef2-413a-a344-81d8093a89e8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/44c74378-cef2-413a-a344-81d8093a89e8-maas-api-xks-ca\") pod \"44c74378-cef2-413a-a344-81d8093a89e8\" (UID: \"44c74378-cef2-413a-a344-81d8093a89e8\") " Oct 30 20:59:48.000774 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:48.000749 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44c74378-cef2-413a-a344-81d8093a89e8-maas-api-openshift-ca" pod "44c74378-cef2-413a-a344-81d8093a89e8" (UID: "44c74378-cef2-413a-a344-81d8093a89e8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:59:48.000995 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:48.000979 4611 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/44c74378-cef2-413a-a344-81d8093a89e8-maas-api-openshift-ca\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:59:48.002552 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:48.002528 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/44c74378-cef2-413a-a344-81d8093a89e8-maas-api-xks-ca" pod "44c74378-cef2-413a-a344-81d8093a89e8" (UID: "44c74378-cef2-413a-a344-81d8093a89e8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:59:48.002642 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:48.002604 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44c74378-cef2-413a-a344-81d8093a89e8-kube-api-access-2q2jh" pod "44c74378-cef2-413a-a344-81d8093a89e8" (UID: "44c74378-cef2-413a-a344-81d8093a89e8"). InnerVolumeSpecName "kube-api-access-2q2jh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:59:48.102259 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:48.102225 4611 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2q2jh\" (UniqueName: \"kubernetes.io/projected/44c74378-cef2-413a-a344-81d8093a89e8-kube-api-access-2q2jh\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:59:48.102259 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:48.102257 4611 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/44c74378-cef2-413a-a344-81d8093a89e8-maas-api-xks-ca\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:59:48.688198 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:48.688157 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:59:48.718875 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:48.718849 4611 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be73052e-plwg2" Oct 30 20:59:48.719384 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:48.718886 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be73052e-plwg2" event={"ID":"44c74378-cef2-413a-a344-81d8093a89e8","Type":"ContainerDied","Data":"579edc9fd5ba437f36de05308bbaa7dee264e9b928c6c8e5e819f185a2ec05ec"} Oct 30 20:59:48.719384 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:48.718934 4611 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="579edc9fd5ba437f36de05308bbaa7dee264e9b928c6c8e5e819f185a2ec05ec" Oct 30 20:59:48.843185 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:48.843153 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:59:48.950251 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:48.950171 4611 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-812bfe65-openshift-default-74647fb5f7-f85fj" Oct 30 20:59:49.457317 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:49.456126 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:59:49.468013 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:49.467967 4611 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w6-3d909d-74bfb44f9b-5f2x8" podUID="46885e54-e3da-4b91-aa7b-433f9b55a0da" containerName="payload-processing" containerID="cri-o://ca32235c39e2d40fc731f627b2cb0f906b6da84bf1a9bf3ecdc1a75dbe43d3d6" gracePeriod=25 Oct 30 20:59:49.489634 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:49.484465 4611 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-3d909d-57b9b6dcf9-l9pwk" podUID="5d963de5-2836-4225-9120-d885791b4f71" containerName="payload-pre-processing" containerID="cri-o://fe58a91cc76f1810321347afc1a4d4adcbdab7c190d7fef96fd2e34ae9f677c1" gracePeriod=25 Oct 30 20:59:49.727356 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:49.727275 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-812bfe65-openshift-default-74647fb5f7-f85fj" Oct 30 20:59:49.728492 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:49.728469 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-812bfe65-openshift-default-74647fb5f7-f85fj" Oct 30 20:59:50.434904 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:50.433594 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:59:50.577401 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:50.577360 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:59:51.210316 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:51.210255 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:59:51.709264 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:51.709227 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:59:51.984272 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:51.984129 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:59:52.133327 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:52.133289 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:59:52.139440 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:52.139403 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-3d909d-57b9b6dcf9-l9pwk" podUID="5d963de5-2836-4225-9120-d885791b4f71" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 20:59:52.139621 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:52.139461 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w6-3d909d-74bfb44f9b-5f2x8" podUID="46885e54-e3da-4b91-aa7b-433f9b55a0da" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 20:59:52.328801 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:52.328707 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:59:53.057760 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:53.057725 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:59:53.848592 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:53.848549 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:59:54.158986 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:54.158950 4611 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-3d909d/distinct-2-w6-3d909d-kserve-59c5d5c595-cjg2m" podUID="e2b4c221-ee94-4131-abd8-2313647c641e" containerName="main" containerID="cri-o://3028a7d3b7a568f38784c3208775409d12f97dba2e31bd2a99bfde0275f82168" gracePeriod=45 Oct 30 20:59:54.177553 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:54.177520 4611 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-3d909d/premium-w6-3d909d-kserve-765648946f-2qvd2" podUID="fca5d697-109d-4e90-ab41-c40ad31af874" containerName="main" containerID="cri-o://65bafa3a9f701bd8a0b3de653b025974cd022196ab21c4096ab2c3222e53b795" gracePeriod=45 Oct 30 20:59:54.435664 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:54.435631 4611 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-3d909d/premium-w6-3d909d-kserve-765648946f-2qvd2" Oct 30 20:59:54.464243 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:54.464215 4611 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-3d909d/distinct-2-w6-3d909d-kserve-59c5d5c595-cjg2m" Oct 30 20:59:54.492889 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:54.492861 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2b4c221-ee94-4131-abd8-2313647c641e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e2b4c221-ee94-4131-abd8-2313647c641e-tmp-dir\") pod \"e2b4c221-ee94-4131-abd8-2313647c641e\" (UID: \"e2b4c221-ee94-4131-abd8-2313647c641e\") " Oct 30 20:59:54.493049 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:54.492933 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2b4c221-ee94-4131-abd8-2313647c641e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e2b4c221-ee94-4131-abd8-2313647c641e-model-cache\") pod \"e2b4c221-ee94-4131-abd8-2313647c641e\" (UID: \"e2b4c221-ee94-4131-abd8-2313647c641e\") " Oct 30 20:59:54.493049 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:54.492986 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2b4c221-ee94-4131-abd8-2313647c641e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e2b4c221-ee94-4131-abd8-2313647c641e-dshm\") pod \"e2b4c221-ee94-4131-abd8-2313647c641e\" (UID: \"e2b4c221-ee94-4131-abd8-2313647c641e\") " Oct 30 20:59:54.493049 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:54.493013 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e2b4c221-ee94-4131-abd8-2313647c641e-tls-certs\" (UniqueName: \"kubernetes.io/secret/e2b4c221-ee94-4131-abd8-2313647c641e-tls-certs\") pod \"e2b4c221-ee94-4131-abd8-2313647c641e\" (UID: \"e2b4c221-ee94-4131-abd8-2313647c641e\") " Oct 30 20:59:54.493232 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:54.493120 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fca5d697-109d-4e90-ab41-c40ad31af874-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fca5d697-109d-4e90-ab41-c40ad31af874-tmp-dir\") pod \"fca5d697-109d-4e90-ab41-c40ad31af874\" (UID: \"fca5d697-109d-4e90-ab41-c40ad31af874\") " Oct 30 20:59:54.493232 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:54.493165 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fca5d697-109d-4e90-ab41-c40ad31af874-home\" (UniqueName: \"kubernetes.io/empty-dir/fca5d697-109d-4e90-ab41-c40ad31af874-home\") pod \"fca5d697-109d-4e90-ab41-c40ad31af874\" (UID: \"fca5d697-109d-4e90-ab41-c40ad31af874\") " Oct 30 20:59:54.493232 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:54.493216 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fca5d697-109d-4e90-ab41-c40ad31af874-tls-certs\" (UniqueName: \"kubernetes.io/secret/fca5d697-109d-4e90-ab41-c40ad31af874-tls-certs\") pod \"fca5d697-109d-4e90-ab41-c40ad31af874\" (UID: \"fca5d697-109d-4e90-ab41-c40ad31af874\") " Oct 30 20:59:54.493394 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:54.493258 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fca5d697-109d-4e90-ab41-c40ad31af874-kube-api-access-t4984\" (UniqueName: \"kubernetes.io/projected/fca5d697-109d-4e90-ab41-c40ad31af874-kube-api-access-t4984\") pod \"fca5d697-109d-4e90-ab41-c40ad31af874\" (UID: \"fca5d697-109d-4e90-ab41-c40ad31af874\") " Oct 30 20:59:54.493394 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:54.493323 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fca5d697-109d-4e90-ab41-c40ad31af874-dshm\" (UniqueName: \"kubernetes.io/empty-dir/fca5d697-109d-4e90-ab41-c40ad31af874-dshm\") pod \"fca5d697-109d-4e90-ab41-c40ad31af874\" (UID: \"fca5d697-109d-4e90-ab41-c40ad31af874\") " Oct 30 20:59:54.493394 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:54.493354 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2b4c221-ee94-4131-abd8-2313647c641e-home\" (UniqueName: \"kubernetes.io/empty-dir/e2b4c221-ee94-4131-abd8-2313647c641e-home\") pod \"e2b4c221-ee94-4131-abd8-2313647c641e\" (UID: \"e2b4c221-ee94-4131-abd8-2313647c641e\") " Oct 30 20:59:54.493567 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:54.493391 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e2b4c221-ee94-4131-abd8-2313647c641e-kube-api-access-gn9vx\" (UniqueName: \"kubernetes.io/projected/e2b4c221-ee94-4131-abd8-2313647c641e-kube-api-access-gn9vx\") pod \"e2b4c221-ee94-4131-abd8-2313647c641e\" (UID: \"e2b4c221-ee94-4131-abd8-2313647c641e\") " Oct 30 20:59:54.493567 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:54.493503 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fca5d697-109d-4e90-ab41-c40ad31af874-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fca5d697-109d-4e90-ab41-c40ad31af874-model-cache\") pod \"fca5d697-109d-4e90-ab41-c40ad31af874\" (UID: \"fca5d697-109d-4e90-ab41-c40ad31af874\") " Oct 30 20:59:54.494409 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:54.494379 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fca5d697-109d-4e90-ab41-c40ad31af874-model-cache" pod "fca5d697-109d-4e90-ab41-c40ad31af874" (UID: "fca5d697-109d-4e90-ab41-c40ad31af874"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:59:54.494704 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:54.494679 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2b4c221-ee94-4131-abd8-2313647c641e-tmp-dir" pod "e2b4c221-ee94-4131-abd8-2313647c641e" (UID: "e2b4c221-ee94-4131-abd8-2313647c641e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:59:54.494895 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:54.494875 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2b4c221-ee94-4131-abd8-2313647c641e-model-cache" pod "e2b4c221-ee94-4131-abd8-2313647c641e" (UID: "e2b4c221-ee94-4131-abd8-2313647c641e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:59:54.497906 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:54.496400 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fca5d697-109d-4e90-ab41-c40ad31af874-tmp-dir" pod "fca5d697-109d-4e90-ab41-c40ad31af874" (UID: "fca5d697-109d-4e90-ab41-c40ad31af874"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:59:54.497906 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:54.497055 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fca5d697-109d-4e90-ab41-c40ad31af874-home" pod "fca5d697-109d-4e90-ab41-c40ad31af874" (UID: "fca5d697-109d-4e90-ab41-c40ad31af874"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:59:54.498468 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:54.498433 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2b4c221-ee94-4131-abd8-2313647c641e-home" pod "e2b4c221-ee94-4131-abd8-2313647c641e" (UID: "e2b4c221-ee94-4131-abd8-2313647c641e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:59:54.503042 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:54.500158 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:59:54.503042 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:54.502896 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2b4c221-ee94-4131-abd8-2313647c641e-kube-api-access-gn9vx" pod "e2b4c221-ee94-4131-abd8-2313647c641e" (UID: "e2b4c221-ee94-4131-abd8-2313647c641e"). InnerVolumeSpecName "kube-api-access-gn9vx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:59:54.504872 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:54.504844 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fca5d697-109d-4e90-ab41-c40ad31af874-tls-certs" pod "fca5d697-109d-4e90-ab41-c40ad31af874" (UID: "fca5d697-109d-4e90-ab41-c40ad31af874"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:59:54.504983 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:54.504867 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e2b4c221-ee94-4131-abd8-2313647c641e-tls-certs" pod "e2b4c221-ee94-4131-abd8-2313647c641e" (UID: "e2b4c221-ee94-4131-abd8-2313647c641e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:59:54.504983 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:54.504848 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2b4c221-ee94-4131-abd8-2313647c641e-dshm" pod "e2b4c221-ee94-4131-abd8-2313647c641e" (UID: "e2b4c221-ee94-4131-abd8-2313647c641e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:59:54.504983 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:54.504852 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fca5d697-109d-4e90-ab41-c40ad31af874-dshm" pod "fca5d697-109d-4e90-ab41-c40ad31af874" (UID: "fca5d697-109d-4e90-ab41-c40ad31af874"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:59:54.506299 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:54.506273 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fca5d697-109d-4e90-ab41-c40ad31af874-kube-api-access-t4984" pod "fca5d697-109d-4e90-ab41-c40ad31af874" (UID: "fca5d697-109d-4e90-ab41-c40ad31af874"). InnerVolumeSpecName "kube-api-access-t4984". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:59:54.595199 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:54.595158 4611 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e2b4c221-ee94-4131-abd8-2313647c641e-dshm\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:59:54.595199 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:54.595189 4611 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e2b4c221-ee94-4131-abd8-2313647c641e-tls-certs\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:59:54.595199 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:54.595198 4611 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fca5d697-109d-4e90-ab41-c40ad31af874-tmp-dir\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:59:54.595199 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:54.595207 4611 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fca5d697-109d-4e90-ab41-c40ad31af874-home\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:59:54.595571 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:54.595215 4611 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fca5d697-109d-4e90-ab41-c40ad31af874-tls-certs\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:59:54.595571 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:54.595224 4611 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t4984\" (UniqueName: \"kubernetes.io/projected/fca5d697-109d-4e90-ab41-c40ad31af874-kube-api-access-t4984\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:59:54.595571 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:54.595233 4611 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fca5d697-109d-4e90-ab41-c40ad31af874-dshm\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:59:54.595571 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:54.595241 4611 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e2b4c221-ee94-4131-abd8-2313647c641e-home\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:59:54.595571 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:54.595249 4611 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gn9vx\" (UniqueName: \"kubernetes.io/projected/e2b4c221-ee94-4131-abd8-2313647c641e-kube-api-access-gn9vx\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:59:54.595571 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:54.595257 4611 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fca5d697-109d-4e90-ab41-c40ad31af874-model-cache\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:59:54.595571 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:54.595266 4611 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e2b4c221-ee94-4131-abd8-2313647c641e-tmp-dir\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:59:54.595571 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:54.595274 4611 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e2b4c221-ee94-4131-abd8-2313647c641e-model-cache\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:59:54.764084 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:54.763995 4611 generic.go:356] "Generic (PLEG): container finished" podID="fca5d697-109d-4e90-ab41-c40ad31af874" containerID="65bafa3a9f701bd8a0b3de653b025974cd022196ab21c4096ab2c3222e53b795" exitCode=0 Oct 30 20:59:54.764084 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:54.764044 4611 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-3d909d/premium-w6-3d909d-kserve-765648946f-2qvd2" Oct 30 20:59:54.764282 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:54.764078 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-3d909d/premium-w6-3d909d-kserve-765648946f-2qvd2" event={"ID":"fca5d697-109d-4e90-ab41-c40ad31af874","Type":"ContainerDied","Data":"65bafa3a9f701bd8a0b3de653b025974cd022196ab21c4096ab2c3222e53b795"} Oct 30 20:59:54.764282 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:54.764131 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-3d909d/premium-w6-3d909d-kserve-765648946f-2qvd2" event={"ID":"fca5d697-109d-4e90-ab41-c40ad31af874","Type":"ContainerDied","Data":"e886a59e35f43dc34aa7fa2211f0bf8bb4818372a50a502b72a2d7be0f68a1df"} Oct 30 20:59:54.764282 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:54.764150 4611 scope.go:122] "RemoveContainer" containerID="65bafa3a9f701bd8a0b3de653b025974cd022196ab21c4096ab2c3222e53b795" Oct 30 20:59:54.765617 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:54.765591 4611 generic.go:356] "Generic (PLEG): container finished" podID="e2b4c221-ee94-4131-abd8-2313647c641e" containerID="3028a7d3b7a568f38784c3208775409d12f97dba2e31bd2a99bfde0275f82168" exitCode=0 Oct 30 20:59:54.765746 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:54.765622 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-3d909d/distinct-2-w6-3d909d-kserve-59c5d5c595-cjg2m" event={"ID":"e2b4c221-ee94-4131-abd8-2313647c641e","Type":"ContainerDied","Data":"3028a7d3b7a568f38784c3208775409d12f97dba2e31bd2a99bfde0275f82168"} Oct 30 20:59:54.765746 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:54.765625 4611 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-3d909d/distinct-2-w6-3d909d-kserve-59c5d5c595-cjg2m" Oct 30 20:59:54.765746 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:54.765644 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-3d909d/distinct-2-w6-3d909d-kserve-59c5d5c595-cjg2m" event={"ID":"e2b4c221-ee94-4131-abd8-2313647c641e","Type":"ContainerDied","Data":"fb3e9e314a5c4e05c7182b2f08f0759eda66de483ad0291abda45713839f68e8"} Oct 30 20:59:54.775167 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:54.775149 4611 scope.go:122] "RemoveContainer" containerID="65bafa3a9f701bd8a0b3de653b025974cd022196ab21c4096ab2c3222e53b795" Oct 30 20:59:54.775483 ip-10-0-133-192 kubenswrapper[4611]: E1030 20:59:54.775459 4611 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"65bafa3a9f701bd8a0b3de653b025974cd022196ab21c4096ab2c3222e53b795\": container with ID starting with 65bafa3a9f701bd8a0b3de653b025974cd022196ab21c4096ab2c3222e53b795 not found: ID does not exist" containerID="65bafa3a9f701bd8a0b3de653b025974cd022196ab21c4096ab2c3222e53b795" Oct 30 20:59:54.775614 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:54.775493 4611 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"65bafa3a9f701bd8a0b3de653b025974cd022196ab21c4096ab2c3222e53b795"} err="failed to get container status \"65bafa3a9f701bd8a0b3de653b025974cd022196ab21c4096ab2c3222e53b795\": rpc error: code = NotFound desc = could not find container \"65bafa3a9f701bd8a0b3de653b025974cd022196ab21c4096ab2c3222e53b795\": container with ID starting with 65bafa3a9f701bd8a0b3de653b025974cd022196ab21c4096ab2c3222e53b795 not found: ID does not exist" Oct 30 20:59:54.775614 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:54.775530 4611 scope.go:122] "RemoveContainer" containerID="3028a7d3b7a568f38784c3208775409d12f97dba2e31bd2a99bfde0275f82168" Oct 30 20:59:54.785518 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:54.785491 4611 scope.go:122] "RemoveContainer" containerID="3028a7d3b7a568f38784c3208775409d12f97dba2e31bd2a99bfde0275f82168" Oct 30 20:59:54.785768 ip-10-0-133-192 kubenswrapper[4611]: E1030 20:59:54.785750 4611 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3028a7d3b7a568f38784c3208775409d12f97dba2e31bd2a99bfde0275f82168\": container with ID starting with 3028a7d3b7a568f38784c3208775409d12f97dba2e31bd2a99bfde0275f82168 not found: ID does not exist" containerID="3028a7d3b7a568f38784c3208775409d12f97dba2e31bd2a99bfde0275f82168" Oct 30 20:59:54.785834 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:54.785774 4611 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3028a7d3b7a568f38784c3208775409d12f97dba2e31bd2a99bfde0275f82168"} err="failed to get container status \"3028a7d3b7a568f38784c3208775409d12f97dba2e31bd2a99bfde0275f82168\": rpc error: code = NotFound desc = could not find container \"3028a7d3b7a568f38784c3208775409d12f97dba2e31bd2a99bfde0275f82168\": container with ID starting with 3028a7d3b7a568f38784c3208775409d12f97dba2e31bd2a99bfde0275f82168 not found: ID does not exist" Oct 30 20:59:54.791123 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:54.791088 4611 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-3d909d/premium-w6-3d909d-kserve-765648946f-2qvd2"] Oct 30 20:59:54.794685 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:54.794665 4611 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-3d909d/premium-w6-3d909d-kserve-765648946f-2qvd2"] Oct 30 20:59:54.806697 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:54.806675 4611 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-3d909d/distinct-2-w6-3d909d-kserve-59c5d5c595-cjg2m"] Oct 30 20:59:54.810819 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:54.810799 4611 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-3d909d/distinct-2-w6-3d909d-kserve-59c5d5c595-cjg2m"] Oct 30 20:59:55.206785 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:55.206749 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:59:55.607092 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:55.607062 4611 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e2b4c221-ee94-4131-abd8-2313647c641e" path="/var/lib/kubelet/pods/e2b4c221-ee94-4131-abd8-2313647c641e/volumes" Oct 30 20:59:55.607480 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:55.607466 4611 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fca5d697-109d-4e90-ab41-c40ad31af874" path="/var/lib/kubelet/pods/fca5d697-109d-4e90-ab41-c40ad31af874/volumes" Oct 30 20:59:55.807574 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:55.802654 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:59:56.592948 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:56.592901 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:59:57.194246 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:57.194210 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:59:57.779298 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:57.779257 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:59:58.383482 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:58.382986 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:59:58.554794 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:58.554746 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:59:58.709501 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:58.709409 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:59:59.092709 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:59.092617 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 20:59:59.506198 ip-10-0-133-192 kubenswrapper[4611]: I1030 20:59:59.506131 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:00:00.153245 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:00.153202 4611 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-812bfe65-29889900-j5fzs"] Oct 30 21:00:00.153860 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:00.153748 4611 cpu_manager.go:414] "containerMap: removing container" podUID="e2b4c221-ee94-4131-abd8-2313647c641e" containerName="main" Oct 30 21:00:00.153860 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:00.153762 4611 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e2b4c221-ee94-4131-abd8-2313647c641e" containerName="main" Oct 30 21:00:00.153860 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:00.153786 4611 cpu_manager.go:414] "containerMap: removing container" podUID="fca5d697-109d-4e90-ab41-c40ad31af874" containerName="main" Oct 30 21:00:00.153860 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:00.153792 4611 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fca5d697-109d-4e90-ab41-c40ad31af874" containerName="main" Oct 30 21:00:00.153860 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:00.153808 4611 cpu_manager.go:414] "containerMap: removing container" podUID="44c74378-cef2-413a-a344-81d8093a89e8" containerName="revoke-keys" Oct 30 21:00:00.153860 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:00.153813 4611 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="44c74378-cef2-413a-a344-81d8093a89e8" containerName="revoke-keys" Oct 30 21:00:00.154241 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:00.153886 4611 memory_manager.go:365] "RemoveStaleState removing state" podUID="fca5d697-109d-4e90-ab41-c40ad31af874" containerName="main" Oct 30 21:00:00.154241 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:00.153896 4611 memory_manager.go:365] "RemoveStaleState removing state" podUID="44c74378-cef2-413a-a344-81d8093a89e8" containerName="revoke-keys" Oct 30 21:00:00.154241 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:00.153907 4611 memory_manager.go:365] "RemoveStaleState removing state" podUID="e2b4c221-ee94-4131-abd8-2313647c641e" containerName="main" Oct 30 21:00:00.160216 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:00.160191 4611 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-f2ed1e-29889900-sctrt"] Oct 30 21:00:00.160558 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:00.160454 4611 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-812bfe65-29889900-j5fzs" Oct 30 21:00:00.182541 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:00.176294 4611 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-812bfe65-29889900-j5fzs"] Oct 30 21:00:00.182541 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:00.176428 4611 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-f2ed1e-29889900-sctrt" Oct 30 21:00:00.196438 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:00.196405 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:00:00.220184 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:00.220136 4611 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-f2ed1e-29889900-sctrt"] Oct 30 21:00:00.264023 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:00.263974 4611 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7hf8r\"" Oct 30 21:00:00.298554 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:00.297889 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5xfhw\" (UniqueName: \"kubernetes.io/projected/9ff515ee-d559-4fe8-8487-5f04e13125bc-kube-api-access-5xfhw\") pod \"maas-api-key-cleanup-e2e-worker-w4-f2ed1e-29889900-sctrt\" (UID: \"9ff515ee-d559-4fe8-8487-5f04e13125bc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-f2ed1e-29889900-sctrt" Oct 30 21:00:00.298554 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:00.298058 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-62fpq\" (UniqueName: \"kubernetes.io/projected/1dd352a3-b3a3-4912-9d9f-723d40415008-kube-api-access-62fpq\") pod \"maas-api-key-cleanup-e2e-mt-812bfe65-29889900-j5fzs\" (UID: \"1dd352a3-b3a3-4912-9d9f-723d40415008\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-812bfe65-29889900-j5fzs" Oct 30 21:00:00.298554 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:00.298088 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1dd352a3-b3a3-4912-9d9f-723d40415008-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-mt-812bfe65-29889900-j5fzs\" (UID: \"1dd352a3-b3a3-4912-9d9f-723d40415008\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-812bfe65-29889900-j5fzs" Oct 30 21:00:00.298554 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:00.298136 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ff515ee-d559-4fe8-8487-5f04e13125bc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-f2ed1e-29889900-sctrt\" (UID: \"9ff515ee-d559-4fe8-8487-5f04e13125bc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-f2ed1e-29889900-sctrt" Oct 30 21:00:00.399697 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:00.399660 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ff515ee-d559-4fe8-8487-5f04e13125bc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-f2ed1e-29889900-sctrt\" (UID: \"9ff515ee-d559-4fe8-8487-5f04e13125bc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-f2ed1e-29889900-sctrt" Oct 30 21:00:00.399896 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:00.399727 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5xfhw\" (UniqueName: \"kubernetes.io/projected/9ff515ee-d559-4fe8-8487-5f04e13125bc-kube-api-access-5xfhw\") pod \"maas-api-key-cleanup-e2e-worker-w4-f2ed1e-29889900-sctrt\" (UID: \"9ff515ee-d559-4fe8-8487-5f04e13125bc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-f2ed1e-29889900-sctrt" Oct 30 21:00:00.399896 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:00.399893 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-62fpq\" (UniqueName: \"kubernetes.io/projected/1dd352a3-b3a3-4912-9d9f-723d40415008-kube-api-access-62fpq\") pod \"maas-api-key-cleanup-e2e-mt-812bfe65-29889900-j5fzs\" (UID: \"1dd352a3-b3a3-4912-9d9f-723d40415008\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-812bfe65-29889900-j5fzs" Oct 30 21:00:00.400033 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:00.399920 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1dd352a3-b3a3-4912-9d9f-723d40415008-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-mt-812bfe65-29889900-j5fzs\" (UID: \"1dd352a3-b3a3-4912-9d9f-723d40415008\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-812bfe65-29889900-j5fzs" Oct 30 21:00:00.400435 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:00.400397 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ff515ee-d559-4fe8-8487-5f04e13125bc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-f2ed1e-29889900-sctrt\" (UID: \"9ff515ee-d559-4fe8-8487-5f04e13125bc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-f2ed1e-29889900-sctrt" Oct 30 21:00:00.400603 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:00.400578 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1dd352a3-b3a3-4912-9d9f-723d40415008-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-mt-812bfe65-29889900-j5fzs\" (UID: \"1dd352a3-b3a3-4912-9d9f-723d40415008\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-812bfe65-29889900-j5fzs" Oct 30 21:00:00.411857 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:00.411806 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5xfhw\" (UniqueName: \"kubernetes.io/projected/9ff515ee-d559-4fe8-8487-5f04e13125bc-kube-api-access-5xfhw\") pod \"maas-api-key-cleanup-e2e-worker-w4-f2ed1e-29889900-sctrt\" (UID: \"9ff515ee-d559-4fe8-8487-5f04e13125bc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-f2ed1e-29889900-sctrt" Oct 30 21:00:00.412084 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:00.412067 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-62fpq\" (UniqueName: \"kubernetes.io/projected/1dd352a3-b3a3-4912-9d9f-723d40415008-kube-api-access-62fpq\") pod \"maas-api-key-cleanup-e2e-mt-812bfe65-29889900-j5fzs\" (UID: \"1dd352a3-b3a3-4912-9d9f-723d40415008\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-812bfe65-29889900-j5fzs" Oct 30 21:00:00.521840 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:00.521804 4611 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-812bfe65-29889900-j5fzs" Oct 30 21:00:00.523962 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:00.523934 4611 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-f2ed1e-29889900-sctrt" Oct 30 21:00:00.837485 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:00.835628 4611 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-812bfe65-29889900-j5fzs"] Oct 30 21:00:00.837887 ip-10-0-133-192 kubenswrapper[4611]: W1030 21:00:00.837857 4611 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1dd352a3_b3a3_4912_9d9f_723d40415008.slice/crio-2cdcee3b2cdda661c1048da9fb4994ffa98a26993f96470f6f7cb542e16804f0 WatchSource:0}: Error finding container 2cdcee3b2cdda661c1048da9fb4994ffa98a26993f96470f6f7cb542e16804f0: Status 404 returned error can't find the container with id 2cdcee3b2cdda661c1048da9fb4994ffa98a26993f96470f6f7cb542e16804f0 Oct 30 21:00:00.840210 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:00.840179 4611 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-f2ed1e-29889900-sctrt"] Oct 30 21:00:00.884534 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:00.884087 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:00:00.915777 ip-10-0-133-192 kubenswrapper[4611]: W1030 21:00:00.915707 4611 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ff515ee_d559_4fe8_8487_5f04e13125bc.slice/crio-0ee70554190138e4c77cc85319aa4e755ea27e8620ae6cf99d1b8519f38cb94c WatchSource:0}: Error finding container 0ee70554190138e4c77cc85319aa4e755ea27e8620ae6cf99d1b8519f38cb94c: Status 404 returned error can't find the container with id 0ee70554190138e4c77cc85319aa4e755ea27e8620ae6cf99d1b8519f38cb94c Oct 30 21:00:01.587624 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:01.587538 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:00:01.826950 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:01.826912 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-f2ed1e-29889900-sctrt" event={"ID":"9ff515ee-d559-4fe8-8487-5f04e13125bc","Type":"ContainerStarted","Data":"12dd349116957d9e85302d854f32f832c9e50408a1ea9c53094bbc196e3ea588"} Oct 30 21:00:01.827110 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:01.826976 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-f2ed1e-29889900-sctrt" event={"ID":"9ff515ee-d559-4fe8-8487-5f04e13125bc","Type":"ContainerStarted","Data":"0ee70554190138e4c77cc85319aa4e755ea27e8620ae6cf99d1b8519f38cb94c"} Oct 30 21:00:01.828708 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:01.828678 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-812bfe65-29889900-j5fzs" event={"ID":"1dd352a3-b3a3-4912-9d9f-723d40415008","Type":"ContainerStarted","Data":"8e51eee9bc1354a9ab4a9a2528eb67992cb47b302013af1dc108d1297f9d5efb"} Oct 30 21:00:01.828838 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:01.828724 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-812bfe65-29889900-j5fzs" event={"ID":"1dd352a3-b3a3-4912-9d9f-723d40415008","Type":"ContainerStarted","Data":"2cdcee3b2cdda661c1048da9fb4994ffa98a26993f96470f6f7cb542e16804f0"} Oct 30 21:00:01.847655 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:01.846430 4611 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-f2ed1e-29889900-sctrt" podStartSLOduration=1.846412832 podStartE2EDuration="1.846412832s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:00:01.844460705 +0000 UTC m=+1256.378487322" watchObservedRunningTime="2026-10-30 21:00:01.846412832 +0000 UTC m=+1256.380439451" Oct 30 21:00:01.878789 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:01.878725 4611 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-812bfe65-29889900-j5fzs" podStartSLOduration=1.878710431 podStartE2EDuration="1.878710431s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:00:01.878381583 +0000 UTC m=+1256.412408200" watchObservedRunningTime="2026-10-30 21:00:01.878710431 +0000 UTC m=+1256.412737088" Oct 30 21:00:02.138755 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:02.138716 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w6-3d909d-74bfb44f9b-5f2x8" podUID="46885e54-e3da-4b91-aa7b-433f9b55a0da" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 21:00:02.138951 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:02.138915 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-3d909d-57b9b6dcf9-l9pwk" podUID="5d963de5-2836-4225-9120-d885791b4f71" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 21:00:02.472590 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:02.464427 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:00:02.869527 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:02.858493 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-f2ed1e-29889900-sctrt" event={"ID":"9ff515ee-d559-4fe8-8487-5f04e13125bc","Type":"ContainerDied","Data":"12dd349116957d9e85302d854f32f832c9e50408a1ea9c53094bbc196e3ea588"} Oct 30 21:00:02.869527 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:02.858296 4611 generic.go:356] "Generic (PLEG): container finished" podID="9ff515ee-d559-4fe8-8487-5f04e13125bc" containerID="12dd349116957d9e85302d854f32f832c9e50408a1ea9c53094bbc196e3ea588" exitCode=60 Oct 30 21:00:02.869527 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:02.868909 4611 scope.go:122] "RemoveContainer" containerID="12dd349116957d9e85302d854f32f832c9e50408a1ea9c53094bbc196e3ea588" Oct 30 21:00:02.896091 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:02.888415 4611 generic.go:356] "Generic (PLEG): container finished" podID="1dd352a3-b3a3-4912-9d9f-723d40415008" containerID="8e51eee9bc1354a9ab4a9a2528eb67992cb47b302013af1dc108d1297f9d5efb" exitCode=60 Oct 30 21:00:02.896091 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:02.888563 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-812bfe65-29889900-j5fzs" event={"ID":"1dd352a3-b3a3-4912-9d9f-723d40415008","Type":"ContainerDied","Data":"8e51eee9bc1354a9ab4a9a2528eb67992cb47b302013af1dc108d1297f9d5efb"} Oct 30 21:00:02.896091 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:02.889009 4611 scope.go:122] "RemoveContainer" containerID="8e51eee9bc1354a9ab4a9a2528eb67992cb47b302013af1dc108d1297f9d5efb" Oct 30 21:00:03.479582 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:03.468983 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:00:04.033657 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:04.033339 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-f2ed1e-29889900-sctrt" event={"ID":"9ff515ee-d559-4fe8-8487-5f04e13125bc","Type":"ContainerStarted","Data":"f1e96aa46218b846126a7aeaf252ddd043415ca79c4d92e2b5fce47ebbbd9fd9"} Oct 30 21:00:04.036628 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:04.036332 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-812bfe65-29889900-j5fzs" event={"ID":"1dd352a3-b3a3-4912-9d9f-723d40415008","Type":"ContainerStarted","Data":"6235d27e5fd8f07010a0a71d511448b4b417288831b01149870de7ad1e8e340e"} Oct 30 21:00:04.151098 ip-10-0-133-192 kubenswrapper[4611]: E1030 21:00:04.151063 4611 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ff515ee_d559_4fe8_8487_5f04e13125bc.slice/crio-f1e96aa46218b846126a7aeaf252ddd043415ca79c4d92e2b5fce47ebbbd9fd9.scope\": RecentStats: unable to find data in memory cache]" Oct 30 21:00:04.214897 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:04.202211 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:00:04.233684 ip-10-0-133-192 kubenswrapper[4611]: E1030 21:00:04.232713 4611 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hwzc6" containerName="extractor" Oct 30 21:00:04.903556 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:04.903482 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:00:05.044697 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:05.044664 4611 generic.go:356] "Generic (PLEG): container finished" podID="9ff515ee-d559-4fe8-8487-5f04e13125bc" containerID="f1e96aa46218b846126a7aeaf252ddd043415ca79c4d92e2b5fce47ebbbd9fd9" exitCode=60 Oct 30 21:00:05.045147 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:05.044742 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-f2ed1e-29889900-sctrt" event={"ID":"9ff515ee-d559-4fe8-8487-5f04e13125bc","Type":"ContainerDied","Data":"f1e96aa46218b846126a7aeaf252ddd043415ca79c4d92e2b5fce47ebbbd9fd9"} Oct 30 21:00:05.045147 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:05.044815 4611 scope.go:122] "RemoveContainer" containerID="12dd349116957d9e85302d854f32f832c9e50408a1ea9c53094bbc196e3ea588" Oct 30 21:00:05.045147 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:05.044994 4611 scope.go:122] "RemoveContainer" containerID="f1e96aa46218b846126a7aeaf252ddd043415ca79c4d92e2b5fce47ebbbd9fd9" Oct 30 21:00:05.045332 ip-10-0-133-192 kubenswrapper[4611]: E1030 21:00:05.045284 4611 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-f2ed1e-29889900-sctrt_odh-ai-gateway-infra(9ff515ee-d559-4fe8-8487-5f04e13125bc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-f2ed1e-29889900-sctrt" podUID="9ff515ee-d559-4fe8-8487-5f04e13125bc" Oct 30 21:00:05.065007 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:05.064488 4611 generic.go:356] "Generic (PLEG): container finished" podID="1dd352a3-b3a3-4912-9d9f-723d40415008" containerID="6235d27e5fd8f07010a0a71d511448b4b417288831b01149870de7ad1e8e340e" exitCode=60 Oct 30 21:00:05.065646 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:05.065625 4611 scope.go:122] "RemoveContainer" containerID="6235d27e5fd8f07010a0a71d511448b4b417288831b01149870de7ad1e8e340e" Oct 30 21:00:05.066542 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:05.064621 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-812bfe65-29889900-j5fzs" event={"ID":"1dd352a3-b3a3-4912-9d9f-723d40415008","Type":"ContainerDied","Data":"6235d27e5fd8f07010a0a71d511448b4b417288831b01149870de7ad1e8e340e"} Oct 30 21:00:05.067147 ip-10-0-133-192 kubenswrapper[4611]: E1030 21:00:05.067121 4611 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-mt-812bfe65-29889900-j5fzs_odh-ai-gateway-infra(1dd352a3-b3a3-4912-9d9f-723d40415008)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-812bfe65-29889900-j5fzs" podUID="1dd352a3-b3a3-4912-9d9f-723d40415008" Oct 30 21:00:05.080612 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:05.080590 4611 scope.go:122] "RemoveContainer" containerID="8e51eee9bc1354a9ab4a9a2528eb67992cb47b302013af1dc108d1297f9d5efb" Oct 30 21:00:05.674150 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:05.674114 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:00:05.832876 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:05.832840 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:00:05.996350 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:05.996284 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:00:06.076772 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:06.076740 4611 scope.go:122] "RemoveContainer" containerID="f1e96aa46218b846126a7aeaf252ddd043415ca79c4d92e2b5fce47ebbbd9fd9" Oct 30 21:00:06.077232 ip-10-0-133-192 kubenswrapper[4611]: E1030 21:00:06.077047 4611 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-f2ed1e-29889900-sctrt_odh-ai-gateway-infra(9ff515ee-d559-4fe8-8487-5f04e13125bc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-f2ed1e-29889900-sctrt" podUID="9ff515ee-d559-4fe8-8487-5f04e13125bc" Oct 30 21:00:06.079714 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:06.079690 4611 scope.go:122] "RemoveContainer" containerID="6235d27e5fd8f07010a0a71d511448b4b417288831b01149870de7ad1e8e340e" Oct 30 21:00:06.080046 ip-10-0-133-192 kubenswrapper[4611]: E1030 21:00:06.080023 4611 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-mt-812bfe65-29889900-j5fzs_odh-ai-gateway-infra(1dd352a3-b3a3-4912-9d9f-723d40415008)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-812bfe65-29889900-j5fzs" podUID="1dd352a3-b3a3-4912-9d9f-723d40415008" Oct 30 21:00:06.146094 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:06.146065 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:00:06.338845 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:06.335909 4611 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-ztxdx"] Oct 30 21:00:06.338845 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:06.336226 4611 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-ztxdx" podUID="5a93b27f-f269-471c-8be2-523f373bca08" containerName="istio-proxy" containerID="cri-o://f8d503a52fd7f6a1f40bb16219178d4eadb387455d394cd382f26d0f645f0dd5" gracePeriod=30 Oct 30 21:00:06.345975 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:06.341558 4611 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-trngw"] Oct 30 21:00:06.345975 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:06.341911 4611 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-trngw" podUID="5ba753c3-5979-4263-8b90-2741747c8592" containerName="istio-proxy" containerID="cri-o://c10f965c9d3fd5d620ad061d397c593e1191596603f3b4b5f8242467c8cab5b5" gracePeriod=30 Oct 30 21:00:06.361957 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:06.361932 4611 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-qkwks"] Oct 30 21:00:06.362128 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:06.362109 4611 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-qkwks" podUID="db1a832a-0df2-4a93-bee2-40c9b99218d9" containerName="istio-proxy" containerID="cri-o://00b4462f302792be026e0808d35c51a456553937259897e483726e2dfb2fbed4" gracePeriod=30 Oct 30 21:00:06.367175 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:06.367151 4611 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-wf8qg"] Oct 30 21:00:06.367359 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:06.367339 4611 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-wf8qg" podUID="c468f3c6-7d73-4fd1-ae34-1212e4d1b387" containerName="istio-proxy" containerID="cri-o://1a52628c1464f529ef397549f13e488f2d96482f8fda373851190beef6458016" gracePeriod=30 Oct 30 21:00:06.402303 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:06.402272 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:00:06.550424 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:06.550384 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:00:06.721485 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:06.721448 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:00:07.325855 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:07.325800 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:00:07.501103 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:07.501070 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:00:07.644531 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:07.644478 4611 scope.go:122] "RemoveContainer" containerID="9aab1ebf1ca04f57868eba0f42a0d2ede0cdd21b941a5105b9671a0dbc642a80" Oct 30 21:00:07.654697 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:07.654671 4611 scope.go:122] "RemoveContainer" containerID="087ddc58167f65b47f6e684ebfdfb7d6fc341e0c4e55750c35fe52dce7ad0473" Oct 30 21:00:08.093189 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:08.093093 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:00:08.804997 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:08.804955 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:00:09.558575 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:09.558539 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:00:09.677620 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:09.677590 4611 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-3d909d-74bfb44f9b-5f2x8" Oct 30 21:00:09.738691 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:09.738619 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46885e54-e3da-4b91-aa7b-433f9b55a0da-kube-api-access-mvm6v\" (UniqueName: \"kubernetes.io/projected/46885e54-e3da-4b91-aa7b-433f9b55a0da-kube-api-access-mvm6v\") pod \"46885e54-e3da-4b91-aa7b-433f9b55a0da\" (UID: \"46885e54-e3da-4b91-aa7b-433f9b55a0da\") " Oct 30 21:00:09.738691 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:09.738660 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/46885e54-e3da-4b91-aa7b-433f9b55a0da-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46885e54-e3da-4b91-aa7b-433f9b55a0da-plugins-config-volume\") pod \"46885e54-e3da-4b91-aa7b-433f9b55a0da\" (UID: \"46885e54-e3da-4b91-aa7b-433f9b55a0da\") " Oct 30 21:00:09.739095 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:09.739067 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46885e54-e3da-4b91-aa7b-433f9b55a0da-plugins-config-volume" pod "46885e54-e3da-4b91-aa7b-433f9b55a0da" (UID: "46885e54-e3da-4b91-aa7b-433f9b55a0da"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:00:09.739362 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:09.739341 4611 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46885e54-e3da-4b91-aa7b-433f9b55a0da-plugins-config-volume\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:00:09.741000 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:09.740963 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46885e54-e3da-4b91-aa7b-433f9b55a0da-kube-api-access-mvm6v" pod "46885e54-e3da-4b91-aa7b-433f9b55a0da" (UID: "46885e54-e3da-4b91-aa7b-433f9b55a0da"). InnerVolumeSpecName "kube-api-access-mvm6v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:00:09.750700 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:09.750681 4611 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-3d909d-57b9b6dcf9-l9pwk" Oct 30 21:00:09.839806 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:09.839778 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d963de5-2836-4225-9120-d885791b4f71-kube-api-access-xg67c\" (UniqueName: \"kubernetes.io/projected/5d963de5-2836-4225-9120-d885791b4f71-kube-api-access-xg67c\") pod \"5d963de5-2836-4225-9120-d885791b4f71\" (UID: \"5d963de5-2836-4225-9120-d885791b4f71\") " Oct 30 21:00:09.840252 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:09.839945 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5d963de5-2836-4225-9120-d885791b4f71-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d963de5-2836-4225-9120-d885791b4f71-plugins-config-volume\") pod \"5d963de5-2836-4225-9120-d885791b4f71\" (UID: \"5d963de5-2836-4225-9120-d885791b4f71\") " Oct 30 21:00:09.840346 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:09.840325 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d963de5-2836-4225-9120-d885791b4f71-plugins-config-volume" pod "5d963de5-2836-4225-9120-d885791b4f71" (UID: "5d963de5-2836-4225-9120-d885791b4f71"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:00:09.840421 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:09.840406 4611 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mvm6v\" (UniqueName: \"kubernetes.io/projected/46885e54-e3da-4b91-aa7b-433f9b55a0da-kube-api-access-mvm6v\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:00:09.841874 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:09.841849 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d963de5-2836-4225-9120-d885791b4f71-kube-api-access-xg67c" pod "5d963de5-2836-4225-9120-d885791b4f71" (UID: "5d963de5-2836-4225-9120-d885791b4f71"). InnerVolumeSpecName "kube-api-access-xg67c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:00:09.941496 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:09.941465 4611 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d963de5-2836-4225-9120-d885791b4f71-plugins-config-volume\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:00:09.941496 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:09.941494 4611 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xg67c\" (UniqueName: \"kubernetes.io/projected/5d963de5-2836-4225-9120-d885791b4f71-kube-api-access-xg67c\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:00:10.113908 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:10.113876 4611 generic.go:356] "Generic (PLEG): container finished" podID="5d963de5-2836-4225-9120-d885791b4f71" containerID="fe58a91cc76f1810321347afc1a4d4adcbdab7c190d7fef96fd2e34ae9f677c1" exitCode=0 Oct 30 21:00:10.114082 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:10.113918 4611 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-3d909d-57b9b6dcf9-l9pwk" Oct 30 21:00:10.114082 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:10.113960 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-3d909d-57b9b6dcf9-l9pwk" event={"ID":"5d963de5-2836-4225-9120-d885791b4f71","Type":"ContainerDied","Data":"fe58a91cc76f1810321347afc1a4d4adcbdab7c190d7fef96fd2e34ae9f677c1"} Oct 30 21:00:10.114082 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:10.114010 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-3d909d-57b9b6dcf9-l9pwk" event={"ID":"5d963de5-2836-4225-9120-d885791b4f71","Type":"ContainerDied","Data":"163dae44e2baad37f2b2c70ce1538eef4db970f635fd74ee9ccf864d62e46c86"} Oct 30 21:00:10.114082 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:10.114033 4611 scope.go:122] "RemoveContainer" containerID="fe58a91cc76f1810321347afc1a4d4adcbdab7c190d7fef96fd2e34ae9f677c1" Oct 30 21:00:10.115479 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:10.115455 4611 generic.go:356] "Generic (PLEG): container finished" podID="46885e54-e3da-4b91-aa7b-433f9b55a0da" containerID="ca32235c39e2d40fc731f627b2cb0f906b6da84bf1a9bf3ecdc1a75dbe43d3d6" exitCode=0 Oct 30 21:00:10.115623 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:10.115490 4611 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-3d909d-74bfb44f9b-5f2x8" Oct 30 21:00:10.115623 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:10.115536 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-3d909d-74bfb44f9b-5f2x8" event={"ID":"46885e54-e3da-4b91-aa7b-433f9b55a0da","Type":"ContainerDied","Data":"ca32235c39e2d40fc731f627b2cb0f906b6da84bf1a9bf3ecdc1a75dbe43d3d6"} Oct 30 21:00:10.115623 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:10.115580 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-3d909d-74bfb44f9b-5f2x8" event={"ID":"46885e54-e3da-4b91-aa7b-433f9b55a0da","Type":"ContainerDied","Data":"5e157bb6d4698ab0d8690569d7f21c73301b0286718525f35f8180addedc96bd"} Oct 30 21:00:10.118301 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:10.118278 4611 generic.go:356] "Generic (PLEG): container finished" podID="918ff269-fb66-4479-9e8d-d0184421bc7c" containerID="374d672a1b8de629149cd6f13a4700bc2904de58b732cc0733a2facef6ba11c4" exitCode=0 Oct 30 21:00:10.118442 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:10.118357 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-812bfe65-openshift-default-74647fb5f7-f85fj" event={"ID":"918ff269-fb66-4479-9e8d-d0184421bc7c","Type":"ContainerDied","Data":"374d672a1b8de629149cd6f13a4700bc2904de58b732cc0733a2facef6ba11c4"} Oct 30 21:00:10.118828 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:10.118808 4611 scope.go:122] "RemoveContainer" containerID="374d672a1b8de629149cd6f13a4700bc2904de58b732cc0733a2facef6ba11c4" Oct 30 21:00:10.120125 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:10.120099 4611 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:00:10.120210 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:10.120167 4611 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:00:10.120258 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:10.120213 4611 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:00:10.126480 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:10.126458 4611 scope.go:122] "RemoveContainer" containerID="fe58a91cc76f1810321347afc1a4d4adcbdab7c190d7fef96fd2e34ae9f677c1" Oct 30 21:00:10.126842 ip-10-0-133-192 kubenswrapper[4611]: E1030 21:00:10.126821 4611 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe58a91cc76f1810321347afc1a4d4adcbdab7c190d7fef96fd2e34ae9f677c1\": container with ID starting with fe58a91cc76f1810321347afc1a4d4adcbdab7c190d7fef96fd2e34ae9f677c1 not found: ID does not exist" containerID="fe58a91cc76f1810321347afc1a4d4adcbdab7c190d7fef96fd2e34ae9f677c1" Oct 30 21:00:10.126915 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:10.126855 4611 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe58a91cc76f1810321347afc1a4d4adcbdab7c190d7fef96fd2e34ae9f677c1"} err="failed to get container status \"fe58a91cc76f1810321347afc1a4d4adcbdab7c190d7fef96fd2e34ae9f677c1\": rpc error: code = NotFound desc = could not find container \"fe58a91cc76f1810321347afc1a4d4adcbdab7c190d7fef96fd2e34ae9f677c1\": container with ID starting with fe58a91cc76f1810321347afc1a4d4adcbdab7c190d7fef96fd2e34ae9f677c1 not found: ID does not exist" Oct 30 21:00:10.126915 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:10.126879 4611 scope.go:122] "RemoveContainer" containerID="ca32235c39e2d40fc731f627b2cb0f906b6da84bf1a9bf3ecdc1a75dbe43d3d6" Oct 30 21:00:10.139466 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:10.139451 4611 scope.go:122] "RemoveContainer" containerID="ca32235c39e2d40fc731f627b2cb0f906b6da84bf1a9bf3ecdc1a75dbe43d3d6" Oct 30 21:00:10.139754 ip-10-0-133-192 kubenswrapper[4611]: E1030 21:00:10.139735 4611 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca32235c39e2d40fc731f627b2cb0f906b6da84bf1a9bf3ecdc1a75dbe43d3d6\": container with ID starting with ca32235c39e2d40fc731f627b2cb0f906b6da84bf1a9bf3ecdc1a75dbe43d3d6 not found: ID does not exist" containerID="ca32235c39e2d40fc731f627b2cb0f906b6da84bf1a9bf3ecdc1a75dbe43d3d6" Oct 30 21:00:10.139825 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:10.139765 4611 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca32235c39e2d40fc731f627b2cb0f906b6da84bf1a9bf3ecdc1a75dbe43d3d6"} err="failed to get container status \"ca32235c39e2d40fc731f627b2cb0f906b6da84bf1a9bf3ecdc1a75dbe43d3d6\": rpc error: code = NotFound desc = could not find container \"ca32235c39e2d40fc731f627b2cb0f906b6da84bf1a9bf3ecdc1a75dbe43d3d6\": container with ID starting with ca32235c39e2d40fc731f627b2cb0f906b6da84bf1a9bf3ecdc1a75dbe43d3d6 not found: ID does not exist" Oct 30 21:00:10.167299 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:10.167271 4611 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-3d909d-74bfb44f9b-5f2x8"] Oct 30 21:00:10.170948 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:10.170922 4611 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-3d909d-74bfb44f9b-5f2x8"] Oct 30 21:00:10.180741 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:10.180704 4611 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-3d909d-57b9b6dcf9-l9pwk"] Oct 30 21:00:10.186479 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:10.186454 4611 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-3d909d-57b9b6dcf9-l9pwk"] Oct 30 21:00:10.304142 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:10.304074 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:00:10.706898 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:10.706824 4611 patch_prober.go:29] interesting pod/e2e-worker-w6-3d909d-openshift-default-77787695b7-qkwks container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 21:00:10.706898 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:10.706876 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-qkwks" podUID="db1a832a-0df2-4a93-bee2-40c9b99218d9" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 21:00:10.992990 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:10.992891 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:00:11.127477 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:11.127444 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-812bfe65-openshift-default-74647fb5f7-f85fj" event={"ID":"918ff269-fb66-4479-9e8d-d0184421bc7c","Type":"ContainerStarted","Data":"01c9b8958f5f15dbc930d892de99b56df1b0df85845bbcd3d55203b2b47b4971"} Oct 30 21:00:11.612530 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:11.611017 4611 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="46885e54-e3da-4b91-aa7b-433f9b55a0da" path="/var/lib/kubelet/pods/46885e54-e3da-4b91-aa7b-433f9b55a0da/volumes" Oct 30 21:00:11.612530 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:11.611650 4611 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5d963de5-2836-4225-9120-d885791b4f71" path="/var/lib/kubelet/pods/5d963de5-2836-4225-9120-d885791b4f71/volumes" Oct 30 21:00:11.729070 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:11.728795 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:00:11.831841 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:11.831539 4611 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-ztxdx" Oct 30 21:00:11.888413 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:11.888388 4611 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-wf8qg" Oct 30 21:00:11.890596 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:11.890545 4611 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-qkwks" Oct 30 21:00:11.892617 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:11.892598 4611 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-trngw" Oct 30 21:00:11.919571 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:11.919527 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a93b27f-f269-471c-8be2-523f373bca08-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a93b27f-f269-471c-8be2-523f373bca08-workload-certs\") pod \"5a93b27f-f269-471c-8be2-523f373bca08\" (UID: \"5a93b27f-f269-471c-8be2-523f373bca08\") " Oct 30 21:00:11.919732 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:11.919613 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a93b27f-f269-471c-8be2-523f373bca08-kube-api-access-zdj5h\" (UniqueName: \"kubernetes.io/projected/5a93b27f-f269-471c-8be2-523f373bca08-kube-api-access-zdj5h\") pod \"5a93b27f-f269-471c-8be2-523f373bca08\" (UID: \"5a93b27f-f269-471c-8be2-523f373bca08\") " Oct 30 21:00:11.919732 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:11.919686 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5a93b27f-f269-471c-8be2-523f373bca08-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a93b27f-f269-471c-8be2-523f373bca08-istiod-ca-cert\") pod \"5a93b27f-f269-471c-8be2-523f373bca08\" (UID: \"5a93b27f-f269-471c-8be2-523f373bca08\") " Oct 30 21:00:11.919869 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:11.919729 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a93b27f-f269-471c-8be2-523f373bca08-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a93b27f-f269-471c-8be2-523f373bca08-credential-socket\") pod \"5a93b27f-f269-471c-8be2-523f373bca08\" (UID: \"5a93b27f-f269-471c-8be2-523f373bca08\") " Oct 30 21:00:11.919869 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:11.919759 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a93b27f-f269-471c-8be2-523f373bca08-workload-certs" pod "5a93b27f-f269-471c-8be2-523f373bca08" (UID: "5a93b27f-f269-471c-8be2-523f373bca08"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:00:11.919869 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:11.919811 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a93b27f-f269-471c-8be2-523f373bca08-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a93b27f-f269-471c-8be2-523f373bca08-istio-envoy\") pod \"5a93b27f-f269-471c-8be2-523f373bca08\" (UID: \"5a93b27f-f269-471c-8be2-523f373bca08\") " Oct 30 21:00:11.919869 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:11.919840 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a93b27f-f269-471c-8be2-523f373bca08-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a93b27f-f269-471c-8be2-523f373bca08-workload-socket\") pod \"5a93b27f-f269-471c-8be2-523f373bca08\" (UID: \"5a93b27f-f269-471c-8be2-523f373bca08\") " Oct 30 21:00:11.920089 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:11.919949 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5a93b27f-f269-471c-8be2-523f373bca08-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a93b27f-f269-471c-8be2-523f373bca08-istio-podinfo\") pod \"5a93b27f-f269-471c-8be2-523f373bca08\" (UID: \"5a93b27f-f269-471c-8be2-523f373bca08\") " Oct 30 21:00:11.920089 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:11.920000 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a93b27f-f269-471c-8be2-523f373bca08-istio-token\" (UniqueName: \"kubernetes.io/projected/5a93b27f-f269-471c-8be2-523f373bca08-istio-token\") pod \"5a93b27f-f269-471c-8be2-523f373bca08\" (UID: \"5a93b27f-f269-471c-8be2-523f373bca08\") " Oct 30 21:00:11.920089 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:11.920063 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a93b27f-f269-471c-8be2-523f373bca08-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a93b27f-f269-471c-8be2-523f373bca08-istio-data\") pod \"5a93b27f-f269-471c-8be2-523f373bca08\" (UID: \"5a93b27f-f269-471c-8be2-523f373bca08\") " Oct 30 21:00:11.920089 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:11.920067 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a93b27f-f269-471c-8be2-523f373bca08-istiod-ca-cert" pod "5a93b27f-f269-471c-8be2-523f373bca08" (UID: "5a93b27f-f269-471c-8be2-523f373bca08"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:00:11.920311 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:11.920285 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a93b27f-f269-471c-8be2-523f373bca08-credential-socket" pod "5a93b27f-f269-471c-8be2-523f373bca08" (UID: "5a93b27f-f269-471c-8be2-523f373bca08"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:00:11.920443 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:11.920311 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a93b27f-f269-471c-8be2-523f373bca08-workload-socket" pod "5a93b27f-f269-471c-8be2-523f373bca08" (UID: "5a93b27f-f269-471c-8be2-523f373bca08"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:00:11.920593 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:11.920569 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a93b27f-f269-471c-8be2-523f373bca08-istio-data" pod "5a93b27f-f269-471c-8be2-523f373bca08" (UID: "5a93b27f-f269-471c-8be2-523f373bca08"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:00:11.920759 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:11.920737 4611 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a93b27f-f269-471c-8be2-523f373bca08-istio-data\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:00:11.920863 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:11.920761 4611 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a93b27f-f269-471c-8be2-523f373bca08-workload-certs\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:00:11.920863 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:11.920797 4611 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a93b27f-f269-471c-8be2-523f373bca08-istiod-ca-cert\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:00:11.920863 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:11.920812 4611 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a93b27f-f269-471c-8be2-523f373bca08-credential-socket\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:00:11.920863 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:11.920826 4611 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a93b27f-f269-471c-8be2-523f373bca08-workload-socket\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:00:11.922865 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:11.922840 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5a93b27f-f269-471c-8be2-523f373bca08-istio-podinfo" pod "5a93b27f-f269-471c-8be2-523f373bca08" (UID: "5a93b27f-f269-471c-8be2-523f373bca08"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:00:11.922865 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:11.922854 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a93b27f-f269-471c-8be2-523f373bca08-kube-api-access-zdj5h" pod "5a93b27f-f269-471c-8be2-523f373bca08" (UID: "5a93b27f-f269-471c-8be2-523f373bca08"). InnerVolumeSpecName "kube-api-access-zdj5h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:00:11.923464 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:11.923443 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a93b27f-f269-471c-8be2-523f373bca08-istio-token" pod "5a93b27f-f269-471c-8be2-523f373bca08" (UID: "5a93b27f-f269-471c-8be2-523f373bca08"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:00:11.924688 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:11.924665 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a93b27f-f269-471c-8be2-523f373bca08-istio-envoy" pod "5a93b27f-f269-471c-8be2-523f373bca08" (UID: "5a93b27f-f269-471c-8be2-523f373bca08"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:00:11.944055 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:11.943993 4611 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-812bfe65-openshift-default-74647fb5f7-f85fj" Oct 30 21:00:11.944186 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:11.944165 4611 patch_prober.go:29] interesting pod/e2e-mt-812bfe65-openshift-default-74647fb5f7-f85fj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body= Oct 30 21:00:11.944254 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:11.944224 4611 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-812bfe65-openshift-default-74647fb5f7-f85fj" podUID="918ff269-fb66-4479-9e8d-d0184421bc7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" Oct 30 21:00:12.022009 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:12.021975 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/db1a832a-0df2-4a93-bee2-40c9b99218d9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db1a832a-0df2-4a93-bee2-40c9b99218d9-istio-podinfo\") pod \"db1a832a-0df2-4a93-bee2-40c9b99218d9\" (UID: \"db1a832a-0df2-4a93-bee2-40c9b99218d9\") " Oct 30 21:00:12.022350 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:12.022046 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ba753c3-5979-4263-8b90-2741747c8592-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ba753c3-5979-4263-8b90-2741747c8592-workload-certs\") pod \"5ba753c3-5979-4263-8b90-2741747c8592\" (UID: \"5ba753c3-5979-4263-8b90-2741747c8592\") " Oct 30 21:00:12.022350 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:12.022078 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db1a832a-0df2-4a93-bee2-40c9b99218d9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db1a832a-0df2-4a93-bee2-40c9b99218d9-istio-envoy\") pod \"db1a832a-0df2-4a93-bee2-40c9b99218d9\" (UID: \"db1a832a-0df2-4a93-bee2-40c9b99218d9\") " Oct 30 21:00:12.022350 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:12.022121 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c468f3c6-7d73-4fd1-ae34-1212e4d1b387-istio-token\" (UniqueName: \"kubernetes.io/projected/c468f3c6-7d73-4fd1-ae34-1212e4d1b387-istio-token\") pod \"c468f3c6-7d73-4fd1-ae34-1212e4d1b387\" (UID: \"c468f3c6-7d73-4fd1-ae34-1212e4d1b387\") " Oct 30 21:00:12.022350 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:12.022150 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ba753c3-5979-4263-8b90-2741747c8592-istio-token\" (UniqueName: \"kubernetes.io/projected/5ba753c3-5979-4263-8b90-2741747c8592-istio-token\") pod \"5ba753c3-5979-4263-8b90-2741747c8592\" (UID: \"5ba753c3-5979-4263-8b90-2741747c8592\") " Oct 30 21:00:12.022350 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:12.022197 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c468f3c6-7d73-4fd1-ae34-1212e4d1b387-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c468f3c6-7d73-4fd1-ae34-1212e4d1b387-credential-socket\") pod \"c468f3c6-7d73-4fd1-ae34-1212e4d1b387\" (UID: \"c468f3c6-7d73-4fd1-ae34-1212e4d1b387\") " Oct 30 21:00:12.022350 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:12.022262 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c468f3c6-7d73-4fd1-ae34-1212e4d1b387-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c468f3c6-7d73-4fd1-ae34-1212e4d1b387-workload-certs\") pod \"c468f3c6-7d73-4fd1-ae34-1212e4d1b387\" (UID: \"c468f3c6-7d73-4fd1-ae34-1212e4d1b387\") " Oct 30 21:00:12.022350 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:12.022292 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db1a832a-0df2-4a93-bee2-40c9b99218d9-kube-api-access-mnf2z\" (UniqueName: \"kubernetes.io/projected/db1a832a-0df2-4a93-bee2-40c9b99218d9-kube-api-access-mnf2z\") pod \"db1a832a-0df2-4a93-bee2-40c9b99218d9\" (UID: \"db1a832a-0df2-4a93-bee2-40c9b99218d9\") " Oct 30 21:00:12.022350 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:12.022341 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ba753c3-5979-4263-8b90-2741747c8592-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ba753c3-5979-4263-8b90-2741747c8592-istio-data\") pod \"5ba753c3-5979-4263-8b90-2741747c8592\" (UID: \"5ba753c3-5979-4263-8b90-2741747c8592\") " Oct 30 21:00:12.022637 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:12.022398 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c468f3c6-7d73-4fd1-ae34-1212e4d1b387-kube-api-access-mm94t\" (UniqueName: \"kubernetes.io/projected/c468f3c6-7d73-4fd1-ae34-1212e4d1b387-kube-api-access-mm94t\") pod \"c468f3c6-7d73-4fd1-ae34-1212e4d1b387\" (UID: \"c468f3c6-7d73-4fd1-ae34-1212e4d1b387\") " Oct 30 21:00:12.022637 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:12.022429 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c468f3c6-7d73-4fd1-ae34-1212e4d1b387-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c468f3c6-7d73-4fd1-ae34-1212e4d1b387-istio-podinfo\") pod \"c468f3c6-7d73-4fd1-ae34-1212e4d1b387\" (UID: \"c468f3c6-7d73-4fd1-ae34-1212e4d1b387\") " Oct 30 21:00:12.022637 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:12.022457 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db1a832a-0df2-4a93-bee2-40c9b99218d9-istio-token\" (UniqueName: \"kubernetes.io/projected/db1a832a-0df2-4a93-bee2-40c9b99218d9-istio-token\") pod \"db1a832a-0df2-4a93-bee2-40c9b99218d9\" (UID: \"db1a832a-0df2-4a93-bee2-40c9b99218d9\") " Oct 30 21:00:12.022637 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:12.022535 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db1a832a-0df2-4a93-bee2-40c9b99218d9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db1a832a-0df2-4a93-bee2-40c9b99218d9-workload-certs\") pod \"db1a832a-0df2-4a93-bee2-40c9b99218d9\" (UID: \"db1a832a-0df2-4a93-bee2-40c9b99218d9\") " Oct 30 21:00:12.022637 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:12.022567 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c468f3c6-7d73-4fd1-ae34-1212e4d1b387-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c468f3c6-7d73-4fd1-ae34-1212e4d1b387-istio-envoy\") pod \"c468f3c6-7d73-4fd1-ae34-1212e4d1b387\" (UID: \"c468f3c6-7d73-4fd1-ae34-1212e4d1b387\") " Oct 30 21:00:12.022792 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:12.022656 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ba753c3-5979-4263-8b90-2741747c8592-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ba753c3-5979-4263-8b90-2741747c8592-istio-envoy\") pod \"5ba753c3-5979-4263-8b90-2741747c8592\" (UID: \"5ba753c3-5979-4263-8b90-2741747c8592\") " Oct 30 21:00:12.022792 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:12.022692 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5ba753c3-5979-4263-8b90-2741747c8592-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ba753c3-5979-4263-8b90-2741747c8592-istio-podinfo\") pod \"5ba753c3-5979-4263-8b90-2741747c8592\" (UID: \"5ba753c3-5979-4263-8b90-2741747c8592\") " Oct 30 21:00:12.022792 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:12.022738 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/db1a832a-0df2-4a93-bee2-40c9b99218d9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db1a832a-0df2-4a93-bee2-40c9b99218d9-istiod-ca-cert\") pod \"db1a832a-0df2-4a93-bee2-40c9b99218d9\" (UID: \"db1a832a-0df2-4a93-bee2-40c9b99218d9\") " Oct 30 21:00:12.022792 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:12.022768 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c468f3c6-7d73-4fd1-ae34-1212e4d1b387-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c468f3c6-7d73-4fd1-ae34-1212e4d1b387-istiod-ca-cert\") pod \"c468f3c6-7d73-4fd1-ae34-1212e4d1b387\" (UID: \"c468f3c6-7d73-4fd1-ae34-1212e4d1b387\") " Oct 30 21:00:12.022904 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:12.022799 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ba753c3-5979-4263-8b90-2741747c8592-kube-api-access-8j5dq\" (UniqueName: \"kubernetes.io/projected/5ba753c3-5979-4263-8b90-2741747c8592-kube-api-access-8j5dq\") pod \"5ba753c3-5979-4263-8b90-2741747c8592\" (UID: \"5ba753c3-5979-4263-8b90-2741747c8592\") " Oct 30 21:00:12.022904 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:12.022826 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5ba753c3-5979-4263-8b90-2741747c8592-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ba753c3-5979-4263-8b90-2741747c8592-istiod-ca-cert\") pod \"5ba753c3-5979-4263-8b90-2741747c8592\" (UID: \"5ba753c3-5979-4263-8b90-2741747c8592\") " Oct 30 21:00:12.022904 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:12.022871 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db1a832a-0df2-4a93-bee2-40c9b99218d9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db1a832a-0df2-4a93-bee2-40c9b99218d9-workload-socket\") pod \"db1a832a-0df2-4a93-bee2-40c9b99218d9\" (UID: \"db1a832a-0df2-4a93-bee2-40c9b99218d9\") " Oct 30 21:00:12.022904 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:12.022896 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ba753c3-5979-4263-8b90-2741747c8592-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ba753c3-5979-4263-8b90-2741747c8592-credential-socket\") pod \"5ba753c3-5979-4263-8b90-2741747c8592\" (UID: \"5ba753c3-5979-4263-8b90-2741747c8592\") " Oct 30 21:00:12.023021 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:12.022948 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c468f3c6-7d73-4fd1-ae34-1212e4d1b387-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c468f3c6-7d73-4fd1-ae34-1212e4d1b387-istio-data\") pod \"c468f3c6-7d73-4fd1-ae34-1212e4d1b387\" (UID: \"c468f3c6-7d73-4fd1-ae34-1212e4d1b387\") " Oct 30 21:00:12.023021 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:12.022975 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db1a832a-0df2-4a93-bee2-40c9b99218d9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db1a832a-0df2-4a93-bee2-40c9b99218d9-istio-data\") pod \"db1a832a-0df2-4a93-bee2-40c9b99218d9\" (UID: \"db1a832a-0df2-4a93-bee2-40c9b99218d9\") " Oct 30 21:00:12.023021 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:12.023013 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ba753c3-5979-4263-8b90-2741747c8592-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ba753c3-5979-4263-8b90-2741747c8592-workload-socket\") pod \"5ba753c3-5979-4263-8b90-2741747c8592\" (UID: \"5ba753c3-5979-4263-8b90-2741747c8592\") " Oct 30 21:00:12.023108 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:12.023052 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db1a832a-0df2-4a93-bee2-40c9b99218d9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db1a832a-0df2-4a93-bee2-40c9b99218d9-credential-socket\") pod \"db1a832a-0df2-4a93-bee2-40c9b99218d9\" (UID: \"db1a832a-0df2-4a93-bee2-40c9b99218d9\") " Oct 30 21:00:12.023108 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:12.023086 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c468f3c6-7d73-4fd1-ae34-1212e4d1b387-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c468f3c6-7d73-4fd1-ae34-1212e4d1b387-workload-socket\") pod \"c468f3c6-7d73-4fd1-ae34-1212e4d1b387\" (UID: \"c468f3c6-7d73-4fd1-ae34-1212e4d1b387\") " Oct 30 21:00:12.023465 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:12.023437 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db1a832a-0df2-4a93-bee2-40c9b99218d9-workload-certs" pod "db1a832a-0df2-4a93-bee2-40c9b99218d9" (UID: "db1a832a-0df2-4a93-bee2-40c9b99218d9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:00:12.024483 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:12.023699 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db1a832a-0df2-4a93-bee2-40c9b99218d9-workload-socket" pod "db1a832a-0df2-4a93-bee2-40c9b99218d9" (UID: "db1a832a-0df2-4a93-bee2-40c9b99218d9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:00:12.024483 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:12.023853 4611 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a93b27f-f269-471c-8be2-523f373bca08-istio-podinfo\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:00:12.024483 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:12.023878 4611 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a93b27f-f269-471c-8be2-523f373bca08-istio-token\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:00:12.024483 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:12.023879 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c468f3c6-7d73-4fd1-ae34-1212e4d1b387-workload-socket" pod "c468f3c6-7d73-4fd1-ae34-1212e4d1b387" (UID: "c468f3c6-7d73-4fd1-ae34-1212e4d1b387"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:00:12.024483 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:12.023895 4611 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zdj5h\" (UniqueName: \"kubernetes.io/projected/5a93b27f-f269-471c-8be2-523f373bca08-kube-api-access-zdj5h\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:00:12.024483 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:12.023910 4611 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db1a832a-0df2-4a93-bee2-40c9b99218d9-workload-certs\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:00:12.024483 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:12.023924 4611 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a93b27f-f269-471c-8be2-523f373bca08-istio-envoy\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:00:12.024483 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:12.023937 4611 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db1a832a-0df2-4a93-bee2-40c9b99218d9-workload-socket\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:00:12.024483 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:12.024003 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ba753c3-5979-4263-8b90-2741747c8592-istio-data" pod "5ba753c3-5979-4263-8b90-2741747c8592" (UID: "5ba753c3-5979-4263-8b90-2741747c8592"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:00:12.024483 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:12.024024 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db1a832a-0df2-4a93-bee2-40c9b99218d9-istiod-ca-cert" pod "db1a832a-0df2-4a93-bee2-40c9b99218d9" (UID: "db1a832a-0df2-4a93-bee2-40c9b99218d9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:00:12.024483 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:12.024108 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db1a832a-0df2-4a93-bee2-40c9b99218d9-istio-data" pod "db1a832a-0df2-4a93-bee2-40c9b99218d9" (UID: "db1a832a-0df2-4a93-bee2-40c9b99218d9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:00:12.024483 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:12.024119 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ba753c3-5979-4263-8b90-2741747c8592-credential-socket" pod "5ba753c3-5979-4263-8b90-2741747c8592" (UID: "5ba753c3-5979-4263-8b90-2741747c8592"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:00:12.024483 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:12.024237 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c468f3c6-7d73-4fd1-ae34-1212e4d1b387-workload-certs" pod "c468f3c6-7d73-4fd1-ae34-1212e4d1b387" (UID: "c468f3c6-7d73-4fd1-ae34-1212e4d1b387"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:00:12.024483 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:12.024374 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c468f3c6-7d73-4fd1-ae34-1212e4d1b387-istiod-ca-cert" pod "c468f3c6-7d73-4fd1-ae34-1212e4d1b387" (UID: "c468f3c6-7d73-4fd1-ae34-1212e4d1b387"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:00:12.025432 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:12.024622 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c468f3c6-7d73-4fd1-ae34-1212e4d1b387-credential-socket" pod "c468f3c6-7d73-4fd1-ae34-1212e4d1b387" (UID: "c468f3c6-7d73-4fd1-ae34-1212e4d1b387"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:00:12.025432 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:12.024905 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ba753c3-5979-4263-8b90-2741747c8592-workload-socket" pod "5ba753c3-5979-4263-8b90-2741747c8592" (UID: "5ba753c3-5979-4263-8b90-2741747c8592"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:00:12.025432 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:12.025145 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db1a832a-0df2-4a93-bee2-40c9b99218d9-credential-socket" pod "db1a832a-0df2-4a93-bee2-40c9b99218d9" (UID: "db1a832a-0df2-4a93-bee2-40c9b99218d9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:00:12.026416 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:12.026386 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c468f3c6-7d73-4fd1-ae34-1212e4d1b387-kube-api-access-mm94t" pod "c468f3c6-7d73-4fd1-ae34-1212e4d1b387" (UID: "c468f3c6-7d73-4fd1-ae34-1212e4d1b387"). InnerVolumeSpecName "kube-api-access-mm94t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:00:12.027807 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:12.027544 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ba753c3-5979-4263-8b90-2741747c8592-workload-certs" pod "5ba753c3-5979-4263-8b90-2741747c8592" (UID: "5ba753c3-5979-4263-8b90-2741747c8592"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:00:12.028214 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:12.028186 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5ba753c3-5979-4263-8b90-2741747c8592-istio-podinfo" pod "5ba753c3-5979-4263-8b90-2741747c8592" (UID: "5ba753c3-5979-4263-8b90-2741747c8592"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:00:12.028335 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:12.028313 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c468f3c6-7d73-4fd1-ae34-1212e4d1b387-istio-envoy" pod "c468f3c6-7d73-4fd1-ae34-1212e4d1b387" (UID: "c468f3c6-7d73-4fd1-ae34-1212e4d1b387"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:00:12.028335 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:12.023449 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c468f3c6-7d73-4fd1-ae34-1212e4d1b387-istio-data" pod "c468f3c6-7d73-4fd1-ae34-1212e4d1b387" (UID: "c468f3c6-7d73-4fd1-ae34-1212e4d1b387"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:00:12.028466 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:12.023581 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ba753c3-5979-4263-8b90-2741747c8592-istiod-ca-cert" pod "5ba753c3-5979-4263-8b90-2741747c8592" (UID: "5ba753c3-5979-4263-8b90-2741747c8592"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:00:12.029175 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:12.028967 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/db1a832a-0df2-4a93-bee2-40c9b99218d9-istio-podinfo" pod "db1a832a-0df2-4a93-bee2-40c9b99218d9" (UID: "db1a832a-0df2-4a93-bee2-40c9b99218d9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:00:12.029175 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:12.029059 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ba753c3-5979-4263-8b90-2741747c8592-kube-api-access-8j5dq" pod "5ba753c3-5979-4263-8b90-2741747c8592" (UID: "5ba753c3-5979-4263-8b90-2741747c8592"). InnerVolumeSpecName "kube-api-access-8j5dq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:00:12.029175 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:12.029135 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db1a832a-0df2-4a93-bee2-40c9b99218d9-kube-api-access-mnf2z" pod "db1a832a-0df2-4a93-bee2-40c9b99218d9" (UID: "db1a832a-0df2-4a93-bee2-40c9b99218d9"). InnerVolumeSpecName "kube-api-access-mnf2z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:00:12.029401 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:12.029234 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c468f3c6-7d73-4fd1-ae34-1212e4d1b387-istio-podinfo" pod "c468f3c6-7d73-4fd1-ae34-1212e4d1b387" (UID: "c468f3c6-7d73-4fd1-ae34-1212e4d1b387"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:00:12.030081 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:12.029992 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db1a832a-0df2-4a93-bee2-40c9b99218d9-istio-envoy" pod "db1a832a-0df2-4a93-bee2-40c9b99218d9" (UID: "db1a832a-0df2-4a93-bee2-40c9b99218d9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:00:12.030081 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:12.030002 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db1a832a-0df2-4a93-bee2-40c9b99218d9-istio-token" pod "db1a832a-0df2-4a93-bee2-40c9b99218d9" (UID: "db1a832a-0df2-4a93-bee2-40c9b99218d9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:00:12.030081 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:12.030059 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ba753c3-5979-4263-8b90-2741747c8592-istio-envoy" pod "5ba753c3-5979-4263-8b90-2741747c8592" (UID: "5ba753c3-5979-4263-8b90-2741747c8592"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:00:12.030338 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:12.030311 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c468f3c6-7d73-4fd1-ae34-1212e4d1b387-istio-token" pod "c468f3c6-7d73-4fd1-ae34-1212e4d1b387" (UID: "c468f3c6-7d73-4fd1-ae34-1212e4d1b387"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:00:12.030602 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:12.030575 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ba753c3-5979-4263-8b90-2741747c8592-istio-token" pod "5ba753c3-5979-4263-8b90-2741747c8592" (UID: "5ba753c3-5979-4263-8b90-2741747c8592"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:00:12.125369 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:12.125335 4611 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ba753c3-5979-4263-8b90-2741747c8592-istio-envoy\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:00:12.125369 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:12.125365 4611 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ba753c3-5979-4263-8b90-2741747c8592-istio-podinfo\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:00:12.125369 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:12.125375 4611 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db1a832a-0df2-4a93-bee2-40c9b99218d9-istiod-ca-cert\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:00:12.125591 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:12.125385 4611 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c468f3c6-7d73-4fd1-ae34-1212e4d1b387-istiod-ca-cert\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:00:12.125591 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:12.125394 4611 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8j5dq\" (UniqueName: \"kubernetes.io/projected/5ba753c3-5979-4263-8b90-2741747c8592-kube-api-access-8j5dq\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:00:12.125591 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:12.125405 4611 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ba753c3-5979-4263-8b90-2741747c8592-istiod-ca-cert\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:00:12.125591 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:12.125413 4611 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ba753c3-5979-4263-8b90-2741747c8592-credential-socket\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:00:12.125591 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:12.125424 4611 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c468f3c6-7d73-4fd1-ae34-1212e4d1b387-istio-data\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:00:12.125591 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:12.125433 4611 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db1a832a-0df2-4a93-bee2-40c9b99218d9-istio-data\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:00:12.125591 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:12.125441 4611 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ba753c3-5979-4263-8b90-2741747c8592-workload-socket\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:00:12.125591 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:12.125450 4611 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db1a832a-0df2-4a93-bee2-40c9b99218d9-credential-socket\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:00:12.125591 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:12.125459 4611 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c468f3c6-7d73-4fd1-ae34-1212e4d1b387-workload-socket\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:00:12.125591 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:12.125467 4611 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db1a832a-0df2-4a93-bee2-40c9b99218d9-istio-podinfo\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:00:12.125591 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:12.125474 4611 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ba753c3-5979-4263-8b90-2741747c8592-workload-certs\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:00:12.125591 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:12.125483 4611 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db1a832a-0df2-4a93-bee2-40c9b99218d9-istio-envoy\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:00:12.125591 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:12.125491 4611 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c468f3c6-7d73-4fd1-ae34-1212e4d1b387-istio-token\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:00:12.125591 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:12.125500 4611 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ba753c3-5979-4263-8b90-2741747c8592-istio-token\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:00:12.125591 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:12.125521 4611 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c468f3c6-7d73-4fd1-ae34-1212e4d1b387-credential-socket\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:00:12.125591 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:12.125529 4611 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c468f3c6-7d73-4fd1-ae34-1212e4d1b387-workload-certs\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:00:12.125591 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:12.125537 4611 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mnf2z\" (UniqueName: \"kubernetes.io/projected/db1a832a-0df2-4a93-bee2-40c9b99218d9-kube-api-access-mnf2z\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:00:12.125591 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:12.125547 4611 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ba753c3-5979-4263-8b90-2741747c8592-istio-data\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:00:12.125591 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:12.125555 4611 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mm94t\" (UniqueName: \"kubernetes.io/projected/c468f3c6-7d73-4fd1-ae34-1212e4d1b387-kube-api-access-mm94t\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:00:12.125591 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:12.125563 4611 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c468f3c6-7d73-4fd1-ae34-1212e4d1b387-istio-podinfo\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:00:12.125591 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:12.125583 4611 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db1a832a-0df2-4a93-bee2-40c9b99218d9-istio-token\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:00:12.125591 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:12.125591 4611 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c468f3c6-7d73-4fd1-ae34-1212e4d1b387-istio-envoy\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:00:12.164479 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:12.164442 4611 generic.go:356] "Generic (PLEG): container finished" podID="5a93b27f-f269-471c-8be2-523f373bca08" containerID="f8d503a52fd7f6a1f40bb16219178d4eadb387455d394cd382f26d0f645f0dd5" exitCode=0 Oct 30 21:00:12.164899 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:12.164768 4611 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-ztxdx" Oct 30 21:00:12.171646 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:12.164791 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-ztxdx" event={"ID":"5a93b27f-f269-471c-8be2-523f373bca08","Type":"ContainerDied","Data":"f8d503a52fd7f6a1f40bb16219178d4eadb387455d394cd382f26d0f645f0dd5"} Oct 30 21:00:12.171890 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:12.171865 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-ztxdx" event={"ID":"5a93b27f-f269-471c-8be2-523f373bca08","Type":"ContainerDied","Data":"0c597908494eba6e21902a88e018b44f15bb5d1a9da340e64c5d0fd2257671ff"} Oct 30 21:00:12.172029 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:12.172013 4611 scope.go:122] "RemoveContainer" containerID="f8d503a52fd7f6a1f40bb16219178d4eadb387455d394cd382f26d0f645f0dd5" Oct 30 21:00:12.181425 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:12.181401 4611 generic.go:356] "Generic (PLEG): container finished" podID="5ba753c3-5979-4263-8b90-2741747c8592" containerID="c10f965c9d3fd5d620ad061d397c593e1191596603f3b4b5f8242467c8cab5b5" exitCode=0 Oct 30 21:00:12.181579 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:12.181469 4611 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-trngw" Oct 30 21:00:12.181641 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:12.181474 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-trngw" event={"ID":"5ba753c3-5979-4263-8b90-2741747c8592","Type":"ContainerDied","Data":"c10f965c9d3fd5d620ad061d397c593e1191596603f3b4b5f8242467c8cab5b5"} Oct 30 21:00:12.181696 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:12.181640 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-trngw" event={"ID":"5ba753c3-5979-4263-8b90-2741747c8592","Type":"ContainerDied","Data":"069cdb2ff8dc5594c013082f64c626b322e05a04a2d51b61ceb128f5b1fdaeb1"} Oct 30 21:00:12.183379 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:12.183352 4611 generic.go:356] "Generic (PLEG): container finished" podID="c468f3c6-7d73-4fd1-ae34-1212e4d1b387" containerID="1a52628c1464f529ef397549f13e488f2d96482f8fda373851190beef6458016" exitCode=0 Oct 30 21:00:12.183482 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:12.183394 4611 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-wf8qg" Oct 30 21:00:12.183577 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:12.183541 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-wf8qg" event={"ID":"c468f3c6-7d73-4fd1-ae34-1212e4d1b387","Type":"ContainerDied","Data":"1a52628c1464f529ef397549f13e488f2d96482f8fda373851190beef6458016"} Oct 30 21:00:12.183644 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:12.183582 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-wf8qg" event={"ID":"c468f3c6-7d73-4fd1-ae34-1212e4d1b387","Type":"ContainerDied","Data":"7a15a8bf690a62f4bda18a6c63b9fa6998ac704d367ccd92af87e66dd6ebc002"} Oct 30 21:00:12.186207 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:12.186187 4611 generic.go:356] "Generic (PLEG): container finished" podID="db1a832a-0df2-4a93-bee2-40c9b99218d9" containerID="00b4462f302792be026e0808d35c51a456553937259897e483726e2dfb2fbed4" exitCode=0 Oct 30 21:00:12.186292 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:12.186247 4611 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-qkwks" Oct 30 21:00:12.186292 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:12.186274 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-qkwks" event={"ID":"db1a832a-0df2-4a93-bee2-40c9b99218d9","Type":"ContainerDied","Data":"00b4462f302792be026e0808d35c51a456553937259897e483726e2dfb2fbed4"} Oct 30 21:00:12.186415 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:12.186324 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-qkwks" event={"ID":"db1a832a-0df2-4a93-bee2-40c9b99218d9","Type":"ContainerDied","Data":"0e26a9f9931834fb2fabf8784ba3ff1ca5f57fc3f16abe5e2eda23b9a3d2d0d3"} Oct 30 21:00:12.193567 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:12.193550 4611 scope.go:122] "RemoveContainer" containerID="f8d503a52fd7f6a1f40bb16219178d4eadb387455d394cd382f26d0f645f0dd5" Oct 30 21:00:12.194288 ip-10-0-133-192 kubenswrapper[4611]: E1030 21:00:12.193886 4611 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f8d503a52fd7f6a1f40bb16219178d4eadb387455d394cd382f26d0f645f0dd5\": container with ID starting with f8d503a52fd7f6a1f40bb16219178d4eadb387455d394cd382f26d0f645f0dd5 not found: ID does not exist" containerID="f8d503a52fd7f6a1f40bb16219178d4eadb387455d394cd382f26d0f645f0dd5" Oct 30 21:00:12.194288 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:12.193930 4611 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8d503a52fd7f6a1f40bb16219178d4eadb387455d394cd382f26d0f645f0dd5"} err="failed to get container status \"f8d503a52fd7f6a1f40bb16219178d4eadb387455d394cd382f26d0f645f0dd5\": rpc error: code = NotFound desc = could not find container \"f8d503a52fd7f6a1f40bb16219178d4eadb387455d394cd382f26d0f645f0dd5\": container with ID starting with f8d503a52fd7f6a1f40bb16219178d4eadb387455d394cd382f26d0f645f0dd5 not found: ID does not exist" Oct 30 21:00:12.194288 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:12.193958 4611 scope.go:122] "RemoveContainer" containerID="c10f965c9d3fd5d620ad061d397c593e1191596603f3b4b5f8242467c8cab5b5" Oct 30 21:00:12.204096 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:12.204080 4611 scope.go:122] "RemoveContainer" containerID="1aa4d02a6e2697b5e2510958ad678aafedd30c89dd21b88538db4d20e053fe11" Oct 30 21:00:12.207925 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:12.207901 4611 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-ztxdx"] Oct 30 21:00:12.211310 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:12.211287 4611 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-ztxdx"] Oct 30 21:00:12.213727 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:12.213711 4611 scope.go:122] "RemoveContainer" containerID="c10f965c9d3fd5d620ad061d397c593e1191596603f3b4b5f8242467c8cab5b5" Oct 30 21:00:12.213969 ip-10-0-133-192 kubenswrapper[4611]: E1030 21:00:12.213949 4611 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c10f965c9d3fd5d620ad061d397c593e1191596603f3b4b5f8242467c8cab5b5\": container with ID starting with c10f965c9d3fd5d620ad061d397c593e1191596603f3b4b5f8242467c8cab5b5 not found: ID does not exist" containerID="c10f965c9d3fd5d620ad061d397c593e1191596603f3b4b5f8242467c8cab5b5" Oct 30 21:00:12.214032 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:12.213976 4611 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c10f965c9d3fd5d620ad061d397c593e1191596603f3b4b5f8242467c8cab5b5"} err="failed to get container status \"c10f965c9d3fd5d620ad061d397c593e1191596603f3b4b5f8242467c8cab5b5\": rpc error: code = NotFound desc = could not find container \"c10f965c9d3fd5d620ad061d397c593e1191596603f3b4b5f8242467c8cab5b5\": container with ID starting with c10f965c9d3fd5d620ad061d397c593e1191596603f3b4b5f8242467c8cab5b5 not found: ID does not exist" Oct 30 21:00:12.214032 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:12.213995 4611 scope.go:122] "RemoveContainer" containerID="1aa4d02a6e2697b5e2510958ad678aafedd30c89dd21b88538db4d20e053fe11" Oct 30 21:00:12.214290 ip-10-0-133-192 kubenswrapper[4611]: E1030 21:00:12.214271 4611 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1aa4d02a6e2697b5e2510958ad678aafedd30c89dd21b88538db4d20e053fe11\": container with ID starting with 1aa4d02a6e2697b5e2510958ad678aafedd30c89dd21b88538db4d20e053fe11 not found: ID does not exist" containerID="1aa4d02a6e2697b5e2510958ad678aafedd30c89dd21b88538db4d20e053fe11" Oct 30 21:00:12.214354 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:12.214296 4611 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1aa4d02a6e2697b5e2510958ad678aafedd30c89dd21b88538db4d20e053fe11"} err="failed to get container status \"1aa4d02a6e2697b5e2510958ad678aafedd30c89dd21b88538db4d20e053fe11\": rpc error: code = NotFound desc = could not find container \"1aa4d02a6e2697b5e2510958ad678aafedd30c89dd21b88538db4d20e053fe11\": container with ID starting with 1aa4d02a6e2697b5e2510958ad678aafedd30c89dd21b88538db4d20e053fe11 not found: ID does not exist" Oct 30 21:00:12.214354 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:12.214311 4611 scope.go:122] "RemoveContainer" containerID="1a52628c1464f529ef397549f13e488f2d96482f8fda373851190beef6458016" Oct 30 21:00:12.223379 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:12.223360 4611 scope.go:122] "RemoveContainer" containerID="1a52628c1464f529ef397549f13e488f2d96482f8fda373851190beef6458016" Oct 30 21:00:12.223641 ip-10-0-133-192 kubenswrapper[4611]: E1030 21:00:12.223620 4611 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a52628c1464f529ef397549f13e488f2d96482f8fda373851190beef6458016\": container with ID starting with 1a52628c1464f529ef397549f13e488f2d96482f8fda373851190beef6458016 not found: ID does not exist" containerID="1a52628c1464f529ef397549f13e488f2d96482f8fda373851190beef6458016" Oct 30 21:00:12.223728 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:12.223646 4611 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a52628c1464f529ef397549f13e488f2d96482f8fda373851190beef6458016"} err="failed to get container status \"1a52628c1464f529ef397549f13e488f2d96482f8fda373851190beef6458016\": rpc error: code = NotFound desc = could not find container \"1a52628c1464f529ef397549f13e488f2d96482f8fda373851190beef6458016\": container with ID starting with 1a52628c1464f529ef397549f13e488f2d96482f8fda373851190beef6458016 not found: ID does not exist" Oct 30 21:00:12.223728 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:12.223666 4611 scope.go:122] "RemoveContainer" containerID="00b4462f302792be026e0808d35c51a456553937259897e483726e2dfb2fbed4" Oct 30 21:00:12.223991 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:12.223971 4611 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-qkwks"] Oct 30 21:00:12.228843 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:12.228820 4611 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-qkwks"] Oct 30 21:00:12.234032 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:12.233998 4611 scope.go:122] "RemoveContainer" containerID="00b4462f302792be026e0808d35c51a456553937259897e483726e2dfb2fbed4" Oct 30 21:00:12.234309 ip-10-0-133-192 kubenswrapper[4611]: E1030 21:00:12.234290 4611 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"00b4462f302792be026e0808d35c51a456553937259897e483726e2dfb2fbed4\": container with ID starting with 00b4462f302792be026e0808d35c51a456553937259897e483726e2dfb2fbed4 not found: ID does not exist" containerID="00b4462f302792be026e0808d35c51a456553937259897e483726e2dfb2fbed4" Oct 30 21:00:12.234387 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:12.234314 4611 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"00b4462f302792be026e0808d35c51a456553937259897e483726e2dfb2fbed4"} err="failed to get container status \"00b4462f302792be026e0808d35c51a456553937259897e483726e2dfb2fbed4\": rpc error: code = NotFound desc = could not find container \"00b4462f302792be026e0808d35c51a456553937259897e483726e2dfb2fbed4\": container with ID starting with 00b4462f302792be026e0808d35c51a456553937259897e483726e2dfb2fbed4 not found: ID does not exist" Oct 30 21:00:12.244186 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:12.244123 4611 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-wf8qg"] Oct 30 21:00:12.272532 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:12.268047 4611 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-wf8qg"] Oct 30 21:00:12.280026 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:12.280001 4611 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-trngw"] Oct 30 21:00:12.284393 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:12.284372 4611 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-3d909d-openshift-default-77787695b7-trngw"] Oct 30 21:00:12.528501 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:12.528447 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:00:12.943736 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:12.943706 4611 patch_prober.go:29] interesting pod/e2e-mt-812bfe65-openshift-default-74647fb5f7-f85fj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body= Oct 30 21:00:12.943914 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:12.943768 4611 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-812bfe65-openshift-default-74647fb5f7-f85fj" podUID="918ff269-fb66-4479-9e8d-d0184421bc7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" Oct 30 21:00:13.022452 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:13.022418 4611 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-94228e47-nfh6n"] Oct 30 21:00:13.025318 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:13.025297 4611 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-94228e47-nfh6n"] Oct 30 21:00:13.279587 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:13.279339 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:00:13.605783 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:13.605751 4611 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5a93b27f-f269-471c-8be2-523f373bca08" path="/var/lib/kubelet/pods/5a93b27f-f269-471c-8be2-523f373bca08/volumes" Oct 30 21:00:13.606249 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:13.606235 4611 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5ba753c3-5979-4263-8b90-2741747c8592" path="/var/lib/kubelet/pods/5ba753c3-5979-4263-8b90-2741747c8592/volumes" Oct 30 21:00:13.606680 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:13.606666 4611 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c468f3c6-7d73-4fd1-ae34-1212e4d1b387" path="/var/lib/kubelet/pods/c468f3c6-7d73-4fd1-ae34-1212e4d1b387/volumes" Oct 30 21:00:13.607385 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:13.607371 4611 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="db1a832a-0df2-4a93-bee2-40c9b99218d9" path="/var/lib/kubelet/pods/db1a832a-0df2-4a93-bee2-40c9b99218d9/volumes" Oct 30 21:00:13.607809 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:13.607795 4611 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fb626b8f-b455-48d5-b436-3f0d9a2f0498" path="/var/lib/kubelet/pods/fb626b8f-b455-48d5-b436-3f0d9a2f0498/volumes" Oct 30 21:00:13.898538 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:13.898436 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:00:13.944731 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:13.944700 4611 patch_prober.go:29] interesting pod/e2e-mt-812bfe65-openshift-default-74647fb5f7-f85fj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body= Oct 30 21:00:13.944936 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:13.944754 4611 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-812bfe65-openshift-default-74647fb5f7-f85fj" podUID="918ff269-fb66-4479-9e8d-d0184421bc7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" Oct 30 21:00:14.599018 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:14.598991 4611 scope.go:122] "RemoveContainer" containerID="f1e96aa46218b846126a7aeaf252ddd043415ca79c4d92e2b5fce47ebbbd9fd9" Oct 30 21:00:14.599488 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:14.599465 4611 scope.go:122] "RemoveContainer" containerID="6235d27e5fd8f07010a0a71d511448b4b417288831b01149870de7ad1e8e340e" Oct 30 21:00:14.699353 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:14.699321 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:00:14.943652 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:14.943619 4611 patch_prober.go:29] interesting pod/e2e-mt-812bfe65-openshift-default-74647fb5f7-f85fj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body= Oct 30 21:00:14.943808 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:14.943678 4611 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-812bfe65-openshift-default-74647fb5f7-f85fj" podUID="918ff269-fb66-4479-9e8d-d0184421bc7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" Oct 30 21:00:15.212773 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:15.212688 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-f2ed1e-29889900-sctrt" event={"ID":"9ff515ee-d559-4fe8-8487-5f04e13125bc","Type":"ContainerStarted","Data":"8d35e5e3d1cb23ead2b661233e1b347645af1b2c6dc928f591aa1c0937344a22"} Oct 30 21:00:15.214781 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:15.214750 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-812bfe65-29889900-j5fzs" event={"ID":"1dd352a3-b3a3-4912-9d9f-723d40415008","Type":"ContainerStarted","Data":"eb8a2733f454d95330603a6e24be6d60933ddd18d2503a0d2ab0203d3139b1dd"} Oct 30 21:00:15.560770 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:15.556444 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:00:15.719106 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:15.719074 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:00:15.943770 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:15.943738 4611 patch_prober.go:29] interesting pod/e2e-mt-812bfe65-openshift-default-74647fb5f7-f85fj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body= Oct 30 21:00:15.943922 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:15.943800 4611 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-812bfe65-openshift-default-74647fb5f7-f85fj" podUID="918ff269-fb66-4479-9e8d-d0184421bc7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" Oct 30 21:00:16.224240 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:16.224138 4611 generic.go:356] "Generic (PLEG): container finished" podID="9ff515ee-d559-4fe8-8487-5f04e13125bc" containerID="8d35e5e3d1cb23ead2b661233e1b347645af1b2c6dc928f591aa1c0937344a22" exitCode=60 Oct 30 21:00:16.224240 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:16.224215 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-f2ed1e-29889900-sctrt" event={"ID":"9ff515ee-d559-4fe8-8487-5f04e13125bc","Type":"ContainerDied","Data":"8d35e5e3d1cb23ead2b661233e1b347645af1b2c6dc928f591aa1c0937344a22"} Oct 30 21:00:16.224475 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:16.224278 4611 scope.go:122] "RemoveContainer" containerID="f1e96aa46218b846126a7aeaf252ddd043415ca79c4d92e2b5fce47ebbbd9fd9" Oct 30 21:00:16.224689 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:16.224671 4611 scope.go:122] "RemoveContainer" containerID="8d35e5e3d1cb23ead2b661233e1b347645af1b2c6dc928f591aa1c0937344a22" Oct 30 21:00:16.224986 ip-10-0-133-192 kubenswrapper[4611]: E1030 21:00:16.224965 4611 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-f2ed1e-29889900-sctrt_odh-ai-gateway-infra(9ff515ee-d559-4fe8-8487-5f04e13125bc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-f2ed1e-29889900-sctrt" podUID="9ff515ee-d559-4fe8-8487-5f04e13125bc" Oct 30 21:00:16.227201 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:16.227177 4611 generic.go:356] "Generic (PLEG): container finished" podID="1dd352a3-b3a3-4912-9d9f-723d40415008" containerID="eb8a2733f454d95330603a6e24be6d60933ddd18d2503a0d2ab0203d3139b1dd" exitCode=60 Oct 30 21:00:16.227324 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:16.227249 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-812bfe65-29889900-j5fzs" event={"ID":"1dd352a3-b3a3-4912-9d9f-723d40415008","Type":"ContainerDied","Data":"eb8a2733f454d95330603a6e24be6d60933ddd18d2503a0d2ab0203d3139b1dd"} Oct 30 21:00:16.227612 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:16.227596 4611 scope.go:122] "RemoveContainer" containerID="eb8a2733f454d95330603a6e24be6d60933ddd18d2503a0d2ab0203d3139b1dd" Oct 30 21:00:16.227867 ip-10-0-133-192 kubenswrapper[4611]: E1030 21:00:16.227847 4611 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-mt-812bfe65-29889900-j5fzs_odh-ai-gateway-infra(1dd352a3-b3a3-4912-9d9f-723d40415008)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-812bfe65-29889900-j5fzs" podUID="1dd352a3-b3a3-4912-9d9f-723d40415008" Oct 30 21:00:16.239677 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:16.239655 4611 scope.go:122] "RemoveContainer" containerID="6235d27e5fd8f07010a0a71d511448b4b417288831b01149870de7ad1e8e340e" Oct 30 21:00:16.243750 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:16.243730 4611 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-f2ed1e-29889900-sctrt"] Oct 30 21:00:16.270647 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:16.270620 4611 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-812bfe65-29889900-j5fzs"] Oct 30 21:00:16.440214 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:16.439819 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:00:16.948881 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:16.948853 4611 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-812bfe65-openshift-default-74647fb5f7-f85fj" Oct 30 21:00:16.949267 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:16.949205 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-812bfe65-openshift-default-74647fb5f7-f85fj" Oct 30 21:00:16.949882 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:16.949861 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-812bfe65-openshift-default-74647fb5f7-f85fj" Oct 30 21:00:17.157063 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:17.157017 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:00:17.420089 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:17.420064 4611 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-812bfe65-29889900-j5fzs" Oct 30 21:00:17.422497 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:17.422479 4611 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-f2ed1e-29889900-sctrt" Oct 30 21:00:17.516339 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:17.516249 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9ff515ee-d559-4fe8-8487-5f04e13125bc-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ff515ee-d559-4fe8-8487-5f04e13125bc-maas-api-service-ca\") pod \"9ff515ee-d559-4fe8-8487-5f04e13125bc\" (UID: \"9ff515ee-d559-4fe8-8487-5f04e13125bc\") " Oct 30 21:00:17.516339 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:17.516329 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1dd352a3-b3a3-4912-9d9f-723d40415008-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1dd352a3-b3a3-4912-9d9f-723d40415008-maas-api-service-ca\") pod \"1dd352a3-b3a3-4912-9d9f-723d40415008\" (UID: \"1dd352a3-b3a3-4912-9d9f-723d40415008\") " Oct 30 21:00:17.516595 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:17.516403 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ff515ee-d559-4fe8-8487-5f04e13125bc-kube-api-access-5xfhw\" (UniqueName: \"kubernetes.io/projected/9ff515ee-d559-4fe8-8487-5f04e13125bc-kube-api-access-5xfhw\") pod \"9ff515ee-d559-4fe8-8487-5f04e13125bc\" (UID: \"9ff515ee-d559-4fe8-8487-5f04e13125bc\") " Oct 30 21:00:17.516595 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:17.516438 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1dd352a3-b3a3-4912-9d9f-723d40415008-kube-api-access-62fpq\" (UniqueName: \"kubernetes.io/projected/1dd352a3-b3a3-4912-9d9f-723d40415008-kube-api-access-62fpq\") pod \"1dd352a3-b3a3-4912-9d9f-723d40415008\" (UID: \"1dd352a3-b3a3-4912-9d9f-723d40415008\") " Oct 30 21:00:17.516708 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:17.516627 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ff515ee-d559-4fe8-8487-5f04e13125bc-maas-api-service-ca" pod "9ff515ee-d559-4fe8-8487-5f04e13125bc" (UID: "9ff515ee-d559-4fe8-8487-5f04e13125bc"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:00:17.516796 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:17.516771 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1dd352a3-b3a3-4912-9d9f-723d40415008-maas-api-service-ca" pod "1dd352a3-b3a3-4912-9d9f-723d40415008" (UID: "1dd352a3-b3a3-4912-9d9f-723d40415008"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:00:17.517033 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:17.517007 4611 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ff515ee-d559-4fe8-8487-5f04e13125bc-maas-api-service-ca\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:00:17.517033 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:17.517034 4611 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1dd352a3-b3a3-4912-9d9f-723d40415008-maas-api-service-ca\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:00:17.518709 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:17.518681 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1dd352a3-b3a3-4912-9d9f-723d40415008-kube-api-access-62fpq" pod "1dd352a3-b3a3-4912-9d9f-723d40415008" (UID: "1dd352a3-b3a3-4912-9d9f-723d40415008"). InnerVolumeSpecName "kube-api-access-62fpq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:00:17.518926 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:17.518899 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ff515ee-d559-4fe8-8487-5f04e13125bc-kube-api-access-5xfhw" pod "9ff515ee-d559-4fe8-8487-5f04e13125bc" (UID: "9ff515ee-d559-4fe8-8487-5f04e13125bc"). InnerVolumeSpecName "kube-api-access-5xfhw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:00:17.617675 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:17.617651 4611 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5xfhw\" (UniqueName: \"kubernetes.io/projected/9ff515ee-d559-4fe8-8487-5f04e13125bc-kube-api-access-5xfhw\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:00:17.617675 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:17.617674 4611 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-62fpq\" (UniqueName: \"kubernetes.io/projected/1dd352a3-b3a3-4912-9d9f-723d40415008-kube-api-access-62fpq\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:00:18.189345 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:18.189293 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:00:18.246159 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:18.246130 4611 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-812bfe65-29889900-j5fzs" Oct 30 21:00:18.246359 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:18.246166 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-812bfe65-29889900-j5fzs" event={"ID":"1dd352a3-b3a3-4912-9d9f-723d40415008","Type":"ContainerDied","Data":"2cdcee3b2cdda661c1048da9fb4994ffa98a26993f96470f6f7cb542e16804f0"} Oct 30 21:00:18.246359 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:18.246242 4611 scope.go:122] "RemoveContainer" containerID="eb8a2733f454d95330603a6e24be6d60933ddd18d2503a0d2ab0203d3139b1dd" Oct 30 21:00:18.247955 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:18.247935 4611 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-f2ed1e-29889900-sctrt" Oct 30 21:00:18.247955 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:18.247939 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-f2ed1e-29889900-sctrt" event={"ID":"9ff515ee-d559-4fe8-8487-5f04e13125bc","Type":"ContainerDied","Data":"0ee70554190138e4c77cc85319aa4e755ea27e8620ae6cf99d1b8519f38cb94c"} Oct 30 21:00:18.255993 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:18.255974 4611 scope.go:122] "RemoveContainer" containerID="8d35e5e3d1cb23ead2b661233e1b347645af1b2c6dc928f591aa1c0937344a22" Oct 30 21:00:18.270070 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:18.269255 4611 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-812bfe65-29889900-j5fzs"] Oct 30 21:00:18.277856 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:18.277832 4611 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-812bfe65-29889900-j5fzs"] Oct 30 21:00:18.292265 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:18.292241 4611 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-f2ed1e-29889900-sctrt"] Oct 30 21:00:18.301302 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:18.301276 4611 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-f2ed1e-29889900-sctrt"] Oct 30 21:00:18.607820 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:18.607784 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:00:18.760124 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:18.760092 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:00:18.913593 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:18.912917 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:00:19.063412 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:19.063370 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:00:19.606322 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:19.606291 4611 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1dd352a3-b3a3-4912-9d9f-723d40415008" path="/var/lib/kubelet/pods/1dd352a3-b3a3-4912-9d9f-723d40415008/volumes" Oct 30 21:00:19.606769 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:19.606753 4611 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9ff515ee-d559-4fe8-8487-5f04e13125bc" path="/var/lib/kubelet/pods/9ff515ee-d559-4fe8-8487-5f04e13125bc/volumes" Oct 30 21:00:19.717255 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:19.717220 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:00:20.321563 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:20.321527 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:00:20.648495 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:20.648465 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:00:21.210612 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:21.210475 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:00:21.908575 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:21.908540 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:00:22.566774 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:22.566739 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:00:22.721413 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:22.721376 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:00:23.380992 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:23.380951 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:00:24.103141 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:24.101768 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:00:24.762726 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:24.762687 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:00:25.543473 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:25.543433 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:00:26.094624 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:26.094564 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:00:26.755000 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:26.754947 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:00:27.528670 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:27.528634 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:00:27.534204 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:27.534179 4611 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-b8aee1/distinct-2-w1-b8aee1-kserve-6bcdd59797-cx6hr"] Oct 30 21:00:27.539768 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:27.539742 4611 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-b8aee1/distinct-w1-b8aee1-kserve-5b857c55cd-x5b7b"] Oct 30 21:00:27.555566 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:27.553743 4611 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-b8aee1/premium-w1-b8aee1-kserve-786d575599-htc8d"] Oct 30 21:00:27.727488 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:27.727450 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:00:27.890630 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:27.890595 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:00:28.195634 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:28.195549 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:00:28.797496 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:28.792160 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:00:29.397590 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:29.397547 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:00:30.096625 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:30.096575 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:00:30.811192 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:30.811149 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:00:31.497380 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:31.497344 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:00:32.023643 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:32.023604 4611 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1ffb1833-hm2pr"] Oct 30 21:00:32.027962 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:32.027932 4611 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1ffb1833-hm2pr"] Oct 30 21:00:32.166088 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:32.166052 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:00:32.852441 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:32.852406 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:00:33.038057 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:33.038016 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:00:33.630949 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:33.630908 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:00:33.631870 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:33.631830 4611 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d22ba0fe-de2a-4ee9-b71e-68d1ee087da5" path="/var/lib/kubelet/pods/d22ba0fe-de2a-4ee9-b71e-68d1ee087da5/volumes" Oct 30 21:00:33.920434 ip-10-0-133-192 kubenswrapper[4611]: E1030 21:00:33.920343 4611 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hwzc6" containerName="extractor" Oct 30 21:00:34.058012 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:34.057972 4611 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4a943ca6-9nm9f"] Oct 30 21:00:34.058739 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:34.058713 4611 cpu_manager.go:414] "containerMap: removing container" podUID="c468f3c6-7d73-4fd1-ae34-1212e4d1b387" containerName="istio-proxy" Oct 30 21:00:34.058739 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:34.058738 4611 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c468f3c6-7d73-4fd1-ae34-1212e4d1b387" containerName="istio-proxy" Oct 30 21:00:34.058948 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:34.058756 4611 cpu_manager.go:414] "containerMap: removing container" podUID="5a93b27f-f269-471c-8be2-523f373bca08" containerName="istio-proxy" Oct 30 21:00:34.058948 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:34.058765 4611 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a93b27f-f269-471c-8be2-523f373bca08" containerName="istio-proxy" Oct 30 21:00:34.058948 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:34.058788 4611 cpu_manager.go:414] "containerMap: removing container" podUID="1dd352a3-b3a3-4912-9d9f-723d40415008" containerName="cleanup" Oct 30 21:00:34.058948 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:34.058796 4611 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1dd352a3-b3a3-4912-9d9f-723d40415008" containerName="cleanup" Oct 30 21:00:34.058948 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:34.058815 4611 cpu_manager.go:414] "containerMap: removing container" podUID="1dd352a3-b3a3-4912-9d9f-723d40415008" containerName="cleanup" Oct 30 21:00:34.058948 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:34.058822 4611 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1dd352a3-b3a3-4912-9d9f-723d40415008" containerName="cleanup" Oct 30 21:00:34.058948 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:34.058837 4611 cpu_manager.go:414] "containerMap: removing container" podUID="46885e54-e3da-4b91-aa7b-433f9b55a0da" containerName="payload-processing" Oct 30 21:00:34.058948 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:34.058845 4611 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46885e54-e3da-4b91-aa7b-433f9b55a0da" containerName="payload-processing" Oct 30 21:00:34.058948 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:34.058858 4611 cpu_manager.go:414] "containerMap: removing container" podUID="5ba753c3-5979-4263-8b90-2741747c8592" containerName="istio-proxy" Oct 30 21:00:34.058948 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:34.058866 4611 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ba753c3-5979-4263-8b90-2741747c8592" containerName="istio-proxy" Oct 30 21:00:34.058948 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:34.058885 4611 cpu_manager.go:414] "containerMap: removing container" podUID="5ba753c3-5979-4263-8b90-2741747c8592" containerName="istio-proxy" Oct 30 21:00:34.058948 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:34.058892 4611 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ba753c3-5979-4263-8b90-2741747c8592" containerName="istio-proxy" Oct 30 21:00:34.058948 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:34.058902 4611 cpu_manager.go:414] "containerMap: removing container" podUID="5d963de5-2836-4225-9120-d885791b4f71" containerName="payload-pre-processing" Oct 30 21:00:34.058948 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:34.058910 4611 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d963de5-2836-4225-9120-d885791b4f71" containerName="payload-pre-processing" Oct 30 21:00:34.058948 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:34.058921 4611 cpu_manager.go:414] "containerMap: removing container" podUID="9ff515ee-d559-4fe8-8487-5f04e13125bc" containerName="cleanup" Oct 30 21:00:34.058948 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:34.058927 4611 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ff515ee-d559-4fe8-8487-5f04e13125bc" containerName="cleanup" Oct 30 21:00:34.058948 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:34.058947 4611 cpu_manager.go:414] "containerMap: removing container" podUID="9ff515ee-d559-4fe8-8487-5f04e13125bc" containerName="cleanup" Oct 30 21:00:34.058948 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:34.058954 4611 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ff515ee-d559-4fe8-8487-5f04e13125bc" containerName="cleanup" Oct 30 21:00:34.060140 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:34.058978 4611 cpu_manager.go:414] "containerMap: removing container" podUID="db1a832a-0df2-4a93-bee2-40c9b99218d9" containerName="istio-proxy" Oct 30 21:00:34.060140 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:34.058986 4611 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db1a832a-0df2-4a93-bee2-40c9b99218d9" containerName="istio-proxy" Oct 30 21:00:34.060140 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:34.059005 4611 cpu_manager.go:414] "containerMap: removing container" podUID="9ff515ee-d559-4fe8-8487-5f04e13125bc" containerName="cleanup" Oct 30 21:00:34.060140 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:34.059012 4611 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ff515ee-d559-4fe8-8487-5f04e13125bc" containerName="cleanup" Oct 30 21:00:34.060140 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:34.059131 4611 memory_manager.go:365] "RemoveStaleState removing state" podUID="db1a832a-0df2-4a93-bee2-40c9b99218d9" containerName="istio-proxy" Oct 30 21:00:34.060140 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:34.059143 4611 memory_manager.go:365] "RemoveStaleState removing state" podUID="9ff515ee-d559-4fe8-8487-5f04e13125bc" containerName="cleanup" Oct 30 21:00:34.060140 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:34.059152 4611 memory_manager.go:365] "RemoveStaleState removing state" podUID="c468f3c6-7d73-4fd1-ae34-1212e4d1b387" containerName="istio-proxy" Oct 30 21:00:34.060140 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:34.059163 4611 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a93b27f-f269-471c-8be2-523f373bca08" containerName="istio-proxy" Oct 30 21:00:34.060140 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:34.059175 4611 memory_manager.go:365] "RemoveStaleState removing state" podUID="1dd352a3-b3a3-4912-9d9f-723d40415008" containerName="cleanup" Oct 30 21:00:34.060140 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:34.059185 4611 memory_manager.go:365] "RemoveStaleState removing state" podUID="1dd352a3-b3a3-4912-9d9f-723d40415008" containerName="cleanup" Oct 30 21:00:34.060140 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:34.059194 4611 memory_manager.go:365] "RemoveStaleState removing state" podUID="46885e54-e3da-4b91-aa7b-433f9b55a0da" containerName="payload-processing" Oct 30 21:00:34.060140 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:34.059204 4611 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ba753c3-5979-4263-8b90-2741747c8592" containerName="istio-proxy" Oct 30 21:00:34.060140 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:34.059213 4611 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ba753c3-5979-4263-8b90-2741747c8592" containerName="istio-proxy" Oct 30 21:00:34.060140 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:34.059221 4611 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d963de5-2836-4225-9120-d885791b4f71" containerName="payload-pre-processing" Oct 30 21:00:34.060140 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:34.059230 4611 memory_manager.go:365] "RemoveStaleState removing state" podUID="9ff515ee-d559-4fe8-8487-5f04e13125bc" containerName="cleanup" Oct 30 21:00:34.060140 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:34.059242 4611 memory_manager.go:365] "RemoveStaleState removing state" podUID="1dd352a3-b3a3-4912-9d9f-723d40415008" containerName="cleanup" Oct 30 21:00:34.060140 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:34.059251 4611 memory_manager.go:365] "RemoveStaleState removing state" podUID="9ff515ee-d559-4fe8-8487-5f04e13125bc" containerName="cleanup" Oct 30 21:00:34.063523 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:34.063489 4611 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a943ca6-9nm9f" Oct 30 21:00:34.079610 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:34.079585 4611 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4a943ca6-9nm9f"] Oct 30 21:00:34.098532 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:34.098122 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6vt2x\" (UniqueName: \"kubernetes.io/projected/0de46c2f-4a48-4d46-be31-cd2173b80b5f-kube-api-access-6vt2x\") pod \"maas-api-revoke-sub-4a943ca6-9nm9f\" (UID: \"0de46c2f-4a48-4d46-be31-cd2173b80b5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a943ca6-9nm9f" Oct 30 21:00:34.098532 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:34.098231 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0de46c2f-4a48-4d46-be31-cd2173b80b5f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4a943ca6-9nm9f\" (UID: \"0de46c2f-4a48-4d46-be31-cd2173b80b5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a943ca6-9nm9f" Oct 30 21:00:34.098532 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:34.098300 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0de46c2f-4a48-4d46-be31-cd2173b80b5f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4a943ca6-9nm9f\" (UID: \"0de46c2f-4a48-4d46-be31-cd2173b80b5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a943ca6-9nm9f" Oct 30 21:00:34.160987 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:34.160959 4611 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:00:34.161208 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:34.161189 4611 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7hf8r\"" Oct 30 21:00:34.199940 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:34.199873 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0de46c2f-4a48-4d46-be31-cd2173b80b5f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4a943ca6-9nm9f\" (UID: \"0de46c2f-4a48-4d46-be31-cd2173b80b5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a943ca6-9nm9f" Oct 30 21:00:34.199940 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:34.199925 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0de46c2f-4a48-4d46-be31-cd2173b80b5f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4a943ca6-9nm9f\" (UID: \"0de46c2f-4a48-4d46-be31-cd2173b80b5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a943ca6-9nm9f" Oct 30 21:00:34.200106 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:34.200065 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6vt2x\" (UniqueName: \"kubernetes.io/projected/0de46c2f-4a48-4d46-be31-cd2173b80b5f-kube-api-access-6vt2x\") pod \"maas-api-revoke-sub-4a943ca6-9nm9f\" (UID: \"0de46c2f-4a48-4d46-be31-cd2173b80b5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a943ca6-9nm9f" Oct 30 21:00:34.200536 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:34.200493 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0de46c2f-4a48-4d46-be31-cd2173b80b5f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4a943ca6-9nm9f\" (UID: \"0de46c2f-4a48-4d46-be31-cd2173b80b5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a943ca6-9nm9f" Oct 30 21:00:34.202295 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:34.202278 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0de46c2f-4a48-4d46-be31-cd2173b80b5f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4a943ca6-9nm9f\" (UID: \"0de46c2f-4a48-4d46-be31-cd2173b80b5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a943ca6-9nm9f" Oct 30 21:00:34.211330 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:34.211307 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6vt2x\" (UniqueName: \"kubernetes.io/projected/0de46c2f-4a48-4d46-be31-cd2173b80b5f-kube-api-access-6vt2x\") pod \"maas-api-revoke-sub-4a943ca6-9nm9f\" (UID: \"0de46c2f-4a48-4d46-be31-cd2173b80b5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a943ca6-9nm9f" Oct 30 21:00:34.394733 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:34.394693 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:00:34.403679 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:34.403645 4611 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a943ca6-9nm9f" Oct 30 21:00:34.692899 ip-10-0-133-192 kubenswrapper[4611]: W1030 21:00:34.692863 4611 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0de46c2f_4a48_4d46_be31_cd2173b80b5f.slice/crio-b348a79164b315ef15b9cc16d859e249cb71f5db543be71b427b29d720aebc1b WatchSource:0}: Error finding container b348a79164b315ef15b9cc16d859e249cb71f5db543be71b427b29d720aebc1b: Status 404 returned error can't find the container with id b348a79164b315ef15b9cc16d859e249cb71f5db543be71b427b29d720aebc1b Oct 30 21:00:34.693016 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:34.692899 4611 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4a943ca6-9nm9f"] Oct 30 21:00:35.071551 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:35.071396 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:00:35.362858 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:35.362822 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a943ca6-9nm9f" event={"ID":"0de46c2f-4a48-4d46-be31-cd2173b80b5f","Type":"ContainerStarted","Data":"5d2677dd32d1d7fe29ac4c776c79a23c2f1bc4c0fa976157275f365d864a23c9"} Oct 30 21:00:35.363035 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:35.362910 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a943ca6-9nm9f" event={"ID":"0de46c2f-4a48-4d46-be31-cd2173b80b5f","Type":"ContainerStarted","Data":"b348a79164b315ef15b9cc16d859e249cb71f5db543be71b427b29d720aebc1b"} Oct 30 21:00:35.384703 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:35.384647 4611 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a943ca6-9nm9f" podStartSLOduration=1.384630455 podStartE2EDuration="1.384630455s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:00:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:00:35.380752596 +0000 UTC m=+1289.914779213" watchObservedRunningTime="2026-10-30 21:00:35.384630455 +0000 UTC m=+1289.918657071" Oct 30 21:00:35.919571 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:35.919532 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:00:36.049257 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:36.049223 4611 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-453cda0b-2krm4"] Oct 30 21:00:36.053146 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:36.053116 4611 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-453cda0b-2krm4"] Oct 30 21:00:36.370358 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:36.370326 4611 generic.go:356] "Generic (PLEG): container finished" podID="0de46c2f-4a48-4d46-be31-cd2173b80b5f" containerID="5d2677dd32d1d7fe29ac4c776c79a23c2f1bc4c0fa976157275f365d864a23c9" exitCode=0 Oct 30 21:00:36.370748 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:36.370382 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a943ca6-9nm9f" event={"ID":"0de46c2f-4a48-4d46-be31-cd2173b80b5f","Type":"ContainerDied","Data":"5d2677dd32d1d7fe29ac4c776c79a23c2f1bc4c0fa976157275f365d864a23c9"} Oct 30 21:00:36.784345 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:36.784251 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:00:37.538125 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:37.538095 4611 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a943ca6-9nm9f" Oct 30 21:00:37.609726 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:37.609684 4611 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="66bfd97d-e97b-41ad-9f4c-dc55ae2cc4c3" path="/var/lib/kubelet/pods/66bfd97d-e97b-41ad-9f4c-dc55ae2cc4c3/volumes" Oct 30 21:00:37.638217 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:37.638143 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0de46c2f-4a48-4d46-be31-cd2173b80b5f-kube-api-access-6vt2x\" (UniqueName: \"kubernetes.io/projected/0de46c2f-4a48-4d46-be31-cd2173b80b5f-kube-api-access-6vt2x\") pod \"0de46c2f-4a48-4d46-be31-cd2173b80b5f\" (UID: \"0de46c2f-4a48-4d46-be31-cd2173b80b5f\") " Oct 30 21:00:37.638382 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:37.638231 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0de46c2f-4a48-4d46-be31-cd2173b80b5f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0de46c2f-4a48-4d46-be31-cd2173b80b5f-maas-api-xks-ca\") pod \"0de46c2f-4a48-4d46-be31-cd2173b80b5f\" (UID: \"0de46c2f-4a48-4d46-be31-cd2173b80b5f\") " Oct 30 21:00:37.638382 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:37.638270 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0de46c2f-4a48-4d46-be31-cd2173b80b5f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0de46c2f-4a48-4d46-be31-cd2173b80b5f-maas-api-openshift-ca\") pod \"0de46c2f-4a48-4d46-be31-cd2173b80b5f\" (UID: \"0de46c2f-4a48-4d46-be31-cd2173b80b5f\") " Oct 30 21:00:37.638711 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:37.638688 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0de46c2f-4a48-4d46-be31-cd2173b80b5f-maas-api-openshift-ca" pod "0de46c2f-4a48-4d46-be31-cd2173b80b5f" (UID: "0de46c2f-4a48-4d46-be31-cd2173b80b5f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:00:37.640466 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:37.640432 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0de46c2f-4a48-4d46-be31-cd2173b80b5f-kube-api-access-6vt2x" pod "0de46c2f-4a48-4d46-be31-cd2173b80b5f" (UID: "0de46c2f-4a48-4d46-be31-cd2173b80b5f"). InnerVolumeSpecName "kube-api-access-6vt2x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:00:37.640650 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:37.640496 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0de46c2f-4a48-4d46-be31-cd2173b80b5f-maas-api-xks-ca" pod "0de46c2f-4a48-4d46-be31-cd2173b80b5f" (UID: "0de46c2f-4a48-4d46-be31-cd2173b80b5f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:00:37.740204 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:37.740163 4611 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6vt2x\" (UniqueName: \"kubernetes.io/projected/0de46c2f-4a48-4d46-be31-cd2173b80b5f-kube-api-access-6vt2x\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:00:37.740204 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:37.740206 4611 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0de46c2f-4a48-4d46-be31-cd2173b80b5f-maas-api-xks-ca\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:00:37.740446 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:37.740224 4611 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0de46c2f-4a48-4d46-be31-cd2173b80b5f-maas-api-openshift-ca\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:00:37.747315 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:37.747285 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:00:38.154455 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:38.154420 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:00:38.386775 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:38.386749 4611 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a943ca6-9nm9f" Oct 30 21:00:38.386968 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:38.386745 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a943ca6-9nm9f" event={"ID":"0de46c2f-4a48-4d46-be31-cd2173b80b5f","Type":"ContainerDied","Data":"b348a79164b315ef15b9cc16d859e249cb71f5db543be71b427b29d720aebc1b"} Oct 30 21:00:38.386968 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:38.386874 4611 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b348a79164b315ef15b9cc16d859e249cb71f5db543be71b427b29d720aebc1b" Oct 30 21:00:38.749335 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:38.749299 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:00:39.394773 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:39.394722 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:00:39.992476 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:39.992426 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:00:40.690368 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:40.690328 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:00:41.315334 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:41.315280 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:00:41.981336 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:41.978199 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:00:42.557731 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:42.557695 4611 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-b8aee1/distinct-2-w1-b8aee1-kserve-6bcdd59797-cx6hr" podUID="286eab7a-81a7-4768-a7f6-be6b3720028a" containerName="main" containerID="cri-o://ff8f2e91da6d2bf856f2e45b8edd50279f8c3f04439f63dc404b0e50e5d54a99" gracePeriod=45 Oct 30 21:00:42.563611 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:42.563585 4611 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-b8aee1/distinct-w1-b8aee1-kserve-5b857c55cd-x5b7b" podUID="1599e588-def8-42e1-b81d-ad0aaf119818" containerName="main" containerID="cri-o://01d7ae87a3fb5dd04f1360b41c674bdf52a66813eea06d10c78c39bc5f491c60" gracePeriod=45 Oct 30 21:00:42.595908 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:42.595879 4611 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-b8aee1/premium-w1-b8aee1-kserve-786d575599-htc8d" podUID="77368afe-edc8-4fa5-8010-d321965fb38e" containerName="main" containerID="cri-o://9446b05e516e9d84fb6c0e27f5f22ca129ff843c39d4c225c6ee73c7219a0892" gracePeriod=45 Oct 30 21:00:42.691006 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:42.690960 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b8aee1/distinct-w1-b8aee1-kserve-5b857c55cd-x5b7b" podUID="1599e588-def8-42e1-b81d-ad0aaf119818" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": dial tcp 10.133.0.71:8000: connect: connection refused" Oct 30 21:00:42.692203 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:42.692156 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:00:42.866993 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:42.866961 4611 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-b8aee1/distinct-2-w1-b8aee1-kserve-6bcdd59797-cx6hr" Oct 30 21:00:42.906275 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:42.906241 4611 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-b8aee1/distinct-w1-b8aee1-kserve-5b857c55cd-x5b7b" Oct 30 21:00:42.938562 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:42.938485 4611 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-b8aee1/premium-w1-b8aee1-kserve-786d575599-htc8d" Oct 30 21:00:43.005332 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:43.005301 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/286eab7a-81a7-4768-a7f6-be6b3720028a-home\" (UniqueName: \"kubernetes.io/empty-dir/286eab7a-81a7-4768-a7f6-be6b3720028a-home\") pod \"286eab7a-81a7-4768-a7f6-be6b3720028a\" (UID: \"286eab7a-81a7-4768-a7f6-be6b3720028a\") " Oct 30 21:00:43.005564 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:43.005340 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/77368afe-edc8-4fa5-8010-d321965fb38e-kube-api-access-5wgpw\" (UniqueName: \"kubernetes.io/projected/77368afe-edc8-4fa5-8010-d321965fb38e-kube-api-access-5wgpw\") pod \"77368afe-edc8-4fa5-8010-d321965fb38e\" (UID: \"77368afe-edc8-4fa5-8010-d321965fb38e\") " Oct 30 21:00:43.005564 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:43.005360 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/77368afe-edc8-4fa5-8010-d321965fb38e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/77368afe-edc8-4fa5-8010-d321965fb38e-tmp-dir\") pod \"77368afe-edc8-4fa5-8010-d321965fb38e\" (UID: \"77368afe-edc8-4fa5-8010-d321965fb38e\") " Oct 30 21:00:43.005564 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:43.005384 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/286eab7a-81a7-4768-a7f6-be6b3720028a-tls-certs\" (UniqueName: \"kubernetes.io/secret/286eab7a-81a7-4768-a7f6-be6b3720028a-tls-certs\") pod \"286eab7a-81a7-4768-a7f6-be6b3720028a\" (UID: \"286eab7a-81a7-4768-a7f6-be6b3720028a\") " Oct 30 21:00:43.005564 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:43.005404 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/286eab7a-81a7-4768-a7f6-be6b3720028a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/286eab7a-81a7-4768-a7f6-be6b3720028a-tmp-dir\") pod \"286eab7a-81a7-4768-a7f6-be6b3720028a\" (UID: \"286eab7a-81a7-4768-a7f6-be6b3720028a\") " Oct 30 21:00:43.005564 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:43.005552 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1599e588-def8-42e1-b81d-ad0aaf119818-dshm\" (UniqueName: \"kubernetes.io/empty-dir/1599e588-def8-42e1-b81d-ad0aaf119818-dshm\") pod \"1599e588-def8-42e1-b81d-ad0aaf119818\" (UID: \"1599e588-def8-42e1-b81d-ad0aaf119818\") " Oct 30 21:00:43.005854 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:43.005596 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/77368afe-edc8-4fa5-8010-d321965fb38e-tls-certs\" (UniqueName: \"kubernetes.io/secret/77368afe-edc8-4fa5-8010-d321965fb38e-tls-certs\") pod \"77368afe-edc8-4fa5-8010-d321965fb38e\" (UID: \"77368afe-edc8-4fa5-8010-d321965fb38e\") " Oct 30 21:00:43.005854 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:43.005665 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1599e588-def8-42e1-b81d-ad0aaf119818-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1599e588-def8-42e1-b81d-ad0aaf119818-tmp-dir\") pod \"1599e588-def8-42e1-b81d-ad0aaf119818\" (UID: \"1599e588-def8-42e1-b81d-ad0aaf119818\") " Oct 30 21:00:43.005854 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:43.005693 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/286eab7a-81a7-4768-a7f6-be6b3720028a-kube-api-access-vphzb\" (UniqueName: \"kubernetes.io/projected/286eab7a-81a7-4768-a7f6-be6b3720028a-kube-api-access-vphzb\") pod \"286eab7a-81a7-4768-a7f6-be6b3720028a\" (UID: \"286eab7a-81a7-4768-a7f6-be6b3720028a\") " Oct 30 21:00:43.005854 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:43.005699 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77368afe-edc8-4fa5-8010-d321965fb38e-tmp-dir" pod "77368afe-edc8-4fa5-8010-d321965fb38e" (UID: "77368afe-edc8-4fa5-8010-d321965fb38e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:00:43.005854 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:43.005724 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/286eab7a-81a7-4768-a7f6-be6b3720028a-tmp-dir" pod "286eab7a-81a7-4768-a7f6-be6b3720028a" (UID: "286eab7a-81a7-4768-a7f6-be6b3720028a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:00:43.005854 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:43.005709 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/286eab7a-81a7-4768-a7f6-be6b3720028a-home" pod "286eab7a-81a7-4768-a7f6-be6b3720028a" (UID: "286eab7a-81a7-4768-a7f6-be6b3720028a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:00:43.005854 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:43.005726 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1599e588-def8-42e1-b81d-ad0aaf119818-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1599e588-def8-42e1-b81d-ad0aaf119818-model-cache\") pod \"1599e588-def8-42e1-b81d-ad0aaf119818\" (UID: \"1599e588-def8-42e1-b81d-ad0aaf119818\") " Oct 30 21:00:43.005854 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:43.005779 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/77368afe-edc8-4fa5-8010-d321965fb38e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/77368afe-edc8-4fa5-8010-d321965fb38e-model-cache\") pod \"77368afe-edc8-4fa5-8010-d321965fb38e\" (UID: \"77368afe-edc8-4fa5-8010-d321965fb38e\") " Oct 30 21:00:43.006278 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:43.005893 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1599e588-def8-42e1-b81d-ad0aaf119818-tmp-dir" pod "1599e588-def8-42e1-b81d-ad0aaf119818" (UID: "1599e588-def8-42e1-b81d-ad0aaf119818"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:00:43.006278 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:43.005902 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/286eab7a-81a7-4768-a7f6-be6b3720028a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/286eab7a-81a7-4768-a7f6-be6b3720028a-model-cache\") pod \"286eab7a-81a7-4768-a7f6-be6b3720028a\" (UID: \"286eab7a-81a7-4768-a7f6-be6b3720028a\") " Oct 30 21:00:43.006278 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:43.005908 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1599e588-def8-42e1-b81d-ad0aaf119818-model-cache" pod "1599e588-def8-42e1-b81d-ad0aaf119818" (UID: "1599e588-def8-42e1-b81d-ad0aaf119818"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:00:43.006278 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:43.005951 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1599e588-def8-42e1-b81d-ad0aaf119818-tls-certs\" (UniqueName: \"kubernetes.io/secret/1599e588-def8-42e1-b81d-ad0aaf119818-tls-certs\") pod \"1599e588-def8-42e1-b81d-ad0aaf119818\" (UID: \"1599e588-def8-42e1-b81d-ad0aaf119818\") " Oct 30 21:00:43.006278 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:43.006105 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77368afe-edc8-4fa5-8010-d321965fb38e-model-cache" pod "77368afe-edc8-4fa5-8010-d321965fb38e" (UID: "77368afe-edc8-4fa5-8010-d321965fb38e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:00:43.006278 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:43.006113 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/77368afe-edc8-4fa5-8010-d321965fb38e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/77368afe-edc8-4fa5-8010-d321965fb38e-dshm\") pod \"77368afe-edc8-4fa5-8010-d321965fb38e\" (UID: \"77368afe-edc8-4fa5-8010-d321965fb38e\") " Oct 30 21:00:43.006278 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:43.006125 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/286eab7a-81a7-4768-a7f6-be6b3720028a-model-cache" pod "286eab7a-81a7-4768-a7f6-be6b3720028a" (UID: "286eab7a-81a7-4768-a7f6-be6b3720028a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:00:43.006278 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:43.006172 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/286eab7a-81a7-4768-a7f6-be6b3720028a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/286eab7a-81a7-4768-a7f6-be6b3720028a-dshm\") pod \"286eab7a-81a7-4768-a7f6-be6b3720028a\" (UID: \"286eab7a-81a7-4768-a7f6-be6b3720028a\") " Oct 30 21:00:43.006278 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:43.006192 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1599e588-def8-42e1-b81d-ad0aaf119818-home\" (UniqueName: \"kubernetes.io/empty-dir/1599e588-def8-42e1-b81d-ad0aaf119818-home\") pod \"1599e588-def8-42e1-b81d-ad0aaf119818\" (UID: \"1599e588-def8-42e1-b81d-ad0aaf119818\") " Oct 30 21:00:43.006278 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:43.006242 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/77368afe-edc8-4fa5-8010-d321965fb38e-home\" (UniqueName: \"kubernetes.io/empty-dir/77368afe-edc8-4fa5-8010-d321965fb38e-home\") pod \"77368afe-edc8-4fa5-8010-d321965fb38e\" (UID: \"77368afe-edc8-4fa5-8010-d321965fb38e\") " Oct 30 21:00:43.006278 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:43.006274 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1599e588-def8-42e1-b81d-ad0aaf119818-kube-api-access-5q5dm\" (UniqueName: \"kubernetes.io/projected/1599e588-def8-42e1-b81d-ad0aaf119818-kube-api-access-5q5dm\") pod \"1599e588-def8-42e1-b81d-ad0aaf119818\" (UID: \"1599e588-def8-42e1-b81d-ad0aaf119818\") " Oct 30 21:00:43.006942 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:43.006916 4611 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/286eab7a-81a7-4768-a7f6-be6b3720028a-home\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:00:43.007004 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:43.006951 4611 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/77368afe-edc8-4fa5-8010-d321965fb38e-tmp-dir\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:00:43.007004 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:43.006967 4611 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/286eab7a-81a7-4768-a7f6-be6b3720028a-tmp-dir\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:00:43.007004 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:43.006982 4611 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1599e588-def8-42e1-b81d-ad0aaf119818-tmp-dir\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:00:43.007004 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:43.006996 4611 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1599e588-def8-42e1-b81d-ad0aaf119818-model-cache\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:00:43.007197 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:43.007010 4611 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/77368afe-edc8-4fa5-8010-d321965fb38e-model-cache\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:00:43.007197 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:43.007024 4611 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/286eab7a-81a7-4768-a7f6-be6b3720028a-model-cache\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:00:43.008344 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:43.008318 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/77368afe-edc8-4fa5-8010-d321965fb38e-tls-certs" pod "77368afe-edc8-4fa5-8010-d321965fb38e" (UID: "77368afe-edc8-4fa5-8010-d321965fb38e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:00:43.008344 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:43.008335 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77368afe-edc8-4fa5-8010-d321965fb38e-kube-api-access-5wgpw" pod "77368afe-edc8-4fa5-8010-d321965fb38e" (UID: "77368afe-edc8-4fa5-8010-d321965fb38e"). InnerVolumeSpecName "kube-api-access-5wgpw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:00:43.008617 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:43.008580 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1599e588-def8-42e1-b81d-ad0aaf119818-home" pod "1599e588-def8-42e1-b81d-ad0aaf119818" (UID: "1599e588-def8-42e1-b81d-ad0aaf119818"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:00:43.008704 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:43.008670 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/286eab7a-81a7-4768-a7f6-be6b3720028a-tls-certs" pod "286eab7a-81a7-4768-a7f6-be6b3720028a" (UID: "286eab7a-81a7-4768-a7f6-be6b3720028a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:00:43.008852 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:43.008830 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77368afe-edc8-4fa5-8010-d321965fb38e-home" pod "77368afe-edc8-4fa5-8010-d321965fb38e" (UID: "77368afe-edc8-4fa5-8010-d321965fb38e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:00:43.008947 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:43.008922 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77368afe-edc8-4fa5-8010-d321965fb38e-dshm" pod "77368afe-edc8-4fa5-8010-d321965fb38e" (UID: "77368afe-edc8-4fa5-8010-d321965fb38e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:00:43.009216 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:43.009193 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/286eab7a-81a7-4768-a7f6-be6b3720028a-kube-api-access-vphzb" pod "286eab7a-81a7-4768-a7f6-be6b3720028a" (UID: "286eab7a-81a7-4768-a7f6-be6b3720028a"). InnerVolumeSpecName "kube-api-access-vphzb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:00:43.009364 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:43.009335 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1599e588-def8-42e1-b81d-ad0aaf119818-dshm" pod "1599e588-def8-42e1-b81d-ad0aaf119818" (UID: "1599e588-def8-42e1-b81d-ad0aaf119818"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:00:43.009364 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:43.009339 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1599e588-def8-42e1-b81d-ad0aaf119818-tls-certs" pod "1599e588-def8-42e1-b81d-ad0aaf119818" (UID: "1599e588-def8-42e1-b81d-ad0aaf119818"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:00:43.009488 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:43.009462 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1599e588-def8-42e1-b81d-ad0aaf119818-kube-api-access-5q5dm" pod "1599e588-def8-42e1-b81d-ad0aaf119818" (UID: "1599e588-def8-42e1-b81d-ad0aaf119818"). InnerVolumeSpecName "kube-api-access-5q5dm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:00:43.010152 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:43.010130 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/286eab7a-81a7-4768-a7f6-be6b3720028a-dshm" pod "286eab7a-81a7-4768-a7f6-be6b3720028a" (UID: "286eab7a-81a7-4768-a7f6-be6b3720028a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:00:43.107901 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:43.107876 4611 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/286eab7a-81a7-4768-a7f6-be6b3720028a-tls-certs\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:00:43.108037 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:43.107903 4611 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1599e588-def8-42e1-b81d-ad0aaf119818-dshm\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:00:43.108037 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:43.107912 4611 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/77368afe-edc8-4fa5-8010-d321965fb38e-tls-certs\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:00:43.108037 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:43.107922 4611 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vphzb\" (UniqueName: \"kubernetes.io/projected/286eab7a-81a7-4768-a7f6-be6b3720028a-kube-api-access-vphzb\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:00:43.108037 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:43.107932 4611 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1599e588-def8-42e1-b81d-ad0aaf119818-tls-certs\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:00:43.108037 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:43.107941 4611 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/77368afe-edc8-4fa5-8010-d321965fb38e-dshm\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:00:43.108037 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:43.107949 4611 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/286eab7a-81a7-4768-a7f6-be6b3720028a-dshm\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:00:43.108037 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:43.107957 4611 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1599e588-def8-42e1-b81d-ad0aaf119818-home\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:00:43.108037 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:43.107965 4611 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/77368afe-edc8-4fa5-8010-d321965fb38e-home\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:00:43.108037 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:43.107973 4611 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5q5dm\" (UniqueName: \"kubernetes.io/projected/1599e588-def8-42e1-b81d-ad0aaf119818-kube-api-access-5q5dm\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:00:43.108037 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:43.107981 4611 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5wgpw\" (UniqueName: \"kubernetes.io/projected/77368afe-edc8-4fa5-8010-d321965fb38e-kube-api-access-5wgpw\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:00:43.425392 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:43.425308 4611 generic.go:356] "Generic (PLEG): container finished" podID="77368afe-edc8-4fa5-8010-d321965fb38e" containerID="9446b05e516e9d84fb6c0e27f5f22ca129ff843c39d4c225c6ee73c7219a0892" exitCode=0 Oct 30 21:00:43.425392 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:43.425351 4611 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-b8aee1/premium-w1-b8aee1-kserve-786d575599-htc8d" Oct 30 21:00:43.425627 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:43.425399 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-b8aee1/premium-w1-b8aee1-kserve-786d575599-htc8d" event={"ID":"77368afe-edc8-4fa5-8010-d321965fb38e","Type":"ContainerDied","Data":"9446b05e516e9d84fb6c0e27f5f22ca129ff843c39d4c225c6ee73c7219a0892"} Oct 30 21:00:43.425627 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:43.425437 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-b8aee1/premium-w1-b8aee1-kserve-786d575599-htc8d" event={"ID":"77368afe-edc8-4fa5-8010-d321965fb38e","Type":"ContainerDied","Data":"f4ae98e2cf300b0abdeedf0e8e61248b7eb03e6a8795aace301d395247412194"} Oct 30 21:00:43.425627 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:43.425459 4611 scope.go:122] "RemoveContainer" containerID="9446b05e516e9d84fb6c0e27f5f22ca129ff843c39d4c225c6ee73c7219a0892" Oct 30 21:00:43.426883 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:43.426859 4611 generic.go:356] "Generic (PLEG): container finished" podID="286eab7a-81a7-4768-a7f6-be6b3720028a" containerID="ff8f2e91da6d2bf856f2e45b8edd50279f8c3f04439f63dc404b0e50e5d54a99" exitCode=0 Oct 30 21:00:43.427001 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:43.426889 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-b8aee1/distinct-2-w1-b8aee1-kserve-6bcdd59797-cx6hr" event={"ID":"286eab7a-81a7-4768-a7f6-be6b3720028a","Type":"ContainerDied","Data":"ff8f2e91da6d2bf856f2e45b8edd50279f8c3f04439f63dc404b0e50e5d54a99"} Oct 30 21:00:43.427001 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:43.426918 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-b8aee1/distinct-2-w1-b8aee1-kserve-6bcdd59797-cx6hr" event={"ID":"286eab7a-81a7-4768-a7f6-be6b3720028a","Type":"ContainerDied","Data":"d932e2d6505e925f627b181ec40f12af3f1d3ddec68dad3d5bc0d329f0fb3adb"} Oct 30 21:00:43.427001 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:43.426918 4611 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-b8aee1/distinct-2-w1-b8aee1-kserve-6bcdd59797-cx6hr" Oct 30 21:00:43.428457 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:43.428438 4611 generic.go:356] "Generic (PLEG): container finished" podID="1599e588-def8-42e1-b81d-ad0aaf119818" containerID="01d7ae87a3fb5dd04f1360b41c674bdf52a66813eea06d10c78c39bc5f491c60" exitCode=0 Oct 30 21:00:43.428606 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:43.428470 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-b8aee1/distinct-w1-b8aee1-kserve-5b857c55cd-x5b7b" event={"ID":"1599e588-def8-42e1-b81d-ad0aaf119818","Type":"ContainerDied","Data":"01d7ae87a3fb5dd04f1360b41c674bdf52a66813eea06d10c78c39bc5f491c60"} Oct 30 21:00:43.428606 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:43.428481 4611 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-b8aee1/distinct-w1-b8aee1-kserve-5b857c55cd-x5b7b" Oct 30 21:00:43.428606 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:43.428490 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-b8aee1/distinct-w1-b8aee1-kserve-5b857c55cd-x5b7b" event={"ID":"1599e588-def8-42e1-b81d-ad0aaf119818","Type":"ContainerDied","Data":"9eb14d5ee13e6b3fc9579fc92ac26c699431277b1f570912ac5a28b4480ac055"} Oct 30 21:00:43.447386 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:43.445994 4611 scope.go:122] "RemoveContainer" containerID="9446b05e516e9d84fb6c0e27f5f22ca129ff843c39d4c225c6ee73c7219a0892" Oct 30 21:00:43.447881 ip-10-0-133-192 kubenswrapper[4611]: E1030 21:00:43.447843 4611 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9446b05e516e9d84fb6c0e27f5f22ca129ff843c39d4c225c6ee73c7219a0892\": container with ID starting with 9446b05e516e9d84fb6c0e27f5f22ca129ff843c39d4c225c6ee73c7219a0892 not found: ID does not exist" containerID="9446b05e516e9d84fb6c0e27f5f22ca129ff843c39d4c225c6ee73c7219a0892" Oct 30 21:00:43.447992 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:43.447893 4611 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9446b05e516e9d84fb6c0e27f5f22ca129ff843c39d4c225c6ee73c7219a0892"} err="failed to get container status \"9446b05e516e9d84fb6c0e27f5f22ca129ff843c39d4c225c6ee73c7219a0892\": rpc error: code = NotFound desc = could not find container \"9446b05e516e9d84fb6c0e27f5f22ca129ff843c39d4c225c6ee73c7219a0892\": container with ID starting with 9446b05e516e9d84fb6c0e27f5f22ca129ff843c39d4c225c6ee73c7219a0892 not found: ID does not exist" Oct 30 21:00:43.447992 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:43.447919 4611 scope.go:122] "RemoveContainer" containerID="ff8f2e91da6d2bf856f2e45b8edd50279f8c3f04439f63dc404b0e50e5d54a99" Oct 30 21:00:43.462694 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:43.462068 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:00:43.465273 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:43.465246 4611 scope.go:122] "RemoveContainer" containerID="ff8f2e91da6d2bf856f2e45b8edd50279f8c3f04439f63dc404b0e50e5d54a99" Oct 30 21:00:43.465640 ip-10-0-133-192 kubenswrapper[4611]: E1030 21:00:43.465616 4611 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff8f2e91da6d2bf856f2e45b8edd50279f8c3f04439f63dc404b0e50e5d54a99\": container with ID starting with ff8f2e91da6d2bf856f2e45b8edd50279f8c3f04439f63dc404b0e50e5d54a99 not found: ID does not exist" containerID="ff8f2e91da6d2bf856f2e45b8edd50279f8c3f04439f63dc404b0e50e5d54a99" Oct 30 21:00:43.465748 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:43.465648 4611 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff8f2e91da6d2bf856f2e45b8edd50279f8c3f04439f63dc404b0e50e5d54a99"} err="failed to get container status \"ff8f2e91da6d2bf856f2e45b8edd50279f8c3f04439f63dc404b0e50e5d54a99\": rpc error: code = NotFound desc = could not find container \"ff8f2e91da6d2bf856f2e45b8edd50279f8c3f04439f63dc404b0e50e5d54a99\": container with ID starting with ff8f2e91da6d2bf856f2e45b8edd50279f8c3f04439f63dc404b0e50e5d54a99 not found: ID does not exist" Oct 30 21:00:43.465748 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:43.465668 4611 scope.go:122] "RemoveContainer" containerID="01d7ae87a3fb5dd04f1360b41c674bdf52a66813eea06d10c78c39bc5f491c60" Oct 30 21:00:43.477372 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:43.477340 4611 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-b8aee1/premium-w1-b8aee1-kserve-786d575599-htc8d"] Oct 30 21:00:43.478132 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:43.478099 4611 scope.go:122] "RemoveContainer" containerID="01d7ae87a3fb5dd04f1360b41c674bdf52a66813eea06d10c78c39bc5f491c60" Oct 30 21:00:43.478913 ip-10-0-133-192 kubenswrapper[4611]: E1030 21:00:43.478836 4611 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"01d7ae87a3fb5dd04f1360b41c674bdf52a66813eea06d10c78c39bc5f491c60\": container with ID starting with 01d7ae87a3fb5dd04f1360b41c674bdf52a66813eea06d10c78c39bc5f491c60 not found: ID does not exist" containerID="01d7ae87a3fb5dd04f1360b41c674bdf52a66813eea06d10c78c39bc5f491c60" Oct 30 21:00:43.478913 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:43.478880 4611 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"01d7ae87a3fb5dd04f1360b41c674bdf52a66813eea06d10c78c39bc5f491c60"} err="failed to get container status \"01d7ae87a3fb5dd04f1360b41c674bdf52a66813eea06d10c78c39bc5f491c60\": rpc error: code = NotFound desc = could not find container \"01d7ae87a3fb5dd04f1360b41c674bdf52a66813eea06d10c78c39bc5f491c60\": container with ID starting with 01d7ae87a3fb5dd04f1360b41c674bdf52a66813eea06d10c78c39bc5f491c60 not found: ID does not exist" Oct 30 21:00:43.481389 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:43.481364 4611 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-b8aee1/premium-w1-b8aee1-kserve-786d575599-htc8d"] Oct 30 21:00:43.495594 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:43.495572 4611 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-b8aee1/distinct-2-w1-b8aee1-kserve-6bcdd59797-cx6hr"] Oct 30 21:00:43.498208 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:43.498186 4611 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-b8aee1/distinct-2-w1-b8aee1-kserve-6bcdd59797-cx6hr"] Oct 30 21:00:43.506954 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:43.506935 4611 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-b8aee1/distinct-w1-b8aee1-kserve-5b857c55cd-x5b7b"] Oct 30 21:00:43.510097 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:43.510077 4611 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-b8aee1/distinct-w1-b8aee1-kserve-5b857c55cd-x5b7b"] Oct 30 21:00:43.604905 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:43.604871 4611 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1599e588-def8-42e1-b81d-ad0aaf119818" path="/var/lib/kubelet/pods/1599e588-def8-42e1-b81d-ad0aaf119818/volumes" Oct 30 21:00:43.605470 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:43.605449 4611 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="286eab7a-81a7-4768-a7f6-be6b3720028a" path="/var/lib/kubelet/pods/286eab7a-81a7-4768-a7f6-be6b3720028a/volumes" Oct 30 21:00:43.605927 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:43.605908 4611 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="77368afe-edc8-4fa5-8010-d321965fb38e" path="/var/lib/kubelet/pods/77368afe-edc8-4fa5-8010-d321965fb38e/volumes" Oct 30 21:00:44.090550 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:44.090345 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:00:44.924079 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:44.924044 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:00:45.174018 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:45.173980 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:00:45.522407 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:45.522371 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:00:46.193466 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:46.193430 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:00:46.962889 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:46.962847 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:00:47.701162 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:47.697857 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:00:48.207184 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:48.207145 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:00:48.448971 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:48.448941 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:00:48.647633 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:48.647133 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:00:49.006585 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:49.006324 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:00:49.176531 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:49.175556 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:00:49.578168 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:49.575776 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:00:50.454541 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:50.454476 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:00:51.305673 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:51.302319 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:00:51.972555 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:51.972472 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:00:52.588303 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:52.588250 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:00:53.194369 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:53.194336 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:00:54.040929 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:54.040881 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:00:54.993343 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:54.993306 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:00:55.730849 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:55.730814 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:00:56.375689 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:56.375647 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:00:57.021935 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:57.021895 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:00:57.180826 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:57.180790 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:00:57.266816 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:57.266783 4611 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-g2gnd"] Oct 30 21:00:57.267399 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:57.267357 4611 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-g2gnd" podUID="ef59c099-cbfc-48c7-9d10-1da01281d7f9" containerName="istio-proxy" containerID="cri-o://f339e661941f927be58d34084d6e8d704e06f578b798e5d4e9a8c5e5f6bd0596" gracePeriod=30 Oct 30 21:00:57.274126 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:57.274064 4611 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-khv95"] Oct 30 21:00:57.274252 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:57.274230 4611 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-khv95" podUID="62c91622-5518-4dc2-9f08-102dd0fcfb84" containerName="istio-proxy" containerID="cri-o://e664a227e37bec15f2ce3cd3f40d2ce51bcbda595bf1acec7ad0dd7e33ef3f53" gracePeriod=30 Oct 30 21:00:57.302875 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:57.302850 4611 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-td2hl"] Oct 30 21:00:57.303031 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:57.302991 4611 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-td2hl" podUID="a08fd3b8-493c-47db-9839-462bcc598e92" containerName="istio-proxy" containerID="cri-o://c3b88cbb89aaf4136d6f91ab75097911dbefa29c114d75768b822fb9f72bb325" gracePeriod=30 Oct 30 21:00:57.305540 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:57.305483 4611 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-6c6h6"] Oct 30 21:00:57.305718 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:57.305689 4611 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-6c6h6" podUID="2042268e-1c64-4fbe-9f2a-b826113d7f30" containerName="istio-proxy" containerID="cri-o://256030229a04a2e5065cc806c24ba748f6ae5c57276bd9ec8a9fef6b19d8abe2" gracePeriod=30 Oct 30 21:00:57.364241 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:57.364206 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:00:57.815750 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:57.815717 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:00:58.499414 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:58.499380 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:00:58.630325 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:58.630293 4611 patch_prober.go:29] interesting pod/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-khv95 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 21:00:58.630480 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:58.630359 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-khv95" podUID="62c91622-5518-4dc2-9f08-102dd0fcfb84" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 21:00:58.658407 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:58.658382 4611 patch_prober.go:29] interesting pod/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-6c6h6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 21:00:58.658608 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:58.658437 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-6c6h6" podUID="2042268e-1c64-4fbe-9f2a-b826113d7f30" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 21:00:59.265849 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:59.265810 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:00:59.891917 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:00:59.891759 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:01:00.686048 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:01:00.685999 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:01:01.307359 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:01:01.307323 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:01:02.082731 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:01:02.082689 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:01:02.574393 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:01:02.574367 4611 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-khv95" Oct 30 21:01:02.578345 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:01:02.578317 4611 generic.go:356] "Generic (PLEG): container finished" podID="2042268e-1c64-4fbe-9f2a-b826113d7f30" containerID="256030229a04a2e5065cc806c24ba748f6ae5c57276bd9ec8a9fef6b19d8abe2" exitCode=0 Oct 30 21:01:02.578475 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:01:02.578391 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-6c6h6" event={"ID":"2042268e-1c64-4fbe-9f2a-b826113d7f30","Type":"ContainerDied","Data":"256030229a04a2e5065cc806c24ba748f6ae5c57276bd9ec8a9fef6b19d8abe2"} Oct 30 21:01:02.580714 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:01:02.580594 4611 generic.go:356] "Generic (PLEG): container finished" podID="62c91622-5518-4dc2-9f08-102dd0fcfb84" containerID="e664a227e37bec15f2ce3cd3f40d2ce51bcbda595bf1acec7ad0dd7e33ef3f53" exitCode=0 Oct 30 21:01:02.580714 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:01:02.580645 4611 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-khv95" Oct 30 21:01:02.580916 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:01:02.580714 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-khv95" event={"ID":"62c91622-5518-4dc2-9f08-102dd0fcfb84","Type":"ContainerDied","Data":"e664a227e37bec15f2ce3cd3f40d2ce51bcbda595bf1acec7ad0dd7e33ef3f53"} Oct 30 21:01:02.580916 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:01:02.580770 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-khv95" event={"ID":"62c91622-5518-4dc2-9f08-102dd0fcfb84","Type":"ContainerDied","Data":"cdbad39e17b4333ea11936107416c37b744e57792aedc37913d55af8ac11907e"} Oct 30 21:01:02.580916 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:01:02.580801 4611 scope.go:122] "RemoveContainer" containerID="e664a227e37bec15f2ce3cd3f40d2ce51bcbda595bf1acec7ad0dd7e33ef3f53" Oct 30 21:01:02.583932 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:01:02.583910 4611 generic.go:356] "Generic (PLEG): container finished" podID="a08fd3b8-493c-47db-9839-462bcc598e92" containerID="c3b88cbb89aaf4136d6f91ab75097911dbefa29c114d75768b822fb9f72bb325" exitCode=0 Oct 30 21:01:02.584078 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:01:02.584058 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-td2hl" event={"ID":"a08fd3b8-493c-47db-9839-462bcc598e92","Type":"ContainerDied","Data":"c3b88cbb89aaf4136d6f91ab75097911dbefa29c114d75768b822fb9f72bb325"} Oct 30 21:01:02.585882 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:01:02.585854 4611 generic.go:356] "Generic (PLEG): container finished" podID="ef59c099-cbfc-48c7-9d10-1da01281d7f9" containerID="f339e661941f927be58d34084d6e8d704e06f578b798e5d4e9a8c5e5f6bd0596" exitCode=0 Oct 30 21:01:02.585999 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:01:02.585903 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-g2gnd" event={"ID":"ef59c099-cbfc-48c7-9d10-1da01281d7f9","Type":"ContainerDied","Data":"f339e661941f927be58d34084d6e8d704e06f578b798e5d4e9a8c5e5f6bd0596"} Oct 30 21:01:02.593753 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:01:02.593732 4611 scope.go:122] "RemoveContainer" containerID="82498596266e320ce3c9b12b22ae7f60164022ccb0b0900e9a013fe7bf58da19" Oct 30 21:01:02.594358 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:01:02.594340 4611 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-g2gnd" Oct 30 21:01:02.608301 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:01:02.608279 4611 scope.go:122] "RemoveContainer" containerID="e664a227e37bec15f2ce3cd3f40d2ce51bcbda595bf1acec7ad0dd7e33ef3f53" Oct 30 21:01:02.608664 ip-10-0-133-192 kubenswrapper[4611]: E1030 21:01:02.608640 4611 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e664a227e37bec15f2ce3cd3f40d2ce51bcbda595bf1acec7ad0dd7e33ef3f53\": container with ID starting with e664a227e37bec15f2ce3cd3f40d2ce51bcbda595bf1acec7ad0dd7e33ef3f53 not found: ID does not exist" containerID="e664a227e37bec15f2ce3cd3f40d2ce51bcbda595bf1acec7ad0dd7e33ef3f53" Oct 30 21:01:02.608786 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:01:02.608673 4611 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e664a227e37bec15f2ce3cd3f40d2ce51bcbda595bf1acec7ad0dd7e33ef3f53"} err="failed to get container status \"e664a227e37bec15f2ce3cd3f40d2ce51bcbda595bf1acec7ad0dd7e33ef3f53\": rpc error: code = NotFound desc = could not find container \"e664a227e37bec15f2ce3cd3f40d2ce51bcbda595bf1acec7ad0dd7e33ef3f53\": container with ID starting with e664a227e37bec15f2ce3cd3f40d2ce51bcbda595bf1acec7ad0dd7e33ef3f53 not found: ID does not exist" Oct 30 21:01:02.608786 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:01:02.608697 4611 scope.go:122] "RemoveContainer" containerID="82498596266e320ce3c9b12b22ae7f60164022ccb0b0900e9a013fe7bf58da19" Oct 30 21:01:02.608987 ip-10-0-133-192 kubenswrapper[4611]: E1030 21:01:02.608968 4611 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"82498596266e320ce3c9b12b22ae7f60164022ccb0b0900e9a013fe7bf58da19\": container with ID starting with 82498596266e320ce3c9b12b22ae7f60164022ccb0b0900e9a013fe7bf58da19 not found: ID does not exist" containerID="82498596266e320ce3c9b12b22ae7f60164022ccb0b0900e9a013fe7bf58da19" Oct 30 21:01:02.609062 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:01:02.608996 4611 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82498596266e320ce3c9b12b22ae7f60164022ccb0b0900e9a013fe7bf58da19"} err="failed to get container status \"82498596266e320ce3c9b12b22ae7f60164022ccb0b0900e9a013fe7bf58da19\": rpc error: code = NotFound desc = could not find container \"82498596266e320ce3c9b12b22ae7f60164022ccb0b0900e9a013fe7bf58da19\": container with ID starting with 82498596266e320ce3c9b12b22ae7f60164022ccb0b0900e9a013fe7bf58da19 not found: ID does not exist" Oct 30 21:01:02.648232 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:01:02.648202 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef59c099-cbfc-48c7-9d10-1da01281d7f9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef59c099-cbfc-48c7-9d10-1da01281d7f9-credential-socket\") pod \"ef59c099-cbfc-48c7-9d10-1da01281d7f9\" (UID: \"ef59c099-cbfc-48c7-9d10-1da01281d7f9\") " Oct 30 21:01:02.648232 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:01:02.648243 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62c91622-5518-4dc2-9f08-102dd0fcfb84-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62c91622-5518-4dc2-9f08-102dd0fcfb84-workload-certs\") pod \"62c91622-5518-4dc2-9f08-102dd0fcfb84\" (UID: \"62c91622-5518-4dc2-9f08-102dd0fcfb84\") " Oct 30 21:01:02.648467 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:01:02.648317 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef59c099-cbfc-48c7-9d10-1da01281d7f9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef59c099-cbfc-48c7-9d10-1da01281d7f9-istio-data\") pod \"ef59c099-cbfc-48c7-9d10-1da01281d7f9\" (UID: \"ef59c099-cbfc-48c7-9d10-1da01281d7f9\") " Oct 30 21:01:02.648467 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:01:02.648382 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/62c91622-5518-4dc2-9f08-102dd0fcfb84-istio-token\" (UniqueName: \"kubernetes.io/projected/62c91622-5518-4dc2-9f08-102dd0fcfb84-istio-token\") pod \"62c91622-5518-4dc2-9f08-102dd0fcfb84\" (UID: \"62c91622-5518-4dc2-9f08-102dd0fcfb84\") " Oct 30 21:01:02.648467 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:01:02.648433 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef59c099-cbfc-48c7-9d10-1da01281d7f9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef59c099-cbfc-48c7-9d10-1da01281d7f9-workload-certs\") pod \"ef59c099-cbfc-48c7-9d10-1da01281d7f9\" (UID: \"ef59c099-cbfc-48c7-9d10-1da01281d7f9\") " Oct 30 21:01:02.648467 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:01:02.648464 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62c91622-5518-4dc2-9f08-102dd0fcfb84-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62c91622-5518-4dc2-9f08-102dd0fcfb84-workload-socket\") pod \"62c91622-5518-4dc2-9f08-102dd0fcfb84\" (UID: \"62c91622-5518-4dc2-9f08-102dd0fcfb84\") " Oct 30 21:01:02.648755 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:01:02.648532 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ef59c099-cbfc-48c7-9d10-1da01281d7f9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef59c099-cbfc-48c7-9d10-1da01281d7f9-istio-podinfo\") pod \"ef59c099-cbfc-48c7-9d10-1da01281d7f9\" (UID: \"ef59c099-cbfc-48c7-9d10-1da01281d7f9\") " Oct 30 21:01:02.648755 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:01:02.648534 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef59c099-cbfc-48c7-9d10-1da01281d7f9-credential-socket" pod "ef59c099-cbfc-48c7-9d10-1da01281d7f9" (UID: "ef59c099-cbfc-48c7-9d10-1da01281d7f9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:01:02.648755 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:01:02.648568 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ef59c099-cbfc-48c7-9d10-1da01281d7f9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef59c099-cbfc-48c7-9d10-1da01281d7f9-istiod-ca-cert\") pod \"ef59c099-cbfc-48c7-9d10-1da01281d7f9\" (UID: \"ef59c099-cbfc-48c7-9d10-1da01281d7f9\") " Oct 30 21:01:02.648755 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:01:02.648609 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62c91622-5518-4dc2-9f08-102dd0fcfb84-workload-certs" pod "62c91622-5518-4dc2-9f08-102dd0fcfb84" (UID: "62c91622-5518-4dc2-9f08-102dd0fcfb84"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:01:02.648755 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:01:02.648636 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef59c099-cbfc-48c7-9d10-1da01281d7f9-istio-token\" (UniqueName: \"kubernetes.io/projected/ef59c099-cbfc-48c7-9d10-1da01281d7f9-istio-token\") pod \"ef59c099-cbfc-48c7-9d10-1da01281d7f9\" (UID: \"ef59c099-cbfc-48c7-9d10-1da01281d7f9\") " Oct 30 21:01:02.649093 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:01:02.648925 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef59c099-cbfc-48c7-9d10-1da01281d7f9-workload-certs" pod "ef59c099-cbfc-48c7-9d10-1da01281d7f9" (UID: "ef59c099-cbfc-48c7-9d10-1da01281d7f9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:01:02.649156 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:01:02.649115 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef59c099-cbfc-48c7-9d10-1da01281d7f9-istio-data" pod "ef59c099-cbfc-48c7-9d10-1da01281d7f9" (UID: "ef59c099-cbfc-48c7-9d10-1da01281d7f9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:01:02.649630 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:01:02.649604 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef59c099-cbfc-48c7-9d10-1da01281d7f9-istiod-ca-cert" pod "ef59c099-cbfc-48c7-9d10-1da01281d7f9" (UID: "ef59c099-cbfc-48c7-9d10-1da01281d7f9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:01:02.652064 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:01:02.649690 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/62c91622-5518-4dc2-9f08-102dd0fcfb84-kube-api-access-cv2j9\" (UniqueName: \"kubernetes.io/projected/62c91622-5518-4dc2-9f08-102dd0fcfb84-kube-api-access-cv2j9\") pod \"62c91622-5518-4dc2-9f08-102dd0fcfb84\" (UID: \"62c91622-5518-4dc2-9f08-102dd0fcfb84\") " Oct 30 21:01:02.652064 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:01:02.649777 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62c91622-5518-4dc2-9f08-102dd0fcfb84-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62c91622-5518-4dc2-9f08-102dd0fcfb84-credential-socket\") pod \"62c91622-5518-4dc2-9f08-102dd0fcfb84\" (UID: \"62c91622-5518-4dc2-9f08-102dd0fcfb84\") " Oct 30 21:01:02.652064 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:01:02.649815 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef59c099-cbfc-48c7-9d10-1da01281d7f9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef59c099-cbfc-48c7-9d10-1da01281d7f9-istio-envoy\") pod \"ef59c099-cbfc-48c7-9d10-1da01281d7f9\" (UID: \"ef59c099-cbfc-48c7-9d10-1da01281d7f9\") " Oct 30 21:01:02.652064 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:01:02.649839 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef59c099-cbfc-48c7-9d10-1da01281d7f9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef59c099-cbfc-48c7-9d10-1da01281d7f9-workload-socket\") pod \"ef59c099-cbfc-48c7-9d10-1da01281d7f9\" (UID: \"ef59c099-cbfc-48c7-9d10-1da01281d7f9\") " Oct 30 21:01:02.652064 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:01:02.649866 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62c91622-5518-4dc2-9f08-102dd0fcfb84-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62c91622-5518-4dc2-9f08-102dd0fcfb84-istio-envoy\") pod \"62c91622-5518-4dc2-9f08-102dd0fcfb84\" (UID: \"62c91622-5518-4dc2-9f08-102dd0fcfb84\") " Oct 30 21:01:02.652064 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:01:02.649911 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/62c91622-5518-4dc2-9f08-102dd0fcfb84-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62c91622-5518-4dc2-9f08-102dd0fcfb84-istio-podinfo\") pod \"62c91622-5518-4dc2-9f08-102dd0fcfb84\" (UID: \"62c91622-5518-4dc2-9f08-102dd0fcfb84\") " Oct 30 21:01:02.652064 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:01:02.649957 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/62c91622-5518-4dc2-9f08-102dd0fcfb84-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62c91622-5518-4dc2-9f08-102dd0fcfb84-istiod-ca-cert\") pod \"62c91622-5518-4dc2-9f08-102dd0fcfb84\" (UID: \"62c91622-5518-4dc2-9f08-102dd0fcfb84\") " Oct 30 21:01:02.652064 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:01:02.649982 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62c91622-5518-4dc2-9f08-102dd0fcfb84-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62c91622-5518-4dc2-9f08-102dd0fcfb84-istio-data\") pod \"62c91622-5518-4dc2-9f08-102dd0fcfb84\" (UID: \"62c91622-5518-4dc2-9f08-102dd0fcfb84\") " Oct 30 21:01:02.652064 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:01:02.650062 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef59c099-cbfc-48c7-9d10-1da01281d7f9-kube-api-access-vrtg7\" (UniqueName: \"kubernetes.io/projected/ef59c099-cbfc-48c7-9d10-1da01281d7f9-kube-api-access-vrtg7\") pod \"ef59c099-cbfc-48c7-9d10-1da01281d7f9\" (UID: \"ef59c099-cbfc-48c7-9d10-1da01281d7f9\") " Oct 30 21:01:02.652064 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:01:02.650435 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62c91622-5518-4dc2-9f08-102dd0fcfb84-credential-socket" pod "62c91622-5518-4dc2-9f08-102dd0fcfb84" (UID: "62c91622-5518-4dc2-9f08-102dd0fcfb84"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:01:02.652064 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:01:02.650733 4611 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef59c099-cbfc-48c7-9d10-1da01281d7f9-workload-certs\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:01:02.652064 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:01:02.650754 4611 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef59c099-cbfc-48c7-9d10-1da01281d7f9-istiod-ca-cert\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:01:02.652064 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:01:02.650768 4611 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62c91622-5518-4dc2-9f08-102dd0fcfb84-credential-socket\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:01:02.652064 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:01:02.650781 4611 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef59c099-cbfc-48c7-9d10-1da01281d7f9-credential-socket\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:01:02.652064 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:01:02.650794 4611 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62c91622-5518-4dc2-9f08-102dd0fcfb84-workload-certs\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:01:02.652064 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:01:02.650806 4611 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef59c099-cbfc-48c7-9d10-1da01281d7f9-istio-data\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:01:02.652064 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:01:02.651116 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62c91622-5518-4dc2-9f08-102dd0fcfb84-istiod-ca-cert" pod "62c91622-5518-4dc2-9f08-102dd0fcfb84" (UID: "62c91622-5518-4dc2-9f08-102dd0fcfb84"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:01:02.654747 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:01:02.652756 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62c91622-5518-4dc2-9f08-102dd0fcfb84-workload-socket" pod "62c91622-5518-4dc2-9f08-102dd0fcfb84" (UID: "62c91622-5518-4dc2-9f08-102dd0fcfb84"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:01:02.654747 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:01:02.652958 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62c91622-5518-4dc2-9f08-102dd0fcfb84-istio-data" pod "62c91622-5518-4dc2-9f08-102dd0fcfb84" (UID: "62c91622-5518-4dc2-9f08-102dd0fcfb84"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:01:02.655468 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:01:02.655437 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62c91622-5518-4dc2-9f08-102dd0fcfb84-kube-api-access-cv2j9" pod "62c91622-5518-4dc2-9f08-102dd0fcfb84" (UID: "62c91622-5518-4dc2-9f08-102dd0fcfb84"). InnerVolumeSpecName "kube-api-access-cv2j9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:01:02.655743 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:01:02.655720 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef59c099-cbfc-48c7-9d10-1da01281d7f9-workload-socket" pod "ef59c099-cbfc-48c7-9d10-1da01281d7f9" (UID: "ef59c099-cbfc-48c7-9d10-1da01281d7f9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:01:02.655831 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:01:02.655745 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/62c91622-5518-4dc2-9f08-102dd0fcfb84-istio-podinfo" pod "62c91622-5518-4dc2-9f08-102dd0fcfb84" (UID: "62c91622-5518-4dc2-9f08-102dd0fcfb84"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:01:02.657044 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:01:02.656988 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef59c099-cbfc-48c7-9d10-1da01281d7f9-istio-envoy" pod "ef59c099-cbfc-48c7-9d10-1da01281d7f9" (UID: "ef59c099-cbfc-48c7-9d10-1da01281d7f9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:01:02.657196 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:01:02.657069 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62c91622-5518-4dc2-9f08-102dd0fcfb84-istio-token" pod "62c91622-5518-4dc2-9f08-102dd0fcfb84" (UID: "62c91622-5518-4dc2-9f08-102dd0fcfb84"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:01:02.657319 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:01:02.657294 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef59c099-cbfc-48c7-9d10-1da01281d7f9-kube-api-access-vrtg7" pod "ef59c099-cbfc-48c7-9d10-1da01281d7f9" (UID: "ef59c099-cbfc-48c7-9d10-1da01281d7f9"). InnerVolumeSpecName "kube-api-access-vrtg7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:01:02.657922 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:01:02.657895 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef59c099-cbfc-48c7-9d10-1da01281d7f9-istio-token" pod "ef59c099-cbfc-48c7-9d10-1da01281d7f9" (UID: "ef59c099-cbfc-48c7-9d10-1da01281d7f9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:01:02.658638 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:01:02.658613 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ef59c099-cbfc-48c7-9d10-1da01281d7f9-istio-podinfo" pod "ef59c099-cbfc-48c7-9d10-1da01281d7f9" (UID: "ef59c099-cbfc-48c7-9d10-1da01281d7f9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:01:02.659355 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:01:02.659286 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62c91622-5518-4dc2-9f08-102dd0fcfb84-istio-envoy" pod "62c91622-5518-4dc2-9f08-102dd0fcfb84" (UID: "62c91622-5518-4dc2-9f08-102dd0fcfb84"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:01:02.667353 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:01:02.667333 4611 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-td2hl" Oct 30 21:01:02.669524 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:01:02.669487 4611 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-6c6h6" Oct 30 21:01:02.751732 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:01:02.751696 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2042268e-1c64-4fbe-9f2a-b826113d7f30-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2042268e-1c64-4fbe-9f2a-b826113d7f30-istiod-ca-cert\") pod \"2042268e-1c64-4fbe-9f2a-b826113d7f30\" (UID: \"2042268e-1c64-4fbe-9f2a-b826113d7f30\") " Oct 30 21:01:02.751913 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:01:02.751745 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a08fd3b8-493c-47db-9839-462bcc598e92-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a08fd3b8-493c-47db-9839-462bcc598e92-workload-socket\") pod \"a08fd3b8-493c-47db-9839-462bcc598e92\" (UID: \"a08fd3b8-493c-47db-9839-462bcc598e92\") " Oct 30 21:01:02.751913 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:01:02.751801 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a08fd3b8-493c-47db-9839-462bcc598e92-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a08fd3b8-493c-47db-9839-462bcc598e92-credential-socket\") pod \"a08fd3b8-493c-47db-9839-462bcc598e92\" (UID: \"a08fd3b8-493c-47db-9839-462bcc598e92\") " Oct 30 21:01:02.751913 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:01:02.751838 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a08fd3b8-493c-47db-9839-462bcc598e92-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a08fd3b8-493c-47db-9839-462bcc598e92-istiod-ca-cert\") pod \"a08fd3b8-493c-47db-9839-462bcc598e92\" (UID: \"a08fd3b8-493c-47db-9839-462bcc598e92\") " Oct 30 21:01:02.751913 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:01:02.751869 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2042268e-1c64-4fbe-9f2a-b826113d7f30-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2042268e-1c64-4fbe-9f2a-b826113d7f30-istio-data\") pod \"2042268e-1c64-4fbe-9f2a-b826113d7f30\" (UID: \"2042268e-1c64-4fbe-9f2a-b826113d7f30\") " Oct 30 21:01:02.751913 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:01:02.751897 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2042268e-1c64-4fbe-9f2a-b826113d7f30-istio-token\" (UniqueName: \"kubernetes.io/projected/2042268e-1c64-4fbe-9f2a-b826113d7f30-istio-token\") pod \"2042268e-1c64-4fbe-9f2a-b826113d7f30\" (UID: \"2042268e-1c64-4fbe-9f2a-b826113d7f30\") " Oct 30 21:01:02.752189 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:01:02.751926 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a08fd3b8-493c-47db-9839-462bcc598e92-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a08fd3b8-493c-47db-9839-462bcc598e92-istio-podinfo\") pod \"a08fd3b8-493c-47db-9839-462bcc598e92\" (UID: \"a08fd3b8-493c-47db-9839-462bcc598e92\") " Oct 30 21:01:02.752189 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:01:02.752000 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a08fd3b8-493c-47db-9839-462bcc598e92-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a08fd3b8-493c-47db-9839-462bcc598e92-workload-certs\") pod \"a08fd3b8-493c-47db-9839-462bcc598e92\" (UID: \"a08fd3b8-493c-47db-9839-462bcc598e92\") " Oct 30 21:01:02.752189 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:01:02.752047 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a08fd3b8-493c-47db-9839-462bcc598e92-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a08fd3b8-493c-47db-9839-462bcc598e92-istio-data\") pod \"a08fd3b8-493c-47db-9839-462bcc598e92\" (UID: \"a08fd3b8-493c-47db-9839-462bcc598e92\") " Oct 30 21:01:02.752189 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:01:02.752086 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2042268e-1c64-4fbe-9f2a-b826113d7f30-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2042268e-1c64-4fbe-9f2a-b826113d7f30-istio-podinfo\") pod \"2042268e-1c64-4fbe-9f2a-b826113d7f30\" (UID: \"2042268e-1c64-4fbe-9f2a-b826113d7f30\") " Oct 30 21:01:02.752189 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:01:02.752085 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a08fd3b8-493c-47db-9839-462bcc598e92-credential-socket" pod "a08fd3b8-493c-47db-9839-462bcc598e92" (UID: "a08fd3b8-493c-47db-9839-462bcc598e92"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:01:02.752189 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:01:02.752126 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a08fd3b8-493c-47db-9839-462bcc598e92-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a08fd3b8-493c-47db-9839-462bcc598e92-istio-envoy\") pod \"a08fd3b8-493c-47db-9839-462bcc598e92\" (UID: \"a08fd3b8-493c-47db-9839-462bcc598e92\") " Oct 30 21:01:02.752189 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:01:02.752150 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2042268e-1c64-4fbe-9f2a-b826113d7f30-istiod-ca-cert" pod "2042268e-1c64-4fbe-9f2a-b826113d7f30" (UID: "2042268e-1c64-4fbe-9f2a-b826113d7f30"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:01:02.752575 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:01:02.752195 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2042268e-1c64-4fbe-9f2a-b826113d7f30-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2042268e-1c64-4fbe-9f2a-b826113d7f30-istio-envoy\") pod \"2042268e-1c64-4fbe-9f2a-b826113d7f30\" (UID: \"2042268e-1c64-4fbe-9f2a-b826113d7f30\") " Oct 30 21:01:02.752575 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:01:02.752236 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2042268e-1c64-4fbe-9f2a-b826113d7f30-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2042268e-1c64-4fbe-9f2a-b826113d7f30-workload-socket\") pod \"2042268e-1c64-4fbe-9f2a-b826113d7f30\" (UID: \"2042268e-1c64-4fbe-9f2a-b826113d7f30\") " Oct 30 21:01:02.752575 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:01:02.752284 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a08fd3b8-493c-47db-9839-462bcc598e92-kube-api-access-d2blq\" (UniqueName: \"kubernetes.io/projected/a08fd3b8-493c-47db-9839-462bcc598e92-kube-api-access-d2blq\") pod \"a08fd3b8-493c-47db-9839-462bcc598e92\" (UID: \"a08fd3b8-493c-47db-9839-462bcc598e92\") " Oct 30 21:01:02.752575 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:01:02.752300 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a08fd3b8-493c-47db-9839-462bcc598e92-istiod-ca-cert" pod "a08fd3b8-493c-47db-9839-462bcc598e92" (UID: "a08fd3b8-493c-47db-9839-462bcc598e92"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:01:02.752575 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:01:02.752315 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2042268e-1c64-4fbe-9f2a-b826113d7f30-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2042268e-1c64-4fbe-9f2a-b826113d7f30-workload-certs\") pod \"2042268e-1c64-4fbe-9f2a-b826113d7f30\" (UID: \"2042268e-1c64-4fbe-9f2a-b826113d7f30\") " Oct 30 21:01:02.752575 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:01:02.752377 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2042268e-1c64-4fbe-9f2a-b826113d7f30-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2042268e-1c64-4fbe-9f2a-b826113d7f30-credential-socket\") pod \"2042268e-1c64-4fbe-9f2a-b826113d7f30\" (UID: \"2042268e-1c64-4fbe-9f2a-b826113d7f30\") " Oct 30 21:01:02.752575 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:01:02.752415 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2042268e-1c64-4fbe-9f2a-b826113d7f30-kube-api-access-sv277\" (UniqueName: \"kubernetes.io/projected/2042268e-1c64-4fbe-9f2a-b826113d7f30-kube-api-access-sv277\") pod \"2042268e-1c64-4fbe-9f2a-b826113d7f30\" (UID: \"2042268e-1c64-4fbe-9f2a-b826113d7f30\") " Oct 30 21:01:02.752575 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:01:02.752442 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a08fd3b8-493c-47db-9839-462bcc598e92-istio-token\" (UniqueName: \"kubernetes.io/projected/a08fd3b8-493c-47db-9839-462bcc598e92-istio-token\") pod \"a08fd3b8-493c-47db-9839-462bcc598e92\" (UID: \"a08fd3b8-493c-47db-9839-462bcc598e92\") " Oct 30 21:01:02.752575 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:01:02.752448 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a08fd3b8-493c-47db-9839-462bcc598e92-workload-socket" pod "a08fd3b8-493c-47db-9839-462bcc598e92" (UID: "a08fd3b8-493c-47db-9839-462bcc598e92"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:01:02.752575 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:01:02.752537 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a08fd3b8-493c-47db-9839-462bcc598e92-istio-data" pod "a08fd3b8-493c-47db-9839-462bcc598e92" (UID: "a08fd3b8-493c-47db-9839-462bcc598e92"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:01:02.753074 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:01:02.752645 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a08fd3b8-493c-47db-9839-462bcc598e92-workload-certs" pod "a08fd3b8-493c-47db-9839-462bcc598e92" (UID: "a08fd3b8-493c-47db-9839-462bcc598e92"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:01:02.753074 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:01:02.752820 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2042268e-1c64-4fbe-9f2a-b826113d7f30-workload-certs" pod "2042268e-1c64-4fbe-9f2a-b826113d7f30" (UID: "2042268e-1c64-4fbe-9f2a-b826113d7f30"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:01:02.753074 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:01:02.752828 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2042268e-1c64-4fbe-9f2a-b826113d7f30-credential-socket" pod "2042268e-1c64-4fbe-9f2a-b826113d7f30" (UID: "2042268e-1c64-4fbe-9f2a-b826113d7f30"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:01:02.753074 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:01:02.753010 4611 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/62c91622-5518-4dc2-9f08-102dd0fcfb84-istio-token\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:01:02.753074 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:01:02.753030 4611 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2042268e-1c64-4fbe-9f2a-b826113d7f30-workload-certs\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:01:02.753074 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:01:02.753046 4611 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62c91622-5518-4dc2-9f08-102dd0fcfb84-workload-socket\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:01:02.753074 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:01:02.753061 4611 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2042268e-1c64-4fbe-9f2a-b826113d7f30-credential-socket\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:01:02.753074 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:01:02.753076 4611 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef59c099-cbfc-48c7-9d10-1da01281d7f9-istio-podinfo\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:01:02.753496 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:01:02.753089 4611 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ef59c099-cbfc-48c7-9d10-1da01281d7f9-istio-token\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:01:02.753496 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:01:02.753104 4611 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cv2j9\" (UniqueName: \"kubernetes.io/projected/62c91622-5518-4dc2-9f08-102dd0fcfb84-kube-api-access-cv2j9\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:01:02.753496 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:01:02.753119 4611 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2042268e-1c64-4fbe-9f2a-b826113d7f30-istiod-ca-cert\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:01:02.753496 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:01:02.753133 4611 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a08fd3b8-493c-47db-9839-462bcc598e92-workload-socket\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:01:02.753496 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:01:02.753146 4611 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a08fd3b8-493c-47db-9839-462bcc598e92-credential-socket\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:01:02.753496 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:01:02.753162 4611 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef59c099-cbfc-48c7-9d10-1da01281d7f9-istio-envoy\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:01:02.753496 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:01:02.753176 4611 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef59c099-cbfc-48c7-9d10-1da01281d7f9-workload-socket\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:01:02.753496 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:01:02.753190 4611 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62c91622-5518-4dc2-9f08-102dd0fcfb84-istio-envoy\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:01:02.753496 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:01:02.753203 4611 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a08fd3b8-493c-47db-9839-462bcc598e92-istiod-ca-cert\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:01:02.753496 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:01:02.753216 4611 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62c91622-5518-4dc2-9f08-102dd0fcfb84-istio-podinfo\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:01:02.753496 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:01:02.753229 4611 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62c91622-5518-4dc2-9f08-102dd0fcfb84-istiod-ca-cert\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:01:02.753496 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:01:02.753245 4611 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62c91622-5518-4dc2-9f08-102dd0fcfb84-istio-data\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:01:02.753496 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:01:02.753260 4611 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vrtg7\" (UniqueName: \"kubernetes.io/projected/ef59c099-cbfc-48c7-9d10-1da01281d7f9-kube-api-access-vrtg7\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:01:02.753496 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:01:02.753274 4611 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a08fd3b8-493c-47db-9839-462bcc598e92-workload-certs\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:01:02.753496 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:01:02.753287 4611 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a08fd3b8-493c-47db-9839-462bcc598e92-istio-data\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:01:02.753496 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:01:02.753525 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2042268e-1c64-4fbe-9f2a-b826113d7f30-istio-data" pod "2042268e-1c64-4fbe-9f2a-b826113d7f30" (UID: "2042268e-1c64-4fbe-9f2a-b826113d7f30"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:01:02.755159 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:01:02.753808 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2042268e-1c64-4fbe-9f2a-b826113d7f30-workload-socket" pod "2042268e-1c64-4fbe-9f2a-b826113d7f30" (UID: "2042268e-1c64-4fbe-9f2a-b826113d7f30"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:01:02.755159 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:01:02.754860 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a08fd3b8-493c-47db-9839-462bcc598e92-istio-podinfo" pod "a08fd3b8-493c-47db-9839-462bcc598e92" (UID: "a08fd3b8-493c-47db-9839-462bcc598e92"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:01:02.755705 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:01:02.755639 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a08fd3b8-493c-47db-9839-462bcc598e92-istio-token" pod "a08fd3b8-493c-47db-9839-462bcc598e92" (UID: "a08fd3b8-493c-47db-9839-462bcc598e92"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:01:02.755948 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:01:02.755909 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a08fd3b8-493c-47db-9839-462bcc598e92-kube-api-access-d2blq" pod "a08fd3b8-493c-47db-9839-462bcc598e92" (UID: "a08fd3b8-493c-47db-9839-462bcc598e92"). InnerVolumeSpecName "kube-api-access-d2blq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:01:02.755948 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:01:02.755924 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2042268e-1c64-4fbe-9f2a-b826113d7f30-istio-token" pod "2042268e-1c64-4fbe-9f2a-b826113d7f30" (UID: "2042268e-1c64-4fbe-9f2a-b826113d7f30"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:01:02.756796 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:01:02.756765 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a08fd3b8-493c-47db-9839-462bcc598e92-istio-envoy" pod "a08fd3b8-493c-47db-9839-462bcc598e92" (UID: "a08fd3b8-493c-47db-9839-462bcc598e92"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:01:02.756914 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:01:02.756892 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2042268e-1c64-4fbe-9f2a-b826113d7f30-istio-podinfo" pod "2042268e-1c64-4fbe-9f2a-b826113d7f30" (UID: "2042268e-1c64-4fbe-9f2a-b826113d7f30"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:01:02.757131 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:01:02.757112 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2042268e-1c64-4fbe-9f2a-b826113d7f30-kube-api-access-sv277" pod "2042268e-1c64-4fbe-9f2a-b826113d7f30" (UID: "2042268e-1c64-4fbe-9f2a-b826113d7f30"). InnerVolumeSpecName "kube-api-access-sv277". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:01:02.757397 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:01:02.757370 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2042268e-1c64-4fbe-9f2a-b826113d7f30-istio-envoy" pod "2042268e-1c64-4fbe-9f2a-b826113d7f30" (UID: "2042268e-1c64-4fbe-9f2a-b826113d7f30"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:01:02.800765 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:01:02.800569 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:01:02.854307 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:01:02.854267 4611 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sv277\" (UniqueName: \"kubernetes.io/projected/2042268e-1c64-4fbe-9f2a-b826113d7f30-kube-api-access-sv277\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:01:02.854307 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:01:02.854302 4611 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a08fd3b8-493c-47db-9839-462bcc598e92-istio-token\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:01:02.854307 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:01:02.854313 4611 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2042268e-1c64-4fbe-9f2a-b826113d7f30-istio-data\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:01:02.854579 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:01:02.854322 4611 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2042268e-1c64-4fbe-9f2a-b826113d7f30-istio-token\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:01:02.854579 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:01:02.854330 4611 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a08fd3b8-493c-47db-9839-462bcc598e92-istio-podinfo\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:01:02.854579 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:01:02.854339 4611 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2042268e-1c64-4fbe-9f2a-b826113d7f30-istio-podinfo\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:01:02.854579 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:01:02.854347 4611 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a08fd3b8-493c-47db-9839-462bcc598e92-istio-envoy\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:01:02.854579 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:01:02.854355 4611 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2042268e-1c64-4fbe-9f2a-b826113d7f30-istio-envoy\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:01:02.854579 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:01:02.854363 4611 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2042268e-1c64-4fbe-9f2a-b826113d7f30-workload-socket\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:01:02.854579 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:01:02.854371 4611 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d2blq\" (UniqueName: \"kubernetes.io/projected/a08fd3b8-493c-47db-9839-462bcc598e92-kube-api-access-d2blq\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:01:02.908319 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:01:02.904964 4611 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-khv95"] Oct 30 21:01:02.912923 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:01:02.912899 4611 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-khv95"] Oct 30 21:01:03.487455 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:01:03.487010 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:01:03.593642 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:01:03.593550 4611 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-td2hl" Oct 30 21:01:03.593807 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:01:03.593547 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-td2hl" event={"ID":"a08fd3b8-493c-47db-9839-462bcc598e92","Type":"ContainerDied","Data":"fc75d9673d8dc9d24c18bd287a7fc8a4ad5c5b716a5df461475b0219c8b6484c"} Oct 30 21:01:03.593807 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:01:03.593750 4611 scope.go:122] "RemoveContainer" containerID="c3b88cbb89aaf4136d6f91ab75097911dbefa29c114d75768b822fb9f72bb325" Oct 30 21:01:03.595166 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:01:03.595115 4611 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-g2gnd" Oct 30 21:01:03.595166 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:01:03.595121 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-g2gnd" event={"ID":"ef59c099-cbfc-48c7-9d10-1da01281d7f9","Type":"ContainerDied","Data":"ab015277a223f742645055afdb399ed5ad9b596d287ef730245f0a9a13f31da7"} Oct 30 21:01:03.596628 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:01:03.596604 4611 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-6c6h6" Oct 30 21:01:03.596816 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:01:03.596637 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-6c6h6" event={"ID":"2042268e-1c64-4fbe-9f2a-b826113d7f30","Type":"ContainerDied","Data":"21acae026f9bc9a30f1954c12f9e4d43846823b62def73ffb7936453893e3bc7"} Oct 30 21:01:03.606689 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:01:03.606667 4611 scope.go:122] "RemoveContainer" containerID="f339e661941f927be58d34084d6e8d704e06f578b798e5d4e9a8c5e5f6bd0596" Oct 30 21:01:03.610356 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:01:03.610242 4611 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="62c91622-5518-4dc2-9f08-102dd0fcfb84" path="/var/lib/kubelet/pods/62c91622-5518-4dc2-9f08-102dd0fcfb84/volumes" Oct 30 21:01:03.617695 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:01:03.617677 4611 scope.go:122] "RemoveContainer" containerID="256030229a04a2e5065cc806c24ba748f6ae5c57276bd9ec8a9fef6b19d8abe2" Oct 30 21:01:03.622086 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:01:03.622067 4611 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-td2hl"] Oct 30 21:01:03.627106 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:01:03.627076 4611 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-td2hl"] Oct 30 21:01:03.636603 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:01:03.636583 4611 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-6c6h6"] Oct 30 21:01:03.649029 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:01:03.649008 4611 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-6c6h6"] Oct 30 21:01:03.702486 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:01:03.702462 4611 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-g2gnd"] Oct 30 21:01:03.739969 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:01:03.739943 4611 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-b8aee1-openshift-default-775cfffcf9-g2gnd"] Oct 30 21:01:03.916282 ip-10-0-133-192 kubenswrapper[4611]: E1030 21:01:03.916248 4611 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hwzc6" containerName="extractor" Oct 30 21:01:04.307250 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:01:04.307138 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:01:04.976438 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:01:04.976409 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:01:05.137742 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:01:05.137705 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:01:05.605814 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:01:05.605782 4611 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2042268e-1c64-4fbe-9f2a-b826113d7f30" path="/var/lib/kubelet/pods/2042268e-1c64-4fbe-9f2a-b826113d7f30/volumes" Oct 30 21:01:05.606237 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:01:05.606223 4611 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a08fd3b8-493c-47db-9839-462bcc598e92" path="/var/lib/kubelet/pods/a08fd3b8-493c-47db-9839-462bcc598e92/volumes" Oct 30 21:01:05.606641 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:01:05.606626 4611 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ef59c099-cbfc-48c7-9d10-1da01281d7f9" path="/var/lib/kubelet/pods/ef59c099-cbfc-48c7-9d10-1da01281d7f9/volumes" Oct 30 21:01:05.811177 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:01:05.811143 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:01:06.592636 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:01:06.592587 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:01:07.151626 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:01:07.151586 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:01:07.351795 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:01:07.351760 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:01:07.709497 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:01:07.709468 4611 scope.go:122] "RemoveContainer" containerID="3b11347347b19a06bec0fbed9f23c6395be0c1630fc7ffd7d8ae1f4716093e23" Oct 30 21:01:07.927652 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:01:07.926821 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:01:08.517700 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:01:08.517661 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:01:09.325888 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:01:09.325851 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:01:10.006080 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:01:10.006000 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:01:10.610419 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:01:10.610383 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:01:11.379715 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:01:11.379672 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:01:12.176574 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:01:12.176532 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:01:12.699558 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:01:12.699520 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:01:13.287228 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:01:13.287191 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:01:13.427004 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:01:13.426964 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:01:14.015115 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:01:14.015079 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:01:14.683081 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:01:14.683034 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:01:15.350083 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:01:15.350045 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:01:16.071618 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:01:16.071589 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:01:16.702262 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:01:16.702222 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:01:17.472818 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:01:17.472782 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:01:18.118393 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:01:18.118355 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:01:18.778785 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:01:18.778743 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:01:19.744900 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:01:19.744863 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:01:20.459745 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:01:20.459703 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:01:21.116205 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:01:21.116152 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:01:21.843849 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:01:21.843808 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:01:22.524033 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:01:22.523995 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:01:23.205348 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:01:23.205316 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:01:23.819900 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:01:23.819855 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:01:24.511333 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:01:24.511298 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:01:25.117267 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:01:25.117226 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:01:28.388129 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:01:28.388097 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:01:28.589052 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:01:28.589010 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:01:29.240329 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:01:29.240296 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:01:29.932413 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:01:29.932377 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:01:30.671161 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:01:30.671121 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:01:31.550230 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:01:31.550190 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:01:32.205901 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:01:32.205863 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:01:32.869873 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:01:32.869837 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:01:33.501489 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:01:33.501455 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:01:33.916519 ip-10-0-133-192 kubenswrapper[4611]: E1030 21:01:33.916474 4611 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hwzc6" containerName="extractor" Oct 30 21:01:34.100359 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:01:34.100304 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:01:34.718374 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:01:34.718338 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:01:35.515142 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:01:35.515099 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:01:36.191462 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:01:36.191420 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:01:36.896278 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:01:36.896228 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:01:37.614592 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:01:37.614559 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:01:38.396147 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:01:38.396100 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:01:39.088701 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:01:39.088656 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:01:39.690243 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:01:39.690203 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:01:40.392791 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:01:40.392738 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:01:41.004257 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:01:41.004225 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:01:41.703120 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:01:41.703083 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:01:42.302646 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:01:42.302608 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:01:42.897194 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:01:42.897155 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:01:43.589058 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:01:43.589019 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:01:44.197338 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:01:44.197297 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:01:44.860545 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:01:44.860467 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:01:45.501332 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:01:45.501292 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:01:46.191138 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:01:46.191103 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:01:46.994448 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:01:46.994413 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:01:47.705414 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:01:47.705357 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:01:48.041405 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:01:48.041323 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:01:48.204751 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:01:48.204444 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:01:48.364115 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:01:48.364074 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:01:48.511876 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:01:48.511827 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:01:48.659559 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:01:48.659456 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:01:48.928543 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:01:48.928441 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:01:49.073014 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:01:49.072973 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:01:49.299190 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:01:49.299087 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:01:50.011142 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:01:50.011104 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:01:50.699167 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:01:50.699128 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:01:51.391915 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:01:51.391879 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:01:52.142141 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:01:52.142106 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:01:52.715050 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:01:52.715007 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:01:53.391566 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:01:53.391528 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:01:54.006774 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:01:54.006713 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:01:54.671427 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:01:54.671393 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:01:55.396344 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:01:55.396311 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:01:56.230728 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:01:56.230689 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:01:56.431079 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:01:56.431047 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:01:56.997642 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:01:56.997603 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:01:57.623855 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:01:57.623824 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:01:58.269248 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:01:58.269212 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:01:59.072130 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:01:59.072088 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:01:59.685806 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:01:59.685773 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:02:00.385544 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:00.385461 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:02:01.085894 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:01.085863 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:02:01.687274 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:01.687232 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:02:02.302008 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:02.301962 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:02:02.984170 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:02.984136 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:02:03.590815 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:03.590604 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:02:03.916572 ip-10-0-133-192 kubenswrapper[4611]: E1030 21:02:03.916539 4611 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hwzc6" containerName="extractor" Oct 30 21:02:04.394797 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:04.394758 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:02:05.089265 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:05.089227 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:02:05.701579 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:05.701535 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:02:06.399353 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:06.399312 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:02:06.606500 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:06.606470 4611 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-812bfe65-openshift-default-74647fb5f7-f85fj"] Oct 30 21:02:06.606740 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:06.606711 4611 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-812bfe65-openshift-default-74647fb5f7-f85fj" podUID="918ff269-fb66-4479-9e8d-d0184421bc7c" containerName="istio-proxy" containerID="cri-o://01c9b8958f5f15dbc930d892de99b56df1b0df85845bbcd3d55203b2b47b4971" gracePeriod=30 Oct 30 21:02:06.691327 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:06.691253 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:02:07.102108 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:07.102072 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:02:07.779888 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:07.779855 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:02:07.795337 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:07.795306 4611 scope.go:122] "RemoveContainer" containerID="0263dc0f6eeecd38f12c665ad11fb9022ddb078ff69fd9d2198a264d2f900478" Oct 30 21:02:07.806428 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:07.806405 4611 scope.go:122] "RemoveContainer" containerID="b5bc767e383fe4a05db6bd506dab86d9d93437887e05cad2be11d894736f71cb" Oct 30 21:02:07.814494 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:07.814464 4611 scope.go:122] "RemoveContainer" containerID="2185b35e2920d87e34091a797b813937841f9c29840ea72fb395f40bce822aec" Oct 30 21:02:08.492762 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:08.492726 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:02:09.120812 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:09.120776 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:02:09.788491 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:09.788457 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:02:10.476862 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:10.476807 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:02:11.304355 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:11.304315 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:02:11.869386 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:11.869327 4611 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-812bfe65-openshift-default-74647fb5f7-f85fj" Oct 30 21:02:11.910194 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:11.910035 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:02:11.977247 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:11.977215 4611 generic.go:356] "Generic (PLEG): container finished" podID="918ff269-fb66-4479-9e8d-d0184421bc7c" containerID="01c9b8958f5f15dbc930d892de99b56df1b0df85845bbcd3d55203b2b47b4971" exitCode=0 Oct 30 21:02:11.977408 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:11.977271 4611 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-812bfe65-openshift-default-74647fb5f7-f85fj" Oct 30 21:02:11.977408 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:11.977303 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-812bfe65-openshift-default-74647fb5f7-f85fj" event={"ID":"918ff269-fb66-4479-9e8d-d0184421bc7c","Type":"ContainerDied","Data":"01c9b8958f5f15dbc930d892de99b56df1b0df85845bbcd3d55203b2b47b4971"} Oct 30 21:02:11.977408 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:11.977357 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-812bfe65-openshift-default-74647fb5f7-f85fj" event={"ID":"918ff269-fb66-4479-9e8d-d0184421bc7c","Type":"ContainerDied","Data":"951a442d5d9efae0eba8c3718c65806b6d63e25eb78670878ad9db1661c76aa4"} Oct 30 21:02:11.977408 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:11.977379 4611 scope.go:122] "RemoveContainer" containerID="01c9b8958f5f15dbc930d892de99b56df1b0df85845bbcd3d55203b2b47b4971" Oct 30 21:02:11.986570 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:11.986550 4611 scope.go:122] "RemoveContainer" containerID="374d672a1b8de629149cd6f13a4700bc2904de58b732cc0733a2facef6ba11c4" Oct 30 21:02:11.988724 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:11.988704 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/918ff269-fb66-4479-9e8d-d0184421bc7c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/918ff269-fb66-4479-9e8d-d0184421bc7c-credential-socket\") pod \"918ff269-fb66-4479-9e8d-d0184421bc7c\" (UID: \"918ff269-fb66-4479-9e8d-d0184421bc7c\") " Oct 30 21:02:11.988840 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:11.988791 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/918ff269-fb66-4479-9e8d-d0184421bc7c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/918ff269-fb66-4479-9e8d-d0184421bc7c-workload-certs\") pod \"918ff269-fb66-4479-9e8d-d0184421bc7c\" (UID: \"918ff269-fb66-4479-9e8d-d0184421bc7c\") " Oct 30 21:02:11.988904 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:11.988883 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/918ff269-fb66-4479-9e8d-d0184421bc7c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/918ff269-fb66-4479-9e8d-d0184421bc7c-istio-envoy\") pod \"918ff269-fb66-4479-9e8d-d0184421bc7c\" (UID: \"918ff269-fb66-4479-9e8d-d0184421bc7c\") " Oct 30 21:02:11.988962 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:11.988921 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/918ff269-fb66-4479-9e8d-d0184421bc7c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/918ff269-fb66-4479-9e8d-d0184421bc7c-istio-podinfo\") pod \"918ff269-fb66-4479-9e8d-d0184421bc7c\" (UID: \"918ff269-fb66-4479-9e8d-d0184421bc7c\") " Oct 30 21:02:11.988962 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:11.988951 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/918ff269-fb66-4479-9e8d-d0184421bc7c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/918ff269-fb66-4479-9e8d-d0184421bc7c-workload-socket\") pod \"918ff269-fb66-4479-9e8d-d0184421bc7c\" (UID: \"918ff269-fb66-4479-9e8d-d0184421bc7c\") " Oct 30 21:02:11.989073 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:11.988969 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/918ff269-fb66-4479-9e8d-d0184421bc7c-credential-socket" pod "918ff269-fb66-4479-9e8d-d0184421bc7c" (UID: "918ff269-fb66-4479-9e8d-d0184421bc7c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:02:11.989073 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:11.988987 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/918ff269-fb66-4479-9e8d-d0184421bc7c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/918ff269-fb66-4479-9e8d-d0184421bc7c-istio-data\") pod \"918ff269-fb66-4479-9e8d-d0184421bc7c\" (UID: \"918ff269-fb66-4479-9e8d-d0184421bc7c\") " Oct 30 21:02:11.989073 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:11.989043 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/918ff269-fb66-4479-9e8d-d0184421bc7c-kube-api-access-hkblh\" (UniqueName: \"kubernetes.io/projected/918ff269-fb66-4479-9e8d-d0184421bc7c-kube-api-access-hkblh\") pod \"918ff269-fb66-4479-9e8d-d0184421bc7c\" (UID: \"918ff269-fb66-4479-9e8d-d0184421bc7c\") " Oct 30 21:02:11.989223 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:11.989088 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/918ff269-fb66-4479-9e8d-d0184421bc7c-workload-certs" pod "918ff269-fb66-4479-9e8d-d0184421bc7c" (UID: "918ff269-fb66-4479-9e8d-d0184421bc7c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:02:11.989223 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:11.989167 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/918ff269-fb66-4479-9e8d-d0184421bc7c-istio-token\" (UniqueName: \"kubernetes.io/projected/918ff269-fb66-4479-9e8d-d0184421bc7c-istio-token\") pod \"918ff269-fb66-4479-9e8d-d0184421bc7c\" (UID: \"918ff269-fb66-4479-9e8d-d0184421bc7c\") " Oct 30 21:02:11.989223 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:11.989183 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/918ff269-fb66-4479-9e8d-d0184421bc7c-workload-socket" pod "918ff269-fb66-4479-9e8d-d0184421bc7c" (UID: "918ff269-fb66-4479-9e8d-d0184421bc7c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:02:11.989223 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:11.989199 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/918ff269-fb66-4479-9e8d-d0184421bc7c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/918ff269-fb66-4479-9e8d-d0184421bc7c-istiod-ca-cert\") pod \"918ff269-fb66-4479-9e8d-d0184421bc7c\" (UID: \"918ff269-fb66-4479-9e8d-d0184421bc7c\") " Oct 30 21:02:11.989415 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:11.989252 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/918ff269-fb66-4479-9e8d-d0184421bc7c-istio-data" pod "918ff269-fb66-4479-9e8d-d0184421bc7c" (UID: "918ff269-fb66-4479-9e8d-d0184421bc7c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:02:11.989656 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:11.989633 4611 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/918ff269-fb66-4479-9e8d-d0184421bc7c-credential-socket\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:02:11.989656 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:11.989657 4611 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/918ff269-fb66-4479-9e8d-d0184421bc7c-workload-certs\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:02:11.989839 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:11.989672 4611 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/918ff269-fb66-4479-9e8d-d0184421bc7c-workload-socket\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:02:11.989839 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:11.989686 4611 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/918ff269-fb66-4479-9e8d-d0184421bc7c-istio-data\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:02:11.989839 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:11.989669 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/918ff269-fb66-4479-9e8d-d0184421bc7c-istiod-ca-cert" pod "918ff269-fb66-4479-9e8d-d0184421bc7c" (UID: "918ff269-fb66-4479-9e8d-d0184421bc7c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:02:11.991239 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:11.991210 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/918ff269-fb66-4479-9e8d-d0184421bc7c-istio-token" pod "918ff269-fb66-4479-9e8d-d0184421bc7c" (UID: "918ff269-fb66-4479-9e8d-d0184421bc7c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:02:11.991328 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:11.991302 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/918ff269-fb66-4479-9e8d-d0184421bc7c-kube-api-access-hkblh" pod "918ff269-fb66-4479-9e8d-d0184421bc7c" (UID: "918ff269-fb66-4479-9e8d-d0184421bc7c"). InnerVolumeSpecName "kube-api-access-hkblh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:02:11.991417 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:11.991401 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/918ff269-fb66-4479-9e8d-d0184421bc7c-istio-podinfo" pod "918ff269-fb66-4479-9e8d-d0184421bc7c" (UID: "918ff269-fb66-4479-9e8d-d0184421bc7c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:02:11.991480 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:11.991452 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/918ff269-fb66-4479-9e8d-d0184421bc7c-istio-envoy" pod "918ff269-fb66-4479-9e8d-d0184421bc7c" (UID: "918ff269-fb66-4479-9e8d-d0184421bc7c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:02:11.996918 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:11.996894 4611 scope.go:122] "RemoveContainer" containerID="01c9b8958f5f15dbc930d892de99b56df1b0df85845bbcd3d55203b2b47b4971" Oct 30 21:02:11.997181 ip-10-0-133-192 kubenswrapper[4611]: E1030 21:02:11.997154 4611 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"01c9b8958f5f15dbc930d892de99b56df1b0df85845bbcd3d55203b2b47b4971\": container with ID starting with 01c9b8958f5f15dbc930d892de99b56df1b0df85845bbcd3d55203b2b47b4971 not found: ID does not exist" containerID="01c9b8958f5f15dbc930d892de99b56df1b0df85845bbcd3d55203b2b47b4971" Oct 30 21:02:11.997250 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:11.997182 4611 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"01c9b8958f5f15dbc930d892de99b56df1b0df85845bbcd3d55203b2b47b4971"} err="failed to get container status \"01c9b8958f5f15dbc930d892de99b56df1b0df85845bbcd3d55203b2b47b4971\": rpc error: code = NotFound desc = could not find container \"01c9b8958f5f15dbc930d892de99b56df1b0df85845bbcd3d55203b2b47b4971\": container with ID starting with 01c9b8958f5f15dbc930d892de99b56df1b0df85845bbcd3d55203b2b47b4971 not found: ID does not exist" Oct 30 21:02:11.997250 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:11.997198 4611 scope.go:122] "RemoveContainer" containerID="374d672a1b8de629149cd6f13a4700bc2904de58b732cc0733a2facef6ba11c4" Oct 30 21:02:11.997473 ip-10-0-133-192 kubenswrapper[4611]: E1030 21:02:11.997451 4611 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"374d672a1b8de629149cd6f13a4700bc2904de58b732cc0733a2facef6ba11c4\": container with ID starting with 374d672a1b8de629149cd6f13a4700bc2904de58b732cc0733a2facef6ba11c4 not found: ID does not exist" containerID="374d672a1b8de629149cd6f13a4700bc2904de58b732cc0733a2facef6ba11c4" Oct 30 21:02:11.997531 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:11.997482 4611 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"374d672a1b8de629149cd6f13a4700bc2904de58b732cc0733a2facef6ba11c4"} err="failed to get container status \"374d672a1b8de629149cd6f13a4700bc2904de58b732cc0733a2facef6ba11c4\": rpc error: code = NotFound desc = could not find container \"374d672a1b8de629149cd6f13a4700bc2904de58b732cc0733a2facef6ba11c4\": container with ID starting with 374d672a1b8de629149cd6f13a4700bc2904de58b732cc0733a2facef6ba11c4 not found: ID does not exist" Oct 30 21:02:12.090897 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:12.090873 4611 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/918ff269-fb66-4479-9e8d-d0184421bc7c-istio-token\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:02:12.090897 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:12.090894 4611 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/918ff269-fb66-4479-9e8d-d0184421bc7c-istiod-ca-cert\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:02:12.091018 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:12.090904 4611 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/918ff269-fb66-4479-9e8d-d0184421bc7c-istio-envoy\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:02:12.091018 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:12.090912 4611 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/918ff269-fb66-4479-9e8d-d0184421bc7c-istio-podinfo\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:02:12.091018 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:12.090922 4611 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hkblh\" (UniqueName: \"kubernetes.io/projected/918ff269-fb66-4479-9e8d-d0184421bc7c-kube-api-access-hkblh\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:02:12.299354 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:12.299323 4611 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-812bfe65-openshift-default-74647fb5f7-f85fj"] Oct 30 21:02:12.303266 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:12.303242 4611 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-812bfe65-openshift-default-74647fb5f7-f85fj"] Oct 30 21:02:12.505206 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:12.505175 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:02:13.183471 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:13.183300 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:02:13.605695 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:13.605666 4611 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="918ff269-fb66-4479-9e8d-d0184421bc7c" path="/var/lib/kubelet/pods/918ff269-fb66-4479-9e8d-d0184421bc7c/volumes" Oct 30 21:02:13.795677 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:13.795638 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:02:14.434491 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:14.434452 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:02:15.144423 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:15.144371 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:02:15.826913 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:15.826875 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:02:16.544333 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:16.544293 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:02:17.262184 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:17.262149 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:02:17.407087 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:17.407054 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:02:17.997526 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:17.997395 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:02:18.691949 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:18.691883 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:02:19.318567 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:19.318500 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:02:19.987599 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:19.987543 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:02:20.613675 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:20.613635 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:02:21.406295 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:21.406260 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:02:22.138729 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:22.138697 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:02:22.174690 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:22.174659 4611 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-f2ed1e/distinct-w4-f2ed1e-kserve-57cff9dcf8-8d5zl"] Oct 30 21:02:22.418399 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:22.418305 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:02:23.015415 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:23.015378 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:02:23.725305 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:23.725268 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:02:23.869347 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:23.869308 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:02:24.405401 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:24.405310 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:02:25.008879 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:25.008843 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:02:25.699303 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:25.699247 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:02:26.322474 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:26.322427 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:02:27.232602 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:27.232563 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:02:27.457639 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:27.457606 4611 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-f2ed1e-66d84b8cff-g2w49"] Oct 30 21:02:27.571516 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:27.571427 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:02:27.735180 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:27.735057 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:02:27.754551 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:27.747256 4611 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2dd9250-9jz97"] Oct 30 21:02:27.754551 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:27.747965 4611 cpu_manager.go:414] "containerMap: removing container" podUID="2042268e-1c64-4fbe-9f2a-b826113d7f30" containerName="istio-proxy" Oct 30 21:02:27.754551 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:27.747981 4611 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2042268e-1c64-4fbe-9f2a-b826113d7f30" containerName="istio-proxy" Oct 30 21:02:27.754551 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:27.748000 4611 cpu_manager.go:414] "containerMap: removing container" podUID="62c91622-5518-4dc2-9f08-102dd0fcfb84" containerName="istio-proxy" Oct 30 21:02:27.754551 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:27.748007 4611 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="62c91622-5518-4dc2-9f08-102dd0fcfb84" containerName="istio-proxy" Oct 30 21:02:27.754551 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:27.748022 4611 cpu_manager.go:414] "containerMap: removing container" podUID="ef59c099-cbfc-48c7-9d10-1da01281d7f9" containerName="istio-proxy" Oct 30 21:02:27.754551 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:27.748030 4611 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef59c099-cbfc-48c7-9d10-1da01281d7f9" containerName="istio-proxy" Oct 30 21:02:27.754551 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:27.748043 4611 cpu_manager.go:414] "containerMap: removing container" podUID="62c91622-5518-4dc2-9f08-102dd0fcfb84" containerName="istio-proxy" Oct 30 21:02:27.754551 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:27.748050 4611 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="62c91622-5518-4dc2-9f08-102dd0fcfb84" containerName="istio-proxy" Oct 30 21:02:27.754551 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:27.748074 4611 cpu_manager.go:414] "containerMap: removing container" podUID="0de46c2f-4a48-4d46-be31-cd2173b80b5f" containerName="revoke-keys" Oct 30 21:02:27.754551 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:27.748081 4611 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0de46c2f-4a48-4d46-be31-cd2173b80b5f" containerName="revoke-keys" Oct 30 21:02:27.754551 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:27.748091 4611 cpu_manager.go:414] "containerMap: removing container" podUID="77368afe-edc8-4fa5-8010-d321965fb38e" containerName="main" Oct 30 21:02:27.754551 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:27.748098 4611 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="77368afe-edc8-4fa5-8010-d321965fb38e" containerName="main" Oct 30 21:02:27.754551 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:27.748109 4611 cpu_manager.go:414] "containerMap: removing container" podUID="286eab7a-81a7-4768-a7f6-be6b3720028a" containerName="main" Oct 30 21:02:27.754551 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:27.748116 4611 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="286eab7a-81a7-4768-a7f6-be6b3720028a" containerName="main" Oct 30 21:02:27.754551 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:27.748130 4611 cpu_manager.go:414] "containerMap: removing container" podUID="918ff269-fb66-4479-9e8d-d0184421bc7c" containerName="istio-proxy" Oct 30 21:02:27.754551 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:27.748137 4611 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="918ff269-fb66-4479-9e8d-d0184421bc7c" containerName="istio-proxy" Oct 30 21:02:27.754551 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:27.748151 4611 cpu_manager.go:414] "containerMap: removing container" podUID="918ff269-fb66-4479-9e8d-d0184421bc7c" containerName="istio-proxy" Oct 30 21:02:27.754551 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:27.748158 4611 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="918ff269-fb66-4479-9e8d-d0184421bc7c" containerName="istio-proxy" Oct 30 21:02:27.754551 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:27.748172 4611 cpu_manager.go:414] "containerMap: removing container" podUID="1dd352a3-b3a3-4912-9d9f-723d40415008" containerName="cleanup" Oct 30 21:02:27.754551 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:27.748179 4611 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1dd352a3-b3a3-4912-9d9f-723d40415008" containerName="cleanup" Oct 30 21:02:27.754551 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:27.748190 4611 cpu_manager.go:414] "containerMap: removing container" podUID="1599e588-def8-42e1-b81d-ad0aaf119818" containerName="main" Oct 30 21:02:27.754551 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:27.748197 4611 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1599e588-def8-42e1-b81d-ad0aaf119818" containerName="main" Oct 30 21:02:27.754551 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:27.748215 4611 cpu_manager.go:414] "containerMap: removing container" podUID="a08fd3b8-493c-47db-9839-462bcc598e92" containerName="istio-proxy" Oct 30 21:02:27.754551 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:27.748222 4611 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a08fd3b8-493c-47db-9839-462bcc598e92" containerName="istio-proxy" Oct 30 21:02:27.754551 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:27.748307 4611 memory_manager.go:365] "RemoveStaleState removing state" podUID="2042268e-1c64-4fbe-9f2a-b826113d7f30" containerName="istio-proxy" Oct 30 21:02:27.754551 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:27.748318 4611 memory_manager.go:365] "RemoveStaleState removing state" podUID="62c91622-5518-4dc2-9f08-102dd0fcfb84" containerName="istio-proxy" Oct 30 21:02:27.754551 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:27.748328 4611 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef59c099-cbfc-48c7-9d10-1da01281d7f9" containerName="istio-proxy" Oct 30 21:02:27.754551 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:27.748339 4611 memory_manager.go:365] "RemoveStaleState removing state" podUID="62c91622-5518-4dc2-9f08-102dd0fcfb84" containerName="istio-proxy" Oct 30 21:02:27.754551 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:27.748352 4611 memory_manager.go:365] "RemoveStaleState removing state" podUID="0de46c2f-4a48-4d46-be31-cd2173b80b5f" containerName="revoke-keys" Oct 30 21:02:27.754551 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:27.748361 4611 memory_manager.go:365] "RemoveStaleState removing state" podUID="77368afe-edc8-4fa5-8010-d321965fb38e" containerName="main" Oct 30 21:02:27.754551 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:27.748370 4611 memory_manager.go:365] "RemoveStaleState removing state" podUID="286eab7a-81a7-4768-a7f6-be6b3720028a" containerName="main" Oct 30 21:02:27.754551 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:27.748379 4611 memory_manager.go:365] "RemoveStaleState removing state" podUID="918ff269-fb66-4479-9e8d-d0184421bc7c" containerName="istio-proxy" Oct 30 21:02:27.754551 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:27.748389 4611 memory_manager.go:365] "RemoveStaleState removing state" podUID="918ff269-fb66-4479-9e8d-d0184421bc7c" containerName="istio-proxy" Oct 30 21:02:27.754551 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:27.748397 4611 memory_manager.go:365] "RemoveStaleState removing state" podUID="1599e588-def8-42e1-b81d-ad0aaf119818" containerName="main" Oct 30 21:02:27.754551 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:27.748407 4611 memory_manager.go:365] "RemoveStaleState removing state" podUID="a08fd3b8-493c-47db-9839-462bcc598e92" containerName="istio-proxy" Oct 30 21:02:27.754551 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:27.753300 4611 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2dd9250-9jz97"] Oct 30 21:02:27.754551 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:27.753413 4611 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2dd9250-9jz97" Oct 30 21:02:27.842151 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:27.842067 4611 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:02:27.842151 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:27.842085 4611 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7hf8r\"" Oct 30 21:02:27.857634 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:27.857611 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e6311907-bd1a-4516-8cfd-78b5b4d5be46-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2dd9250-9jz97\" (UID: \"e6311907-bd1a-4516-8cfd-78b5b4d5be46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2dd9250-9jz97" Oct 30 21:02:27.857746 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:27.857652 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ccqgc\" (UniqueName: \"kubernetes.io/projected/e6311907-bd1a-4516-8cfd-78b5b4d5be46-kube-api-access-ccqgc\") pod \"maas-api-revoke-sub-e2dd9250-9jz97\" (UID: \"e6311907-bd1a-4516-8cfd-78b5b4d5be46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2dd9250-9jz97" Oct 30 21:02:27.857816 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:27.857782 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e6311907-bd1a-4516-8cfd-78b5b4d5be46-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2dd9250-9jz97\" (UID: \"e6311907-bd1a-4516-8cfd-78b5b4d5be46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2dd9250-9jz97" Oct 30 21:02:27.959341 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:27.959313 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e6311907-bd1a-4516-8cfd-78b5b4d5be46-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2dd9250-9jz97\" (UID: \"e6311907-bd1a-4516-8cfd-78b5b4d5be46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2dd9250-9jz97" Oct 30 21:02:27.959498 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:27.959361 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ccqgc\" (UniqueName: \"kubernetes.io/projected/e6311907-bd1a-4516-8cfd-78b5b4d5be46-kube-api-access-ccqgc\") pod \"maas-api-revoke-sub-e2dd9250-9jz97\" (UID: \"e6311907-bd1a-4516-8cfd-78b5b4d5be46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2dd9250-9jz97" Oct 30 21:02:27.959498 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:27.959415 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e6311907-bd1a-4516-8cfd-78b5b4d5be46-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2dd9250-9jz97\" (UID: \"e6311907-bd1a-4516-8cfd-78b5b4d5be46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2dd9250-9jz97" Oct 30 21:02:27.959993 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:27.959971 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e6311907-bd1a-4516-8cfd-78b5b4d5be46-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2dd9250-9jz97\" (UID: \"e6311907-bd1a-4516-8cfd-78b5b4d5be46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2dd9250-9jz97" Oct 30 21:02:27.961791 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:27.961770 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e6311907-bd1a-4516-8cfd-78b5b4d5be46-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2dd9250-9jz97\" (UID: \"e6311907-bd1a-4516-8cfd-78b5b4d5be46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2dd9250-9jz97" Oct 30 21:02:27.971072 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:27.971045 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ccqgc\" (UniqueName: \"kubernetes.io/projected/e6311907-bd1a-4516-8cfd-78b5b4d5be46-kube-api-access-ccqgc\") pod \"maas-api-revoke-sub-e2dd9250-9jz97\" (UID: \"e6311907-bd1a-4516-8cfd-78b5b4d5be46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2dd9250-9jz97" Oct 30 21:02:28.072866 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:28.072838 4611 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2dd9250-9jz97" Oct 30 21:02:28.349575 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:28.349545 4611 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2dd9250-9jz97"] Oct 30 21:02:28.351270 ip-10-0-133-192 kubenswrapper[4611]: W1030 21:02:28.351240 4611 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6311907_bd1a_4516_8cfd_78b5b4d5be46.slice/crio-d4b52d0337db35bee5d40d68c1cdecc5e14202017b3c90dc47e10a794833044d WatchSource:0}: Error finding container d4b52d0337db35bee5d40d68c1cdecc5e14202017b3c90dc47e10a794833044d: Status 404 returned error can't find the container with id d4b52d0337db35bee5d40d68c1cdecc5e14202017b3c90dc47e10a794833044d Oct 30 21:02:29.073019 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:29.072931 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2dd9250-9jz97" event={"ID":"e6311907-bd1a-4516-8cfd-78b5b4d5be46","Type":"ContainerStarted","Data":"f2ab44a8a2c6c2bf7927c564c71649d6389d31d4aaf0b10b27605206eb51a886"} Oct 30 21:02:29.073019 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:29.072998 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2dd9250-9jz97" event={"ID":"e6311907-bd1a-4516-8cfd-78b5b4d5be46","Type":"ContainerStarted","Data":"d4b52d0337db35bee5d40d68c1cdecc5e14202017b3c90dc47e10a794833044d"} Oct 30 21:02:29.088942 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:29.088888 4611 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2dd9250-9jz97" podStartSLOduration=2.088872811 podStartE2EDuration="2.088872811s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:02:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:02:29.088496968 +0000 UTC m=+1403.622523596" watchObservedRunningTime="2026-10-30 21:02:29.088872811 +0000 UTC m=+1403.622899427" Oct 30 21:02:30.079986 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:30.079943 4611 generic.go:356] "Generic (PLEG): container finished" podID="e6311907-bd1a-4516-8cfd-78b5b4d5be46" containerID="f2ab44a8a2c6c2bf7927c564c71649d6389d31d4aaf0b10b27605206eb51a886" exitCode=0 Oct 30 21:02:30.080435 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:30.080027 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2dd9250-9jz97" event={"ID":"e6311907-bd1a-4516-8cfd-78b5b4d5be46","Type":"ContainerDied","Data":"f2ab44a8a2c6c2bf7927c564c71649d6389d31d4aaf0b10b27605206eb51a886"} Oct 30 21:02:31.226661 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:31.226639 4611 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2dd9250-9jz97" Oct 30 21:02:31.391835 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:31.391805 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e6311907-bd1a-4516-8cfd-78b5b4d5be46-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e6311907-bd1a-4516-8cfd-78b5b4d5be46-maas-api-xks-ca\") pod \"e6311907-bd1a-4516-8cfd-78b5b4d5be46\" (UID: \"e6311907-bd1a-4516-8cfd-78b5b4d5be46\") " Oct 30 21:02:31.391991 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:31.391952 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e6311907-bd1a-4516-8cfd-78b5b4d5be46-kube-api-access-ccqgc\" (UniqueName: \"kubernetes.io/projected/e6311907-bd1a-4516-8cfd-78b5b4d5be46-kube-api-access-ccqgc\") pod \"e6311907-bd1a-4516-8cfd-78b5b4d5be46\" (UID: \"e6311907-bd1a-4516-8cfd-78b5b4d5be46\") " Oct 30 21:02:31.391991 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:31.391975 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e6311907-bd1a-4516-8cfd-78b5b4d5be46-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e6311907-bd1a-4516-8cfd-78b5b4d5be46-maas-api-openshift-ca\") pod \"e6311907-bd1a-4516-8cfd-78b5b4d5be46\" (UID: \"e6311907-bd1a-4516-8cfd-78b5b4d5be46\") " Oct 30 21:02:31.392368 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:31.392345 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e6311907-bd1a-4516-8cfd-78b5b4d5be46-maas-api-openshift-ca" pod "e6311907-bd1a-4516-8cfd-78b5b4d5be46" (UID: "e6311907-bd1a-4516-8cfd-78b5b4d5be46"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:02:31.393966 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:31.393942 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e6311907-bd1a-4516-8cfd-78b5b4d5be46-maas-api-xks-ca" pod "e6311907-bd1a-4516-8cfd-78b5b4d5be46" (UID: "e6311907-bd1a-4516-8cfd-78b5b4d5be46"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:02:31.393966 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:31.393941 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6311907-bd1a-4516-8cfd-78b5b4d5be46-kube-api-access-ccqgc" pod "e6311907-bd1a-4516-8cfd-78b5b4d5be46" (UID: "e6311907-bd1a-4516-8cfd-78b5b4d5be46"). InnerVolumeSpecName "kube-api-access-ccqgc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:02:31.493397 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:31.493373 4611 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ccqgc\" (UniqueName: \"kubernetes.io/projected/e6311907-bd1a-4516-8cfd-78b5b4d5be46-kube-api-access-ccqgc\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:02:31.493397 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:31.493396 4611 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e6311907-bd1a-4516-8cfd-78b5b4d5be46-maas-api-openshift-ca\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:02:31.493579 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:31.493405 4611 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e6311907-bd1a-4516-8cfd-78b5b4d5be46-maas-api-xks-ca\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:02:32.091761 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:32.091730 4611 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2dd9250-9jz97" Oct 30 21:02:32.091928 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:32.091762 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2dd9250-9jz97" event={"ID":"e6311907-bd1a-4516-8cfd-78b5b4d5be46","Type":"ContainerDied","Data":"d4b52d0337db35bee5d40d68c1cdecc5e14202017b3c90dc47e10a794833044d"} Oct 30 21:02:32.091928 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:32.091809 4611 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d4b52d0337db35bee5d40d68c1cdecc5e14202017b3c90dc47e10a794833044d" Oct 30 21:02:32.458908 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:32.458873 4611 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-f2ed1e-66d84b8cff-g2w49" podUID="d9298078-08c1-40a5-9117-ced750d46bd8" containerName="payload-pre-processing" containerID="cri-o://5abfa4a71e3d2a9caa482c69607cb70c5e2edb08b54218a4fc94637fffa80b1d" gracePeriod=25 Oct 30 21:02:33.917213 ip-10-0-133-192 kubenswrapper[4611]: E1030 21:02:33.917172 4611 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hwzc6" containerName="extractor" Oct 30 21:02:37.194870 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:37.194816 4611 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-f2ed1e/distinct-w4-f2ed1e-kserve-57cff9dcf8-8d5zl" podUID="653e3ec7-2147-4813-8946-1f25cfa425f6" containerName="main" containerID="cri-o://880b13fdc8d0d02d089a94a05e01562d0e2b604022bb39fe0938b3ef30ccee2a" gracePeriod=45 Oct 30 21:02:37.444850 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:37.444827 4611 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-f2ed1e/distinct-w4-f2ed1e-kserve-57cff9dcf8-8d5zl" Oct 30 21:02:37.560862 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:37.560782 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/653e3ec7-2147-4813-8946-1f25cfa425f6-dshm\" (UniqueName: \"kubernetes.io/empty-dir/653e3ec7-2147-4813-8946-1f25cfa425f6-dshm\") pod \"653e3ec7-2147-4813-8946-1f25cfa425f6\" (UID: \"653e3ec7-2147-4813-8946-1f25cfa425f6\") " Oct 30 21:02:37.560862 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:37.560823 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/653e3ec7-2147-4813-8946-1f25cfa425f6-home\" (UniqueName: \"kubernetes.io/empty-dir/653e3ec7-2147-4813-8946-1f25cfa425f6-home\") pod \"653e3ec7-2147-4813-8946-1f25cfa425f6\" (UID: \"653e3ec7-2147-4813-8946-1f25cfa425f6\") " Oct 30 21:02:37.561061 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:37.560886 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/653e3ec7-2147-4813-8946-1f25cfa425f6-kube-api-access-j7zwk\" (UniqueName: \"kubernetes.io/projected/653e3ec7-2147-4813-8946-1f25cfa425f6-kube-api-access-j7zwk\") pod \"653e3ec7-2147-4813-8946-1f25cfa425f6\" (UID: \"653e3ec7-2147-4813-8946-1f25cfa425f6\") " Oct 30 21:02:37.561061 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:37.560983 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/653e3ec7-2147-4813-8946-1f25cfa425f6-tls-certs\" (UniqueName: \"kubernetes.io/secret/653e3ec7-2147-4813-8946-1f25cfa425f6-tls-certs\") pod \"653e3ec7-2147-4813-8946-1f25cfa425f6\" (UID: \"653e3ec7-2147-4813-8946-1f25cfa425f6\") " Oct 30 21:02:37.561186 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:37.561168 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/653e3ec7-2147-4813-8946-1f25cfa425f6-home" pod "653e3ec7-2147-4813-8946-1f25cfa425f6" (UID: "653e3ec7-2147-4813-8946-1f25cfa425f6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:02:37.561284 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:37.561255 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/653e3ec7-2147-4813-8946-1f25cfa425f6-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/653e3ec7-2147-4813-8946-1f25cfa425f6-model-cache\") pod \"653e3ec7-2147-4813-8946-1f25cfa425f6\" (UID: \"653e3ec7-2147-4813-8946-1f25cfa425f6\") " Oct 30 21:02:37.561382 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:37.561365 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/653e3ec7-2147-4813-8946-1f25cfa425f6-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/653e3ec7-2147-4813-8946-1f25cfa425f6-tmp-dir\") pod \"653e3ec7-2147-4813-8946-1f25cfa425f6\" (UID: \"653e3ec7-2147-4813-8946-1f25cfa425f6\") " Oct 30 21:02:37.561565 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:37.561541 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/653e3ec7-2147-4813-8946-1f25cfa425f6-model-cache" pod "653e3ec7-2147-4813-8946-1f25cfa425f6" (UID: "653e3ec7-2147-4813-8946-1f25cfa425f6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:02:37.561797 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:37.561768 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/653e3ec7-2147-4813-8946-1f25cfa425f6-tmp-dir" pod "653e3ec7-2147-4813-8946-1f25cfa425f6" (UID: "653e3ec7-2147-4813-8946-1f25cfa425f6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:02:37.561927 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:37.561898 4611 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/653e3ec7-2147-4813-8946-1f25cfa425f6-tmp-dir\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:02:37.561927 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:37.561919 4611 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/653e3ec7-2147-4813-8946-1f25cfa425f6-home\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:02:37.562048 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:37.561933 4611 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/653e3ec7-2147-4813-8946-1f25cfa425f6-model-cache\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:02:37.563215 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:37.563187 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/653e3ec7-2147-4813-8946-1f25cfa425f6-dshm" pod "653e3ec7-2147-4813-8946-1f25cfa425f6" (UID: "653e3ec7-2147-4813-8946-1f25cfa425f6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:02:37.563338 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:37.563316 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/653e3ec7-2147-4813-8946-1f25cfa425f6-tls-certs" pod "653e3ec7-2147-4813-8946-1f25cfa425f6" (UID: "653e3ec7-2147-4813-8946-1f25cfa425f6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:02:37.563453 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:37.563436 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/653e3ec7-2147-4813-8946-1f25cfa425f6-kube-api-access-j7zwk" pod "653e3ec7-2147-4813-8946-1f25cfa425f6" (UID: "653e3ec7-2147-4813-8946-1f25cfa425f6"). InnerVolumeSpecName "kube-api-access-j7zwk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:02:37.663211 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:37.663182 4611 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/653e3ec7-2147-4813-8946-1f25cfa425f6-tls-certs\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:02:37.663211 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:37.663210 4611 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/653e3ec7-2147-4813-8946-1f25cfa425f6-dshm\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:02:37.663379 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:37.663220 4611 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j7zwk\" (UniqueName: \"kubernetes.io/projected/653e3ec7-2147-4813-8946-1f25cfa425f6-kube-api-access-j7zwk\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:02:38.124388 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:38.124348 4611 generic.go:356] "Generic (PLEG): container finished" podID="653e3ec7-2147-4813-8946-1f25cfa425f6" containerID="880b13fdc8d0d02d089a94a05e01562d0e2b604022bb39fe0938b3ef30ccee2a" exitCode=0 Oct 30 21:02:38.124629 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:38.124395 4611 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-f2ed1e/distinct-w4-f2ed1e-kserve-57cff9dcf8-8d5zl" Oct 30 21:02:38.124629 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:38.124438 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-f2ed1e/distinct-w4-f2ed1e-kserve-57cff9dcf8-8d5zl" event={"ID":"653e3ec7-2147-4813-8946-1f25cfa425f6","Type":"ContainerDied","Data":"880b13fdc8d0d02d089a94a05e01562d0e2b604022bb39fe0938b3ef30ccee2a"} Oct 30 21:02:38.124629 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:38.124485 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-f2ed1e/distinct-w4-f2ed1e-kserve-57cff9dcf8-8d5zl" event={"ID":"653e3ec7-2147-4813-8946-1f25cfa425f6","Type":"ContainerDied","Data":"5616fdc6af3f8b43a6b4dafef0901dc015f7baa74ecd3dab04422331604d1b7b"} Oct 30 21:02:38.124629 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:38.124534 4611 scope.go:122] "RemoveContainer" containerID="880b13fdc8d0d02d089a94a05e01562d0e2b604022bb39fe0938b3ef30ccee2a" Oct 30 21:02:38.133329 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:38.133306 4611 scope.go:122] "RemoveContainer" containerID="880b13fdc8d0d02d089a94a05e01562d0e2b604022bb39fe0938b3ef30ccee2a" Oct 30 21:02:38.133647 ip-10-0-133-192 kubenswrapper[4611]: E1030 21:02:38.133623 4611 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"880b13fdc8d0d02d089a94a05e01562d0e2b604022bb39fe0938b3ef30ccee2a\": container with ID starting with 880b13fdc8d0d02d089a94a05e01562d0e2b604022bb39fe0938b3ef30ccee2a not found: ID does not exist" containerID="880b13fdc8d0d02d089a94a05e01562d0e2b604022bb39fe0938b3ef30ccee2a" Oct 30 21:02:38.133733 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:38.133654 4611 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"880b13fdc8d0d02d089a94a05e01562d0e2b604022bb39fe0938b3ef30ccee2a"} err="failed to get container status \"880b13fdc8d0d02d089a94a05e01562d0e2b604022bb39fe0938b3ef30ccee2a\": rpc error: code = NotFound desc = could not find container \"880b13fdc8d0d02d089a94a05e01562d0e2b604022bb39fe0938b3ef30ccee2a\": container with ID starting with 880b13fdc8d0d02d089a94a05e01562d0e2b604022bb39fe0938b3ef30ccee2a not found: ID does not exist" Oct 30 21:02:38.140761 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:38.140736 4611 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-f2ed1e/distinct-w4-f2ed1e-kserve-57cff9dcf8-8d5zl"] Oct 30 21:02:38.143112 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:38.143091 4611 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-f2ed1e/distinct-w4-f2ed1e-kserve-57cff9dcf8-8d5zl"] Oct 30 21:02:39.605431 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:39.605397 4611 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="653e3ec7-2147-4813-8946-1f25cfa425f6" path="/var/lib/kubelet/pods/653e3ec7-2147-4813-8946-1f25cfa425f6/volumes" Oct 30 21:02:42.139148 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:42.139109 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-f2ed1e-66d84b8cff-g2w49" podUID="d9298078-08c1-40a5-9117-ced750d46bd8" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 21:02:42.732759 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:42.732726 4611 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-f2ed1e-c97788dc8-hnqvv"] Oct 30 21:02:42.733069 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:42.732967 4611 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-f2ed1e-c97788dc8-hnqvv" podUID="5052ef1d-3134-4545-8f2e-2c6ef046ccf6" containerName="maas-api" containerID="cri-o://75fb518252c8b2154f6c0306becaa50e1400b2d57efc9eb8f087e731dcfb3755" gracePeriod=30 Oct 30 21:02:42.833034 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:42.833004 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:02:42.985500 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:42.985432 4611 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-f2ed1e-c97788dc8-hnqvv" Oct 30 21:02:42.987963 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:42.987919 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:02:43.035430 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:43.035378 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5052ef1d-3134-4545-8f2e-2c6ef046ccf6-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5052ef1d-3134-4545-8f2e-2c6ef046ccf6-maas-api-tls\") pod \"5052ef1d-3134-4545-8f2e-2c6ef046ccf6\" (UID: \"5052ef1d-3134-4545-8f2e-2c6ef046ccf6\") " Oct 30 21:02:43.035649 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:43.035439 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5052ef1d-3134-4545-8f2e-2c6ef046ccf6-kube-api-access-x74kg\" (UniqueName: \"kubernetes.io/projected/5052ef1d-3134-4545-8f2e-2c6ef046ccf6-kube-api-access-x74kg\") pod \"5052ef1d-3134-4545-8f2e-2c6ef046ccf6\" (UID: \"5052ef1d-3134-4545-8f2e-2c6ef046ccf6\") " Oct 30 21:02:43.035649 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:43.035563 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5052ef1d-3134-4545-8f2e-2c6ef046ccf6-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5052ef1d-3134-4545-8f2e-2c6ef046ccf6-openshift-service-ca\") pod \"5052ef1d-3134-4545-8f2e-2c6ef046ccf6\" (UID: \"5052ef1d-3134-4545-8f2e-2c6ef046ccf6\") " Oct 30 21:02:43.035649 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:43.035589 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5052ef1d-3134-4545-8f2e-2c6ef046ccf6-metrics-certs\" (UniqueName: \"kubernetes.io/secret/5052ef1d-3134-4545-8f2e-2c6ef046ccf6-metrics-certs\") pod \"5052ef1d-3134-4545-8f2e-2c6ef046ccf6\" (UID: \"5052ef1d-3134-4545-8f2e-2c6ef046ccf6\") " Oct 30 21:02:43.036962 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:43.036919 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5052ef1d-3134-4545-8f2e-2c6ef046ccf6-openshift-service-ca" pod "5052ef1d-3134-4545-8f2e-2c6ef046ccf6" (UID: "5052ef1d-3134-4545-8f2e-2c6ef046ccf6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:02:43.042098 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:43.042065 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5052ef1d-3134-4545-8f2e-2c6ef046ccf6-kube-api-access-x74kg" pod "5052ef1d-3134-4545-8f2e-2c6ef046ccf6" (UID: "5052ef1d-3134-4545-8f2e-2c6ef046ccf6"). InnerVolumeSpecName "kube-api-access-x74kg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:02:43.042098 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:43.042087 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5052ef1d-3134-4545-8f2e-2c6ef046ccf6-maas-api-tls" pod "5052ef1d-3134-4545-8f2e-2c6ef046ccf6" (UID: "5052ef1d-3134-4545-8f2e-2c6ef046ccf6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:02:43.042264 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:43.042137 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5052ef1d-3134-4545-8f2e-2c6ef046ccf6-metrics-certs" pod "5052ef1d-3134-4545-8f2e-2c6ef046ccf6" (UID: "5052ef1d-3134-4545-8f2e-2c6ef046ccf6"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:02:43.137399 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:43.137365 4611 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5052ef1d-3134-4545-8f2e-2c6ef046ccf6-maas-api-tls\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:02:43.137399 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:43.137397 4611 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x74kg\" (UniqueName: \"kubernetes.io/projected/5052ef1d-3134-4545-8f2e-2c6ef046ccf6-kube-api-access-x74kg\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:02:43.137654 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:43.137411 4611 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5052ef1d-3134-4545-8f2e-2c6ef046ccf6-openshift-service-ca\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:02:43.137654 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:43.137425 4611 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5052ef1d-3134-4545-8f2e-2c6ef046ccf6-metrics-certs\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:02:43.154649 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:43.154618 4611 generic.go:356] "Generic (PLEG): container finished" podID="5052ef1d-3134-4545-8f2e-2c6ef046ccf6" containerID="75fb518252c8b2154f6c0306becaa50e1400b2d57efc9eb8f087e731dcfb3755" exitCode=0 Oct 30 21:02:43.155050 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:43.154655 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-f2ed1e-c97788dc8-hnqvv" event={"ID":"5052ef1d-3134-4545-8f2e-2c6ef046ccf6","Type":"ContainerDied","Data":"75fb518252c8b2154f6c0306becaa50e1400b2d57efc9eb8f087e731dcfb3755"} Oct 30 21:02:43.155050 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:43.154660 4611 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-f2ed1e-c97788dc8-hnqvv" Oct 30 21:02:43.155050 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:43.154702 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-f2ed1e-c97788dc8-hnqvv" event={"ID":"5052ef1d-3134-4545-8f2e-2c6ef046ccf6","Type":"ContainerDied","Data":"abb64141453fe03188fd2e696a0c51c60500325cacb72fa46e46d4a16153d05a"} Oct 30 21:02:43.155050 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:43.154724 4611 scope.go:122] "RemoveContainer" containerID="75fb518252c8b2154f6c0306becaa50e1400b2d57efc9eb8f087e731dcfb3755" Oct 30 21:02:43.165999 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:43.165980 4611 scope.go:122] "RemoveContainer" containerID="75fb518252c8b2154f6c0306becaa50e1400b2d57efc9eb8f087e731dcfb3755" Oct 30 21:02:43.166265 ip-10-0-133-192 kubenswrapper[4611]: E1030 21:02:43.166243 4611 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"75fb518252c8b2154f6c0306becaa50e1400b2d57efc9eb8f087e731dcfb3755\": container with ID starting with 75fb518252c8b2154f6c0306becaa50e1400b2d57efc9eb8f087e731dcfb3755 not found: ID does not exist" containerID="75fb518252c8b2154f6c0306becaa50e1400b2d57efc9eb8f087e731dcfb3755" Oct 30 21:02:43.166329 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:43.166274 4611 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"75fb518252c8b2154f6c0306becaa50e1400b2d57efc9eb8f087e731dcfb3755"} err="failed to get container status \"75fb518252c8b2154f6c0306becaa50e1400b2d57efc9eb8f087e731dcfb3755\": rpc error: code = NotFound desc = could not find container \"75fb518252c8b2154f6c0306becaa50e1400b2d57efc9eb8f087e731dcfb3755\": container with ID starting with 75fb518252c8b2154f6c0306becaa50e1400b2d57efc9eb8f087e731dcfb3755 not found: ID does not exist" Oct 30 21:02:43.179655 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:43.179631 4611 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-f2ed1e-c97788dc8-hnqvv"] Oct 30 21:02:43.190078 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:43.190056 4611 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-f2ed1e-c97788dc8-hnqvv"] Oct 30 21:02:43.199336 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:43.199294 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:02:43.561656 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:43.561618 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:02:43.607668 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:43.607631 4611 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5052ef1d-3134-4545-8f2e-2c6ef046ccf6" path="/var/lib/kubelet/pods/5052ef1d-3134-4545-8f2e-2c6ef046ccf6/volumes" Oct 30 21:02:49.120347 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:49.120312 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:02:49.659288 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:49.659251 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:02:49.825357 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:49.825318 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:02:49.970950 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:49.970866 4611 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-hx7mv"] Oct 30 21:02:49.971123 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:49.971085 4611 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-hx7mv" podUID="52f7cad0-6a73-4573-9789-59ac6df33da4" containerName="istio-proxy" containerID="cri-o://0028b06d562d6e065e5b330b2875605dc7238ba219b2e2f402fbd9a06b916e9f" gracePeriod=30 Oct 30 21:02:49.978464 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:49.978436 4611 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-vpbzr"] Oct 30 21:02:49.978720 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:49.978666 4611 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-vpbzr" podUID="b6fdc9f9-8be5-4dfa-9390-3ca43a74b6be" containerName="istio-proxy" containerID="cri-o://2bc814836d17daacf5ccf26f67b858b82e8a583d3005c1ab3df30b51d13ef359" gracePeriod=30 Oct 30 21:02:49.982482 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:49.982457 4611 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-qvfk5"] Oct 30 21:02:49.982719 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:49.982691 4611 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-qvfk5" podUID="527a4fc6-4bfe-49c5-9352-d5b82a74e7ff" containerName="istio-proxy" containerID="cri-o://07f6f4705c682af2e74dcdad8363c9ce1eee6a59a57f010b76fd28ad40d54386" gracePeriod=30 Oct 30 21:02:50.076060 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:50.075953 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:02:52.138950 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:52.138911 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-f2ed1e-66d84b8cff-g2w49" podUID="d9298078-08c1-40a5-9117-ced750d46bd8" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 21:02:52.711937 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:52.711904 4611 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-f2ed1e-66d84b8cff-g2w49" Oct 30 21:02:52.855317 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:52.855294 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d9298078-08c1-40a5-9117-ced750d46bd8-kube-api-access-wbxlv\" (UniqueName: \"kubernetes.io/projected/d9298078-08c1-40a5-9117-ced750d46bd8-kube-api-access-wbxlv\") pod \"d9298078-08c1-40a5-9117-ced750d46bd8\" (UID: \"d9298078-08c1-40a5-9117-ced750d46bd8\") " Oct 30 21:02:52.855470 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:52.855343 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d9298078-08c1-40a5-9117-ced750d46bd8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9298078-08c1-40a5-9117-ced750d46bd8-plugins-config-volume\") pod \"d9298078-08c1-40a5-9117-ced750d46bd8\" (UID: \"d9298078-08c1-40a5-9117-ced750d46bd8\") " Oct 30 21:02:52.855820 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:52.855788 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9298078-08c1-40a5-9117-ced750d46bd8-plugins-config-volume" pod "d9298078-08c1-40a5-9117-ced750d46bd8" (UID: "d9298078-08c1-40a5-9117-ced750d46bd8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:02:52.857372 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:52.857347 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9298078-08c1-40a5-9117-ced750d46bd8-kube-api-access-wbxlv" pod "d9298078-08c1-40a5-9117-ced750d46bd8" (UID: "d9298078-08c1-40a5-9117-ced750d46bd8"). InnerVolumeSpecName "kube-api-access-wbxlv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:02:52.956748 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:52.956718 4611 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wbxlv\" (UniqueName: \"kubernetes.io/projected/d9298078-08c1-40a5-9117-ced750d46bd8-kube-api-access-wbxlv\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:02:52.956748 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:52.956744 4611 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9298078-08c1-40a5-9117-ced750d46bd8-plugins-config-volume\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:02:53.246424 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:53.246327 4611 generic.go:356] "Generic (PLEG): container finished" podID="d9298078-08c1-40a5-9117-ced750d46bd8" containerID="5abfa4a71e3d2a9caa482c69607cb70c5e2edb08b54218a4fc94637fffa80b1d" exitCode=0 Oct 30 21:02:53.246424 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:53.246371 4611 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-f2ed1e-66d84b8cff-g2w49" Oct 30 21:02:53.246424 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:53.246406 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-f2ed1e-66d84b8cff-g2w49" event={"ID":"d9298078-08c1-40a5-9117-ced750d46bd8","Type":"ContainerDied","Data":"5abfa4a71e3d2a9caa482c69607cb70c5e2edb08b54218a4fc94637fffa80b1d"} Oct 30 21:02:53.247026 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:53.246461 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-f2ed1e-66d84b8cff-g2w49" event={"ID":"d9298078-08c1-40a5-9117-ced750d46bd8","Type":"ContainerDied","Data":"515c987bcb9b42118a95a267c0dfd3ad8ebe4c23cc564cb243e8d849a900ef9e"} Oct 30 21:02:53.247026 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:53.246483 4611 scope.go:122] "RemoveContainer" containerID="5abfa4a71e3d2a9caa482c69607cb70c5e2edb08b54218a4fc94637fffa80b1d" Oct 30 21:02:53.255590 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:53.255575 4611 scope.go:122] "RemoveContainer" containerID="5abfa4a71e3d2a9caa482c69607cb70c5e2edb08b54218a4fc94637fffa80b1d" Oct 30 21:02:53.255857 ip-10-0-133-192 kubenswrapper[4611]: E1030 21:02:53.255836 4611 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5abfa4a71e3d2a9caa482c69607cb70c5e2edb08b54218a4fc94637fffa80b1d\": container with ID starting with 5abfa4a71e3d2a9caa482c69607cb70c5e2edb08b54218a4fc94637fffa80b1d not found: ID does not exist" containerID="5abfa4a71e3d2a9caa482c69607cb70c5e2edb08b54218a4fc94637fffa80b1d" Oct 30 21:02:53.255910 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:53.255865 4611 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5abfa4a71e3d2a9caa482c69607cb70c5e2edb08b54218a4fc94637fffa80b1d"} err="failed to get container status \"5abfa4a71e3d2a9caa482c69607cb70c5e2edb08b54218a4fc94637fffa80b1d\": rpc error: code = NotFound desc = could not find container \"5abfa4a71e3d2a9caa482c69607cb70c5e2edb08b54218a4fc94637fffa80b1d\": container with ID starting with 5abfa4a71e3d2a9caa482c69607cb70c5e2edb08b54218a4fc94637fffa80b1d not found: ID does not exist" Oct 30 21:02:53.267420 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:53.267390 4611 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-f2ed1e-66d84b8cff-g2w49"] Oct 30 21:02:53.274050 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:53.274031 4611 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-f2ed1e-66d84b8cff-g2w49"] Oct 30 21:02:53.604862 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:53.604833 4611 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d9298078-08c1-40a5-9117-ced750d46bd8" path="/var/lib/kubelet/pods/d9298078-08c1-40a5-9117-ced750d46bd8/volumes" Oct 30 21:02:53.808216 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:53.808184 4611 patch_prober.go:29] interesting pod/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-qvfk5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 21:02:53.808457 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:53.808237 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-qvfk5" podUID="527a4fc6-4bfe-49c5-9352-d5b82a74e7ff" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 21:02:54.822072 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:54.822039 4611 patch_prober.go:29] interesting pod/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-vpbzr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 21:02:54.822496 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:54.822098 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-vpbzr" podUID="b6fdc9f9-8be5-4dfa-9390-3ca43a74b6be" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 21:02:55.236706 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:55.236674 4611 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-vpbzr" Oct 30 21:02:55.261248 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:55.260129 4611 generic.go:356] "Generic (PLEG): container finished" podID="527a4fc6-4bfe-49c5-9352-d5b82a74e7ff" containerID="07f6f4705c682af2e74dcdad8363c9ce1eee6a59a57f010b76fd28ad40d54386" exitCode=0 Oct 30 21:02:55.261385 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:55.261364 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-qvfk5" event={"ID":"527a4fc6-4bfe-49c5-9352-d5b82a74e7ff","Type":"ContainerDied","Data":"07f6f4705c682af2e74dcdad8363c9ce1eee6a59a57f010b76fd28ad40d54386"} Oct 30 21:02:55.263558 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:55.263493 4611 generic.go:356] "Generic (PLEG): container finished" podID="b6fdc9f9-8be5-4dfa-9390-3ca43a74b6be" containerID="2bc814836d17daacf5ccf26f67b858b82e8a583d3005c1ab3df30b51d13ef359" exitCode=0 Oct 30 21:02:55.263712 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:55.263599 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-vpbzr" event={"ID":"b6fdc9f9-8be5-4dfa-9390-3ca43a74b6be","Type":"ContainerDied","Data":"2bc814836d17daacf5ccf26f67b858b82e8a583d3005c1ab3df30b51d13ef359"} Oct 30 21:02:55.263712 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:55.263638 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-vpbzr" event={"ID":"b6fdc9f9-8be5-4dfa-9390-3ca43a74b6be","Type":"ContainerDied","Data":"e90c4d6b384e65bdb28d8389ea63c41a6760d1a9cf1dc486dcbdd84ae41b3b0e"} Oct 30 21:02:55.263712 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:55.263662 4611 scope.go:122] "RemoveContainer" containerID="2bc814836d17daacf5ccf26f67b858b82e8a583d3005c1ab3df30b51d13ef359" Oct 30 21:02:55.263887 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:55.263781 4611 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-vpbzr" Oct 30 21:02:55.266120 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:55.266097 4611 generic.go:356] "Generic (PLEG): container finished" podID="52f7cad0-6a73-4573-9789-59ac6df33da4" containerID="0028b06d562d6e065e5b330b2875605dc7238ba219b2e2f402fbd9a06b916e9f" exitCode=0 Oct 30 21:02:55.266213 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:55.266134 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-hx7mv" event={"ID":"52f7cad0-6a73-4573-9789-59ac6df33da4","Type":"ContainerDied","Data":"0028b06d562d6e065e5b330b2875605dc7238ba219b2e2f402fbd9a06b916e9f"} Oct 30 21:02:55.266689 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:55.266673 4611 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-hx7mv" Oct 30 21:02:55.274572 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:55.274307 4611 scope.go:122] "RemoveContainer" containerID="2bc814836d17daacf5ccf26f67b858b82e8a583d3005c1ab3df30b51d13ef359" Oct 30 21:02:55.274684 ip-10-0-133-192 kubenswrapper[4611]: E1030 21:02:55.274647 4611 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2bc814836d17daacf5ccf26f67b858b82e8a583d3005c1ab3df30b51d13ef359\": container with ID starting with 2bc814836d17daacf5ccf26f67b858b82e8a583d3005c1ab3df30b51d13ef359 not found: ID does not exist" containerID="2bc814836d17daacf5ccf26f67b858b82e8a583d3005c1ab3df30b51d13ef359" Oct 30 21:02:55.274762 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:55.274675 4611 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2bc814836d17daacf5ccf26f67b858b82e8a583d3005c1ab3df30b51d13ef359"} err="failed to get container status \"2bc814836d17daacf5ccf26f67b858b82e8a583d3005c1ab3df30b51d13ef359\": rpc error: code = NotFound desc = could not find container \"2bc814836d17daacf5ccf26f67b858b82e8a583d3005c1ab3df30b51d13ef359\": container with ID starting with 2bc814836d17daacf5ccf26f67b858b82e8a583d3005c1ab3df30b51d13ef359 not found: ID does not exist" Oct 30 21:02:55.274762 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:55.274696 4611 scope.go:122] "RemoveContainer" containerID="91d09dc6096a9bc89e3ea13fab3ce3ddb29cc39689b61711253cc60ce5c9db06" Oct 30 21:02:55.284913 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:55.284894 4611 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-qvfk5" Oct 30 21:02:55.378993 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:55.378958 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/527a4fc6-4bfe-49c5-9352-d5b82a74e7ff-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/527a4fc6-4bfe-49c5-9352-d5b82a74e7ff-istiod-ca-cert\") pod \"527a4fc6-4bfe-49c5-9352-d5b82a74e7ff\" (UID: \"527a4fc6-4bfe-49c5-9352-d5b82a74e7ff\") " Oct 30 21:02:55.379190 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:55.379000 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6fdc9f9-8be5-4dfa-9390-3ca43a74b6be-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6fdc9f9-8be5-4dfa-9390-3ca43a74b6be-workload-socket\") pod \"b6fdc9f9-8be5-4dfa-9390-3ca43a74b6be\" (UID: \"b6fdc9f9-8be5-4dfa-9390-3ca43a74b6be\") " Oct 30 21:02:55.379190 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:55.379021 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52f7cad0-6a73-4573-9789-59ac6df33da4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52f7cad0-6a73-4573-9789-59ac6df33da4-workload-socket\") pod \"52f7cad0-6a73-4573-9789-59ac6df33da4\" (UID: \"52f7cad0-6a73-4573-9789-59ac6df33da4\") " Oct 30 21:02:55.379190 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:55.379045 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52f7cad0-6a73-4573-9789-59ac6df33da4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52f7cad0-6a73-4573-9789-59ac6df33da4-credential-socket\") pod \"52f7cad0-6a73-4573-9789-59ac6df33da4\" (UID: \"52f7cad0-6a73-4573-9789-59ac6df33da4\") " Oct 30 21:02:55.379190 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:55.379064 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/527a4fc6-4bfe-49c5-9352-d5b82a74e7ff-istio-token\" (UniqueName: \"kubernetes.io/projected/527a4fc6-4bfe-49c5-9352-d5b82a74e7ff-istio-token\") pod \"527a4fc6-4bfe-49c5-9352-d5b82a74e7ff\" (UID: \"527a4fc6-4bfe-49c5-9352-d5b82a74e7ff\") " Oct 30 21:02:55.379190 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:55.379084 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/527a4fc6-4bfe-49c5-9352-d5b82a74e7ff-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/527a4fc6-4bfe-49c5-9352-d5b82a74e7ff-istio-podinfo\") pod \"527a4fc6-4bfe-49c5-9352-d5b82a74e7ff\" (UID: \"527a4fc6-4bfe-49c5-9352-d5b82a74e7ff\") " Oct 30 21:02:55.379190 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:55.379107 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6fdc9f9-8be5-4dfa-9390-3ca43a74b6be-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b6fdc9f9-8be5-4dfa-9390-3ca43a74b6be-workload-certs\") pod \"b6fdc9f9-8be5-4dfa-9390-3ca43a74b6be\" (UID: \"b6fdc9f9-8be5-4dfa-9390-3ca43a74b6be\") " Oct 30 21:02:55.379190 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:55.379146 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52f7cad0-6a73-4573-9789-59ac6df33da4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52f7cad0-6a73-4573-9789-59ac6df33da4-istio-envoy\") pod \"52f7cad0-6a73-4573-9789-59ac6df33da4\" (UID: \"52f7cad0-6a73-4573-9789-59ac6df33da4\") " Oct 30 21:02:55.379190 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:55.379190 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52f7cad0-6a73-4573-9789-59ac6df33da4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52f7cad0-6a73-4573-9789-59ac6df33da4-workload-certs\") pod \"52f7cad0-6a73-4573-9789-59ac6df33da4\" (UID: \"52f7cad0-6a73-4573-9789-59ac6df33da4\") " Oct 30 21:02:55.379671 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:55.379230 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/527a4fc6-4bfe-49c5-9352-d5b82a74e7ff-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/527a4fc6-4bfe-49c5-9352-d5b82a74e7ff-workload-socket\") pod \"527a4fc6-4bfe-49c5-9352-d5b82a74e7ff\" (UID: \"527a4fc6-4bfe-49c5-9352-d5b82a74e7ff\") " Oct 30 21:02:55.379671 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:55.379268 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/527a4fc6-4bfe-49c5-9352-d5b82a74e7ff-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/527a4fc6-4bfe-49c5-9352-d5b82a74e7ff-istio-data\") pod \"527a4fc6-4bfe-49c5-9352-d5b82a74e7ff\" (UID: \"527a4fc6-4bfe-49c5-9352-d5b82a74e7ff\") " Oct 30 21:02:55.379671 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:55.379291 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b6fdc9f9-8be5-4dfa-9390-3ca43a74b6be-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b6fdc9f9-8be5-4dfa-9390-3ca43a74b6be-istio-podinfo\") pod \"b6fdc9f9-8be5-4dfa-9390-3ca43a74b6be\" (UID: \"b6fdc9f9-8be5-4dfa-9390-3ca43a74b6be\") " Oct 30 21:02:55.379671 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:55.379319 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b6fdc9f9-8be5-4dfa-9390-3ca43a74b6be-kube-api-access-72mrx\" (UniqueName: \"kubernetes.io/projected/b6fdc9f9-8be5-4dfa-9390-3ca43a74b6be-kube-api-access-72mrx\") pod \"b6fdc9f9-8be5-4dfa-9390-3ca43a74b6be\" (UID: \"b6fdc9f9-8be5-4dfa-9390-3ca43a74b6be\") " Oct 30 21:02:55.379671 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:55.379322 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6fdc9f9-8be5-4dfa-9390-3ca43a74b6be-workload-socket" pod "b6fdc9f9-8be5-4dfa-9390-3ca43a74b6be" (UID: "b6fdc9f9-8be5-4dfa-9390-3ca43a74b6be"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:02:55.379671 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:55.379344 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52f7cad0-6a73-4573-9789-59ac6df33da4-credential-socket" pod "52f7cad0-6a73-4573-9789-59ac6df33da4" (UID: "52f7cad0-6a73-4573-9789-59ac6df33da4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:02:55.379671 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:55.379352 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52f7cad0-6a73-4573-9789-59ac6df33da4-istio-token\" (UniqueName: \"kubernetes.io/projected/52f7cad0-6a73-4573-9789-59ac6df33da4-istio-token\") pod \"52f7cad0-6a73-4573-9789-59ac6df33da4\" (UID: \"52f7cad0-6a73-4573-9789-59ac6df33da4\") " Oct 30 21:02:55.379671 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:55.379465 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b6fdc9f9-8be5-4dfa-9390-3ca43a74b6be-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b6fdc9f9-8be5-4dfa-9390-3ca43a74b6be-istiod-ca-cert\") pod \"b6fdc9f9-8be5-4dfa-9390-3ca43a74b6be\" (UID: \"b6fdc9f9-8be5-4dfa-9390-3ca43a74b6be\") " Oct 30 21:02:55.379671 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:55.379496 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52f7cad0-6a73-4573-9789-59ac6df33da4-workload-certs" pod "52f7cad0-6a73-4573-9789-59ac6df33da4" (UID: "52f7cad0-6a73-4573-9789-59ac6df33da4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:02:55.379671 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:55.379503 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52f7cad0-6a73-4573-9789-59ac6df33da4-kube-api-access-7ktlz\" (UniqueName: \"kubernetes.io/projected/52f7cad0-6a73-4573-9789-59ac6df33da4-kube-api-access-7ktlz\") pod \"52f7cad0-6a73-4573-9789-59ac6df33da4\" (UID: \"52f7cad0-6a73-4573-9789-59ac6df33da4\") " Oct 30 21:02:55.379671 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:55.379535 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/527a4fc6-4bfe-49c5-9352-d5b82a74e7ff-istiod-ca-cert" pod "527a4fc6-4bfe-49c5-9352-d5b82a74e7ff" (UID: "527a4fc6-4bfe-49c5-9352-d5b82a74e7ff"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:02:55.379671 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:55.379563 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6fdc9f9-8be5-4dfa-9390-3ca43a74b6be-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b6fdc9f9-8be5-4dfa-9390-3ca43a74b6be-istio-envoy\") pod \"b6fdc9f9-8be5-4dfa-9390-3ca43a74b6be\" (UID: \"b6fdc9f9-8be5-4dfa-9390-3ca43a74b6be\") " Oct 30 21:02:55.379671 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:55.379602 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52f7cad0-6a73-4573-9789-59ac6df33da4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52f7cad0-6a73-4573-9789-59ac6df33da4-istio-data\") pod \"52f7cad0-6a73-4573-9789-59ac6df33da4\" (UID: \"52f7cad0-6a73-4573-9789-59ac6df33da4\") " Oct 30 21:02:55.379671 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:55.379614 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6fdc9f9-8be5-4dfa-9390-3ca43a74b6be-workload-certs" pod "b6fdc9f9-8be5-4dfa-9390-3ca43a74b6be" (UID: "b6fdc9f9-8be5-4dfa-9390-3ca43a74b6be"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:02:55.379671 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:55.379636 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/527a4fc6-4bfe-49c5-9352-d5b82a74e7ff-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/527a4fc6-4bfe-49c5-9352-d5b82a74e7ff-workload-certs\") pod \"527a4fc6-4bfe-49c5-9352-d5b82a74e7ff\" (UID: \"527a4fc6-4bfe-49c5-9352-d5b82a74e7ff\") " Oct 30 21:02:55.380431 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:55.379711 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/52f7cad0-6a73-4573-9789-59ac6df33da4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52f7cad0-6a73-4573-9789-59ac6df33da4-istio-podinfo\") pod \"52f7cad0-6a73-4573-9789-59ac6df33da4\" (UID: \"52f7cad0-6a73-4573-9789-59ac6df33da4\") " Oct 30 21:02:55.380431 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:55.379769 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6fdc9f9-8be5-4dfa-9390-3ca43a74b6be-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b6fdc9f9-8be5-4dfa-9390-3ca43a74b6be-istio-data\") pod \"b6fdc9f9-8be5-4dfa-9390-3ca43a74b6be\" (UID: \"b6fdc9f9-8be5-4dfa-9390-3ca43a74b6be\") " Oct 30 21:02:55.380431 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:55.379801 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6fdc9f9-8be5-4dfa-9390-3ca43a74b6be-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6fdc9f9-8be5-4dfa-9390-3ca43a74b6be-credential-socket\") pod \"b6fdc9f9-8be5-4dfa-9390-3ca43a74b6be\" (UID: \"b6fdc9f9-8be5-4dfa-9390-3ca43a74b6be\") " Oct 30 21:02:55.380431 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:55.379818 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/527a4fc6-4bfe-49c5-9352-d5b82a74e7ff-istio-data" pod "527a4fc6-4bfe-49c5-9352-d5b82a74e7ff" (UID: "527a4fc6-4bfe-49c5-9352-d5b82a74e7ff"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:02:55.380431 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:55.379850 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b6fdc9f9-8be5-4dfa-9390-3ca43a74b6be-istio-token\" (UniqueName: \"kubernetes.io/projected/b6fdc9f9-8be5-4dfa-9390-3ca43a74b6be-istio-token\") pod \"b6fdc9f9-8be5-4dfa-9390-3ca43a74b6be\" (UID: \"b6fdc9f9-8be5-4dfa-9390-3ca43a74b6be\") " Oct 30 21:02:55.380431 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:55.379856 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52f7cad0-6a73-4573-9789-59ac6df33da4-workload-socket" pod "52f7cad0-6a73-4573-9789-59ac6df33da4" (UID: "52f7cad0-6a73-4573-9789-59ac6df33da4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:02:55.380431 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:55.379884 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6fdc9f9-8be5-4dfa-9390-3ca43a74b6be-istiod-ca-cert" pod "b6fdc9f9-8be5-4dfa-9390-3ca43a74b6be" (UID: "b6fdc9f9-8be5-4dfa-9390-3ca43a74b6be"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:02:55.380431 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:55.379889 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/52f7cad0-6a73-4573-9789-59ac6df33da4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52f7cad0-6a73-4573-9789-59ac6df33da4-istiod-ca-cert\") pod \"52f7cad0-6a73-4573-9789-59ac6df33da4\" (UID: \"52f7cad0-6a73-4573-9789-59ac6df33da4\") " Oct 30 21:02:55.380431 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:55.379962 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/527a4fc6-4bfe-49c5-9352-d5b82a74e7ff-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/527a4fc6-4bfe-49c5-9352-d5b82a74e7ff-credential-socket\") pod \"527a4fc6-4bfe-49c5-9352-d5b82a74e7ff\" (UID: \"527a4fc6-4bfe-49c5-9352-d5b82a74e7ff\") " Oct 30 21:02:55.380431 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:55.380104 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/527a4fc6-4bfe-49c5-9352-d5b82a74e7ff-kube-api-access-5smv6\" (UniqueName: \"kubernetes.io/projected/527a4fc6-4bfe-49c5-9352-d5b82a74e7ff-kube-api-access-5smv6\") pod \"527a4fc6-4bfe-49c5-9352-d5b82a74e7ff\" (UID: \"527a4fc6-4bfe-49c5-9352-d5b82a74e7ff\") " Oct 30 21:02:55.380431 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:55.380108 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/527a4fc6-4bfe-49c5-9352-d5b82a74e7ff-workload-certs" pod "527a4fc6-4bfe-49c5-9352-d5b82a74e7ff" (UID: "527a4fc6-4bfe-49c5-9352-d5b82a74e7ff"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:02:55.380431 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:55.380143 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/527a4fc6-4bfe-49c5-9352-d5b82a74e7ff-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/527a4fc6-4bfe-49c5-9352-d5b82a74e7ff-istio-envoy\") pod \"527a4fc6-4bfe-49c5-9352-d5b82a74e7ff\" (UID: \"527a4fc6-4bfe-49c5-9352-d5b82a74e7ff\") " Oct 30 21:02:55.380431 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:55.380181 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52f7cad0-6a73-4573-9789-59ac6df33da4-istiod-ca-cert" pod "52f7cad0-6a73-4573-9789-59ac6df33da4" (UID: "52f7cad0-6a73-4573-9789-59ac6df33da4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:02:55.381127 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:55.380470 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/527a4fc6-4bfe-49c5-9352-d5b82a74e7ff-workload-socket" pod "527a4fc6-4bfe-49c5-9352-d5b82a74e7ff" (UID: "527a4fc6-4bfe-49c5-9352-d5b82a74e7ff"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:02:55.381127 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:55.380663 4611 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52f7cad0-6a73-4573-9789-59ac6df33da4-workload-certs\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:02:55.381127 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:55.380682 4611 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/527a4fc6-4bfe-49c5-9352-d5b82a74e7ff-workload-socket\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:02:55.381127 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:55.380697 4611 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/527a4fc6-4bfe-49c5-9352-d5b82a74e7ff-istio-data\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:02:55.381127 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:55.380713 4611 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b6fdc9f9-8be5-4dfa-9390-3ca43a74b6be-istiod-ca-cert\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:02:55.381127 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:55.380727 4611 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/527a4fc6-4bfe-49c5-9352-d5b82a74e7ff-workload-certs\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:02:55.381127 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:55.380741 4611 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52f7cad0-6a73-4573-9789-59ac6df33da4-istiod-ca-cert\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:02:55.381127 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:55.380755 4611 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/527a4fc6-4bfe-49c5-9352-d5b82a74e7ff-istiod-ca-cert\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:02:55.381127 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:55.380770 4611 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6fdc9f9-8be5-4dfa-9390-3ca43a74b6be-workload-socket\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:02:55.381127 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:55.380784 4611 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52f7cad0-6a73-4573-9789-59ac6df33da4-workload-socket\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:02:55.381127 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:55.380798 4611 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52f7cad0-6a73-4573-9789-59ac6df33da4-credential-socket\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:02:55.381127 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:55.380812 4611 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b6fdc9f9-8be5-4dfa-9390-3ca43a74b6be-workload-certs\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:02:55.381127 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:55.381032 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6fdc9f9-8be5-4dfa-9390-3ca43a74b6be-credential-socket" pod "b6fdc9f9-8be5-4dfa-9390-3ca43a74b6be" (UID: "b6fdc9f9-8be5-4dfa-9390-3ca43a74b6be"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:02:55.382523 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:55.381210 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6fdc9f9-8be5-4dfa-9390-3ca43a74b6be-istio-data" pod "b6fdc9f9-8be5-4dfa-9390-3ca43a74b6be" (UID: "b6fdc9f9-8be5-4dfa-9390-3ca43a74b6be"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:02:55.382523 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:55.381854 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52f7cad0-6a73-4573-9789-59ac6df33da4-istio-data" pod "52f7cad0-6a73-4573-9789-59ac6df33da4" (UID: "52f7cad0-6a73-4573-9789-59ac6df33da4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:02:55.382523 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:55.381925 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/527a4fc6-4bfe-49c5-9352-d5b82a74e7ff-credential-socket" pod "527a4fc6-4bfe-49c5-9352-d5b82a74e7ff" (UID: "527a4fc6-4bfe-49c5-9352-d5b82a74e7ff"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:02:55.382523 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:55.382090 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/527a4fc6-4bfe-49c5-9352-d5b82a74e7ff-istio-podinfo" pod "527a4fc6-4bfe-49c5-9352-d5b82a74e7ff" (UID: "527a4fc6-4bfe-49c5-9352-d5b82a74e7ff"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:02:55.382523 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:55.382453 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52f7cad0-6a73-4573-9789-59ac6df33da4-istio-envoy" pod "52f7cad0-6a73-4573-9789-59ac6df33da4" (UID: "52f7cad0-6a73-4573-9789-59ac6df33da4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:02:55.382792 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:55.382680 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52f7cad0-6a73-4573-9789-59ac6df33da4-istio-token" pod "52f7cad0-6a73-4573-9789-59ac6df33da4" (UID: "52f7cad0-6a73-4573-9789-59ac6df33da4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:02:55.382871 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:55.382848 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/527a4fc6-4bfe-49c5-9352-d5b82a74e7ff-istio-envoy" pod "527a4fc6-4bfe-49c5-9352-d5b82a74e7ff" (UID: "527a4fc6-4bfe-49c5-9352-d5b82a74e7ff"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:02:55.383171 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:55.383148 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/52f7cad0-6a73-4573-9789-59ac6df33da4-istio-podinfo" pod "52f7cad0-6a73-4573-9789-59ac6df33da4" (UID: "52f7cad0-6a73-4573-9789-59ac6df33da4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:02:55.383654 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:55.383628 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b6fdc9f9-8be5-4dfa-9390-3ca43a74b6be-istio-podinfo" pod "b6fdc9f9-8be5-4dfa-9390-3ca43a74b6be" (UID: "b6fdc9f9-8be5-4dfa-9390-3ca43a74b6be"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:02:55.383742 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:55.383655 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6fdc9f9-8be5-4dfa-9390-3ca43a74b6be-istio-token" pod "b6fdc9f9-8be5-4dfa-9390-3ca43a74b6be" (UID: "b6fdc9f9-8be5-4dfa-9390-3ca43a74b6be"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:02:55.384349 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:55.384320 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/527a4fc6-4bfe-49c5-9352-d5b82a74e7ff-istio-token" pod "527a4fc6-4bfe-49c5-9352-d5b82a74e7ff" (UID: "527a4fc6-4bfe-49c5-9352-d5b82a74e7ff"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:02:55.384549 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:55.384526 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6fdc9f9-8be5-4dfa-9390-3ca43a74b6be-kube-api-access-72mrx" pod "b6fdc9f9-8be5-4dfa-9390-3ca43a74b6be" (UID: "b6fdc9f9-8be5-4dfa-9390-3ca43a74b6be"). InnerVolumeSpecName "kube-api-access-72mrx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:02:55.384615 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:55.384594 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6fdc9f9-8be5-4dfa-9390-3ca43a74b6be-istio-envoy" pod "b6fdc9f9-8be5-4dfa-9390-3ca43a74b6be" (UID: "b6fdc9f9-8be5-4dfa-9390-3ca43a74b6be"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:02:55.384660 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:55.384637 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52f7cad0-6a73-4573-9789-59ac6df33da4-kube-api-access-7ktlz" pod "52f7cad0-6a73-4573-9789-59ac6df33da4" (UID: "52f7cad0-6a73-4573-9789-59ac6df33da4"). InnerVolumeSpecName "kube-api-access-7ktlz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:02:55.384808 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:55.384793 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/527a4fc6-4bfe-49c5-9352-d5b82a74e7ff-kube-api-access-5smv6" pod "527a4fc6-4bfe-49c5-9352-d5b82a74e7ff" (UID: "527a4fc6-4bfe-49c5-9352-d5b82a74e7ff"). InnerVolumeSpecName "kube-api-access-5smv6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:02:55.481734 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:55.481712 4611 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5smv6\" (UniqueName: \"kubernetes.io/projected/527a4fc6-4bfe-49c5-9352-d5b82a74e7ff-kube-api-access-5smv6\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:02:55.481734 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:55.481734 4611 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/527a4fc6-4bfe-49c5-9352-d5b82a74e7ff-istio-envoy\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:02:55.481860 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:55.481743 4611 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/527a4fc6-4bfe-49c5-9352-d5b82a74e7ff-istio-token\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:02:55.481860 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:55.481752 4611 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/527a4fc6-4bfe-49c5-9352-d5b82a74e7ff-istio-podinfo\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:02:55.481860 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:55.481761 4611 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52f7cad0-6a73-4573-9789-59ac6df33da4-istio-envoy\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:02:55.481860 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:55.481769 4611 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b6fdc9f9-8be5-4dfa-9390-3ca43a74b6be-istio-podinfo\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:02:55.481860 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:55.481778 4611 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-72mrx\" (UniqueName: \"kubernetes.io/projected/b6fdc9f9-8be5-4dfa-9390-3ca43a74b6be-kube-api-access-72mrx\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:02:55.481860 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:55.481787 4611 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52f7cad0-6a73-4573-9789-59ac6df33da4-istio-token\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:02:55.481860 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:55.481795 4611 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7ktlz\" (UniqueName: \"kubernetes.io/projected/52f7cad0-6a73-4573-9789-59ac6df33da4-kube-api-access-7ktlz\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:02:55.481860 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:55.481803 4611 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b6fdc9f9-8be5-4dfa-9390-3ca43a74b6be-istio-envoy\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:02:55.481860 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:55.481811 4611 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52f7cad0-6a73-4573-9789-59ac6df33da4-istio-data\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:02:55.481860 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:55.481818 4611 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52f7cad0-6a73-4573-9789-59ac6df33da4-istio-podinfo\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:02:55.481860 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:55.481826 4611 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b6fdc9f9-8be5-4dfa-9390-3ca43a74b6be-istio-data\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:02:55.481860 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:55.481834 4611 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6fdc9f9-8be5-4dfa-9390-3ca43a74b6be-credential-socket\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:02:55.481860 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:55.481841 4611 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b6fdc9f9-8be5-4dfa-9390-3ca43a74b6be-istio-token\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:02:55.481860 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:55.481849 4611 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/527a4fc6-4bfe-49c5-9352-d5b82a74e7ff-credential-socket\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:02:55.587558 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:55.587530 4611 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-vpbzr"] Oct 30 21:02:55.595825 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:55.595798 4611 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-vpbzr"] Oct 30 21:02:55.605716 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:55.605693 4611 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b6fdc9f9-8be5-4dfa-9390-3ca43a74b6be" path="/var/lib/kubelet/pods/b6fdc9f9-8be5-4dfa-9390-3ca43a74b6be/volumes" Oct 30 21:02:56.273141 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:56.273066 4611 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-qvfk5" Oct 30 21:02:56.273141 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:56.273093 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-qvfk5" event={"ID":"527a4fc6-4bfe-49c5-9352-d5b82a74e7ff","Type":"ContainerDied","Data":"d1397b503ba4140455a1da1c7696506684d9fc3d6fcdfb2f8b09e7fa9c3b6c80"} Oct 30 21:02:56.273676 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:56.273156 4611 scope.go:122] "RemoveContainer" containerID="07f6f4705c682af2e74dcdad8363c9ce1eee6a59a57f010b76fd28ad40d54386" Oct 30 21:02:56.275979 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:56.275963 4611 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-hx7mv" Oct 30 21:02:56.276124 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:56.275989 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-hx7mv" event={"ID":"52f7cad0-6a73-4573-9789-59ac6df33da4","Type":"ContainerDied","Data":"b99d8fe48920544fea38344e40bb203fd45545c58d9ba322cd794a53f7760182"} Oct 30 21:02:56.282643 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:56.282625 4611 scope.go:122] "RemoveContainer" containerID="0028b06d562d6e065e5b330b2875605dc7238ba219b2e2f402fbd9a06b916e9f" Oct 30 21:02:56.314944 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:56.314922 4611 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-qvfk5"] Oct 30 21:02:56.318965 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:56.318941 4611 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-qvfk5"] Oct 30 21:02:56.331121 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:56.331097 4611 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-hx7mv"] Oct 30 21:02:56.337272 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:56.337250 4611 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-f2ed1e-openshift-default-7694b65f87-hx7mv"] Oct 30 21:02:57.604718 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:57.604678 4611 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="527a4fc6-4bfe-49c5-9352-d5b82a74e7ff" path="/var/lib/kubelet/pods/527a4fc6-4bfe-49c5-9352-d5b82a74e7ff/volumes" Oct 30 21:02:57.605292 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:57.605270 4611 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="52f7cad0-6a73-4573-9789-59ac6df33da4" path="/var/lib/kubelet/pods/52f7cad0-6a73-4573-9789-59ac6df33da4/volumes" Oct 30 21:02:59.105852 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:59.105804 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:02:59.306928 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:02:59.306891 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:03:00.032758 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:03:00.032722 4611 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ee99e329-9w8fj"] Oct 30 21:03:00.036003 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:03:00.035974 4611 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ee99e329-9w8fj"] Oct 30 21:03:01.241618 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:03:01.241586 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:03:01.604602 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:03:01.604573 4611 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="763cbacf-a524-4397-bf3a-2878d0e15a6d" path="/var/lib/kubelet/pods/763cbacf-a524-4397-bf3a-2878d0e15a6d/volumes" Oct 30 21:03:03.917562 ip-10-0-133-192 kubenswrapper[4611]: E1030 21:03:03.917501 4611 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hwzc6" containerName="extractor" Oct 30 21:03:09.010545 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:03:09.009337 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:03:24.666294 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:03:24.666257 4611 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-xv2z4"] Oct 30 21:03:24.666893 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:03:24.666847 4611 cpu_manager.go:414] "containerMap: removing container" podUID="52f7cad0-6a73-4573-9789-59ac6df33da4" containerName="istio-proxy" Oct 30 21:03:24.666893 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:03:24.666868 4611 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="52f7cad0-6a73-4573-9789-59ac6df33da4" containerName="istio-proxy" Oct 30 21:03:24.666893 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:03:24.666885 4611 cpu_manager.go:414] "containerMap: removing container" podUID="d9298078-08c1-40a5-9117-ced750d46bd8" containerName="payload-pre-processing" Oct 30 21:03:24.667080 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:03:24.666894 4611 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d9298078-08c1-40a5-9117-ced750d46bd8" containerName="payload-pre-processing" Oct 30 21:03:24.667080 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:03:24.666915 4611 cpu_manager.go:414] "containerMap: removing container" podUID="527a4fc6-4bfe-49c5-9352-d5b82a74e7ff" containerName="istio-proxy" Oct 30 21:03:24.667080 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:03:24.666923 4611 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="527a4fc6-4bfe-49c5-9352-d5b82a74e7ff" containerName="istio-proxy" Oct 30 21:03:24.667080 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:03:24.666941 4611 cpu_manager.go:414] "containerMap: removing container" podUID="e6311907-bd1a-4516-8cfd-78b5b4d5be46" containerName="revoke-keys" Oct 30 21:03:24.667080 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:03:24.666949 4611 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e6311907-bd1a-4516-8cfd-78b5b4d5be46" containerName="revoke-keys" Oct 30 21:03:24.667080 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:03:24.666957 4611 cpu_manager.go:414] "containerMap: removing container" podUID="653e3ec7-2147-4813-8946-1f25cfa425f6" containerName="main" Oct 30 21:03:24.667080 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:03:24.666966 4611 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="653e3ec7-2147-4813-8946-1f25cfa425f6" containerName="main" Oct 30 21:03:24.667080 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:03:24.666994 4611 cpu_manager.go:414] "containerMap: removing container" podUID="b6fdc9f9-8be5-4dfa-9390-3ca43a74b6be" containerName="istio-proxy" Oct 30 21:03:24.667080 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:03:24.667001 4611 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b6fdc9f9-8be5-4dfa-9390-3ca43a74b6be" containerName="istio-proxy" Oct 30 21:03:24.667080 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:03:24.667017 4611 cpu_manager.go:414] "containerMap: removing container" podUID="5052ef1d-3134-4545-8f2e-2c6ef046ccf6" containerName="maas-api" Oct 30 21:03:24.667080 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:03:24.667024 4611 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5052ef1d-3134-4545-8f2e-2c6ef046ccf6" containerName="maas-api" Oct 30 21:03:24.667629 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:03:24.667117 4611 memory_manager.go:365] "RemoveStaleState removing state" podUID="527a4fc6-4bfe-49c5-9352-d5b82a74e7ff" containerName="istio-proxy" Oct 30 21:03:24.667629 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:03:24.667129 4611 memory_manager.go:365] "RemoveStaleState removing state" podUID="52f7cad0-6a73-4573-9789-59ac6df33da4" containerName="istio-proxy" Oct 30 21:03:24.667629 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:03:24.667138 4611 memory_manager.go:365] "RemoveStaleState removing state" podUID="e6311907-bd1a-4516-8cfd-78b5b4d5be46" containerName="revoke-keys" Oct 30 21:03:24.667629 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:03:24.667147 4611 memory_manager.go:365] "RemoveStaleState removing state" podUID="653e3ec7-2147-4813-8946-1f25cfa425f6" containerName="main" Oct 30 21:03:24.667629 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:03:24.667159 4611 memory_manager.go:365] "RemoveStaleState removing state" podUID="b6fdc9f9-8be5-4dfa-9390-3ca43a74b6be" containerName="istio-proxy" Oct 30 21:03:24.667629 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:03:24.667169 4611 memory_manager.go:365] "RemoveStaleState removing state" podUID="5052ef1d-3134-4545-8f2e-2c6ef046ccf6" containerName="maas-api" Oct 30 21:03:24.667629 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:03:24.667181 4611 memory_manager.go:365] "RemoveStaleState removing state" podUID="52f7cad0-6a73-4573-9789-59ac6df33da4" containerName="istio-proxy" Oct 30 21:03:24.667629 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:03:24.667190 4611 memory_manager.go:365] "RemoveStaleState removing state" podUID="d9298078-08c1-40a5-9117-ced750d46bd8" containerName="payload-pre-processing" Oct 30 21:03:24.672327 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:03:24.672304 4611 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-xv2z4" Oct 30 21:03:24.677206 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:03:24.677179 4611 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:03:24.677206 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:03:24.677190 4611 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7hf8r\"" Oct 30 21:03:24.677354 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:03:24.677261 4611 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:03:24.677354 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:03:24.677334 4611 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:03:24.679884 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:03:24.679861 4611 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-xv2z4"] Oct 30 21:03:24.736921 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:03:24.736895 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2c7s8\" (UniqueName: \"kubernetes.io/projected/147ccaa0-2bf7-41d9-abd5-45757ea45d96-kube-api-access-2c7s8\") pod \"maas-api-revoke-sub-85237e45-xv2z4\" (UID: \"147ccaa0-2bf7-41d9-abd5-45757ea45d96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-xv2z4" Oct 30 21:03:24.737042 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:03:24.736960 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/147ccaa0-2bf7-41d9-abd5-45757ea45d96-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-xv2z4\" (UID: \"147ccaa0-2bf7-41d9-abd5-45757ea45d96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-xv2z4" Oct 30 21:03:24.737096 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:03:24.737058 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/147ccaa0-2bf7-41d9-abd5-45757ea45d96-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-xv2z4\" (UID: \"147ccaa0-2bf7-41d9-abd5-45757ea45d96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-xv2z4" Oct 30 21:03:24.837726 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:03:24.837687 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2c7s8\" (UniqueName: \"kubernetes.io/projected/147ccaa0-2bf7-41d9-abd5-45757ea45d96-kube-api-access-2c7s8\") pod \"maas-api-revoke-sub-85237e45-xv2z4\" (UID: \"147ccaa0-2bf7-41d9-abd5-45757ea45d96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-xv2z4" Oct 30 21:03:24.837847 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:03:24.837787 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/147ccaa0-2bf7-41d9-abd5-45757ea45d96-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-xv2z4\" (UID: \"147ccaa0-2bf7-41d9-abd5-45757ea45d96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-xv2z4" Oct 30 21:03:24.837919 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:03:24.837848 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/147ccaa0-2bf7-41d9-abd5-45757ea45d96-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-xv2z4\" (UID: \"147ccaa0-2bf7-41d9-abd5-45757ea45d96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-xv2z4" Oct 30 21:03:24.838368 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:03:24.838344 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/147ccaa0-2bf7-41d9-abd5-45757ea45d96-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-xv2z4\" (UID: \"147ccaa0-2bf7-41d9-abd5-45757ea45d96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-xv2z4" Oct 30 21:03:24.840043 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:03:24.840022 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/147ccaa0-2bf7-41d9-abd5-45757ea45d96-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-xv2z4\" (UID: \"147ccaa0-2bf7-41d9-abd5-45757ea45d96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-xv2z4" Oct 30 21:03:24.849120 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:03:24.849099 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2c7s8\" (UniqueName: \"kubernetes.io/projected/147ccaa0-2bf7-41d9-abd5-45757ea45d96-kube-api-access-2c7s8\") pod \"maas-api-revoke-sub-85237e45-xv2z4\" (UID: \"147ccaa0-2bf7-41d9-abd5-45757ea45d96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-xv2z4" Oct 30 21:03:24.989324 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:03:24.989276 4611 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-xv2z4" Oct 30 21:03:25.266583 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:03:25.266558 4611 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-xv2z4"] Oct 30 21:03:25.268910 ip-10-0-133-192 kubenswrapper[4611]: W1030 21:03:25.268882 4611 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod147ccaa0_2bf7_41d9_abd5_45757ea45d96.slice/crio-2b3f4cdb6815c7333d0e9f92f6f5a9ce00a7ae9b2b74167e524df223e5a79fc1 WatchSource:0}: Error finding container 2b3f4cdb6815c7333d0e9f92f6f5a9ce00a7ae9b2b74167e524df223e5a79fc1: Status 404 returned error can't find the container with id 2b3f4cdb6815c7333d0e9f92f6f5a9ce00a7ae9b2b74167e524df223e5a79fc1 Oct 30 21:03:25.466662 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:03:25.466634 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-xv2z4" event={"ID":"147ccaa0-2bf7-41d9-abd5-45757ea45d96","Type":"ContainerStarted","Data":"dcb78882b314ca558a12ad39710e0f92ac64daa36ce2e361e5af22c570a885dd"} Oct 30 21:03:25.466785 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:03:25.466690 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-xv2z4" event={"ID":"147ccaa0-2bf7-41d9-abd5-45757ea45d96","Type":"ContainerStarted","Data":"2b3f4cdb6815c7333d0e9f92f6f5a9ce00a7ae9b2b74167e524df223e5a79fc1"} Oct 30 21:03:25.483284 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:03:25.483239 4611 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-xv2z4" podStartSLOduration=1.4832234899999999 podStartE2EDuration="1.48322349s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:03:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:03:25.482881366 +0000 UTC m=+1460.016907983" watchObservedRunningTime="2026-10-30 21:03:25.48322349 +0000 UTC m=+1460.017250108" Oct 30 21:03:26.480450 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:03:26.480414 4611 generic.go:356] "Generic (PLEG): container finished" podID="147ccaa0-2bf7-41d9-abd5-45757ea45d96" containerID="dcb78882b314ca558a12ad39710e0f92ac64daa36ce2e361e5af22c570a885dd" exitCode=0 Oct 30 21:03:26.480871 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:03:26.480483 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-xv2z4" event={"ID":"147ccaa0-2bf7-41d9-abd5-45757ea45d96","Type":"ContainerDied","Data":"dcb78882b314ca558a12ad39710e0f92ac64daa36ce2e361e5af22c570a885dd"} Oct 30 21:03:27.616065 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:03:27.616042 4611 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-xv2z4" Oct 30 21:03:27.765682 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:03:27.765610 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/147ccaa0-2bf7-41d9-abd5-45757ea45d96-kube-api-access-2c7s8\" (UniqueName: \"kubernetes.io/projected/147ccaa0-2bf7-41d9-abd5-45757ea45d96-kube-api-access-2c7s8\") pod \"147ccaa0-2bf7-41d9-abd5-45757ea45d96\" (UID: \"147ccaa0-2bf7-41d9-abd5-45757ea45d96\") " Oct 30 21:03:27.765682 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:03:27.765662 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/147ccaa0-2bf7-41d9-abd5-45757ea45d96-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/147ccaa0-2bf7-41d9-abd5-45757ea45d96-maas-api-openshift-ca\") pod \"147ccaa0-2bf7-41d9-abd5-45757ea45d96\" (UID: \"147ccaa0-2bf7-41d9-abd5-45757ea45d96\") " Oct 30 21:03:27.765878 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:03:27.765790 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/147ccaa0-2bf7-41d9-abd5-45757ea45d96-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/147ccaa0-2bf7-41d9-abd5-45757ea45d96-maas-api-xks-ca\") pod \"147ccaa0-2bf7-41d9-abd5-45757ea45d96\" (UID: \"147ccaa0-2bf7-41d9-abd5-45757ea45d96\") " Oct 30 21:03:27.766089 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:03:27.766061 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/147ccaa0-2bf7-41d9-abd5-45757ea45d96-maas-api-openshift-ca" pod "147ccaa0-2bf7-41d9-abd5-45757ea45d96" (UID: "147ccaa0-2bf7-41d9-abd5-45757ea45d96"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:03:27.767581 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:03:27.767557 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/147ccaa0-2bf7-41d9-abd5-45757ea45d96-kube-api-access-2c7s8" pod "147ccaa0-2bf7-41d9-abd5-45757ea45d96" (UID: "147ccaa0-2bf7-41d9-abd5-45757ea45d96"). InnerVolumeSpecName "kube-api-access-2c7s8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:03:27.767757 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:03:27.767740 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/147ccaa0-2bf7-41d9-abd5-45757ea45d96-maas-api-xks-ca" pod "147ccaa0-2bf7-41d9-abd5-45757ea45d96" (UID: "147ccaa0-2bf7-41d9-abd5-45757ea45d96"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:03:27.867378 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:03:27.867352 4611 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/147ccaa0-2bf7-41d9-abd5-45757ea45d96-maas-api-openshift-ca\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:03:27.867378 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:03:27.867377 4611 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/147ccaa0-2bf7-41d9-abd5-45757ea45d96-maas-api-xks-ca\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:03:27.867541 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:03:27.867387 4611 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2c7s8\" (UniqueName: \"kubernetes.io/projected/147ccaa0-2bf7-41d9-abd5-45757ea45d96-kube-api-access-2c7s8\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:03:28.491627 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:03:28.491596 4611 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-xv2z4" Oct 30 21:03:28.491802 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:03:28.491628 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-xv2z4" event={"ID":"147ccaa0-2bf7-41d9-abd5-45757ea45d96","Type":"ContainerDied","Data":"2b3f4cdb6815c7333d0e9f92f6f5a9ce00a7ae9b2b74167e524df223e5a79fc1"} Oct 30 21:03:28.491802 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:03:28.491675 4611 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2b3f4cdb6815c7333d0e9f92f6f5a9ce00a7ae9b2b74167e524df223e5a79fc1" Oct 30 21:03:31.054839 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:03:31.054800 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:03:33.917036 ip-10-0-133-192 kubenswrapper[4611]: E1030 21:03:33.916991 4611 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hwzc6" containerName="extractor" Oct 30 21:03:34.322797 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:03:34.322709 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:03:36.400900 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:03:36.400863 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:03:37.695186 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:03:37.695149 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:03:48.779300 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:03:48.779222 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:03:49.038704 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:03:49.038626 4611 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d1eb66ad-jjwfx"] Oct 30 21:03:49.042243 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:03:49.042218 4611 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d1eb66ad-jjwfx"] Oct 30 21:03:49.104696 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:03:49.104654 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:03:49.604130 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:03:49.604099 4611 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="78c5ded8-a0ed-4176-bc86-f6e3e233b455" path="/var/lib/kubelet/pods/78c5ded8-a0ed-4176-bc86-f6e3e233b455/volumes" Oct 30 21:04:00.738016 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:04:00.737981 4611 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-czzct"] Oct 30 21:04:00.738414 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:04:00.738186 4611 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-czzct" podUID="6e28fa18-8c44-4272-b3a6-9448e957b02a" containerName="discovery" containerID="cri-o://5d9cb080123e0e5bb89149b2b258f90deebe74499536f29788f136cb15125472" gracePeriod=30 Oct 30 21:04:00.991202 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:04:00.991147 4611 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-czzct" Oct 30 21:04:01.062647 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:04:01.062619 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6e28fa18-8c44-4272-b3a6-9448e957b02a-istio-token\" (UniqueName: \"kubernetes.io/projected/6e28fa18-8c44-4272-b3a6-9448e957b02a-istio-token\") pod \"6e28fa18-8c44-4272-b3a6-9448e957b02a\" (UID: \"6e28fa18-8c44-4272-b3a6-9448e957b02a\") " Oct 30 21:04:01.062808 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:04:01.062700 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6e28fa18-8c44-4272-b3a6-9448e957b02a-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/6e28fa18-8c44-4272-b3a6-9448e957b02a-istio-kubeconfig\") pod \"6e28fa18-8c44-4272-b3a6-9448e957b02a\" (UID: \"6e28fa18-8c44-4272-b3a6-9448e957b02a\") " Oct 30 21:04:01.062808 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:04:01.062759 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6e28fa18-8c44-4272-b3a6-9448e957b02a-cacerts\" (UniqueName: \"kubernetes.io/secret/6e28fa18-8c44-4272-b3a6-9448e957b02a-cacerts\") pod \"6e28fa18-8c44-4272-b3a6-9448e957b02a\" (UID: \"6e28fa18-8c44-4272-b3a6-9448e957b02a\") " Oct 30 21:04:01.062932 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:04:01.062835 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6e28fa18-8c44-4272-b3a6-9448e957b02a-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/6e28fa18-8c44-4272-b3a6-9448e957b02a-istio-csr-dns-cert\") pod \"6e28fa18-8c44-4272-b3a6-9448e957b02a\" (UID: \"6e28fa18-8c44-4272-b3a6-9448e957b02a\") " Oct 30 21:04:01.062932 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:04:01.062866 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6e28fa18-8c44-4272-b3a6-9448e957b02a-kube-api-access-jq8hn\" (UniqueName: \"kubernetes.io/projected/6e28fa18-8c44-4272-b3a6-9448e957b02a-kube-api-access-jq8hn\") pod \"6e28fa18-8c44-4272-b3a6-9448e957b02a\" (UID: \"6e28fa18-8c44-4272-b3a6-9448e957b02a\") " Oct 30 21:04:01.062932 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:04:01.062917 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6e28fa18-8c44-4272-b3a6-9448e957b02a-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/6e28fa18-8c44-4272-b3a6-9448e957b02a-istio-csr-ca-configmap\") pod \"6e28fa18-8c44-4272-b3a6-9448e957b02a\" (UID: \"6e28fa18-8c44-4272-b3a6-9448e957b02a\") " Oct 30 21:04:01.063097 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:04:01.062962 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6e28fa18-8c44-4272-b3a6-9448e957b02a-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/6e28fa18-8c44-4272-b3a6-9448e957b02a-local-certs\") pod \"6e28fa18-8c44-4272-b3a6-9448e957b02a\" (UID: \"6e28fa18-8c44-4272-b3a6-9448e957b02a\") " Oct 30 21:04:01.063442 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:04:01.063407 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e28fa18-8c44-4272-b3a6-9448e957b02a-istio-csr-ca-configmap" pod "6e28fa18-8c44-4272-b3a6-9448e957b02a" (UID: "6e28fa18-8c44-4272-b3a6-9448e957b02a"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:04:01.065059 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:04:01.065031 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e28fa18-8c44-4272-b3a6-9448e957b02a-cacerts" pod "6e28fa18-8c44-4272-b3a6-9448e957b02a" (UID: "6e28fa18-8c44-4272-b3a6-9448e957b02a"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:04:01.065197 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:04:01.065032 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e28fa18-8c44-4272-b3a6-9448e957b02a-kube-api-access-jq8hn" pod "6e28fa18-8c44-4272-b3a6-9448e957b02a" (UID: "6e28fa18-8c44-4272-b3a6-9448e957b02a"). InnerVolumeSpecName "kube-api-access-jq8hn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:04:01.065302 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:04:01.065282 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e28fa18-8c44-4272-b3a6-9448e957b02a-istio-csr-dns-cert" pod "6e28fa18-8c44-4272-b3a6-9448e957b02a" (UID: "6e28fa18-8c44-4272-b3a6-9448e957b02a"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:04:01.065302 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:04:01.065295 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e28fa18-8c44-4272-b3a6-9448e957b02a-istio-kubeconfig" pod "6e28fa18-8c44-4272-b3a6-9448e957b02a" (UID: "6e28fa18-8c44-4272-b3a6-9448e957b02a"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:04:01.065567 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:04:01.065548 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e28fa18-8c44-4272-b3a6-9448e957b02a-local-certs" pod "6e28fa18-8c44-4272-b3a6-9448e957b02a" (UID: "6e28fa18-8c44-4272-b3a6-9448e957b02a"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:04:01.065673 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:04:01.065660 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e28fa18-8c44-4272-b3a6-9448e957b02a-istio-token" pod "6e28fa18-8c44-4272-b3a6-9448e957b02a" (UID: "6e28fa18-8c44-4272-b3a6-9448e957b02a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:04:01.163974 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:04:01.163946 4611 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/6e28fa18-8c44-4272-b3a6-9448e957b02a-istio-csr-dns-cert\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:04:01.163974 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:04:01.163972 4611 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jq8hn\" (UniqueName: \"kubernetes.io/projected/6e28fa18-8c44-4272-b3a6-9448e957b02a-kube-api-access-jq8hn\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:04:01.164149 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:04:01.163983 4611 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/6e28fa18-8c44-4272-b3a6-9448e957b02a-istio-csr-ca-configmap\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:04:01.164149 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:04:01.163992 4611 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/6e28fa18-8c44-4272-b3a6-9448e957b02a-local-certs\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:04:01.164149 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:04:01.164000 4611 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6e28fa18-8c44-4272-b3a6-9448e957b02a-istio-token\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:04:01.164149 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:04:01.164009 4611 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/6e28fa18-8c44-4272-b3a6-9448e957b02a-istio-kubeconfig\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:04:01.164149 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:04:01.164017 4611 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/6e28fa18-8c44-4272-b3a6-9448e957b02a-cacerts\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:04:01.650467 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:04:01.650434 4611 generic.go:356] "Generic (PLEG): container finished" podID="6e28fa18-8c44-4272-b3a6-9448e957b02a" containerID="5d9cb080123e0e5bb89149b2b258f90deebe74499536f29788f136cb15125472" exitCode=0 Oct 30 21:04:01.650613 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:04:01.650481 4611 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-czzct" Oct 30 21:04:01.650613 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:04:01.650538 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-czzct" event={"ID":"6e28fa18-8c44-4272-b3a6-9448e957b02a","Type":"ContainerDied","Data":"5d9cb080123e0e5bb89149b2b258f90deebe74499536f29788f136cb15125472"} Oct 30 21:04:01.650613 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:04:01.650591 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-czzct" event={"ID":"6e28fa18-8c44-4272-b3a6-9448e957b02a","Type":"ContainerDied","Data":"38f1edf942d90f86a3db980d0800e3da251d31d4bd138a57ede1f35e03c26f20"} Oct 30 21:04:01.650613 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:04:01.650611 4611 scope.go:122] "RemoveContainer" containerID="5d9cb080123e0e5bb89149b2b258f90deebe74499536f29788f136cb15125472" Oct 30 21:04:01.660018 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:04:01.659998 4611 scope.go:122] "RemoveContainer" containerID="5d9cb080123e0e5bb89149b2b258f90deebe74499536f29788f136cb15125472" Oct 30 21:04:01.660286 ip-10-0-133-192 kubenswrapper[4611]: E1030 21:04:01.660265 4611 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d9cb080123e0e5bb89149b2b258f90deebe74499536f29788f136cb15125472\": container with ID starting with 5d9cb080123e0e5bb89149b2b258f90deebe74499536f29788f136cb15125472 not found: ID does not exist" containerID="5d9cb080123e0e5bb89149b2b258f90deebe74499536f29788f136cb15125472" Oct 30 21:04:01.660355 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:04:01.660293 4611 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d9cb080123e0e5bb89149b2b258f90deebe74499536f29788f136cb15125472"} err="failed to get container status \"5d9cb080123e0e5bb89149b2b258f90deebe74499536f29788f136cb15125472\": rpc error: code = NotFound desc = could not find container \"5d9cb080123e0e5bb89149b2b258f90deebe74499536f29788f136cb15125472\": container with ID starting with 5d9cb080123e0e5bb89149b2b258f90deebe74499536f29788f136cb15125472 not found: ID does not exist" Oct 30 21:04:01.667714 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:04:01.667691 4611 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-czzct"] Oct 30 21:04:01.671848 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:04:01.671828 4611 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-czzct"] Oct 30 21:04:03.604486 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:04:03.604446 4611 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6e28fa18-8c44-4272-b3a6-9448e957b02a" path="/var/lib/kubelet/pods/6e28fa18-8c44-4272-b3a6-9448e957b02a/volumes" Oct 30 21:04:03.916314 ip-10-0-133-192 kubenswrapper[4611]: E1030 21:04:03.916237 4611 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hwzc6" containerName="extractor" Oct 30 21:04:05.812176 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:04:05.812147 4611 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7w5wz_7e2574c1-b59c-453c-9501-25ec13a1ccad/ovn-acl-logging/0.log" Oct 30 21:04:05.816147 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:04:05.816120 4611 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7w5wz_7e2574c1-b59c-453c-9501-25ec13a1ccad/ovn-acl-logging/0.log" Oct 30 21:04:07.886473 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:04:07.886442 4611 scope.go:122] "RemoveContainer" containerID="be94e0c3a5311bf034bb48181a805f887425335488aa69afe5ab33cc0cceb696" Oct 30 21:04:24.122601 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:04:24.122564 4611 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-pxn6c"] Oct 30 21:04:24.123086 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:04:24.122961 4611 cpu_manager.go:414] "containerMap: removing container" podUID="6e28fa18-8c44-4272-b3a6-9448e957b02a" containerName="discovery" Oct 30 21:04:24.123086 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:04:24.122974 4611 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e28fa18-8c44-4272-b3a6-9448e957b02a" containerName="discovery" Oct 30 21:04:24.123086 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:04:24.122996 4611 cpu_manager.go:414] "containerMap: removing container" podUID="147ccaa0-2bf7-41d9-abd5-45757ea45d96" containerName="revoke-keys" Oct 30 21:04:24.123086 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:04:24.123001 4611 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="147ccaa0-2bf7-41d9-abd5-45757ea45d96" containerName="revoke-keys" Oct 30 21:04:24.123086 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:04:24.123018 4611 cpu_manager.go:414] "containerMap: removing container" podUID="52f7cad0-6a73-4573-9789-59ac6df33da4" containerName="istio-proxy" Oct 30 21:04:24.123086 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:04:24.123024 4611 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="52f7cad0-6a73-4573-9789-59ac6df33da4" containerName="istio-proxy" Oct 30 21:04:24.123086 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:04:24.123087 4611 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e28fa18-8c44-4272-b3a6-9448e957b02a" containerName="discovery" Oct 30 21:04:24.123309 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:04:24.123098 4611 memory_manager.go:365] "RemoveStaleState removing state" podUID="147ccaa0-2bf7-41d9-abd5-45757ea45d96" containerName="revoke-keys" Oct 30 21:04:24.127845 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:04:24.127826 4611 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-pxn6c" Oct 30 21:04:24.133788 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:04:24.133768 4611 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:04:24.134035 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:04:24.134016 4611 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7hf8r\"" Oct 30 21:04:24.134858 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:04:24.134840 4611 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:04:24.134858 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:04:24.134849 4611 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:04:24.142520 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:04:24.142482 4611 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-pxn6c"] Oct 30 21:04:24.298568 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:04:24.298533 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/382d2eac-35bc-4c38-b87b-0140afd66f24-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-pxn6c\" (UID: \"382d2eac-35bc-4c38-b87b-0140afd66f24\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-pxn6c" Oct 30 21:04:24.298717 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:04:24.298576 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/382d2eac-35bc-4c38-b87b-0140afd66f24-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-pxn6c\" (UID: \"382d2eac-35bc-4c38-b87b-0140afd66f24\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-pxn6c" Oct 30 21:04:24.298717 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:04:24.298592 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j4t7f\" (UniqueName: \"kubernetes.io/projected/382d2eac-35bc-4c38-b87b-0140afd66f24-kube-api-access-j4t7f\") pod \"maas-api-revoke-sub-c1e271a9-pxn6c\" (UID: \"382d2eac-35bc-4c38-b87b-0140afd66f24\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-pxn6c" Oct 30 21:04:24.399610 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:04:24.399531 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/382d2eac-35bc-4c38-b87b-0140afd66f24-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-pxn6c\" (UID: \"382d2eac-35bc-4c38-b87b-0140afd66f24\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-pxn6c" Oct 30 21:04:24.399610 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:04:24.399575 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/382d2eac-35bc-4c38-b87b-0140afd66f24-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-pxn6c\" (UID: \"382d2eac-35bc-4c38-b87b-0140afd66f24\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-pxn6c" Oct 30 21:04:24.399610 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:04:24.399595 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j4t7f\" (UniqueName: \"kubernetes.io/projected/382d2eac-35bc-4c38-b87b-0140afd66f24-kube-api-access-j4t7f\") pod \"maas-api-revoke-sub-c1e271a9-pxn6c\" (UID: \"382d2eac-35bc-4c38-b87b-0140afd66f24\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-pxn6c" Oct 30 21:04:24.400282 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:04:24.400258 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/382d2eac-35bc-4c38-b87b-0140afd66f24-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-pxn6c\" (UID: \"382d2eac-35bc-4c38-b87b-0140afd66f24\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-pxn6c" Oct 30 21:04:24.401846 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:04:24.401822 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/382d2eac-35bc-4c38-b87b-0140afd66f24-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-pxn6c\" (UID: \"382d2eac-35bc-4c38-b87b-0140afd66f24\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-pxn6c" Oct 30 21:04:24.411491 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:04:24.411466 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j4t7f\" (UniqueName: \"kubernetes.io/projected/382d2eac-35bc-4c38-b87b-0140afd66f24-kube-api-access-j4t7f\") pod \"maas-api-revoke-sub-c1e271a9-pxn6c\" (UID: \"382d2eac-35bc-4c38-b87b-0140afd66f24\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-pxn6c" Oct 30 21:04:24.440577 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:04:24.440552 4611 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-pxn6c" Oct 30 21:04:24.720431 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:04:24.720408 4611 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-pxn6c"] Oct 30 21:04:24.722526 ip-10-0-133-192 kubenswrapper[4611]: W1030 21:04:24.722485 4611 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod382d2eac_35bc_4c38_b87b_0140afd66f24.slice/crio-0fdba3b3a404179cf16d28754fb856b23e6f1a4079313d4f49e2dbfe2fefaa24 WatchSource:0}: Error finding container 0fdba3b3a404179cf16d28754fb856b23e6f1a4079313d4f49e2dbfe2fefaa24: Status 404 returned error can't find the container with id 0fdba3b3a404179cf16d28754fb856b23e6f1a4079313d4f49e2dbfe2fefaa24 Oct 30 21:04:24.780351 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:04:24.780317 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-pxn6c" event={"ID":"382d2eac-35bc-4c38-b87b-0140afd66f24","Type":"ContainerStarted","Data":"0fdba3b3a404179cf16d28754fb856b23e6f1a4079313d4f49e2dbfe2fefaa24"} Oct 30 21:04:25.786499 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:04:25.786466 4611 generic.go:356] "Generic (PLEG): container finished" podID="382d2eac-35bc-4c38-b87b-0140afd66f24" containerID="a2ac1015e9bd303fdcabf13a1419179ca3dbd77a4946cb0b23240dc410b6467c" exitCode=0 Oct 30 21:04:25.786901 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:04:25.786530 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-pxn6c" event={"ID":"382d2eac-35bc-4c38-b87b-0140afd66f24","Type":"ContainerDied","Data":"a2ac1015e9bd303fdcabf13a1419179ca3dbd77a4946cb0b23240dc410b6467c"} Oct 30 21:04:26.927075 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:04:26.927048 4611 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-pxn6c" Oct 30 21:04:27.025227 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:04:27.025184 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/382d2eac-35bc-4c38-b87b-0140afd66f24-kube-api-access-j4t7f\" (UniqueName: \"kubernetes.io/projected/382d2eac-35bc-4c38-b87b-0140afd66f24-kube-api-access-j4t7f\") pod \"382d2eac-35bc-4c38-b87b-0140afd66f24\" (UID: \"382d2eac-35bc-4c38-b87b-0140afd66f24\") " Oct 30 21:04:27.025387 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:04:27.025241 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/382d2eac-35bc-4c38-b87b-0140afd66f24-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/382d2eac-35bc-4c38-b87b-0140afd66f24-maas-api-xks-ca\") pod \"382d2eac-35bc-4c38-b87b-0140afd66f24\" (UID: \"382d2eac-35bc-4c38-b87b-0140afd66f24\") " Oct 30 21:04:27.025387 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:04:27.025277 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/382d2eac-35bc-4c38-b87b-0140afd66f24-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/382d2eac-35bc-4c38-b87b-0140afd66f24-maas-api-openshift-ca\") pod \"382d2eac-35bc-4c38-b87b-0140afd66f24\" (UID: \"382d2eac-35bc-4c38-b87b-0140afd66f24\") " Oct 30 21:04:27.025731 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:04:27.025708 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/382d2eac-35bc-4c38-b87b-0140afd66f24-maas-api-openshift-ca" pod "382d2eac-35bc-4c38-b87b-0140afd66f24" (UID: "382d2eac-35bc-4c38-b87b-0140afd66f24"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:04:27.027280 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:04:27.027261 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/382d2eac-35bc-4c38-b87b-0140afd66f24-maas-api-xks-ca" pod "382d2eac-35bc-4c38-b87b-0140afd66f24" (UID: "382d2eac-35bc-4c38-b87b-0140afd66f24"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:04:27.027331 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:04:27.027280 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/382d2eac-35bc-4c38-b87b-0140afd66f24-kube-api-access-j4t7f" pod "382d2eac-35bc-4c38-b87b-0140afd66f24" (UID: "382d2eac-35bc-4c38-b87b-0140afd66f24"). InnerVolumeSpecName "kube-api-access-j4t7f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:04:27.126941 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:04:27.126911 4611 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/382d2eac-35bc-4c38-b87b-0140afd66f24-maas-api-xks-ca\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:04:27.126941 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:04:27.126941 4611 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/382d2eac-35bc-4c38-b87b-0140afd66f24-maas-api-openshift-ca\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:04:27.127087 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:04:27.126952 4611 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j4t7f\" (UniqueName: \"kubernetes.io/projected/382d2eac-35bc-4c38-b87b-0140afd66f24-kube-api-access-j4t7f\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:04:27.796978 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:04:27.796950 4611 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-pxn6c" Oct 30 21:04:27.796978 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:04:27.796955 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-pxn6c" event={"ID":"382d2eac-35bc-4c38-b87b-0140afd66f24","Type":"ContainerDied","Data":"0fdba3b3a404179cf16d28754fb856b23e6f1a4079313d4f49e2dbfe2fefaa24"} Oct 30 21:04:27.797182 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:04:27.797002 4611 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0fdba3b3a404179cf16d28754fb856b23e6f1a4079313d4f49e2dbfe2fefaa24" Oct 30 21:04:33.916630 ip-10-0-133-192 kubenswrapper[4611]: E1030 21:04:33.916591 4611 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hwzc6" containerName="extractor" Oct 30 21:04:37.105535 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:04:37.105473 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:04:42.935689 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:04:42.935639 4611 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-kvgmg"] Oct 30 21:04:42.937340 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:04:42.937305 4611 cpu_manager.go:414] "containerMap: removing container" podUID="382d2eac-35bc-4c38-b87b-0140afd66f24" containerName="revoke-keys" Oct 30 21:04:42.937536 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:04:42.937502 4611 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="382d2eac-35bc-4c38-b87b-0140afd66f24" containerName="revoke-keys" Oct 30 21:04:42.937934 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:04:42.937909 4611 memory_manager.go:365] "RemoveStaleState removing state" podUID="382d2eac-35bc-4c38-b87b-0140afd66f24" containerName="revoke-keys" Oct 30 21:04:42.942752 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:04:42.942731 4611 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-kvgmg" Oct 30 21:04:42.946735 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:04:42.946704 4611 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-kvgmg"] Oct 30 21:04:42.947205 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:04:42.947178 4611 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:04:42.947312 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:04:42.947261 4611 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:04:42.947312 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:04:42.947307 4611 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:04:42.947593 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:04:42.947568 4611 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7hf8r\"" Oct 30 21:04:43.011890 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:04:43.011862 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vjz5x\" (UniqueName: \"kubernetes.io/projected/3909d445-54f4-43ae-b2f4-1bae843188d8-kube-api-access-vjz5x\") pod \"maas-api-revoke-sub-2e1f5b5f-kvgmg\" (UID: \"3909d445-54f4-43ae-b2f4-1bae843188d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-kvgmg" Oct 30 21:04:43.012064 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:04:43.011922 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3909d445-54f4-43ae-b2f4-1bae843188d8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-kvgmg\" (UID: \"3909d445-54f4-43ae-b2f4-1bae843188d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-kvgmg" Oct 30 21:04:43.012064 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:04:43.012028 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3909d445-54f4-43ae-b2f4-1bae843188d8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-kvgmg\" (UID: \"3909d445-54f4-43ae-b2f4-1bae843188d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-kvgmg" Oct 30 21:04:43.112791 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:04:43.112756 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjz5x\" (UniqueName: \"kubernetes.io/projected/3909d445-54f4-43ae-b2f4-1bae843188d8-kube-api-access-vjz5x\") pod \"maas-api-revoke-sub-2e1f5b5f-kvgmg\" (UID: \"3909d445-54f4-43ae-b2f4-1bae843188d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-kvgmg" Oct 30 21:04:43.112971 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:04:43.112828 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3909d445-54f4-43ae-b2f4-1bae843188d8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-kvgmg\" (UID: \"3909d445-54f4-43ae-b2f4-1bae843188d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-kvgmg" Oct 30 21:04:43.112971 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:04:43.112870 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3909d445-54f4-43ae-b2f4-1bae843188d8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-kvgmg\" (UID: \"3909d445-54f4-43ae-b2f4-1bae843188d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-kvgmg" Oct 30 21:04:43.113661 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:04:43.113640 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3909d445-54f4-43ae-b2f4-1bae843188d8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-kvgmg\" (UID: \"3909d445-54f4-43ae-b2f4-1bae843188d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-kvgmg" Oct 30 21:04:43.115190 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:04:43.115173 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3909d445-54f4-43ae-b2f4-1bae843188d8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-kvgmg\" (UID: \"3909d445-54f4-43ae-b2f4-1bae843188d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-kvgmg" Oct 30 21:04:43.128738 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:04:43.128714 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vjz5x\" (UniqueName: \"kubernetes.io/projected/3909d445-54f4-43ae-b2f4-1bae843188d8-kube-api-access-vjz5x\") pod \"maas-api-revoke-sub-2e1f5b5f-kvgmg\" (UID: \"3909d445-54f4-43ae-b2f4-1bae843188d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-kvgmg" Oct 30 21:04:43.261584 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:04:43.261487 4611 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-kvgmg" Oct 30 21:04:43.537926 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:04:43.537889 4611 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-kvgmg"] Oct 30 21:04:43.540488 ip-10-0-133-192 kubenswrapper[4611]: W1030 21:04:43.540456 4611 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3909d445_54f4_43ae_b2f4_1bae843188d8.slice/crio-a3a16ca95111d5a6647900f87393b8b250a8a8c287a5ba090adaceebd4da28ac WatchSource:0}: Error finding container a3a16ca95111d5a6647900f87393b8b250a8a8c287a5ba090adaceebd4da28ac: Status 404 returned error can't find the container with id a3a16ca95111d5a6647900f87393b8b250a8a8c287a5ba090adaceebd4da28ac Oct 30 21:04:43.904667 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:04:43.904634 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-kvgmg" event={"ID":"3909d445-54f4-43ae-b2f4-1bae843188d8","Type":"ContainerStarted","Data":"4ff0ac7b0038c429ee9d56e92d59cc563181405975e8706c7edcde3b84d4000d"} Oct 30 21:04:43.904791 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:04:43.904686 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-kvgmg" event={"ID":"3909d445-54f4-43ae-b2f4-1bae843188d8","Type":"ContainerStarted","Data":"a3a16ca95111d5a6647900f87393b8b250a8a8c287a5ba090adaceebd4da28ac"} Oct 30 21:04:43.922722 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:04:43.922678 4611 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-kvgmg" podStartSLOduration=1.9226637279999998 podStartE2EDuration="1.922663728s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:04:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:04:43.921394761 +0000 UTC m=+1538.455421379" watchObservedRunningTime="2026-10-30 21:04:43.922663728 +0000 UTC m=+1538.456690344" Oct 30 21:04:44.912316 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:04:44.912279 4611 generic.go:356] "Generic (PLEG): container finished" podID="3909d445-54f4-43ae-b2f4-1bae843188d8" containerID="4ff0ac7b0038c429ee9d56e92d59cc563181405975e8706c7edcde3b84d4000d" exitCode=0 Oct 30 21:04:44.912774 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:04:44.912356 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-kvgmg" event={"ID":"3909d445-54f4-43ae-b2f4-1bae843188d8","Type":"ContainerDied","Data":"4ff0ac7b0038c429ee9d56e92d59cc563181405975e8706c7edcde3b84d4000d"} Oct 30 21:04:46.052062 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:04:46.052036 4611 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-kvgmg" Oct 30 21:04:46.140455 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:04:46.140427 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3909d445-54f4-43ae-b2f4-1bae843188d8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3909d445-54f4-43ae-b2f4-1bae843188d8-maas-api-xks-ca\") pod \"3909d445-54f4-43ae-b2f4-1bae843188d8\" (UID: \"3909d445-54f4-43ae-b2f4-1bae843188d8\") " Oct 30 21:04:46.140724 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:04:46.140473 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3909d445-54f4-43ae-b2f4-1bae843188d8-kube-api-access-vjz5x\" (UniqueName: \"kubernetes.io/projected/3909d445-54f4-43ae-b2f4-1bae843188d8-kube-api-access-vjz5x\") pod \"3909d445-54f4-43ae-b2f4-1bae843188d8\" (UID: \"3909d445-54f4-43ae-b2f4-1bae843188d8\") " Oct 30 21:04:46.140724 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:04:46.140663 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3909d445-54f4-43ae-b2f4-1bae843188d8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3909d445-54f4-43ae-b2f4-1bae843188d8-maas-api-openshift-ca\") pod \"3909d445-54f4-43ae-b2f4-1bae843188d8\" (UID: \"3909d445-54f4-43ae-b2f4-1bae843188d8\") " Oct 30 21:04:46.141022 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:04:46.140985 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3909d445-54f4-43ae-b2f4-1bae843188d8-maas-api-openshift-ca" pod "3909d445-54f4-43ae-b2f4-1bae843188d8" (UID: "3909d445-54f4-43ae-b2f4-1bae843188d8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:04:46.142581 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:04:46.142556 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3909d445-54f4-43ae-b2f4-1bae843188d8-maas-api-xks-ca" pod "3909d445-54f4-43ae-b2f4-1bae843188d8" (UID: "3909d445-54f4-43ae-b2f4-1bae843188d8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:04:46.142660 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:04:46.142635 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3909d445-54f4-43ae-b2f4-1bae843188d8-kube-api-access-vjz5x" pod "3909d445-54f4-43ae-b2f4-1bae843188d8" (UID: "3909d445-54f4-43ae-b2f4-1bae843188d8"). InnerVolumeSpecName "kube-api-access-vjz5x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:04:46.242152 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:04:46.242098 4611 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3909d445-54f4-43ae-b2f4-1bae843188d8-maas-api-xks-ca\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:04:46.242152 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:04:46.242120 4611 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vjz5x\" (UniqueName: \"kubernetes.io/projected/3909d445-54f4-43ae-b2f4-1bae843188d8-kube-api-access-vjz5x\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:04:46.242152 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:04:46.242130 4611 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3909d445-54f4-43ae-b2f4-1bae843188d8-maas-api-openshift-ca\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:04:46.922965 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:04:46.922934 4611 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-kvgmg" Oct 30 21:04:46.923146 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:04:46.922967 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-kvgmg" event={"ID":"3909d445-54f4-43ae-b2f4-1bae843188d8","Type":"ContainerDied","Data":"a3a16ca95111d5a6647900f87393b8b250a8a8c287a5ba090adaceebd4da28ac"} Oct 30 21:04:46.923146 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:04:46.923013 4611 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a3a16ca95111d5a6647900f87393b8b250a8a8c287a5ba090adaceebd4da28ac" Oct 30 21:04:48.022363 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:04:48.022321 4611 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-be73052e-plwg2"] Oct 30 21:04:48.027148 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:04:48.027122 4611 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-be73052e-plwg2"] Oct 30 21:04:49.123055 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:04:49.123021 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:04:49.604970 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:04:49.604930 4611 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="44c74378-cef2-413a-a344-81d8093a89e8" path="/var/lib/kubelet/pods/44c74378-cef2-413a-a344-81d8093a89e8/volumes" Oct 30 21:04:53.850367 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:04:53.850329 4611 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-w5w78"] Oct 30 21:04:53.850787 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:04:53.850748 4611 cpu_manager.go:414] "containerMap: removing container" podUID="3909d445-54f4-43ae-b2f4-1bae843188d8" containerName="revoke-keys" Oct 30 21:04:53.850787 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:04:53.850762 4611 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3909d445-54f4-43ae-b2f4-1bae843188d8" containerName="revoke-keys" Oct 30 21:04:53.850866 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:04:53.850855 4611 memory_manager.go:365] "RemoveStaleState removing state" podUID="3909d445-54f4-43ae-b2f4-1bae843188d8" containerName="revoke-keys" Oct 30 21:04:53.853756 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:04:53.853741 4611 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-w5w78" Oct 30 21:04:53.857765 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:04:53.857739 4611 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:04:53.857911 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:04:53.857768 4611 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7hf8r\"" Oct 30 21:04:53.857911 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:04:53.857807 4611 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:04:53.857911 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:04:53.857809 4611 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:04:53.863681 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:04:53.863647 4611 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-w5w78"] Oct 30 21:04:54.013141 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:04:54.013110 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2qlx2\" (UniqueName: \"kubernetes.io/projected/235db0c6-6afd-470d-8a49-3cea7316a82a-kube-api-access-2qlx2\") pod \"maas-api-revoke-sub-ec843018-w5w78\" (UID: \"235db0c6-6afd-470d-8a49-3cea7316a82a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-w5w78" Oct 30 21:04:54.013291 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:04:54.013224 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/235db0c6-6afd-470d-8a49-3cea7316a82a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-w5w78\" (UID: \"235db0c6-6afd-470d-8a49-3cea7316a82a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-w5w78" Oct 30 21:04:54.013291 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:04:54.013267 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/235db0c6-6afd-470d-8a49-3cea7316a82a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-w5w78\" (UID: \"235db0c6-6afd-470d-8a49-3cea7316a82a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-w5w78" Oct 30 21:04:54.113748 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:04:54.113725 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/235db0c6-6afd-470d-8a49-3cea7316a82a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-w5w78\" (UID: \"235db0c6-6afd-470d-8a49-3cea7316a82a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-w5w78" Oct 30 21:04:54.113920 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:04:54.113772 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/235db0c6-6afd-470d-8a49-3cea7316a82a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-w5w78\" (UID: \"235db0c6-6afd-470d-8a49-3cea7316a82a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-w5w78" Oct 30 21:04:54.113920 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:04:54.113821 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2qlx2\" (UniqueName: \"kubernetes.io/projected/235db0c6-6afd-470d-8a49-3cea7316a82a-kube-api-access-2qlx2\") pod \"maas-api-revoke-sub-ec843018-w5w78\" (UID: \"235db0c6-6afd-470d-8a49-3cea7316a82a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-w5w78" Oct 30 21:04:54.114395 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:04:54.114367 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/235db0c6-6afd-470d-8a49-3cea7316a82a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-w5w78\" (UID: \"235db0c6-6afd-470d-8a49-3cea7316a82a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-w5w78" Oct 30 21:04:54.116034 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:04:54.116014 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/235db0c6-6afd-470d-8a49-3cea7316a82a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-w5w78\" (UID: \"235db0c6-6afd-470d-8a49-3cea7316a82a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-w5w78" Oct 30 21:04:54.127067 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:04:54.127046 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2qlx2\" (UniqueName: \"kubernetes.io/projected/235db0c6-6afd-470d-8a49-3cea7316a82a-kube-api-access-2qlx2\") pod \"maas-api-revoke-sub-ec843018-w5w78\" (UID: \"235db0c6-6afd-470d-8a49-3cea7316a82a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-w5w78" Oct 30 21:04:54.167461 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:04:54.167430 4611 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-w5w78" Oct 30 21:04:54.468860 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:04:54.468836 4611 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-w5w78"] Oct 30 21:04:54.471124 ip-10-0-133-192 kubenswrapper[4611]: W1030 21:04:54.471081 4611 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod235db0c6_6afd_470d_8a49_3cea7316a82a.slice/crio-45a92eca3614027eae4d702854b86f35ec4d162de7ab16cfd02b21900bad501c WatchSource:0}: Error finding container 45a92eca3614027eae4d702854b86f35ec4d162de7ab16cfd02b21900bad501c: Status 404 returned error can't find the container with id 45a92eca3614027eae4d702854b86f35ec4d162de7ab16cfd02b21900bad501c Oct 30 21:04:54.967032 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:04:54.967000 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-w5w78" event={"ID":"235db0c6-6afd-470d-8a49-3cea7316a82a","Type":"ContainerStarted","Data":"95f514df524b461231af54589071497892056ea6cf17e094a06fbc648a6befdf"} Oct 30 21:04:54.967383 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:04:54.967052 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-w5w78" event={"ID":"235db0c6-6afd-470d-8a49-3cea7316a82a","Type":"ContainerStarted","Data":"45a92eca3614027eae4d702854b86f35ec4d162de7ab16cfd02b21900bad501c"} Oct 30 21:04:54.985232 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:04:54.985186 4611 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-w5w78" podStartSLOduration=1.985173839 podStartE2EDuration="1.985173839s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:04:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:04:54.983487121 +0000 UTC m=+1549.517513739" watchObservedRunningTime="2026-10-30 21:04:54.985173839 +0000 UTC m=+1549.519200455" Oct 30 21:04:55.973244 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:04:55.973208 4611 generic.go:356] "Generic (PLEG): container finished" podID="235db0c6-6afd-470d-8a49-3cea7316a82a" containerID="95f514df524b461231af54589071497892056ea6cf17e094a06fbc648a6befdf" exitCode=0 Oct 30 21:04:55.973669 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:04:55.973265 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-w5w78" event={"ID":"235db0c6-6afd-470d-8a49-3cea7316a82a","Type":"ContainerDied","Data":"95f514df524b461231af54589071497892056ea6cf17e094a06fbc648a6befdf"} Oct 30 21:04:57.740287 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:04:57.740260 4611 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-w5w78" Oct 30 21:04:57.771691 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:04:57.770859 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/235db0c6-6afd-470d-8a49-3cea7316a82a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/235db0c6-6afd-470d-8a49-3cea7316a82a-maas-api-openshift-ca\") pod \"235db0c6-6afd-470d-8a49-3cea7316a82a\" (UID: \"235db0c6-6afd-470d-8a49-3cea7316a82a\") " Oct 30 21:04:57.771691 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:04:57.770946 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/235db0c6-6afd-470d-8a49-3cea7316a82a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/235db0c6-6afd-470d-8a49-3cea7316a82a-maas-api-xks-ca\") pod \"235db0c6-6afd-470d-8a49-3cea7316a82a\" (UID: \"235db0c6-6afd-470d-8a49-3cea7316a82a\") " Oct 30 21:04:57.771691 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:04:57.770982 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/235db0c6-6afd-470d-8a49-3cea7316a82a-kube-api-access-2qlx2\" (UniqueName: \"kubernetes.io/projected/235db0c6-6afd-470d-8a49-3cea7316a82a-kube-api-access-2qlx2\") pod \"235db0c6-6afd-470d-8a49-3cea7316a82a\" (UID: \"235db0c6-6afd-470d-8a49-3cea7316a82a\") " Oct 30 21:04:57.792038 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:04:57.791994 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/235db0c6-6afd-470d-8a49-3cea7316a82a-kube-api-access-2qlx2" pod "235db0c6-6afd-470d-8a49-3cea7316a82a" (UID: "235db0c6-6afd-470d-8a49-3cea7316a82a"). InnerVolumeSpecName "kube-api-access-2qlx2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:04:57.792300 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:04:57.792279 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/235db0c6-6afd-470d-8a49-3cea7316a82a-maas-api-xks-ca" pod "235db0c6-6afd-470d-8a49-3cea7316a82a" (UID: "235db0c6-6afd-470d-8a49-3cea7316a82a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:04:57.792657 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:04:57.792624 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/235db0c6-6afd-470d-8a49-3cea7316a82a-maas-api-openshift-ca" pod "235db0c6-6afd-470d-8a49-3cea7316a82a" (UID: "235db0c6-6afd-470d-8a49-3cea7316a82a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:04:57.903005 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:04:57.902870 4611 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/235db0c6-6afd-470d-8a49-3cea7316a82a-maas-api-openshift-ca\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:04:57.903005 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:04:57.902907 4611 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/235db0c6-6afd-470d-8a49-3cea7316a82a-maas-api-xks-ca\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:04:57.903005 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:04:57.902924 4611 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2qlx2\" (UniqueName: \"kubernetes.io/projected/235db0c6-6afd-470d-8a49-3cea7316a82a-kube-api-access-2qlx2\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:04:58.137208 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:04:58.137168 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-w5w78" event={"ID":"235db0c6-6afd-470d-8a49-3cea7316a82a","Type":"ContainerDied","Data":"45a92eca3614027eae4d702854b86f35ec4d162de7ab16cfd02b21900bad501c"} Oct 30 21:04:58.137527 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:04:58.137489 4611 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="45a92eca3614027eae4d702854b86f35ec4d162de7ab16cfd02b21900bad501c" Oct 30 21:04:58.137758 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:04:58.137740 4611 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-w5w78" Oct 30 21:05:03.916870 ip-10-0-133-192 kubenswrapper[4611]: E1030 21:05:03.916834 4611 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hwzc6" containerName="extractor" Oct 30 21:05:07.933014 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:05:07.932975 4611 scope.go:122] "RemoveContainer" containerID="af320cd17fbeb1e6606e834c84593d4e5f8b21dbfbf265ac2b860e75e69b8c76" Oct 30 21:05:27.311273 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:05:27.311183 4611 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-pbg7g"] Oct 30 21:05:27.313715 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:05:27.311598 4611 cpu_manager.go:414] "containerMap: removing container" podUID="235db0c6-6afd-470d-8a49-3cea7316a82a" containerName="revoke-keys" Oct 30 21:05:27.313715 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:05:27.311611 4611 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="235db0c6-6afd-470d-8a49-3cea7316a82a" containerName="revoke-keys" Oct 30 21:05:27.313715 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:05:27.311672 4611 memory_manager.go:365] "RemoveStaleState removing state" podUID="235db0c6-6afd-470d-8a49-3cea7316a82a" containerName="revoke-keys" Oct 30 21:05:27.314699 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:05:27.314680 4611 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-pbg7g" Oct 30 21:05:27.317525 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:05:27.317492 4611 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:05:27.317737 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:05:27.317720 4611 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:05:27.320538 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:05:27.320501 4611 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:05:27.320783 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:05:27.320763 4611 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7hf8r\"" Oct 30 21:05:27.324180 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:05:27.324157 4611 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-pbg7g"] Oct 30 21:05:27.405487 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:05:27.405456 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x9dkc\" (UniqueName: \"kubernetes.io/projected/9a6184dd-ccdd-4f76-8f61-02f92047efcf-kube-api-access-x9dkc\") pod \"maas-api-revoke-sub-046e1343-pbg7g\" (UID: \"9a6184dd-ccdd-4f76-8f61-02f92047efcf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-pbg7g" Oct 30 21:05:27.405669 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:05:27.405608 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9a6184dd-ccdd-4f76-8f61-02f92047efcf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-pbg7g\" (UID: \"9a6184dd-ccdd-4f76-8f61-02f92047efcf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-pbg7g" Oct 30 21:05:27.405743 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:05:27.405678 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9a6184dd-ccdd-4f76-8f61-02f92047efcf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-pbg7g\" (UID: \"9a6184dd-ccdd-4f76-8f61-02f92047efcf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-pbg7g" Oct 30 21:05:27.506475 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:05:27.506440 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9a6184dd-ccdd-4f76-8f61-02f92047efcf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-pbg7g\" (UID: \"9a6184dd-ccdd-4f76-8f61-02f92047efcf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-pbg7g" Oct 30 21:05:27.506693 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:05:27.506532 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9a6184dd-ccdd-4f76-8f61-02f92047efcf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-pbg7g\" (UID: \"9a6184dd-ccdd-4f76-8f61-02f92047efcf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-pbg7g" Oct 30 21:05:27.506693 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:05:27.506592 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x9dkc\" (UniqueName: \"kubernetes.io/projected/9a6184dd-ccdd-4f76-8f61-02f92047efcf-kube-api-access-x9dkc\") pod \"maas-api-revoke-sub-046e1343-pbg7g\" (UID: \"9a6184dd-ccdd-4f76-8f61-02f92047efcf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-pbg7g" Oct 30 21:05:27.507233 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:05:27.507211 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9a6184dd-ccdd-4f76-8f61-02f92047efcf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-pbg7g\" (UID: \"9a6184dd-ccdd-4f76-8f61-02f92047efcf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-pbg7g" Oct 30 21:05:27.508834 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:05:27.508810 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9a6184dd-ccdd-4f76-8f61-02f92047efcf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-pbg7g\" (UID: \"9a6184dd-ccdd-4f76-8f61-02f92047efcf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-pbg7g" Oct 30 21:05:27.517879 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:05:27.517849 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x9dkc\" (UniqueName: \"kubernetes.io/projected/9a6184dd-ccdd-4f76-8f61-02f92047efcf-kube-api-access-x9dkc\") pod \"maas-api-revoke-sub-046e1343-pbg7g\" (UID: \"9a6184dd-ccdd-4f76-8f61-02f92047efcf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-pbg7g" Oct 30 21:05:27.629142 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:05:27.629121 4611 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-pbg7g" Oct 30 21:05:27.903474 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:05:27.903447 4611 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-pbg7g"] Oct 30 21:05:27.906643 ip-10-0-133-192 kubenswrapper[4611]: W1030 21:05:27.906611 4611 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a6184dd_ccdd_4f76_8f61_02f92047efcf.slice/crio-efbc02163b5ee34285db36539b880196717471718f58286c98c718a18a9281da WatchSource:0}: Error finding container efbc02163b5ee34285db36539b880196717471718f58286c98c718a18a9281da: Status 404 returned error can't find the container with id efbc02163b5ee34285db36539b880196717471718f58286c98c718a18a9281da Oct 30 21:05:28.319173 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:05:28.319086 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-pbg7g" event={"ID":"9a6184dd-ccdd-4f76-8f61-02f92047efcf","Type":"ContainerStarted","Data":"3b8879221448b0b50275051e14a71070d57b45857299705ec87f77acc89e6f2d"} Oct 30 21:05:28.319173 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:05:28.319148 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-pbg7g" event={"ID":"9a6184dd-ccdd-4f76-8f61-02f92047efcf","Type":"ContainerStarted","Data":"efbc02163b5ee34285db36539b880196717471718f58286c98c718a18a9281da"} Oct 30 21:05:28.342028 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:05:28.341987 4611 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-pbg7g" podStartSLOduration=1.341974567 podStartE2EDuration="1.341974567s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:05:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:05:28.34028859 +0000 UTC m=+1582.874315305" watchObservedRunningTime="2026-10-30 21:05:28.341974567 +0000 UTC m=+1582.876001182" Oct 30 21:05:29.325471 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:05:29.325436 4611 generic.go:356] "Generic (PLEG): container finished" podID="9a6184dd-ccdd-4f76-8f61-02f92047efcf" containerID="3b8879221448b0b50275051e14a71070d57b45857299705ec87f77acc89e6f2d" exitCode=0 Oct 30 21:05:29.325868 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:05:29.325539 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-pbg7g" event={"ID":"9a6184dd-ccdd-4f76-8f61-02f92047efcf","Type":"ContainerDied","Data":"3b8879221448b0b50275051e14a71070d57b45857299705ec87f77acc89e6f2d"} Oct 30 21:05:30.460861 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:05:30.460839 4611 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-pbg7g" Oct 30 21:05:30.536453 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:05:30.536426 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9a6184dd-ccdd-4f76-8f61-02f92047efcf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9a6184dd-ccdd-4f76-8f61-02f92047efcf-maas-api-openshift-ca\") pod \"9a6184dd-ccdd-4f76-8f61-02f92047efcf\" (UID: \"9a6184dd-ccdd-4f76-8f61-02f92047efcf\") " Oct 30 21:05:30.536606 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:05:30.536585 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a6184dd-ccdd-4f76-8f61-02f92047efcf-kube-api-access-x9dkc\" (UniqueName: \"kubernetes.io/projected/9a6184dd-ccdd-4f76-8f61-02f92047efcf-kube-api-access-x9dkc\") pod \"9a6184dd-ccdd-4f76-8f61-02f92047efcf\" (UID: \"9a6184dd-ccdd-4f76-8f61-02f92047efcf\") " Oct 30 21:05:30.536750 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:05:30.536730 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9a6184dd-ccdd-4f76-8f61-02f92047efcf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9a6184dd-ccdd-4f76-8f61-02f92047efcf-maas-api-xks-ca\") pod \"9a6184dd-ccdd-4f76-8f61-02f92047efcf\" (UID: \"9a6184dd-ccdd-4f76-8f61-02f92047efcf\") " Oct 30 21:05:30.536885 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:05:30.536859 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a6184dd-ccdd-4f76-8f61-02f92047efcf-maas-api-openshift-ca" pod "9a6184dd-ccdd-4f76-8f61-02f92047efcf" (UID: "9a6184dd-ccdd-4f76-8f61-02f92047efcf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:05:30.537055 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:05:30.537039 4611 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9a6184dd-ccdd-4f76-8f61-02f92047efcf-maas-api-openshift-ca\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:05:30.538628 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:05:30.538606 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9a6184dd-ccdd-4f76-8f61-02f92047efcf-maas-api-xks-ca" pod "9a6184dd-ccdd-4f76-8f61-02f92047efcf" (UID: "9a6184dd-ccdd-4f76-8f61-02f92047efcf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:05:30.538701 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:05:30.538626 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a6184dd-ccdd-4f76-8f61-02f92047efcf-kube-api-access-x9dkc" pod "9a6184dd-ccdd-4f76-8f61-02f92047efcf" (UID: "9a6184dd-ccdd-4f76-8f61-02f92047efcf"). InnerVolumeSpecName "kube-api-access-x9dkc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:05:30.638421 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:05:30.638394 4611 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x9dkc\" (UniqueName: \"kubernetes.io/projected/9a6184dd-ccdd-4f76-8f61-02f92047efcf-kube-api-access-x9dkc\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:05:30.638421 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:05:30.638420 4611 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9a6184dd-ccdd-4f76-8f61-02f92047efcf-maas-api-xks-ca\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:05:31.336339 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:05:31.336307 4611 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-pbg7g" Oct 30 21:05:31.336494 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:05:31.336339 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-pbg7g" event={"ID":"9a6184dd-ccdd-4f76-8f61-02f92047efcf","Type":"ContainerDied","Data":"efbc02163b5ee34285db36539b880196717471718f58286c98c718a18a9281da"} Oct 30 21:05:31.336494 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:05:31.336379 4611 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="efbc02163b5ee34285db36539b880196717471718f58286c98c718a18a9281da" Oct 30 21:05:33.916773 ip-10-0-133-192 kubenswrapper[4611]: E1030 21:05:33.916735 4611 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hwzc6" containerName="extractor" Oct 30 21:05:37.530957 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:05:37.530919 4611 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-mtvnz"] Oct 30 21:05:37.531379 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:05:37.531340 4611 cpu_manager.go:414] "containerMap: removing container" podUID="9a6184dd-ccdd-4f76-8f61-02f92047efcf" containerName="revoke-keys" Oct 30 21:05:37.531379 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:05:37.531351 4611 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a6184dd-ccdd-4f76-8f61-02f92047efcf" containerName="revoke-keys" Oct 30 21:05:37.531460 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:05:37.531420 4611 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a6184dd-ccdd-4f76-8f61-02f92047efcf" containerName="revoke-keys" Oct 30 21:05:37.534621 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:05:37.534605 4611 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-mtvnz" Oct 30 21:05:37.537610 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:05:37.537563 4611 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:05:37.537743 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:05:37.537726 4611 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:05:37.544693 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:05:37.544664 4611 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-mtvnz"] Oct 30 21:05:37.602023 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:05:37.601992 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-scll2\" (UniqueName: \"kubernetes.io/projected/4792453f-2774-409b-8dc4-a9b8ce0f7bf7-kube-api-access-scll2\") pod \"maas-api-revoke-sub-d3fba056-mtvnz\" (UID: \"4792453f-2774-409b-8dc4-a9b8ce0f7bf7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-mtvnz" Oct 30 21:05:37.602203 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:05:37.602134 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4792453f-2774-409b-8dc4-a9b8ce0f7bf7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-mtvnz\" (UID: \"4792453f-2774-409b-8dc4-a9b8ce0f7bf7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-mtvnz" Oct 30 21:05:37.602291 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:05:37.602269 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4792453f-2774-409b-8dc4-a9b8ce0f7bf7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-mtvnz\" (UID: \"4792453f-2774-409b-8dc4-a9b8ce0f7bf7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-mtvnz" Oct 30 21:05:37.621745 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:05:37.621723 4611 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:05:37.621851 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:05:37.621797 4611 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7hf8r\"" Oct 30 21:05:37.702918 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:05:37.702890 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4792453f-2774-409b-8dc4-a9b8ce0f7bf7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-mtvnz\" (UID: \"4792453f-2774-409b-8dc4-a9b8ce0f7bf7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-mtvnz" Oct 30 21:05:37.703067 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:05:37.702928 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-scll2\" (UniqueName: \"kubernetes.io/projected/4792453f-2774-409b-8dc4-a9b8ce0f7bf7-kube-api-access-scll2\") pod \"maas-api-revoke-sub-d3fba056-mtvnz\" (UID: \"4792453f-2774-409b-8dc4-a9b8ce0f7bf7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-mtvnz" Oct 30 21:05:37.703067 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:05:37.702987 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4792453f-2774-409b-8dc4-a9b8ce0f7bf7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-mtvnz\" (UID: \"4792453f-2774-409b-8dc4-a9b8ce0f7bf7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-mtvnz" Oct 30 21:05:37.703537 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:05:37.703492 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4792453f-2774-409b-8dc4-a9b8ce0f7bf7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-mtvnz\" (UID: \"4792453f-2774-409b-8dc4-a9b8ce0f7bf7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-mtvnz" Oct 30 21:05:37.705330 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:05:37.705309 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4792453f-2774-409b-8dc4-a9b8ce0f7bf7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-mtvnz\" (UID: \"4792453f-2774-409b-8dc4-a9b8ce0f7bf7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-mtvnz" Oct 30 21:05:37.714460 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:05:37.714435 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-scll2\" (UniqueName: \"kubernetes.io/projected/4792453f-2774-409b-8dc4-a9b8ce0f7bf7-kube-api-access-scll2\") pod \"maas-api-revoke-sub-d3fba056-mtvnz\" (UID: \"4792453f-2774-409b-8dc4-a9b8ce0f7bf7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-mtvnz" Oct 30 21:05:37.848797 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:05:37.848719 4611 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-mtvnz" Oct 30 21:05:38.030073 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:05:38.030039 4611 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4a943ca6-9nm9f"] Oct 30 21:05:38.035684 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:05:38.035651 4611 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4a943ca6-9nm9f"] Oct 30 21:05:38.125650 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:05:38.125628 4611 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-mtvnz"] Oct 30 21:05:38.127545 ip-10-0-133-192 kubenswrapper[4611]: W1030 21:05:38.127491 4611 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4792453f_2774_409b_8dc4_a9b8ce0f7bf7.slice/crio-c1656ac6ae32a9b9925d202e246dc03f1690a20a77d0cf4337097e161744ac9f WatchSource:0}: Error finding container c1656ac6ae32a9b9925d202e246dc03f1690a20a77d0cf4337097e161744ac9f: Status 404 returned error can't find the container with id c1656ac6ae32a9b9925d202e246dc03f1690a20a77d0cf4337097e161744ac9f Oct 30 21:05:38.372243 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:05:38.372212 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-mtvnz" event={"ID":"4792453f-2774-409b-8dc4-a9b8ce0f7bf7","Type":"ContainerStarted","Data":"c1656ac6ae32a9b9925d202e246dc03f1690a20a77d0cf4337097e161744ac9f"} Oct 30 21:05:39.382361 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:05:39.382320 4611 generic.go:356] "Generic (PLEG): container finished" podID="4792453f-2774-409b-8dc4-a9b8ce0f7bf7" containerID="55c2edc295b367157a85b69b7e9679018546cabb9f4bbb7d116e94c7cda79786" exitCode=0 Oct 30 21:05:39.383022 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:05:39.382411 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-mtvnz" event={"ID":"4792453f-2774-409b-8dc4-a9b8ce0f7bf7","Type":"ContainerDied","Data":"55c2edc295b367157a85b69b7e9679018546cabb9f4bbb7d116e94c7cda79786"} Oct 30 21:05:39.625968 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:05:39.625932 4611 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0de46c2f-4a48-4d46-be31-cd2173b80b5f" path="/var/lib/kubelet/pods/0de46c2f-4a48-4d46-be31-cd2173b80b5f/volumes" Oct 30 21:05:41.041533 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:05:41.040517 4611 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-mtvnz" Oct 30 21:05:41.155591 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:05:41.146792 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4792453f-2774-409b-8dc4-a9b8ce0f7bf7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4792453f-2774-409b-8dc4-a9b8ce0f7bf7-maas-api-openshift-ca\") pod \"4792453f-2774-409b-8dc4-a9b8ce0f7bf7\" (UID: \"4792453f-2774-409b-8dc4-a9b8ce0f7bf7\") " Oct 30 21:05:41.155591 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:05:41.146903 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4792453f-2774-409b-8dc4-a9b8ce0f7bf7-kube-api-access-scll2\" (UniqueName: \"kubernetes.io/projected/4792453f-2774-409b-8dc4-a9b8ce0f7bf7-kube-api-access-scll2\") pod \"4792453f-2774-409b-8dc4-a9b8ce0f7bf7\" (UID: \"4792453f-2774-409b-8dc4-a9b8ce0f7bf7\") " Oct 30 21:05:41.155591 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:05:41.146945 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4792453f-2774-409b-8dc4-a9b8ce0f7bf7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4792453f-2774-409b-8dc4-a9b8ce0f7bf7-maas-api-xks-ca\") pod \"4792453f-2774-409b-8dc4-a9b8ce0f7bf7\" (UID: \"4792453f-2774-409b-8dc4-a9b8ce0f7bf7\") " Oct 30 21:05:41.188650 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:05:41.188613 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4792453f-2774-409b-8dc4-a9b8ce0f7bf7-maas-api-openshift-ca" pod "4792453f-2774-409b-8dc4-a9b8ce0f7bf7" (UID: "4792453f-2774-409b-8dc4-a9b8ce0f7bf7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:05:41.195697 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:05:41.195662 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4792453f-2774-409b-8dc4-a9b8ce0f7bf7-maas-api-xks-ca" pod "4792453f-2774-409b-8dc4-a9b8ce0f7bf7" (UID: "4792453f-2774-409b-8dc4-a9b8ce0f7bf7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:05:41.200697 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:05:41.200660 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4792453f-2774-409b-8dc4-a9b8ce0f7bf7-kube-api-access-scll2" pod "4792453f-2774-409b-8dc4-a9b8ce0f7bf7" (UID: "4792453f-2774-409b-8dc4-a9b8ce0f7bf7"). InnerVolumeSpecName "kube-api-access-scll2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:05:41.252194 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:05:41.252123 4611 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-scll2\" (UniqueName: \"kubernetes.io/projected/4792453f-2774-409b-8dc4-a9b8ce0f7bf7-kube-api-access-scll2\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:05:41.252194 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:05:41.252152 4611 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4792453f-2774-409b-8dc4-a9b8ce0f7bf7-maas-api-xks-ca\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:05:41.252194 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:05:41.252171 4611 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4792453f-2774-409b-8dc4-a9b8ce0f7bf7-maas-api-openshift-ca\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:05:41.431694 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:05:41.431581 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-mtvnz" event={"ID":"4792453f-2774-409b-8dc4-a9b8ce0f7bf7","Type":"ContainerDied","Data":"c1656ac6ae32a9b9925d202e246dc03f1690a20a77d0cf4337097e161744ac9f"} Oct 30 21:05:41.431959 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:05:41.431925 4611 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c1656ac6ae32a9b9925d202e246dc03f1690a20a77d0cf4337097e161744ac9f" Oct 30 21:05:41.433542 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:05:41.432918 4611 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-mtvnz" Oct 30 21:05:49.143730 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:05:49.143693 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:05:58.486273 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:05:58.486235 4611 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-nf5px"] Oct 30 21:05:58.486772 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:05:58.486648 4611 cpu_manager.go:414] "containerMap: removing container" podUID="4792453f-2774-409b-8dc4-a9b8ce0f7bf7" containerName="revoke-keys" Oct 30 21:05:58.486772 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:05:58.486661 4611 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4792453f-2774-409b-8dc4-a9b8ce0f7bf7" containerName="revoke-keys" Oct 30 21:05:58.486772 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:05:58.486755 4611 memory_manager.go:365] "RemoveStaleState removing state" podUID="4792453f-2774-409b-8dc4-a9b8ce0f7bf7" containerName="revoke-keys" Oct 30 21:05:58.489950 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:05:58.489930 4611 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-nf5px" Oct 30 21:05:58.493116 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:05:58.493090 4611 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:05:58.493116 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:05:58.493107 4611 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7hf8r\"" Oct 30 21:05:58.493271 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:05:58.493096 4611 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:05:58.495756 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:05:58.495736 4611 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:05:58.499603 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:05:58.499574 4611 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-nf5px"] Oct 30 21:05:58.563218 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:05:58.563191 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dcb51fc1-b1d9-4ca4-9ea4-32a8f21f2c93-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-nf5px\" (UID: \"dcb51fc1-b1d9-4ca4-9ea4-32a8f21f2c93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-nf5px" Oct 30 21:05:58.563374 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:05:58.563228 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-22gjp\" (UniqueName: \"kubernetes.io/projected/dcb51fc1-b1d9-4ca4-9ea4-32a8f21f2c93-kube-api-access-22gjp\") pod \"maas-api-revoke-sub-ae594979-nf5px\" (UID: \"dcb51fc1-b1d9-4ca4-9ea4-32a8f21f2c93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-nf5px" Oct 30 21:05:58.563416 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:05:58.563374 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dcb51fc1-b1d9-4ca4-9ea4-32a8f21f2c93-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-nf5px\" (UID: \"dcb51fc1-b1d9-4ca4-9ea4-32a8f21f2c93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-nf5px" Oct 30 21:05:58.663810 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:05:58.663781 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dcb51fc1-b1d9-4ca4-9ea4-32a8f21f2c93-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-nf5px\" (UID: \"dcb51fc1-b1d9-4ca4-9ea4-32a8f21f2c93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-nf5px" Oct 30 21:05:58.663943 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:05:58.663836 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dcb51fc1-b1d9-4ca4-9ea4-32a8f21f2c93-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-nf5px\" (UID: \"dcb51fc1-b1d9-4ca4-9ea4-32a8f21f2c93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-nf5px" Oct 30 21:05:58.663943 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:05:58.663863 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-22gjp\" (UniqueName: \"kubernetes.io/projected/dcb51fc1-b1d9-4ca4-9ea4-32a8f21f2c93-kube-api-access-22gjp\") pod \"maas-api-revoke-sub-ae594979-nf5px\" (UID: \"dcb51fc1-b1d9-4ca4-9ea4-32a8f21f2c93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-nf5px" Oct 30 21:05:58.664377 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:05:58.664358 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dcb51fc1-b1d9-4ca4-9ea4-32a8f21f2c93-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-nf5px\" (UID: \"dcb51fc1-b1d9-4ca4-9ea4-32a8f21f2c93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-nf5px" Oct 30 21:05:58.666107 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:05:58.666087 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dcb51fc1-b1d9-4ca4-9ea4-32a8f21f2c93-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-nf5px\" (UID: \"dcb51fc1-b1d9-4ca4-9ea4-32a8f21f2c93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-nf5px" Oct 30 21:05:58.675483 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:05:58.675462 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-22gjp\" (UniqueName: \"kubernetes.io/projected/dcb51fc1-b1d9-4ca4-9ea4-32a8f21f2c93-kube-api-access-22gjp\") pod \"maas-api-revoke-sub-ae594979-nf5px\" (UID: \"dcb51fc1-b1d9-4ca4-9ea4-32a8f21f2c93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-nf5px" Oct 30 21:05:58.805298 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:05:58.805247 4611 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-nf5px" Oct 30 21:05:59.079187 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:05:59.079156 4611 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-nf5px"] Oct 30 21:05:59.081178 ip-10-0-133-192 kubenswrapper[4611]: W1030 21:05:59.081150 4611 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddcb51fc1_b1d9_4ca4_9ea4_32a8f21f2c93.slice/crio-0f18215c4a386060b00d55a899697497d1f7e19470a5eeedd3a290bd97fea834 WatchSource:0}: Error finding container 0f18215c4a386060b00d55a899697497d1f7e19470a5eeedd3a290bd97fea834: Status 404 returned error can't find the container with id 0f18215c4a386060b00d55a899697497d1f7e19470a5eeedd3a290bd97fea834 Oct 30 21:05:59.549112 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:05:59.549083 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-nf5px" event={"ID":"dcb51fc1-b1d9-4ca4-9ea4-32a8f21f2c93","Type":"ContainerStarted","Data":"3cc02d14f136f926b51ddca3758f741f2e2b4616a5a0805f32b1c1c5d0060a60"} Oct 30 21:05:59.549441 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:05:59.549128 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-nf5px" event={"ID":"dcb51fc1-b1d9-4ca4-9ea4-32a8f21f2c93","Type":"ContainerStarted","Data":"0f18215c4a386060b00d55a899697497d1f7e19470a5eeedd3a290bd97fea834"} Oct 30 21:05:59.567019 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:05:59.566972 4611 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-nf5px" podStartSLOduration=1.5669588779999999 podStartE2EDuration="1.566958878s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:05:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:05:59.565280726 +0000 UTC m=+1614.099307343" watchObservedRunningTime="2026-10-30 21:05:59.566958878 +0000 UTC m=+1614.100985517" Oct 30 21:06:00.555592 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:06:00.555558 4611 generic.go:356] "Generic (PLEG): container finished" podID="dcb51fc1-b1d9-4ca4-9ea4-32a8f21f2c93" containerID="3cc02d14f136f926b51ddca3758f741f2e2b4616a5a0805f32b1c1c5d0060a60" exitCode=0 Oct 30 21:06:00.555974 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:06:00.555599 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-nf5px" event={"ID":"dcb51fc1-b1d9-4ca4-9ea4-32a8f21f2c93","Type":"ContainerDied","Data":"3cc02d14f136f926b51ddca3758f741f2e2b4616a5a0805f32b1c1c5d0060a60"} Oct 30 21:06:01.689688 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:06:01.689656 4611 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-nf5px" Oct 30 21:06:01.797451 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:06:01.797420 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dcb51fc1-b1d9-4ca4-9ea4-32a8f21f2c93-kube-api-access-22gjp\" (UniqueName: \"kubernetes.io/projected/dcb51fc1-b1d9-4ca4-9ea4-32a8f21f2c93-kube-api-access-22gjp\") pod \"dcb51fc1-b1d9-4ca4-9ea4-32a8f21f2c93\" (UID: \"dcb51fc1-b1d9-4ca4-9ea4-32a8f21f2c93\") " Oct 30 21:06:01.797629 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:06:01.797531 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dcb51fc1-b1d9-4ca4-9ea4-32a8f21f2c93-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dcb51fc1-b1d9-4ca4-9ea4-32a8f21f2c93-maas-api-xks-ca\") pod \"dcb51fc1-b1d9-4ca4-9ea4-32a8f21f2c93\" (UID: \"dcb51fc1-b1d9-4ca4-9ea4-32a8f21f2c93\") " Oct 30 21:06:01.797728 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:06:01.797711 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dcb51fc1-b1d9-4ca4-9ea4-32a8f21f2c93-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dcb51fc1-b1d9-4ca4-9ea4-32a8f21f2c93-maas-api-openshift-ca\") pod \"dcb51fc1-b1d9-4ca4-9ea4-32a8f21f2c93\" (UID: \"dcb51fc1-b1d9-4ca4-9ea4-32a8f21f2c93\") " Oct 30 21:06:01.798051 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:06:01.798023 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dcb51fc1-b1d9-4ca4-9ea4-32a8f21f2c93-maas-api-openshift-ca" pod "dcb51fc1-b1d9-4ca4-9ea4-32a8f21f2c93" (UID: "dcb51fc1-b1d9-4ca4-9ea4-32a8f21f2c93"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:06:01.799534 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:06:01.799499 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dcb51fc1-b1d9-4ca4-9ea4-32a8f21f2c93-kube-api-access-22gjp" pod "dcb51fc1-b1d9-4ca4-9ea4-32a8f21f2c93" (UID: "dcb51fc1-b1d9-4ca4-9ea4-32a8f21f2c93"). InnerVolumeSpecName "kube-api-access-22gjp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:06:01.799624 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:06:01.799596 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dcb51fc1-b1d9-4ca4-9ea4-32a8f21f2c93-maas-api-xks-ca" pod "dcb51fc1-b1d9-4ca4-9ea4-32a8f21f2c93" (UID: "dcb51fc1-b1d9-4ca4-9ea4-32a8f21f2c93"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:06:01.899273 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:06:01.899243 4611 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dcb51fc1-b1d9-4ca4-9ea4-32a8f21f2c93-maas-api-openshift-ca\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:06:01.899273 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:06:01.899270 4611 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-22gjp\" (UniqueName: \"kubernetes.io/projected/dcb51fc1-b1d9-4ca4-9ea4-32a8f21f2c93-kube-api-access-22gjp\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:06:01.899465 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:06:01.899280 4611 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dcb51fc1-b1d9-4ca4-9ea4-32a8f21f2c93-maas-api-xks-ca\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:06:02.566989 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:06:02.566957 4611 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-nf5px" Oct 30 21:06:02.567153 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:06:02.566957 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-nf5px" event={"ID":"dcb51fc1-b1d9-4ca4-9ea4-32a8f21f2c93","Type":"ContainerDied","Data":"0f18215c4a386060b00d55a899697497d1f7e19470a5eeedd3a290bd97fea834"} Oct 30 21:06:02.567153 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:06:02.567083 4611 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0f18215c4a386060b00d55a899697497d1f7e19470a5eeedd3a290bd97fea834" Oct 30 21:06:03.916853 ip-10-0-133-192 kubenswrapper[4611]: E1030 21:06:03.916820 4611 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hwzc6" containerName="extractor" Oct 30 21:06:08.062614 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:06:08.062580 4611 scope.go:122] "RemoveContainer" containerID="78e0e5c6a863eb41854621bf79532a02aa17184d3e242af256f658a0b93aa5d1" Oct 30 21:06:16.776375 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:06:16.776337 4611 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-7t8l5"] Oct 30 21:06:16.776879 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:06:16.776759 4611 cpu_manager.go:414] "containerMap: removing container" podUID="dcb51fc1-b1d9-4ca4-9ea4-32a8f21f2c93" containerName="revoke-keys" Oct 30 21:06:16.776879 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:06:16.776771 4611 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dcb51fc1-b1d9-4ca4-9ea4-32a8f21f2c93" containerName="revoke-keys" Oct 30 21:06:16.776879 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:06:16.776831 4611 memory_manager.go:365] "RemoveStaleState removing state" podUID="dcb51fc1-b1d9-4ca4-9ea4-32a8f21f2c93" containerName="revoke-keys" Oct 30 21:06:16.781281 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:06:16.781264 4611 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-7t8l5" Oct 30 21:06:16.794272 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:06:16.794243 4611 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-7t8l5"] Oct 30 21:06:16.794447 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:06:16.794424 4611 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:06:16.794581 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:06:16.794446 4611 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:06:16.794581 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:06:16.794545 4611 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:06:16.794581 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:06:16.794568 4611 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7hf8r\"" Oct 30 21:06:16.909877 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:06:16.909842 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22f1a1fc-7bc9-42d6-9f36-ecbc3279cb06-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-7t8l5\" (UID: \"22f1a1fc-7bc9-42d6-9f36-ecbc3279cb06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-7t8l5" Oct 30 21:06:16.909877 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:06:16.909880 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22f1a1fc-7bc9-42d6-9f36-ecbc3279cb06-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-7t8l5\" (UID: \"22f1a1fc-7bc9-42d6-9f36-ecbc3279cb06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-7t8l5" Oct 30 21:06:16.910087 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:06:16.910023 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kh6zl\" (UniqueName: \"kubernetes.io/projected/22f1a1fc-7bc9-42d6-9f36-ecbc3279cb06-kube-api-access-kh6zl\") pod \"maas-api-revoke-sub-a4aef7d7-7t8l5\" (UID: \"22f1a1fc-7bc9-42d6-9f36-ecbc3279cb06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-7t8l5" Oct 30 21:06:17.011476 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:06:17.011447 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22f1a1fc-7bc9-42d6-9f36-ecbc3279cb06-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-7t8l5\" (UID: \"22f1a1fc-7bc9-42d6-9f36-ecbc3279cb06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-7t8l5" Oct 30 21:06:17.011665 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:06:17.011480 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22f1a1fc-7bc9-42d6-9f36-ecbc3279cb06-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-7t8l5\" (UID: \"22f1a1fc-7bc9-42d6-9f36-ecbc3279cb06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-7t8l5" Oct 30 21:06:17.011665 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:06:17.011633 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kh6zl\" (UniqueName: \"kubernetes.io/projected/22f1a1fc-7bc9-42d6-9f36-ecbc3279cb06-kube-api-access-kh6zl\") pod \"maas-api-revoke-sub-a4aef7d7-7t8l5\" (UID: \"22f1a1fc-7bc9-42d6-9f36-ecbc3279cb06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-7t8l5" Oct 30 21:06:17.012265 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:06:17.012238 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22f1a1fc-7bc9-42d6-9f36-ecbc3279cb06-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-7t8l5\" (UID: \"22f1a1fc-7bc9-42d6-9f36-ecbc3279cb06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-7t8l5" Oct 30 21:06:17.013793 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:06:17.013773 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22f1a1fc-7bc9-42d6-9f36-ecbc3279cb06-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-7t8l5\" (UID: \"22f1a1fc-7bc9-42d6-9f36-ecbc3279cb06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-7t8l5" Oct 30 21:06:17.027688 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:06:17.027631 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kh6zl\" (UniqueName: \"kubernetes.io/projected/22f1a1fc-7bc9-42d6-9f36-ecbc3279cb06-kube-api-access-kh6zl\") pod \"maas-api-revoke-sub-a4aef7d7-7t8l5\" (UID: \"22f1a1fc-7bc9-42d6-9f36-ecbc3279cb06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-7t8l5" Oct 30 21:06:17.095039 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:06:17.095012 4611 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-7t8l5" Oct 30 21:06:17.373362 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:06:17.373337 4611 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-7t8l5"] Oct 30 21:06:17.376193 ip-10-0-133-192 kubenswrapper[4611]: W1030 21:06:17.376165 4611 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22f1a1fc_7bc9_42d6_9f36_ecbc3279cb06.slice/crio-5b2832cc2e0cf3668013f332066518f23b916084e0e2b839d61dfa8a949239e0 WatchSource:0}: Error finding container 5b2832cc2e0cf3668013f332066518f23b916084e0e2b839d61dfa8a949239e0: Status 404 returned error can't find the container with id 5b2832cc2e0cf3668013f332066518f23b916084e0e2b839d61dfa8a949239e0 Oct 30 21:06:17.666055 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:06:17.666025 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-7t8l5" event={"ID":"22f1a1fc-7bc9-42d6-9f36-ecbc3279cb06","Type":"ContainerStarted","Data":"a1ed426187dc69dfc6a3d6d7eb2c6a9e9261674e9973d14f47134ca1ed0221a9"} Oct 30 21:06:17.666192 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:06:17.666074 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-7t8l5" event={"ID":"22f1a1fc-7bc9-42d6-9f36-ecbc3279cb06","Type":"ContainerStarted","Data":"5b2832cc2e0cf3668013f332066518f23b916084e0e2b839d61dfa8a949239e0"} Oct 30 21:06:17.683179 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:06:17.683132 4611 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-7t8l5" podStartSLOduration=1.6831186219999998 podStartE2EDuration="1.683118622s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:06:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:06:17.682136702 +0000 UTC m=+1632.216163317" watchObservedRunningTime="2026-10-30 21:06:17.683118622 +0000 UTC m=+1632.217145238" Oct 30 21:06:18.673785 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:06:18.673755 4611 generic.go:356] "Generic (PLEG): container finished" podID="22f1a1fc-7bc9-42d6-9f36-ecbc3279cb06" containerID="a1ed426187dc69dfc6a3d6d7eb2c6a9e9261674e9973d14f47134ca1ed0221a9" exitCode=0 Oct 30 21:06:18.674167 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:06:18.673828 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-7t8l5" event={"ID":"22f1a1fc-7bc9-42d6-9f36-ecbc3279cb06","Type":"ContainerDied","Data":"a1ed426187dc69dfc6a3d6d7eb2c6a9e9261674e9973d14f47134ca1ed0221a9"} Oct 30 21:06:19.808825 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:06:19.808800 4611 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-7t8l5" Oct 30 21:06:19.940629 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:06:19.940556 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/22f1a1fc-7bc9-42d6-9f36-ecbc3279cb06-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22f1a1fc-7bc9-42d6-9f36-ecbc3279cb06-maas-api-xks-ca\") pod \"22f1a1fc-7bc9-42d6-9f36-ecbc3279cb06\" (UID: \"22f1a1fc-7bc9-42d6-9f36-ecbc3279cb06\") " Oct 30 21:06:19.940806 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:06:19.940664 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/22f1a1fc-7bc9-42d6-9f36-ecbc3279cb06-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22f1a1fc-7bc9-42d6-9f36-ecbc3279cb06-maas-api-openshift-ca\") pod \"22f1a1fc-7bc9-42d6-9f36-ecbc3279cb06\" (UID: \"22f1a1fc-7bc9-42d6-9f36-ecbc3279cb06\") " Oct 30 21:06:19.940806 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:06:19.940700 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22f1a1fc-7bc9-42d6-9f36-ecbc3279cb06-kube-api-access-kh6zl\" (UniqueName: \"kubernetes.io/projected/22f1a1fc-7bc9-42d6-9f36-ecbc3279cb06-kube-api-access-kh6zl\") pod \"22f1a1fc-7bc9-42d6-9f36-ecbc3279cb06\" (UID: \"22f1a1fc-7bc9-42d6-9f36-ecbc3279cb06\") " Oct 30 21:06:19.941041 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:06:19.941017 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22f1a1fc-7bc9-42d6-9f36-ecbc3279cb06-maas-api-openshift-ca" pod "22f1a1fc-7bc9-42d6-9f36-ecbc3279cb06" (UID: "22f1a1fc-7bc9-42d6-9f36-ecbc3279cb06"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:06:19.942679 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:06:19.942641 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/22f1a1fc-7bc9-42d6-9f36-ecbc3279cb06-maas-api-xks-ca" pod "22f1a1fc-7bc9-42d6-9f36-ecbc3279cb06" (UID: "22f1a1fc-7bc9-42d6-9f36-ecbc3279cb06"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:06:19.942777 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:06:19.942688 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22f1a1fc-7bc9-42d6-9f36-ecbc3279cb06-kube-api-access-kh6zl" pod "22f1a1fc-7bc9-42d6-9f36-ecbc3279cb06" (UID: "22f1a1fc-7bc9-42d6-9f36-ecbc3279cb06"). InnerVolumeSpecName "kube-api-access-kh6zl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:06:20.041821 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:06:20.041800 4611 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22f1a1fc-7bc9-42d6-9f36-ecbc3279cb06-maas-api-openshift-ca\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:06:20.041821 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:06:20.041821 4611 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kh6zl\" (UniqueName: \"kubernetes.io/projected/22f1a1fc-7bc9-42d6-9f36-ecbc3279cb06-kube-api-access-kh6zl\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:06:20.041963 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:06:20.041831 4611 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22f1a1fc-7bc9-42d6-9f36-ecbc3279cb06-maas-api-xks-ca\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:06:20.684374 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:06:20.684340 4611 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-7t8l5" Oct 30 21:06:20.684571 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:06:20.684376 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-7t8l5" event={"ID":"22f1a1fc-7bc9-42d6-9f36-ecbc3279cb06","Type":"ContainerDied","Data":"5b2832cc2e0cf3668013f332066518f23b916084e0e2b839d61dfa8a949239e0"} Oct 30 21:06:20.684571 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:06:20.684430 4611 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5b2832cc2e0cf3668013f332066518f23b916084e0e2b839d61dfa8a949239e0" Oct 30 21:06:33.121531 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:06:33.120600 4611 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-8tvvl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 21:06:33.121531 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:06:33.120680 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8tvvl" podUID="46ae03da-7a80-4f78-8abd-391ebc697f2f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:06:33.194672 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:06:33.194622 4611 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-w44qs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 21:06:33.194977 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:06:33.194945 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-w44qs" podUID="6d6fd4b2-ad0a-4ce3-b5b3-966de34bd8b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:06:33.920387 ip-10-0-133-192 kubenswrapper[4611]: E1030 21:06:33.920339 4611 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hwzc6" containerName="extractor" Oct 30 21:06:35.605829 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:06:35.605788 4611 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-b26lj"] Oct 30 21:06:35.606457 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:06:35.606433 4611 cpu_manager.go:414] "containerMap: removing container" podUID="22f1a1fc-7bc9-42d6-9f36-ecbc3279cb06" containerName="revoke-keys" Oct 30 21:06:35.606501 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:06:35.606462 4611 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="22f1a1fc-7bc9-42d6-9f36-ecbc3279cb06" containerName="revoke-keys" Oct 30 21:06:35.606653 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:06:35.606634 4611 memory_manager.go:365] "RemoveStaleState removing state" podUID="22f1a1fc-7bc9-42d6-9f36-ecbc3279cb06" containerName="revoke-keys" Oct 30 21:06:35.610573 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:06:35.610546 4611 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-b26lj" Oct 30 21:06:35.615752 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:06:35.615725 4611 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:06:35.616609 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:06:35.615745 4611 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:06:35.616792 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:06:35.616348 4611 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7hf8r\"" Oct 30 21:06:35.616960 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:06:35.616375 4611 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:06:35.620564 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:06:35.618883 4611 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-b26lj"] Oct 30 21:06:35.678391 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:06:35.678349 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eaa6301f-743d-46d5-b4b7-84d36896eec7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-b26lj\" (UID: \"eaa6301f-743d-46d5-b4b7-84d36896eec7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-b26lj" Oct 30 21:06:35.678557 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:06:35.678478 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eaa6301f-743d-46d5-b4b7-84d36896eec7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-b26lj\" (UID: \"eaa6301f-743d-46d5-b4b7-84d36896eec7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-b26lj" Oct 30 21:06:35.678609 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:06:35.678562 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l5c2d\" (UniqueName: \"kubernetes.io/projected/eaa6301f-743d-46d5-b4b7-84d36896eec7-kube-api-access-l5c2d\") pod \"maas-api-revoke-sub-79805ca9-b26lj\" (UID: \"eaa6301f-743d-46d5-b4b7-84d36896eec7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-b26lj" Oct 30 21:06:35.779916 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:06:35.779885 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eaa6301f-743d-46d5-b4b7-84d36896eec7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-b26lj\" (UID: \"eaa6301f-743d-46d5-b4b7-84d36896eec7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-b26lj" Oct 30 21:06:35.780065 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:06:35.779946 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eaa6301f-743d-46d5-b4b7-84d36896eec7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-b26lj\" (UID: \"eaa6301f-743d-46d5-b4b7-84d36896eec7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-b26lj" Oct 30 21:06:35.780065 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:06:35.780000 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l5c2d\" (UniqueName: \"kubernetes.io/projected/eaa6301f-743d-46d5-b4b7-84d36896eec7-kube-api-access-l5c2d\") pod \"maas-api-revoke-sub-79805ca9-b26lj\" (UID: \"eaa6301f-743d-46d5-b4b7-84d36896eec7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-b26lj" Oct 30 21:06:35.780663 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:06:35.780610 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eaa6301f-743d-46d5-b4b7-84d36896eec7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-b26lj\" (UID: \"eaa6301f-743d-46d5-b4b7-84d36896eec7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-b26lj" Oct 30 21:06:35.782240 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:06:35.782219 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eaa6301f-743d-46d5-b4b7-84d36896eec7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-b26lj\" (UID: \"eaa6301f-743d-46d5-b4b7-84d36896eec7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-b26lj" Oct 30 21:06:35.791731 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:06:35.791708 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l5c2d\" (UniqueName: \"kubernetes.io/projected/eaa6301f-743d-46d5-b4b7-84d36896eec7-kube-api-access-l5c2d\") pod \"maas-api-revoke-sub-79805ca9-b26lj\" (UID: \"eaa6301f-743d-46d5-b4b7-84d36896eec7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-b26lj" Oct 30 21:06:35.933314 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:06:35.933258 4611 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-b26lj" Oct 30 21:06:36.209341 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:06:36.209317 4611 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-b26lj"] Oct 30 21:06:36.211666 ip-10-0-133-192 kubenswrapper[4611]: W1030 21:06:36.211633 4611 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeaa6301f_743d_46d5_b4b7_84d36896eec7.slice/crio-7e9e5ebfcf158647c2ed89218d467116c2efa102e9ac52b31fde9c9f997a6a7b WatchSource:0}: Error finding container 7e9e5ebfcf158647c2ed89218d467116c2efa102e9ac52b31fde9c9f997a6a7b: Status 404 returned error can't find the container with id 7e9e5ebfcf158647c2ed89218d467116c2efa102e9ac52b31fde9c9f997a6a7b Oct 30 21:06:36.779306 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:06:36.779227 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-b26lj" event={"ID":"eaa6301f-743d-46d5-b4b7-84d36896eec7","Type":"ContainerStarted","Data":"ad481ae2fdcccb97669ce22f9621029d69c5219d3213a54487117958b2d140b7"} Oct 30 21:06:36.779306 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:06:36.779275 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-b26lj" event={"ID":"eaa6301f-743d-46d5-b4b7-84d36896eec7","Type":"ContainerStarted","Data":"7e9e5ebfcf158647c2ed89218d467116c2efa102e9ac52b31fde9c9f997a6a7b"} Oct 30 21:06:36.798528 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:06:36.798462 4611 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-b26lj" podStartSLOduration=1.798449326 podStartE2EDuration="1.798449326s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:06:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:06:36.796171801 +0000 UTC m=+1651.330198417" watchObservedRunningTime="2026-10-30 21:06:36.798449326 +0000 UTC m=+1651.332475942" Oct 30 21:06:37.786026 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:06:37.785995 4611 generic.go:356] "Generic (PLEG): container finished" podID="eaa6301f-743d-46d5-b4b7-84d36896eec7" containerID="ad481ae2fdcccb97669ce22f9621029d69c5219d3213a54487117958b2d140b7" exitCode=0 Oct 30 21:06:37.786424 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:06:37.786083 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-b26lj" event={"ID":"eaa6301f-743d-46d5-b4b7-84d36896eec7","Type":"ContainerDied","Data":"ad481ae2fdcccb97669ce22f9621029d69c5219d3213a54487117958b2d140b7"} Oct 30 21:06:38.927066 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:06:38.927042 4611 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-b26lj" Oct 30 21:06:39.010385 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:06:39.010341 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eaa6301f-743d-46d5-b4b7-84d36896eec7-kube-api-access-l5c2d\" (UniqueName: \"kubernetes.io/projected/eaa6301f-743d-46d5-b4b7-84d36896eec7-kube-api-access-l5c2d\") pod \"eaa6301f-743d-46d5-b4b7-84d36896eec7\" (UID: \"eaa6301f-743d-46d5-b4b7-84d36896eec7\") " Oct 30 21:06:39.010562 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:06:39.010401 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eaa6301f-743d-46d5-b4b7-84d36896eec7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eaa6301f-743d-46d5-b4b7-84d36896eec7-maas-api-xks-ca\") pod \"eaa6301f-743d-46d5-b4b7-84d36896eec7\" (UID: \"eaa6301f-743d-46d5-b4b7-84d36896eec7\") " Oct 30 21:06:39.010562 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:06:39.010443 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eaa6301f-743d-46d5-b4b7-84d36896eec7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eaa6301f-743d-46d5-b4b7-84d36896eec7-maas-api-openshift-ca\") pod \"eaa6301f-743d-46d5-b4b7-84d36896eec7\" (UID: \"eaa6301f-743d-46d5-b4b7-84d36896eec7\") " Oct 30 21:06:39.010863 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:06:39.010835 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eaa6301f-743d-46d5-b4b7-84d36896eec7-maas-api-openshift-ca" pod "eaa6301f-743d-46d5-b4b7-84d36896eec7" (UID: "eaa6301f-743d-46d5-b4b7-84d36896eec7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:06:39.012419 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:06:39.012393 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eaa6301f-743d-46d5-b4b7-84d36896eec7-kube-api-access-l5c2d" pod "eaa6301f-743d-46d5-b4b7-84d36896eec7" (UID: "eaa6301f-743d-46d5-b4b7-84d36896eec7"). InnerVolumeSpecName "kube-api-access-l5c2d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:06:39.012495 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:06:39.012441 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eaa6301f-743d-46d5-b4b7-84d36896eec7-maas-api-xks-ca" pod "eaa6301f-743d-46d5-b4b7-84d36896eec7" (UID: "eaa6301f-743d-46d5-b4b7-84d36896eec7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:06:39.111444 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:06:39.111412 4611 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eaa6301f-743d-46d5-b4b7-84d36896eec7-maas-api-openshift-ca\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:06:39.111444 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:06:39.111445 4611 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l5c2d\" (UniqueName: \"kubernetes.io/projected/eaa6301f-743d-46d5-b4b7-84d36896eec7-kube-api-access-l5c2d\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:06:39.111603 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:06:39.111456 4611 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eaa6301f-743d-46d5-b4b7-84d36896eec7-maas-api-xks-ca\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:06:39.797474 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:06:39.797438 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-b26lj" event={"ID":"eaa6301f-743d-46d5-b4b7-84d36896eec7","Type":"ContainerDied","Data":"7e9e5ebfcf158647c2ed89218d467116c2efa102e9ac52b31fde9c9f997a6a7b"} Oct 30 21:06:39.797474 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:06:39.797460 4611 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-b26lj" Oct 30 21:06:39.797677 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:06:39.797491 4611 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7e9e5ebfcf158647c2ed89218d467116c2efa102e9ac52b31fde9c9f997a6a7b" Oct 30 21:06:49.208092 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:06:49.207883 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:06:49.692120 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:06:49.692081 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:06:49.879587 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:06:49.878664 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:06:58.584996 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:06:58.584961 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:07:03.445431 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:07:03.445395 4611 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-6nhlh"] Oct 30 21:07:03.446024 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:07:03.446001 4611 cpu_manager.go:414] "containerMap: removing container" podUID="eaa6301f-743d-46d5-b4b7-84d36896eec7" containerName="revoke-keys" Oct 30 21:07:03.446109 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:07:03.446027 4611 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eaa6301f-743d-46d5-b4b7-84d36896eec7" containerName="revoke-keys" Oct 30 21:07:03.446165 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:07:03.446135 4611 memory_manager.go:365] "RemoveStaleState removing state" podUID="eaa6301f-743d-46d5-b4b7-84d36896eec7" containerName="revoke-keys" Oct 30 21:07:03.449264 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:07:03.449240 4611 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-6nhlh" Oct 30 21:07:03.455435 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:07:03.455408 4611 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:07:03.457451 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:07:03.457426 4611 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:07:03.457580 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:07:03.457535 4611 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7hf8r\"" Oct 30 21:07:03.457641 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:07:03.457583 4611 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:07:03.459035 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:07:03.459012 4611 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-6nhlh"] Oct 30 21:07:03.475127 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:07:03.475105 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43120b8c-224c-474e-858f-12ab24ef455a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-6nhlh\" (UID: \"43120b8c-224c-474e-858f-12ab24ef455a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-6nhlh" Oct 30 21:07:03.475231 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:07:03.475158 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8rs25\" (UniqueName: \"kubernetes.io/projected/43120b8c-224c-474e-858f-12ab24ef455a-kube-api-access-8rs25\") pod \"maas-api-revoke-sub-0b7e8edb-6nhlh\" (UID: \"43120b8c-224c-474e-858f-12ab24ef455a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-6nhlh" Oct 30 21:07:03.475338 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:07:03.475314 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43120b8c-224c-474e-858f-12ab24ef455a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-6nhlh\" (UID: \"43120b8c-224c-474e-858f-12ab24ef455a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-6nhlh" Oct 30 21:07:03.576715 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:07:03.576686 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43120b8c-224c-474e-858f-12ab24ef455a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-6nhlh\" (UID: \"43120b8c-224c-474e-858f-12ab24ef455a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-6nhlh" Oct 30 21:07:03.576865 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:07:03.576737 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43120b8c-224c-474e-858f-12ab24ef455a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-6nhlh\" (UID: \"43120b8c-224c-474e-858f-12ab24ef455a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-6nhlh" Oct 30 21:07:03.576865 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:07:03.576784 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8rs25\" (UniqueName: \"kubernetes.io/projected/43120b8c-224c-474e-858f-12ab24ef455a-kube-api-access-8rs25\") pod \"maas-api-revoke-sub-0b7e8edb-6nhlh\" (UID: \"43120b8c-224c-474e-858f-12ab24ef455a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-6nhlh" Oct 30 21:07:03.577332 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:07:03.577310 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43120b8c-224c-474e-858f-12ab24ef455a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-6nhlh\" (UID: \"43120b8c-224c-474e-858f-12ab24ef455a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-6nhlh" Oct 30 21:07:03.579042 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:07:03.579017 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43120b8c-224c-474e-858f-12ab24ef455a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-6nhlh\" (UID: \"43120b8c-224c-474e-858f-12ab24ef455a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-6nhlh" Oct 30 21:07:03.588190 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:07:03.588170 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8rs25\" (UniqueName: \"kubernetes.io/projected/43120b8c-224c-474e-858f-12ab24ef455a-kube-api-access-8rs25\") pod \"maas-api-revoke-sub-0b7e8edb-6nhlh\" (UID: \"43120b8c-224c-474e-858f-12ab24ef455a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-6nhlh" Oct 30 21:07:03.764155 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:07:03.764086 4611 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-6nhlh" Oct 30 21:07:03.917705 ip-10-0-133-192 kubenswrapper[4611]: E1030 21:07:03.917669 4611 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hwzc6" containerName="extractor" Oct 30 21:07:04.039376 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:07:04.039353 4611 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-6nhlh"] Oct 30 21:07:04.041150 ip-10-0-133-192 kubenswrapper[4611]: W1030 21:07:04.041117 4611 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43120b8c_224c_474e_858f_12ab24ef455a.slice/crio-9df7229e92410c02fbbea6ceee6015372b389ac37be953c49780274258fc1fd7 WatchSource:0}: Error finding container 9df7229e92410c02fbbea6ceee6015372b389ac37be953c49780274258fc1fd7: Status 404 returned error can't find the container with id 9df7229e92410c02fbbea6ceee6015372b389ac37be953c49780274258fc1fd7 Oct 30 21:07:04.999101 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:07:04.999069 4611 generic.go:356] "Generic (PLEG): container finished" podID="43120b8c-224c-474e-858f-12ab24ef455a" containerID="3bf57584ca0cc4264d8e372e1b09ba939d1d43dc56d0117d081cc10f1b6cd69a" exitCode=0 Oct 30 21:07:04.999391 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:07:04.999117 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-6nhlh" event={"ID":"43120b8c-224c-474e-858f-12ab24ef455a","Type":"ContainerDied","Data":"3bf57584ca0cc4264d8e372e1b09ba939d1d43dc56d0117d081cc10f1b6cd69a"} Oct 30 21:07:04.999391 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:07:04.999169 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-6nhlh" event={"ID":"43120b8c-224c-474e-858f-12ab24ef455a","Type":"ContainerStarted","Data":"9df7229e92410c02fbbea6ceee6015372b389ac37be953c49780274258fc1fd7"} Oct 30 21:07:06.137630 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:07:06.137607 4611 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-6nhlh" Oct 30 21:07:06.203902 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:07:06.203872 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/43120b8c-224c-474e-858f-12ab24ef455a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43120b8c-224c-474e-858f-12ab24ef455a-maas-api-xks-ca\") pod \"43120b8c-224c-474e-858f-12ab24ef455a\" (UID: \"43120b8c-224c-474e-858f-12ab24ef455a\") " Oct 30 21:07:06.204045 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:07:06.203938 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43120b8c-224c-474e-858f-12ab24ef455a-kube-api-access-8rs25\" (UniqueName: \"kubernetes.io/projected/43120b8c-224c-474e-858f-12ab24ef455a-kube-api-access-8rs25\") pod \"43120b8c-224c-474e-858f-12ab24ef455a\" (UID: \"43120b8c-224c-474e-858f-12ab24ef455a\") " Oct 30 21:07:06.204377 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:07:06.204077 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/43120b8c-224c-474e-858f-12ab24ef455a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43120b8c-224c-474e-858f-12ab24ef455a-maas-api-openshift-ca\") pod \"43120b8c-224c-474e-858f-12ab24ef455a\" (UID: \"43120b8c-224c-474e-858f-12ab24ef455a\") " Oct 30 21:07:06.204467 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:07:06.204440 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43120b8c-224c-474e-858f-12ab24ef455a-maas-api-openshift-ca" pod "43120b8c-224c-474e-858f-12ab24ef455a" (UID: "43120b8c-224c-474e-858f-12ab24ef455a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:07:06.205994 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:07:06.205968 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/43120b8c-224c-474e-858f-12ab24ef455a-maas-api-xks-ca" pod "43120b8c-224c-474e-858f-12ab24ef455a" (UID: "43120b8c-224c-474e-858f-12ab24ef455a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:07:06.206108 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:07:06.206029 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43120b8c-224c-474e-858f-12ab24ef455a-kube-api-access-8rs25" pod "43120b8c-224c-474e-858f-12ab24ef455a" (UID: "43120b8c-224c-474e-858f-12ab24ef455a"). InnerVolumeSpecName "kube-api-access-8rs25". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:07:06.305001 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:07:06.304926 4611 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43120b8c-224c-474e-858f-12ab24ef455a-maas-api-openshift-ca\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:07:06.305001 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:07:06.304961 4611 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43120b8c-224c-474e-858f-12ab24ef455a-maas-api-xks-ca\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:07:06.305001 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:07:06.304975 4611 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8rs25\" (UniqueName: \"kubernetes.io/projected/43120b8c-224c-474e-858f-12ab24ef455a-kube-api-access-8rs25\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:07:06.725973 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:07:06.725932 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:07:06.888404 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:07:06.888372 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:07:07.011307 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:07:07.011231 4611 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-6nhlh" Oct 30 21:07:07.011307 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:07:07.011264 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-6nhlh" event={"ID":"43120b8c-224c-474e-858f-12ab24ef455a","Type":"ContainerDied","Data":"9df7229e92410c02fbbea6ceee6015372b389ac37be953c49780274258fc1fd7"} Oct 30 21:07:07.011484 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:07:07.011308 4611 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9df7229e92410c02fbbea6ceee6015372b389ac37be953c49780274258fc1fd7" Oct 30 21:07:08.082588 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:07:08.082558 4611 scope.go:122] "RemoveContainer" containerID="5d2677dd32d1d7fe29ac4c776c79a23c2f1bc4c0fa976157275f365d864a23c9" Oct 30 21:07:10.636714 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:07:10.636674 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:07:14.120459 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:07:14.120414 4611 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-mmgwd"] Oct 30 21:07:14.121018 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:07:14.120997 4611 cpu_manager.go:414] "containerMap: removing container" podUID="43120b8c-224c-474e-858f-12ab24ef455a" containerName="revoke-keys" Oct 30 21:07:14.121107 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:07:14.121020 4611 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43120b8c-224c-474e-858f-12ab24ef455a" containerName="revoke-keys" Oct 30 21:07:14.121185 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:07:14.121172 4611 memory_manager.go:365] "RemoveStaleState removing state" podUID="43120b8c-224c-474e-858f-12ab24ef455a" containerName="revoke-keys" Oct 30 21:07:14.125464 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:07:14.125442 4611 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-mmgwd" Oct 30 21:07:14.129658 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:07:14.129632 4611 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:07:14.134284 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:07:14.129673 4611 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:07:14.134410 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:07:14.129843 4611 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7hf8r\"" Oct 30 21:07:14.134410 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:07:14.129938 4611 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:07:14.134410 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:07:14.134214 4611 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-mmgwd"] Oct 30 21:07:14.176068 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:07:14.176034 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3ca132a-fffe-4547-895b-d12c487d51f4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-mmgwd\" (UID: \"a3ca132a-fffe-4547-895b-d12c487d51f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-mmgwd" Oct 30 21:07:14.176186 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:07:14.176094 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3ca132a-fffe-4547-895b-d12c487d51f4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-mmgwd\" (UID: \"a3ca132a-fffe-4547-895b-d12c487d51f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-mmgwd" Oct 30 21:07:14.176237 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:07:14.176206 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h5vm8\" (UniqueName: \"kubernetes.io/projected/a3ca132a-fffe-4547-895b-d12c487d51f4-kube-api-access-h5vm8\") pod \"maas-api-revoke-sub-3d4156f1-mmgwd\" (UID: \"a3ca132a-fffe-4547-895b-d12c487d51f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-mmgwd" Oct 30 21:07:14.277160 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:07:14.277134 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h5vm8\" (UniqueName: \"kubernetes.io/projected/a3ca132a-fffe-4547-895b-d12c487d51f4-kube-api-access-h5vm8\") pod \"maas-api-revoke-sub-3d4156f1-mmgwd\" (UID: \"a3ca132a-fffe-4547-895b-d12c487d51f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-mmgwd" Oct 30 21:07:14.277297 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:07:14.277202 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3ca132a-fffe-4547-895b-d12c487d51f4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-mmgwd\" (UID: \"a3ca132a-fffe-4547-895b-d12c487d51f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-mmgwd" Oct 30 21:07:14.277297 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:07:14.277225 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3ca132a-fffe-4547-895b-d12c487d51f4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-mmgwd\" (UID: \"a3ca132a-fffe-4547-895b-d12c487d51f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-mmgwd" Oct 30 21:07:14.277921 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:07:14.277900 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3ca132a-fffe-4547-895b-d12c487d51f4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-mmgwd\" (UID: \"a3ca132a-fffe-4547-895b-d12c487d51f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-mmgwd" Oct 30 21:07:14.279360 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:07:14.279340 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3ca132a-fffe-4547-895b-d12c487d51f4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-mmgwd\" (UID: \"a3ca132a-fffe-4547-895b-d12c487d51f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-mmgwd" Oct 30 21:07:14.288722 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:07:14.288701 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h5vm8\" (UniqueName: \"kubernetes.io/projected/a3ca132a-fffe-4547-895b-d12c487d51f4-kube-api-access-h5vm8\") pod \"maas-api-revoke-sub-3d4156f1-mmgwd\" (UID: \"a3ca132a-fffe-4547-895b-d12c487d51f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-mmgwd" Oct 30 21:07:14.440839 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:07:14.440781 4611 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-mmgwd" Oct 30 21:07:14.720823 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:07:14.718869 4611 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-mmgwd"] Oct 30 21:07:14.724195 ip-10-0-133-192 kubenswrapper[4611]: W1030 21:07:14.724168 4611 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3ca132a_fffe_4547_895b_d12c487d51f4.slice/crio-52bcc01ae5214ecfafdb2d23532fcba73f8d5d45f178acddb5c74887b1979405 WatchSource:0}: Error finding container 52bcc01ae5214ecfafdb2d23532fcba73f8d5d45f178acddb5c74887b1979405: Status 404 returned error can't find the container with id 52bcc01ae5214ecfafdb2d23532fcba73f8d5d45f178acddb5c74887b1979405 Oct 30 21:07:15.053687 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:07:15.053656 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-mmgwd" event={"ID":"a3ca132a-fffe-4547-895b-d12c487d51f4","Type":"ContainerStarted","Data":"c787c074b85f2ce8b6b686e0cc2bd9959d1c1008295ef063f64941f62fced449"} Oct 30 21:07:15.053817 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:07:15.053720 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-mmgwd" event={"ID":"a3ca132a-fffe-4547-895b-d12c487d51f4","Type":"ContainerStarted","Data":"52bcc01ae5214ecfafdb2d23532fcba73f8d5d45f178acddb5c74887b1979405"} Oct 30 21:07:15.072049 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:07:15.072002 4611 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-mmgwd" podStartSLOduration=1.071988017 podStartE2EDuration="1.071988017s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:07:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:07:15.06953418 +0000 UTC m=+1689.603560793" watchObservedRunningTime="2026-10-30 21:07:15.071988017 +0000 UTC m=+1689.606014627" Oct 30 21:07:16.066112 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:07:16.066070 4611 generic.go:356] "Generic (PLEG): container finished" podID="a3ca132a-fffe-4547-895b-d12c487d51f4" containerID="c787c074b85f2ce8b6b686e0cc2bd9959d1c1008295ef063f64941f62fced449" exitCode=0 Oct 30 21:07:16.066546 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:07:16.066150 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-mmgwd" event={"ID":"a3ca132a-fffe-4547-895b-d12c487d51f4","Type":"ContainerDied","Data":"c787c074b85f2ce8b6b686e0cc2bd9959d1c1008295ef063f64941f62fced449"} Oct 30 21:07:17.084311 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:07:17.084265 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:07:17.204875 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:07:17.204852 4611 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-mmgwd" Oct 30 21:07:17.282897 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:07:17.282859 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:07:17.302849 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:07:17.302819 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3ca132a-fffe-4547-895b-d12c487d51f4-kube-api-access-h5vm8\" (UniqueName: \"kubernetes.io/projected/a3ca132a-fffe-4547-895b-d12c487d51f4-kube-api-access-h5vm8\") pod \"a3ca132a-fffe-4547-895b-d12c487d51f4\" (UID: \"a3ca132a-fffe-4547-895b-d12c487d51f4\") " Oct 30 21:07:17.302994 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:07:17.302859 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a3ca132a-fffe-4547-895b-d12c487d51f4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3ca132a-fffe-4547-895b-d12c487d51f4-maas-api-xks-ca\") pod \"a3ca132a-fffe-4547-895b-d12c487d51f4\" (UID: \"a3ca132a-fffe-4547-895b-d12c487d51f4\") " Oct 30 21:07:17.302994 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:07:17.302919 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a3ca132a-fffe-4547-895b-d12c487d51f4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3ca132a-fffe-4547-895b-d12c487d51f4-maas-api-openshift-ca\") pod \"a3ca132a-fffe-4547-895b-d12c487d51f4\" (UID: \"a3ca132a-fffe-4547-895b-d12c487d51f4\") " Oct 30 21:07:17.303331 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:07:17.303308 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3ca132a-fffe-4547-895b-d12c487d51f4-maas-api-openshift-ca" pod "a3ca132a-fffe-4547-895b-d12c487d51f4" (UID: "a3ca132a-fffe-4547-895b-d12c487d51f4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:07:17.305218 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:07:17.305191 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a3ca132a-fffe-4547-895b-d12c487d51f4-maas-api-xks-ca" pod "a3ca132a-fffe-4547-895b-d12c487d51f4" (UID: "a3ca132a-fffe-4547-895b-d12c487d51f4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:07:17.305336 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:07:17.305289 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3ca132a-fffe-4547-895b-d12c487d51f4-kube-api-access-h5vm8" pod "a3ca132a-fffe-4547-895b-d12c487d51f4" (UID: "a3ca132a-fffe-4547-895b-d12c487d51f4"). InnerVolumeSpecName "kube-api-access-h5vm8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:07:17.404707 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:07:17.404681 4611 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3ca132a-fffe-4547-895b-d12c487d51f4-maas-api-openshift-ca\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:07:17.404707 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:07:17.404706 4611 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h5vm8\" (UniqueName: \"kubernetes.io/projected/a3ca132a-fffe-4547-895b-d12c487d51f4-kube-api-access-h5vm8\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:07:17.404861 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:07:17.404716 4611 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3ca132a-fffe-4547-895b-d12c487d51f4-maas-api-xks-ca\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:07:18.077202 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:07:18.077171 4611 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-mmgwd" Oct 30 21:07:18.077349 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:07:18.077204 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-mmgwd" event={"ID":"a3ca132a-fffe-4547-895b-d12c487d51f4","Type":"ContainerDied","Data":"52bcc01ae5214ecfafdb2d23532fcba73f8d5d45f178acddb5c74887b1979405"} Oct 30 21:07:18.077349 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:07:18.077254 4611 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="52bcc01ae5214ecfafdb2d23532fcba73f8d5d45f178acddb5c74887b1979405" Oct 30 21:07:21.587592 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:07:21.587556 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:07:22.537323 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:07:22.537290 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:07:32.029444 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:07:32.029409 4611 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2dd9250-9jz97"] Oct 30 21:07:32.032745 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:07:32.032717 4611 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2dd9250-9jz97"] Oct 30 21:07:33.604457 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:07:33.604427 4611 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e6311907-bd1a-4516-8cfd-78b5b4d5be46" path="/var/lib/kubelet/pods/e6311907-bd1a-4516-8cfd-78b5b4d5be46/volumes" Oct 30 21:07:33.916693 ip-10-0-133-192 kubenswrapper[4611]: E1030 21:07:33.916607 4611 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hwzc6" containerName="extractor" Oct 30 21:07:35.422617 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:07:35.422578 4611 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-mmgwd"] Oct 30 21:07:35.437579 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:07:35.437547 4611 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-mmgwd"] Oct 30 21:07:35.448180 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:07:35.448141 4611 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ncgpg"] Oct 30 21:07:35.448734 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:07:35.448715 4611 cpu_manager.go:414] "containerMap: removing container" podUID="a3ca132a-fffe-4547-895b-d12c487d51f4" containerName="revoke-keys" Oct 30 21:07:35.448786 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:07:35.448738 4611 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3ca132a-fffe-4547-895b-d12c487d51f4" containerName="revoke-keys" Oct 30 21:07:35.448894 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:07:35.448883 4611 memory_manager.go:365] "RemoveStaleState removing state" podUID="a3ca132a-fffe-4547-895b-d12c487d51f4" containerName="revoke-keys" Oct 30 21:07:35.452610 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:07:35.452590 4611 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ncgpg" Oct 30 21:07:35.456090 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:07:35.456071 4611 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:07:35.456181 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:07:35.456125 4611 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7hf8r\"" Oct 30 21:07:35.456181 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:07:35.456143 4611 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:07:35.459085 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:07:35.459065 4611 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:07:35.462871 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:07:35.462845 4611 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ncgpg"] Oct 30 21:07:35.515001 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:07:35.514971 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1d95e20-e7c3-49ee-a348-9507d2ce858f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-ncgpg\" (UID: \"f1d95e20-e7c3-49ee-a348-9507d2ce858f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ncgpg" Oct 30 21:07:35.515128 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:07:35.515006 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nkkgc\" (UniqueName: \"kubernetes.io/projected/f1d95e20-e7c3-49ee-a348-9507d2ce858f-kube-api-access-nkkgc\") pod \"maas-api-revoke-sub-3d4156f1-ncgpg\" (UID: \"f1d95e20-e7c3-49ee-a348-9507d2ce858f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ncgpg" Oct 30 21:07:35.515128 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:07:35.515113 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1d95e20-e7c3-49ee-a348-9507d2ce858f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-ncgpg\" (UID: \"f1d95e20-e7c3-49ee-a348-9507d2ce858f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ncgpg" Oct 30 21:07:35.610735 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:07:35.610704 4611 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a3ca132a-fffe-4547-895b-d12c487d51f4" path="/var/lib/kubelet/pods/a3ca132a-fffe-4547-895b-d12c487d51f4/volumes" Oct 30 21:07:35.615946 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:07:35.615925 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1d95e20-e7c3-49ee-a348-9507d2ce858f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-ncgpg\" (UID: \"f1d95e20-e7c3-49ee-a348-9507d2ce858f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ncgpg" Oct 30 21:07:35.616039 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:07:35.616021 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1d95e20-e7c3-49ee-a348-9507d2ce858f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-ncgpg\" (UID: \"f1d95e20-e7c3-49ee-a348-9507d2ce858f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ncgpg" Oct 30 21:07:35.616094 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:07:35.616059 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nkkgc\" (UniqueName: \"kubernetes.io/projected/f1d95e20-e7c3-49ee-a348-9507d2ce858f-kube-api-access-nkkgc\") pod \"maas-api-revoke-sub-3d4156f1-ncgpg\" (UID: \"f1d95e20-e7c3-49ee-a348-9507d2ce858f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ncgpg" Oct 30 21:07:35.616689 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:07:35.616665 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1d95e20-e7c3-49ee-a348-9507d2ce858f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-ncgpg\" (UID: \"f1d95e20-e7c3-49ee-a348-9507d2ce858f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ncgpg" Oct 30 21:07:35.618324 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:07:35.618300 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1d95e20-e7c3-49ee-a348-9507d2ce858f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-ncgpg\" (UID: \"f1d95e20-e7c3-49ee-a348-9507d2ce858f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ncgpg" Oct 30 21:07:35.627937 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:07:35.627918 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nkkgc\" (UniqueName: \"kubernetes.io/projected/f1d95e20-e7c3-49ee-a348-9507d2ce858f-kube-api-access-nkkgc\") pod \"maas-api-revoke-sub-3d4156f1-ncgpg\" (UID: \"f1d95e20-e7c3-49ee-a348-9507d2ce858f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ncgpg" Oct 30 21:07:35.769050 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:07:35.768993 4611 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ncgpg" Oct 30 21:07:36.047536 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:07:36.047494 4611 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ncgpg"] Oct 30 21:07:36.049577 ip-10-0-133-192 kubenswrapper[4611]: W1030 21:07:36.049548 4611 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1d95e20_e7c3_49ee_a348_9507d2ce858f.slice/crio-05674c536e86a5011cd44d72276d82f40d73b907df43ac588c248a6befe5016c WatchSource:0}: Error finding container 05674c536e86a5011cd44d72276d82f40d73b907df43ac588c248a6befe5016c: Status 404 returned error can't find the container with id 05674c536e86a5011cd44d72276d82f40d73b907df43ac588c248a6befe5016c Oct 30 21:07:36.218394 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:07:36.218360 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:07:36.263355 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:07:36.263327 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ncgpg" event={"ID":"f1d95e20-e7c3-49ee-a348-9507d2ce858f","Type":"ContainerStarted","Data":"05674c536e86a5011cd44d72276d82f40d73b907df43ac588c248a6befe5016c"} Oct 30 21:07:37.270979 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:07:37.270893 4611 generic.go:356] "Generic (PLEG): container finished" podID="f1d95e20-e7c3-49ee-a348-9507d2ce858f" containerID="0603157d4809fe0dd66ded80eb78bf01c70638061cd19f7392a67c9e7dcd86b0" exitCode=0 Oct 30 21:07:37.270979 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:07:37.270945 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ncgpg" event={"ID":"f1d95e20-e7c3-49ee-a348-9507d2ce858f","Type":"ContainerDied","Data":"0603157d4809fe0dd66ded80eb78bf01c70638061cd19f7392a67c9e7dcd86b0"} Oct 30 21:07:38.401422 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:07:38.401395 4611 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ncgpg" Oct 30 21:07:38.545835 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:07:38.545765 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f1d95e20-e7c3-49ee-a348-9507d2ce858f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1d95e20-e7c3-49ee-a348-9507d2ce858f-maas-api-xks-ca\") pod \"f1d95e20-e7c3-49ee-a348-9507d2ce858f\" (UID: \"f1d95e20-e7c3-49ee-a348-9507d2ce858f\") " Oct 30 21:07:38.545835 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:07:38.545818 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f1d95e20-e7c3-49ee-a348-9507d2ce858f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1d95e20-e7c3-49ee-a348-9507d2ce858f-maas-api-openshift-ca\") pod \"f1d95e20-e7c3-49ee-a348-9507d2ce858f\" (UID: \"f1d95e20-e7c3-49ee-a348-9507d2ce858f\") " Oct 30 21:07:38.546056 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:07:38.545853 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1d95e20-e7c3-49ee-a348-9507d2ce858f-kube-api-access-nkkgc\" (UniqueName: \"kubernetes.io/projected/f1d95e20-e7c3-49ee-a348-9507d2ce858f-kube-api-access-nkkgc\") pod \"f1d95e20-e7c3-49ee-a348-9507d2ce858f\" (UID: \"f1d95e20-e7c3-49ee-a348-9507d2ce858f\") " Oct 30 21:07:38.546213 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:07:38.546188 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1d95e20-e7c3-49ee-a348-9507d2ce858f-maas-api-openshift-ca" pod "f1d95e20-e7c3-49ee-a348-9507d2ce858f" (UID: "f1d95e20-e7c3-49ee-a348-9507d2ce858f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:07:38.547888 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:07:38.547867 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f1d95e20-e7c3-49ee-a348-9507d2ce858f-maas-api-xks-ca" pod "f1d95e20-e7c3-49ee-a348-9507d2ce858f" (UID: "f1d95e20-e7c3-49ee-a348-9507d2ce858f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:07:38.548102 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:07:38.548086 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1d95e20-e7c3-49ee-a348-9507d2ce858f-kube-api-access-nkkgc" pod "f1d95e20-e7c3-49ee-a348-9507d2ce858f" (UID: "f1d95e20-e7c3-49ee-a348-9507d2ce858f"). InnerVolumeSpecName "kube-api-access-nkkgc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:07:38.646786 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:07:38.646764 4611 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1d95e20-e7c3-49ee-a348-9507d2ce858f-maas-api-xks-ca\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:07:38.646786 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:07:38.646785 4611 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1d95e20-e7c3-49ee-a348-9507d2ce858f-maas-api-openshift-ca\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:07:38.646914 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:07:38.646795 4611 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nkkgc\" (UniqueName: \"kubernetes.io/projected/f1d95e20-e7c3-49ee-a348-9507d2ce858f-kube-api-access-nkkgc\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:07:39.288551 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:07:39.288496 4611 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ncgpg" Oct 30 21:07:39.288551 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:07:39.288540 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ncgpg" event={"ID":"f1d95e20-e7c3-49ee-a348-9507d2ce858f","Type":"ContainerDied","Data":"05674c536e86a5011cd44d72276d82f40d73b907df43ac588c248a6befe5016c"} Oct 30 21:07:39.288799 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:07:39.288619 4611 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="05674c536e86a5011cd44d72276d82f40d73b907df43ac588c248a6befe5016c" Oct 30 21:07:49.283550 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:07:49.283483 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:08:02.209104 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:08:02.209068 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:08:03.916362 ip-10-0-133-192 kubenswrapper[4611]: E1030 21:08:03.916325 4611 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hwzc6" containerName="extractor" Oct 30 21:08:14.393125 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:08:14.393088 4611 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-hxr75"] Oct 30 21:08:14.393552 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:08:14.393474 4611 cpu_manager.go:414] "containerMap: removing container" podUID="f1d95e20-e7c3-49ee-a348-9507d2ce858f" containerName="revoke-keys" Oct 30 21:08:14.393552 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:08:14.393485 4611 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1d95e20-e7c3-49ee-a348-9507d2ce858f" containerName="revoke-keys" Oct 30 21:08:14.393645 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:08:14.393591 4611 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1d95e20-e7c3-49ee-a348-9507d2ce858f" containerName="revoke-keys" Oct 30 21:08:14.396935 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:08:14.396918 4611 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-hxr75" Oct 30 21:08:14.400243 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:08:14.400213 4611 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:08:14.400243 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:08:14.400213 4611 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7hf8r\"" Oct 30 21:08:14.400464 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:08:14.400257 4611 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:08:14.402333 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:08:14.402310 4611 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:08:14.406330 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:08:14.406307 4611 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-hxr75"] Oct 30 21:08:14.519821 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:08:14.519786 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/395d5b9b-8e6a-4219-97dd-f550ed328d6a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-hxr75\" (UID: \"395d5b9b-8e6a-4219-97dd-f550ed328d6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-hxr75" Oct 30 21:08:14.519990 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:08:14.519831 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/395d5b9b-8e6a-4219-97dd-f550ed328d6a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-hxr75\" (UID: \"395d5b9b-8e6a-4219-97dd-f550ed328d6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-hxr75" Oct 30 21:08:14.519990 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:08:14.519969 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qxx2x\" (UniqueName: \"kubernetes.io/projected/395d5b9b-8e6a-4219-97dd-f550ed328d6a-kube-api-access-qxx2x\") pod \"maas-api-revoke-sub-e2fc9c5b-hxr75\" (UID: \"395d5b9b-8e6a-4219-97dd-f550ed328d6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-hxr75" Oct 30 21:08:14.621154 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:08:14.621120 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qxx2x\" (UniqueName: \"kubernetes.io/projected/395d5b9b-8e6a-4219-97dd-f550ed328d6a-kube-api-access-qxx2x\") pod \"maas-api-revoke-sub-e2fc9c5b-hxr75\" (UID: \"395d5b9b-8e6a-4219-97dd-f550ed328d6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-hxr75" Oct 30 21:08:14.621325 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:08:14.621209 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/395d5b9b-8e6a-4219-97dd-f550ed328d6a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-hxr75\" (UID: \"395d5b9b-8e6a-4219-97dd-f550ed328d6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-hxr75" Oct 30 21:08:14.621325 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:08:14.621245 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/395d5b9b-8e6a-4219-97dd-f550ed328d6a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-hxr75\" (UID: \"395d5b9b-8e6a-4219-97dd-f550ed328d6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-hxr75" Oct 30 21:08:14.621898 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:08:14.621872 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/395d5b9b-8e6a-4219-97dd-f550ed328d6a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-hxr75\" (UID: \"395d5b9b-8e6a-4219-97dd-f550ed328d6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-hxr75" Oct 30 21:08:14.623537 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:08:14.623502 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/395d5b9b-8e6a-4219-97dd-f550ed328d6a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-hxr75\" (UID: \"395d5b9b-8e6a-4219-97dd-f550ed328d6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-hxr75" Oct 30 21:08:14.632723 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:08:14.632705 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qxx2x\" (UniqueName: \"kubernetes.io/projected/395d5b9b-8e6a-4219-97dd-f550ed328d6a-kube-api-access-qxx2x\") pod \"maas-api-revoke-sub-e2fc9c5b-hxr75\" (UID: \"395d5b9b-8e6a-4219-97dd-f550ed328d6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-hxr75" Oct 30 21:08:14.712696 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:08:14.712645 4611 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-hxr75" Oct 30 21:08:14.989848 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:08:14.989824 4611 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-hxr75"] Oct 30 21:08:14.991911 ip-10-0-133-192 kubenswrapper[4611]: W1030 21:08:14.991881 4611 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod395d5b9b_8e6a_4219_97dd_f550ed328d6a.slice/crio-c509db3e687fee317902a9cd0efac67c8bc7dc00a948db811ed7de73ec8ddb84 WatchSource:0}: Error finding container c509db3e687fee317902a9cd0efac67c8bc7dc00a948db811ed7de73ec8ddb84: Status 404 returned error can't find the container with id c509db3e687fee317902a9cd0efac67c8bc7dc00a948db811ed7de73ec8ddb84 Oct 30 21:08:15.502249 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:08:15.502215 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-hxr75" event={"ID":"395d5b9b-8e6a-4219-97dd-f550ed328d6a","Type":"ContainerStarted","Data":"fa1ad561e8cf07be219281b5506d7ca076056d0434d66b545d16ef4f37a3e7ed"} Oct 30 21:08:15.502710 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:08:15.502265 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-hxr75" event={"ID":"395d5b9b-8e6a-4219-97dd-f550ed328d6a","Type":"ContainerStarted","Data":"c509db3e687fee317902a9cd0efac67c8bc7dc00a948db811ed7de73ec8ddb84"} Oct 30 21:08:15.521842 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:08:15.521758 4611 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-hxr75" podStartSLOduration=1.521746543 podStartE2EDuration="1.521746543s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:08:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:08:15.519154859 +0000 UTC m=+1750.053181476" watchObservedRunningTime="2026-10-30 21:08:15.521746543 +0000 UTC m=+1750.055773211" Oct 30 21:08:16.509381 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:08:16.509347 4611 generic.go:356] "Generic (PLEG): container finished" podID="395d5b9b-8e6a-4219-97dd-f550ed328d6a" containerID="fa1ad561e8cf07be219281b5506d7ca076056d0434d66b545d16ef4f37a3e7ed" exitCode=0 Oct 30 21:08:16.509781 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:08:16.509438 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-hxr75" event={"ID":"395d5b9b-8e6a-4219-97dd-f550ed328d6a","Type":"ContainerDied","Data":"fa1ad561e8cf07be219281b5506d7ca076056d0434d66b545d16ef4f37a3e7ed"} Oct 30 21:08:17.473776 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:08:17.473734 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:08:17.645273 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:08:17.645243 4611 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-hxr75" Oct 30 21:08:17.649539 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:08:17.649484 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:08:17.750927 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:08:17.750856 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/395d5b9b-8e6a-4219-97dd-f550ed328d6a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/395d5b9b-8e6a-4219-97dd-f550ed328d6a-maas-api-xks-ca\") pod \"395d5b9b-8e6a-4219-97dd-f550ed328d6a\" (UID: \"395d5b9b-8e6a-4219-97dd-f550ed328d6a\") " Oct 30 21:08:17.750927 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:08:17.750910 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/395d5b9b-8e6a-4219-97dd-f550ed328d6a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/395d5b9b-8e6a-4219-97dd-f550ed328d6a-maas-api-openshift-ca\") pod \"395d5b9b-8e6a-4219-97dd-f550ed328d6a\" (UID: \"395d5b9b-8e6a-4219-97dd-f550ed328d6a\") " Oct 30 21:08:17.751101 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:08:17.750943 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/395d5b9b-8e6a-4219-97dd-f550ed328d6a-kube-api-access-qxx2x\" (UniqueName: \"kubernetes.io/projected/395d5b9b-8e6a-4219-97dd-f550ed328d6a-kube-api-access-qxx2x\") pod \"395d5b9b-8e6a-4219-97dd-f550ed328d6a\" (UID: \"395d5b9b-8e6a-4219-97dd-f550ed328d6a\") " Oct 30 21:08:17.751562 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:08:17.751528 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/395d5b9b-8e6a-4219-97dd-f550ed328d6a-maas-api-openshift-ca" pod "395d5b9b-8e6a-4219-97dd-f550ed328d6a" (UID: "395d5b9b-8e6a-4219-97dd-f550ed328d6a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:08:17.751913 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:08:17.751892 4611 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/395d5b9b-8e6a-4219-97dd-f550ed328d6a-maas-api-openshift-ca\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:08:17.753576 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:08:17.753541 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/395d5b9b-8e6a-4219-97dd-f550ed328d6a-maas-api-xks-ca" pod "395d5b9b-8e6a-4219-97dd-f550ed328d6a" (UID: "395d5b9b-8e6a-4219-97dd-f550ed328d6a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:08:17.755176 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:08:17.755137 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/395d5b9b-8e6a-4219-97dd-f550ed328d6a-kube-api-access-qxx2x" pod "395d5b9b-8e6a-4219-97dd-f550ed328d6a" (UID: "395d5b9b-8e6a-4219-97dd-f550ed328d6a"). InnerVolumeSpecName "kube-api-access-qxx2x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:08:17.852959 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:08:17.852932 4611 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/395d5b9b-8e6a-4219-97dd-f550ed328d6a-maas-api-xks-ca\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:08:17.852959 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:08:17.852956 4611 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qxx2x\" (UniqueName: \"kubernetes.io/projected/395d5b9b-8e6a-4219-97dd-f550ed328d6a-kube-api-access-qxx2x\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:08:18.520254 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:08:18.520224 4611 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-hxr75" Oct 30 21:08:18.520420 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:08:18.520247 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-hxr75" event={"ID":"395d5b9b-8e6a-4219-97dd-f550ed328d6a","Type":"ContainerDied","Data":"c509db3e687fee317902a9cd0efac67c8bc7dc00a948db811ed7de73ec8ddb84"} Oct 30 21:08:18.520420 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:08:18.520318 4611 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c509db3e687fee317902a9cd0efac67c8bc7dc00a948db811ed7de73ec8ddb84" Oct 30 21:08:24.604351 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:08:24.604317 4611 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-55fn8"] Oct 30 21:08:24.604763 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:08:24.604733 4611 cpu_manager.go:414] "containerMap: removing container" podUID="395d5b9b-8e6a-4219-97dd-f550ed328d6a" containerName="revoke-keys" Oct 30 21:08:24.604763 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:08:24.604745 4611 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="395d5b9b-8e6a-4219-97dd-f550ed328d6a" containerName="revoke-keys" Oct 30 21:08:24.604850 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:08:24.604801 4611 memory_manager.go:365] "RemoveStaleState removing state" podUID="395d5b9b-8e6a-4219-97dd-f550ed328d6a" containerName="revoke-keys" Oct 30 21:08:24.608223 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:08:24.608200 4611 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-55fn8" Oct 30 21:08:24.613564 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:08:24.613544 4611 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:08:24.614231 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:08:24.614202 4611 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7hf8r\"" Oct 30 21:08:24.617185 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:08:24.617163 4611 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-55fn8"] Oct 30 21:08:24.667911 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:08:24.667882 4611 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:08:24.668040 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:08:24.668003 4611 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:08:24.711122 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:08:24.711095 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/edb49403-8ee6-481d-b874-67c0d802f201-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-55fn8\" (UID: \"edb49403-8ee6-481d-b874-67c0d802f201\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-55fn8" Oct 30 21:08:24.711231 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:08:24.711151 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/edb49403-8ee6-481d-b874-67c0d802f201-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-55fn8\" (UID: \"edb49403-8ee6-481d-b874-67c0d802f201\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-55fn8" Oct 30 21:08:24.711306 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:08:24.711286 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n7qgs\" (UniqueName: \"kubernetes.io/projected/edb49403-8ee6-481d-b874-67c0d802f201-kube-api-access-n7qgs\") pod \"maas-api-revoke-sub-5a29efb3-55fn8\" (UID: \"edb49403-8ee6-481d-b874-67c0d802f201\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-55fn8" Oct 30 21:08:24.812310 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:08:24.812284 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/edb49403-8ee6-481d-b874-67c0d802f201-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-55fn8\" (UID: \"edb49403-8ee6-481d-b874-67c0d802f201\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-55fn8" Oct 30 21:08:24.812422 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:08:24.812340 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n7qgs\" (UniqueName: \"kubernetes.io/projected/edb49403-8ee6-481d-b874-67c0d802f201-kube-api-access-n7qgs\") pod \"maas-api-revoke-sub-5a29efb3-55fn8\" (UID: \"edb49403-8ee6-481d-b874-67c0d802f201\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-55fn8" Oct 30 21:08:24.812422 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:08:24.812375 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/edb49403-8ee6-481d-b874-67c0d802f201-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-55fn8\" (UID: \"edb49403-8ee6-481d-b874-67c0d802f201\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-55fn8" Oct 30 21:08:24.812933 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:08:24.812915 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/edb49403-8ee6-481d-b874-67c0d802f201-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-55fn8\" (UID: \"edb49403-8ee6-481d-b874-67c0d802f201\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-55fn8" Oct 30 21:08:24.814635 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:08:24.814607 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/edb49403-8ee6-481d-b874-67c0d802f201-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-55fn8\" (UID: \"edb49403-8ee6-481d-b874-67c0d802f201\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-55fn8" Oct 30 21:08:24.824264 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:08:24.824244 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n7qgs\" (UniqueName: \"kubernetes.io/projected/edb49403-8ee6-481d-b874-67c0d802f201-kube-api-access-n7qgs\") pod \"maas-api-revoke-sub-5a29efb3-55fn8\" (UID: \"edb49403-8ee6-481d-b874-67c0d802f201\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-55fn8" Oct 30 21:08:24.927567 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:08:24.927490 4611 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-55fn8" Oct 30 21:08:25.207733 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:08:25.207709 4611 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-55fn8"] Oct 30 21:08:25.210013 ip-10-0-133-192 kubenswrapper[4611]: W1030 21:08:25.209981 4611 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podedb49403_8ee6_481d_b874_67c0d802f201.slice/crio-39a9433d74b177eab42a46bcde58231421f25f983c1acab7822d3aaa09332f51 WatchSource:0}: Error finding container 39a9433d74b177eab42a46bcde58231421f25f983c1acab7822d3aaa09332f51: Status 404 returned error can't find the container with id 39a9433d74b177eab42a46bcde58231421f25f983c1acab7822d3aaa09332f51 Oct 30 21:08:25.558479 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:08:25.558403 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-55fn8" event={"ID":"edb49403-8ee6-481d-b874-67c0d802f201","Type":"ContainerStarted","Data":"15bb295c7703f215a1f7a45271e993fbf813050b4e84a5628f92f22f105441ee"} Oct 30 21:08:25.558479 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:08:25.558454 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-55fn8" event={"ID":"edb49403-8ee6-481d-b874-67c0d802f201","Type":"ContainerStarted","Data":"39a9433d74b177eab42a46bcde58231421f25f983c1acab7822d3aaa09332f51"} Oct 30 21:08:25.580349 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:08:25.580292 4611 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-55fn8" podStartSLOduration=1.580275205 podStartE2EDuration="1.580275205s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:08:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:08:25.579185596 +0000 UTC m=+1760.113212214" watchObservedRunningTime="2026-10-30 21:08:25.580275205 +0000 UTC m=+1760.114301822" Oct 30 21:08:26.567578 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:08:26.567534 4611 generic.go:356] "Generic (PLEG): container finished" podID="edb49403-8ee6-481d-b874-67c0d802f201" containerID="15bb295c7703f215a1f7a45271e993fbf813050b4e84a5628f92f22f105441ee" exitCode=0 Oct 30 21:08:26.568097 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:08:26.567697 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-55fn8" event={"ID":"edb49403-8ee6-481d-b874-67c0d802f201","Type":"ContainerDied","Data":"15bb295c7703f215a1f7a45271e993fbf813050b4e84a5628f92f22f105441ee"} Oct 30 21:08:27.039049 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:08:27.038995 4611 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-xv2z4"] Oct 30 21:08:27.044343 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:08:27.044318 4611 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-xv2z4"] Oct 30 21:08:27.637555 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:08:27.626766 4611 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="147ccaa0-2bf7-41d9-abd5-45757ea45d96" path="/var/lib/kubelet/pods/147ccaa0-2bf7-41d9-abd5-45757ea45d96/volumes" Oct 30 21:08:28.444405 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:08:28.444379 4611 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-55fn8" Oct 30 21:08:28.495373 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:08:28.495337 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/edb49403-8ee6-481d-b874-67c0d802f201-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/edb49403-8ee6-481d-b874-67c0d802f201-maas-api-xks-ca\") pod \"edb49403-8ee6-481d-b874-67c0d802f201\" (UID: \"edb49403-8ee6-481d-b874-67c0d802f201\") " Oct 30 21:08:28.495700 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:08:28.495673 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/edb49403-8ee6-481d-b874-67c0d802f201-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/edb49403-8ee6-481d-b874-67c0d802f201-maas-api-openshift-ca\") pod \"edb49403-8ee6-481d-b874-67c0d802f201\" (UID: \"edb49403-8ee6-481d-b874-67c0d802f201\") " Oct 30 21:08:28.495864 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:08:28.495846 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/edb49403-8ee6-481d-b874-67c0d802f201-kube-api-access-n7qgs\" (UniqueName: \"kubernetes.io/projected/edb49403-8ee6-481d-b874-67c0d802f201-kube-api-access-n7qgs\") pod \"edb49403-8ee6-481d-b874-67c0d802f201\" (UID: \"edb49403-8ee6-481d-b874-67c0d802f201\") " Oct 30 21:08:28.498359 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:08:28.498333 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/edb49403-8ee6-481d-b874-67c0d802f201-maas-api-openshift-ca" pod "edb49403-8ee6-481d-b874-67c0d802f201" (UID: "edb49403-8ee6-481d-b874-67c0d802f201"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:08:28.508537 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:08:28.505260 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/edb49403-8ee6-481d-b874-67c0d802f201-kube-api-access-n7qgs" pod "edb49403-8ee6-481d-b874-67c0d802f201" (UID: "edb49403-8ee6-481d-b874-67c0d802f201"). InnerVolumeSpecName "kube-api-access-n7qgs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:08:28.508537 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:08:28.505634 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/edb49403-8ee6-481d-b874-67c0d802f201-maas-api-xks-ca" pod "edb49403-8ee6-481d-b874-67c0d802f201" (UID: "edb49403-8ee6-481d-b874-67c0d802f201"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:08:28.591176 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:08:28.591137 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-55fn8" event={"ID":"edb49403-8ee6-481d-b874-67c0d802f201","Type":"ContainerDied","Data":"39a9433d74b177eab42a46bcde58231421f25f983c1acab7822d3aaa09332f51"} Oct 30 21:08:28.591466 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:08:28.591446 4611 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="39a9433d74b177eab42a46bcde58231421f25f983c1acab7822d3aaa09332f51" Oct 30 21:08:28.591723 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:08:28.591704 4611 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-55fn8" Oct 30 21:08:28.597793 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:08:28.597759 4611 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/edb49403-8ee6-481d-b874-67c0d802f201-maas-api-openshift-ca\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:08:28.597901 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:08:28.597797 4611 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n7qgs\" (UniqueName: \"kubernetes.io/projected/edb49403-8ee6-481d-b874-67c0d802f201-kube-api-access-n7qgs\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:08:28.597901 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:08:28.597815 4611 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/edb49403-8ee6-481d-b874-67c0d802f201-maas-api-xks-ca\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:08:31.207961 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:08:31.207922 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:08:31.363232 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:08:31.363198 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:08:33.916170 ip-10-0-133-192 kubenswrapper[4611]: E1030 21:08:33.916136 4611 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hwzc6" containerName="extractor" Oct 30 21:08:49.309920 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:08:49.309883 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:08:50.588788 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:08:50.588755 4611 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ncgpg"] Oct 30 21:08:50.594251 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:08:50.594219 4611 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ncgpg"] Oct 30 21:08:50.613989 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:08:50.613961 4611 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-hjrht"] Oct 30 21:08:50.614324 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:08:50.614312 4611 cpu_manager.go:414] "containerMap: removing container" podUID="edb49403-8ee6-481d-b874-67c0d802f201" containerName="revoke-keys" Oct 30 21:08:50.614378 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:08:50.614325 4611 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="edb49403-8ee6-481d-b874-67c0d802f201" containerName="revoke-keys" Oct 30 21:08:50.614415 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:08:50.614408 4611 memory_manager.go:365] "RemoveStaleState removing state" podUID="edb49403-8ee6-481d-b874-67c0d802f201" containerName="revoke-keys" Oct 30 21:08:50.617636 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:08:50.617620 4611 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-hjrht" Oct 30 21:08:50.625540 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:08:50.625241 4611 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:08:50.625540 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:08:50.625490 4611 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7hf8r\"" Oct 30 21:08:50.625736 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:08:50.625552 4611 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:08:50.625736 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:08:50.625629 4611 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:08:50.628071 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:08:50.628045 4611 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-hjrht"] Oct 30 21:08:50.696221 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:08:50.696181 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kjjhd\" (UniqueName: \"kubernetes.io/projected/58003568-802b-479c-aa1e-04276bff15cc-kube-api-access-kjjhd\") pod \"maas-api-revoke-sub-3d4156f1-hjrht\" (UID: \"58003568-802b-479c-aa1e-04276bff15cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-hjrht" Oct 30 21:08:50.696789 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:08:50.696658 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58003568-802b-479c-aa1e-04276bff15cc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-hjrht\" (UID: \"58003568-802b-479c-aa1e-04276bff15cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-hjrht" Oct 30 21:08:50.696789 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:08:50.696767 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58003568-802b-479c-aa1e-04276bff15cc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-hjrht\" (UID: \"58003568-802b-479c-aa1e-04276bff15cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-hjrht" Oct 30 21:08:50.798407 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:08:50.798382 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58003568-802b-479c-aa1e-04276bff15cc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-hjrht\" (UID: \"58003568-802b-479c-aa1e-04276bff15cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-hjrht" Oct 30 21:08:50.798561 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:08:50.798416 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58003568-802b-479c-aa1e-04276bff15cc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-hjrht\" (UID: \"58003568-802b-479c-aa1e-04276bff15cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-hjrht" Oct 30 21:08:50.798561 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:08:50.798470 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kjjhd\" (UniqueName: \"kubernetes.io/projected/58003568-802b-479c-aa1e-04276bff15cc-kube-api-access-kjjhd\") pod \"maas-api-revoke-sub-3d4156f1-hjrht\" (UID: \"58003568-802b-479c-aa1e-04276bff15cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-hjrht" Oct 30 21:08:50.799153 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:08:50.799134 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58003568-802b-479c-aa1e-04276bff15cc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-hjrht\" (UID: \"58003568-802b-479c-aa1e-04276bff15cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-hjrht" Oct 30 21:08:50.800686 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:08:50.800664 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58003568-802b-479c-aa1e-04276bff15cc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-hjrht\" (UID: \"58003568-802b-479c-aa1e-04276bff15cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-hjrht" Oct 30 21:08:50.809906 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:08:50.809883 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kjjhd\" (UniqueName: \"kubernetes.io/projected/58003568-802b-479c-aa1e-04276bff15cc-kube-api-access-kjjhd\") pod \"maas-api-revoke-sub-3d4156f1-hjrht\" (UID: \"58003568-802b-479c-aa1e-04276bff15cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-hjrht" Oct 30 21:08:50.932213 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:08:50.932190 4611 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-hjrht" Oct 30 21:08:51.208792 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:08:51.207986 4611 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-hjrht"] Oct 30 21:08:51.211598 ip-10-0-133-192 kubenswrapper[4611]: W1030 21:08:51.211564 4611 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58003568_802b_479c_aa1e_04276bff15cc.slice/crio-48cbbe2db7677482bc0ce30f70fc0cf44d38c2020ce30220369a3d7e544885b8 WatchSource:0}: Error finding container 48cbbe2db7677482bc0ce30f70fc0cf44d38c2020ce30220369a3d7e544885b8: Status 404 returned error can't find the container with id 48cbbe2db7677482bc0ce30f70fc0cf44d38c2020ce30220369a3d7e544885b8 Oct 30 21:08:51.604687 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:08:51.604651 4611 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f1d95e20-e7c3-49ee-a348-9507d2ce858f" path="/var/lib/kubelet/pods/f1d95e20-e7c3-49ee-a348-9507d2ce858f/volumes" Oct 30 21:08:51.708144 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:08:51.708115 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-hjrht" event={"ID":"58003568-802b-479c-aa1e-04276bff15cc","Type":"ContainerStarted","Data":"d8ba9ad13d7d452bc0b44e598e8b7567a25cac49c340d022a8f98dff62cc035a"} Oct 30 21:08:51.708261 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:08:51.708176 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-hjrht" event={"ID":"58003568-802b-479c-aa1e-04276bff15cc","Type":"ContainerStarted","Data":"48cbbe2db7677482bc0ce30f70fc0cf44d38c2020ce30220369a3d7e544885b8"} Oct 30 21:08:51.726109 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:08:51.726048 4611 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-hjrht" podStartSLOduration=1.726028514 podStartE2EDuration="1.726028514s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:08:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:08:51.724849617 +0000 UTC m=+1786.258876232" watchObservedRunningTime="2026-10-30 21:08:51.726028514 +0000 UTC m=+1786.260055130" Oct 30 21:08:52.714820 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:08:52.714784 4611 generic.go:356] "Generic (PLEG): container finished" podID="58003568-802b-479c-aa1e-04276bff15cc" containerID="d8ba9ad13d7d452bc0b44e598e8b7567a25cac49c340d022a8f98dff62cc035a" exitCode=0 Oct 30 21:08:52.715196 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:08:52.714862 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-hjrht" event={"ID":"58003568-802b-479c-aa1e-04276bff15cc","Type":"ContainerDied","Data":"d8ba9ad13d7d452bc0b44e598e8b7567a25cac49c340d022a8f98dff62cc035a"} Oct 30 21:08:53.859469 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:08:53.859443 4611 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-hjrht" Oct 30 21:08:53.924681 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:08:53.924653 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/58003568-802b-479c-aa1e-04276bff15cc-kube-api-access-kjjhd\" (UniqueName: \"kubernetes.io/projected/58003568-802b-479c-aa1e-04276bff15cc-kube-api-access-kjjhd\") pod \"58003568-802b-479c-aa1e-04276bff15cc\" (UID: \"58003568-802b-479c-aa1e-04276bff15cc\") " Oct 30 21:08:53.924850 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:08:53.924766 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/58003568-802b-479c-aa1e-04276bff15cc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58003568-802b-479c-aa1e-04276bff15cc-maas-api-xks-ca\") pod \"58003568-802b-479c-aa1e-04276bff15cc\" (UID: \"58003568-802b-479c-aa1e-04276bff15cc\") " Oct 30 21:08:53.924850 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:08:53.924821 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/58003568-802b-479c-aa1e-04276bff15cc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58003568-802b-479c-aa1e-04276bff15cc-maas-api-openshift-ca\") pod \"58003568-802b-479c-aa1e-04276bff15cc\" (UID: \"58003568-802b-479c-aa1e-04276bff15cc\") " Oct 30 21:08:53.925188 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:08:53.925156 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58003568-802b-479c-aa1e-04276bff15cc-maas-api-openshift-ca" pod "58003568-802b-479c-aa1e-04276bff15cc" (UID: "58003568-802b-479c-aa1e-04276bff15cc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:08:53.926791 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:08:53.926766 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/58003568-802b-479c-aa1e-04276bff15cc-maas-api-xks-ca" pod "58003568-802b-479c-aa1e-04276bff15cc" (UID: "58003568-802b-479c-aa1e-04276bff15cc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:08:53.926895 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:08:53.926873 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58003568-802b-479c-aa1e-04276bff15cc-kube-api-access-kjjhd" pod "58003568-802b-479c-aa1e-04276bff15cc" (UID: "58003568-802b-479c-aa1e-04276bff15cc"). InnerVolumeSpecName "kube-api-access-kjjhd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:08:54.026167 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:08:54.026111 4611 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58003568-802b-479c-aa1e-04276bff15cc-maas-api-xks-ca\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:08:54.026167 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:08:54.026135 4611 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58003568-802b-479c-aa1e-04276bff15cc-maas-api-openshift-ca\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:08:54.026167 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:08:54.026145 4611 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kjjhd\" (UniqueName: \"kubernetes.io/projected/58003568-802b-479c-aa1e-04276bff15cc-kube-api-access-kjjhd\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:08:54.725650 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:08:54.725615 4611 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-hjrht" Oct 30 21:08:54.725818 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:08:54.725618 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-hjrht" event={"ID":"58003568-802b-479c-aa1e-04276bff15cc","Type":"ContainerDied","Data":"48cbbe2db7677482bc0ce30f70fc0cf44d38c2020ce30220369a3d7e544885b8"} Oct 30 21:08:54.725818 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:08:54.725750 4611 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="48cbbe2db7677482bc0ce30f70fc0cf44d38c2020ce30220369a3d7e544885b8" Oct 30 21:09:04.081007 ip-10-0-133-192 kubenswrapper[4611]: E1030 21:09:04.080964 4611 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hwzc6" containerName="extractor" Oct 30 21:09:05.857096 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:09:05.857060 4611 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7w5wz_7e2574c1-b59c-453c-9501-25ec13a1ccad/ovn-acl-logging/0.log" Oct 30 21:09:05.863176 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:09:05.863154 4611 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7w5wz_7e2574c1-b59c-453c-9501-25ec13a1ccad/ovn-acl-logging/0.log" Oct 30 21:09:07.590801 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:09:07.590762 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:09:07.779228 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:09:07.779192 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:09:08.117152 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:09:08.117123 4611 scope.go:122] "RemoveContainer" containerID="f2ab44a8a2c6c2bf7927c564c71649d6389d31d4aaf0b10b27605206eb51a886" Oct 30 21:09:12.132236 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:09:12.132202 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:09:12.512607 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:09:12.512536 4611 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-single-main-1c4642-openshift-default-6784bc96d9fswv6"] Oct 30 21:09:12.512939 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:09:12.512926 4611 cpu_manager.go:414] "containerMap: removing container" podUID="58003568-802b-479c-aa1e-04276bff15cc" containerName="revoke-keys" Oct 30 21:09:12.512983 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:09:12.512939 4611 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="58003568-802b-479c-aa1e-04276bff15cc" containerName="revoke-keys" Oct 30 21:09:12.513026 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:09:12.513010 4611 memory_manager.go:365] "RemoveStaleState removing state" podUID="58003568-802b-479c-aa1e-04276bff15cc" containerName="revoke-keys" Oct 30 21:09:12.517828 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:09:12.517808 4611 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-1c4642-openshift-default-6784bc96d9fswv6" Oct 30 21:09:12.521267 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:09:12.521246 4611 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-single-main-1c4642-openshift-default-dockercfg-6m8rx\"" Oct 30 21:09:12.528820 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:09:12.528796 4611 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-1c4642-openshift-default-6784bc96d9fswv6"] Oct 30 21:09:12.593924 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:09:12.593895 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:09:12.619164 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:09:12.619123 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ef2c7e4-e9e7-4d0d-91b0-026b1f9cef76-istiod-ca-cert\") pod \"e2e-worker-single-main-1c4642-openshift-default-6784bc96d9fswv6\" (UID: \"4ef2c7e4-e9e7-4d0d-91b0-026b1f9cef76\") " pod="openshift-ingress/e2e-worker-single-main-1c4642-openshift-default-6784bc96d9fswv6" Oct 30 21:09:12.619519 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:09:12.619475 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ef2c7e4-e9e7-4d0d-91b0-026b1f9cef76-credential-socket\") pod \"e2e-worker-single-main-1c4642-openshift-default-6784bc96d9fswv6\" (UID: \"4ef2c7e4-e9e7-4d0d-91b0-026b1f9cef76\") " pod="openshift-ingress/e2e-worker-single-main-1c4642-openshift-default-6784bc96d9fswv6" Oct 30 21:09:12.619655 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:09:12.619532 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-prsdq\" (UniqueName: \"kubernetes.io/projected/4ef2c7e4-e9e7-4d0d-91b0-026b1f9cef76-kube-api-access-prsdq\") pod \"e2e-worker-single-main-1c4642-openshift-default-6784bc96d9fswv6\" (UID: \"4ef2c7e4-e9e7-4d0d-91b0-026b1f9cef76\") " pod="openshift-ingress/e2e-worker-single-main-1c4642-openshift-default-6784bc96d9fswv6" Oct 30 21:09:12.619735 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:09:12.619654 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ef2c7e4-e9e7-4d0d-91b0-026b1f9cef76-istio-podinfo\") pod \"e2e-worker-single-main-1c4642-openshift-default-6784bc96d9fswv6\" (UID: \"4ef2c7e4-e9e7-4d0d-91b0-026b1f9cef76\") " pod="openshift-ingress/e2e-worker-single-main-1c4642-openshift-default-6784bc96d9fswv6" Oct 30 21:09:12.619735 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:09:12.619703 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ef2c7e4-e9e7-4d0d-91b0-026b1f9cef76-istio-token\") pod \"e2e-worker-single-main-1c4642-openshift-default-6784bc96d9fswv6\" (UID: \"4ef2c7e4-e9e7-4d0d-91b0-026b1f9cef76\") " pod="openshift-ingress/e2e-worker-single-main-1c4642-openshift-default-6784bc96d9fswv6" Oct 30 21:09:12.619735 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:09:12.619729 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ef2c7e4-e9e7-4d0d-91b0-026b1f9cef76-workload-certs\") pod \"e2e-worker-single-main-1c4642-openshift-default-6784bc96d9fswv6\" (UID: \"4ef2c7e4-e9e7-4d0d-91b0-026b1f9cef76\") " pod="openshift-ingress/e2e-worker-single-main-1c4642-openshift-default-6784bc96d9fswv6" Oct 30 21:09:12.619959 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:09:12.619806 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ef2c7e4-e9e7-4d0d-91b0-026b1f9cef76-workload-socket\") pod \"e2e-worker-single-main-1c4642-openshift-default-6784bc96d9fswv6\" (UID: \"4ef2c7e4-e9e7-4d0d-91b0-026b1f9cef76\") " pod="openshift-ingress/e2e-worker-single-main-1c4642-openshift-default-6784bc96d9fswv6" Oct 30 21:09:12.619959 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:09:12.619846 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ef2c7e4-e9e7-4d0d-91b0-026b1f9cef76-istio-envoy\") pod \"e2e-worker-single-main-1c4642-openshift-default-6784bc96d9fswv6\" (UID: \"4ef2c7e4-e9e7-4d0d-91b0-026b1f9cef76\") " pod="openshift-ingress/e2e-worker-single-main-1c4642-openshift-default-6784bc96d9fswv6" Oct 30 21:09:12.619959 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:09:12.619912 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ef2c7e4-e9e7-4d0d-91b0-026b1f9cef76-istio-data\") pod \"e2e-worker-single-main-1c4642-openshift-default-6784bc96d9fswv6\" (UID: \"4ef2c7e4-e9e7-4d0d-91b0-026b1f9cef76\") " pod="openshift-ingress/e2e-worker-single-main-1c4642-openshift-default-6784bc96d9fswv6" Oct 30 21:09:12.722382 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:09:12.722339 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ef2c7e4-e9e7-4d0d-91b0-026b1f9cef76-istiod-ca-cert\") pod \"e2e-worker-single-main-1c4642-openshift-default-6784bc96d9fswv6\" (UID: \"4ef2c7e4-e9e7-4d0d-91b0-026b1f9cef76\") " pod="openshift-ingress/e2e-worker-single-main-1c4642-openshift-default-6784bc96d9fswv6" Oct 30 21:09:12.722645 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:09:12.722416 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ef2c7e4-e9e7-4d0d-91b0-026b1f9cef76-credential-socket\") pod \"e2e-worker-single-main-1c4642-openshift-default-6784bc96d9fswv6\" (UID: \"4ef2c7e4-e9e7-4d0d-91b0-026b1f9cef76\") " pod="openshift-ingress/e2e-worker-single-main-1c4642-openshift-default-6784bc96d9fswv6" Oct 30 21:09:12.722645 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:09:12.722458 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-prsdq\" (UniqueName: \"kubernetes.io/projected/4ef2c7e4-e9e7-4d0d-91b0-026b1f9cef76-kube-api-access-prsdq\") pod \"e2e-worker-single-main-1c4642-openshift-default-6784bc96d9fswv6\" (UID: \"4ef2c7e4-e9e7-4d0d-91b0-026b1f9cef76\") " pod="openshift-ingress/e2e-worker-single-main-1c4642-openshift-default-6784bc96d9fswv6" Oct 30 21:09:12.722645 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:09:12.722556 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ef2c7e4-e9e7-4d0d-91b0-026b1f9cef76-istio-podinfo\") pod \"e2e-worker-single-main-1c4642-openshift-default-6784bc96d9fswv6\" (UID: \"4ef2c7e4-e9e7-4d0d-91b0-026b1f9cef76\") " pod="openshift-ingress/e2e-worker-single-main-1c4642-openshift-default-6784bc96d9fswv6" Oct 30 21:09:12.723196 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:09:12.722593 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ef2c7e4-e9e7-4d0d-91b0-026b1f9cef76-istio-token\") pod \"e2e-worker-single-main-1c4642-openshift-default-6784bc96d9fswv6\" (UID: \"4ef2c7e4-e9e7-4d0d-91b0-026b1f9cef76\") " pod="openshift-ingress/e2e-worker-single-main-1c4642-openshift-default-6784bc96d9fswv6" Oct 30 21:09:12.723357 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:09:12.723203 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ef2c7e4-e9e7-4d0d-91b0-026b1f9cef76-credential-socket\") pod \"e2e-worker-single-main-1c4642-openshift-default-6784bc96d9fswv6\" (UID: \"4ef2c7e4-e9e7-4d0d-91b0-026b1f9cef76\") " pod="openshift-ingress/e2e-worker-single-main-1c4642-openshift-default-6784bc96d9fswv6" Oct 30 21:09:12.723357 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:09:12.723226 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ef2c7e4-e9e7-4d0d-91b0-026b1f9cef76-workload-certs\") pod \"e2e-worker-single-main-1c4642-openshift-default-6784bc96d9fswv6\" (UID: \"4ef2c7e4-e9e7-4d0d-91b0-026b1f9cef76\") " pod="openshift-ingress/e2e-worker-single-main-1c4642-openshift-default-6784bc96d9fswv6" Oct 30 21:09:12.723357 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:09:12.723263 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ef2c7e4-e9e7-4d0d-91b0-026b1f9cef76-istiod-ca-cert\") pod \"e2e-worker-single-main-1c4642-openshift-default-6784bc96d9fswv6\" (UID: \"4ef2c7e4-e9e7-4d0d-91b0-026b1f9cef76\") " pod="openshift-ingress/e2e-worker-single-main-1c4642-openshift-default-6784bc96d9fswv6" Oct 30 21:09:12.723357 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:09:12.723312 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ef2c7e4-e9e7-4d0d-91b0-026b1f9cef76-workload-socket\") pod \"e2e-worker-single-main-1c4642-openshift-default-6784bc96d9fswv6\" (UID: \"4ef2c7e4-e9e7-4d0d-91b0-026b1f9cef76\") " pod="openshift-ingress/e2e-worker-single-main-1c4642-openshift-default-6784bc96d9fswv6" Oct 30 21:09:12.723357 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:09:12.723341 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ef2c7e4-e9e7-4d0d-91b0-026b1f9cef76-istio-envoy\") pod \"e2e-worker-single-main-1c4642-openshift-default-6784bc96d9fswv6\" (UID: \"4ef2c7e4-e9e7-4d0d-91b0-026b1f9cef76\") " pod="openshift-ingress/e2e-worker-single-main-1c4642-openshift-default-6784bc96d9fswv6" Oct 30 21:09:12.723708 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:09:12.723389 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ef2c7e4-e9e7-4d0d-91b0-026b1f9cef76-istio-data\") pod \"e2e-worker-single-main-1c4642-openshift-default-6784bc96d9fswv6\" (UID: \"4ef2c7e4-e9e7-4d0d-91b0-026b1f9cef76\") " pod="openshift-ingress/e2e-worker-single-main-1c4642-openshift-default-6784bc96d9fswv6" Oct 30 21:09:12.723708 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:09:12.723630 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ef2c7e4-e9e7-4d0d-91b0-026b1f9cef76-workload-socket\") pod \"e2e-worker-single-main-1c4642-openshift-default-6784bc96d9fswv6\" (UID: \"4ef2c7e4-e9e7-4d0d-91b0-026b1f9cef76\") " pod="openshift-ingress/e2e-worker-single-main-1c4642-openshift-default-6784bc96d9fswv6" Oct 30 21:09:12.723708 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:09:12.723629 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ef2c7e4-e9e7-4d0d-91b0-026b1f9cef76-workload-certs\") pod \"e2e-worker-single-main-1c4642-openshift-default-6784bc96d9fswv6\" (UID: \"4ef2c7e4-e9e7-4d0d-91b0-026b1f9cef76\") " pod="openshift-ingress/e2e-worker-single-main-1c4642-openshift-default-6784bc96d9fswv6" Oct 30 21:09:12.723871 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:09:12.723719 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ef2c7e4-e9e7-4d0d-91b0-026b1f9cef76-istio-data\") pod \"e2e-worker-single-main-1c4642-openshift-default-6784bc96d9fswv6\" (UID: \"4ef2c7e4-e9e7-4d0d-91b0-026b1f9cef76\") " pod="openshift-ingress/e2e-worker-single-main-1c4642-openshift-default-6784bc96d9fswv6" Oct 30 21:09:12.725949 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:09:12.725899 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ef2c7e4-e9e7-4d0d-91b0-026b1f9cef76-istio-podinfo\") pod \"e2e-worker-single-main-1c4642-openshift-default-6784bc96d9fswv6\" (UID: \"4ef2c7e4-e9e7-4d0d-91b0-026b1f9cef76\") " pod="openshift-ingress/e2e-worker-single-main-1c4642-openshift-default-6784bc96d9fswv6" Oct 30 21:09:12.726355 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:09:12.726338 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ef2c7e4-e9e7-4d0d-91b0-026b1f9cef76-istio-envoy\") pod \"e2e-worker-single-main-1c4642-openshift-default-6784bc96d9fswv6\" (UID: \"4ef2c7e4-e9e7-4d0d-91b0-026b1f9cef76\") " pod="openshift-ingress/e2e-worker-single-main-1c4642-openshift-default-6784bc96d9fswv6" Oct 30 21:09:12.734620 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:09:12.734575 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-prsdq\" (UniqueName: \"kubernetes.io/projected/4ef2c7e4-e9e7-4d0d-91b0-026b1f9cef76-kube-api-access-prsdq\") pod \"e2e-worker-single-main-1c4642-openshift-default-6784bc96d9fswv6\" (UID: \"4ef2c7e4-e9e7-4d0d-91b0-026b1f9cef76\") " pod="openshift-ingress/e2e-worker-single-main-1c4642-openshift-default-6784bc96d9fswv6" Oct 30 21:09:12.735594 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:09:12.735569 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ef2c7e4-e9e7-4d0d-91b0-026b1f9cef76-istio-token\") pod \"e2e-worker-single-main-1c4642-openshift-default-6784bc96d9fswv6\" (UID: \"4ef2c7e4-e9e7-4d0d-91b0-026b1f9cef76\") " pod="openshift-ingress/e2e-worker-single-main-1c4642-openshift-default-6784bc96d9fswv6" Oct 30 21:09:12.744805 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:09:12.744771 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:09:12.831776 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:09:12.831721 4611 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-1c4642-openshift-default-6784bc96d9fswv6" Oct 30 21:09:13.118467 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:09:13.118441 4611 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-1c4642-openshift-default-6784bc96d9fswv6"] Oct 30 21:09:13.120739 ip-10-0-133-192 kubenswrapper[4611]: W1030 21:09:13.120710 4611 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ef2c7e4_e9e7_4d0d_91b0_026b1f9cef76.slice/crio-f8ade14db29be3eb12cd8c1527249283f06ec2b872b00a4f70f9b396b10b7fe8 WatchSource:0}: Error finding container f8ade14db29be3eb12cd8c1527249283f06ec2b872b00a4f70f9b396b10b7fe8: Status 404 returned error can't find the container with id f8ade14db29be3eb12cd8c1527249283f06ec2b872b00a4f70f9b396b10b7fe8 Oct 30 21:09:13.122990 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:09:13.122957 4611 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:09:13.123068 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:09:13.123024 4611 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:09:13.123068 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:09:13.123055 4611 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:09:13.872143 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:09:13.872109 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-1c4642-openshift-default-6784bc96d9fswv6" event={"ID":"4ef2c7e4-e9e7-4d0d-91b0-026b1f9cef76","Type":"ContainerStarted","Data":"0df373cd1e01b6da9c8183c930636c153b5a820d1032218de833214526580bb1"} Oct 30 21:09:13.872565 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:09:13.872162 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-1c4642-openshift-default-6784bc96d9fswv6" event={"ID":"4ef2c7e4-e9e7-4d0d-91b0-026b1f9cef76","Type":"ContainerStarted","Data":"f8ade14db29be3eb12cd8c1527249283f06ec2b872b00a4f70f9b396b10b7fe8"} Oct 30 21:09:13.896770 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:09:13.896721 4611 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-single-main-1c4642-openshift-default-6784bc96d9fswv6" podStartSLOduration=1.896707742 podStartE2EDuration="1.896707742s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:09:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:09:13.893705459 +0000 UTC m=+1808.427732077" watchObservedRunningTime="2026-10-30 21:09:13.896707742 +0000 UTC m=+1808.430734357" Oct 30 21:09:14.832300 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:09:14.832269 4611 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-1c4642-openshift-default-6784bc96d9fswv6" Oct 30 21:09:14.836745 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:09:14.836718 4611 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-1c4642-openshift-default-6784bc96d9fswv6" Oct 30 21:09:14.877137 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:09:14.877107 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-1c4642-openshift-default-6784bc96d9fswv6" Oct 30 21:09:14.877994 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:09:14.877972 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-1c4642-openshift-default-6784bc96d9fswv6" Oct 30 21:09:16.306991 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:09:16.306956 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:09:16.570795 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:09:16.570671 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:09:17.499809 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:09:17.499756 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:09:17.543321 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:09:17.543292 4611 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-1c4642-7bf468k4cr"] Oct 30 21:09:17.547758 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:09:17.547738 4611 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-1c4642-7bf468k4cr" Oct 30 21:09:17.555873 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:09:17.555847 4611 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-88afd8f1-dockercfg-7jq2z\"" Oct 30 21:09:17.556105 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:09:17.555964 4611 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-1c4642\"" Oct 30 21:09:17.557778 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:09:17.557756 4611 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-1c4642-7bf468k4cr"] Oct 30 21:09:17.575933 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:09:17.575899 4611 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-1c4642-64f579d6bttr5r"] Oct 30 21:09:17.579887 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:09:17.579867 4611 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-1c4642-64f579d6bttr5r" Oct 30 21:09:17.589176 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:09:17.589149 4611 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-1c4642-64f579d6bttr5r"] Oct 30 21:09:17.670533 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:09:17.659032 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:09:17.682540 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:09:17.678966 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fba16a3f-2530-4780-be32-2f72358c7e47-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-1c4642-64f579d6bttr5r\" (UID: \"fba16a3f-2530-4780-be32-2f72358c7e47\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-1c4642-64f579d6bttr5r" Oct 30 21:09:17.682540 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:09:17.679066 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec926cab-0bca-4a28-bada-cc067d70d3b9-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-1c4642-7bf468k4cr\" (UID: \"ec926cab-0bca-4a28-bada-cc067d70d3b9\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-1c4642-7bf468k4cr" Oct 30 21:09:17.682540 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:09:17.679209 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jsrkf\" (UniqueName: \"kubernetes.io/projected/ec926cab-0bca-4a28-bada-cc067d70d3b9-kube-api-access-jsrkf\") pod \"payload-pre-processing-e2e-worker-single-main-1c4642-7bf468k4cr\" (UID: \"ec926cab-0bca-4a28-bada-cc067d70d3b9\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-1c4642-7bf468k4cr" Oct 30 21:09:17.682540 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:09:17.679275 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bwrzr\" (UniqueName: \"kubernetes.io/projected/fba16a3f-2530-4780-be32-2f72358c7e47-kube-api-access-bwrzr\") pod \"payload-processing-e2e-worker-single-main-1c4642-64f579d6bttr5r\" (UID: \"fba16a3f-2530-4780-be32-2f72358c7e47\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-1c4642-64f579d6bttr5r" Oct 30 21:09:17.780984 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:09:17.780896 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jsrkf\" (UniqueName: \"kubernetes.io/projected/ec926cab-0bca-4a28-bada-cc067d70d3b9-kube-api-access-jsrkf\") pod \"payload-pre-processing-e2e-worker-single-main-1c4642-7bf468k4cr\" (UID: \"ec926cab-0bca-4a28-bada-cc067d70d3b9\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-1c4642-7bf468k4cr" Oct 30 21:09:17.780984 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:09:17.780973 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bwrzr\" (UniqueName: \"kubernetes.io/projected/fba16a3f-2530-4780-be32-2f72358c7e47-kube-api-access-bwrzr\") pod \"payload-processing-e2e-worker-single-main-1c4642-64f579d6bttr5r\" (UID: \"fba16a3f-2530-4780-be32-2f72358c7e47\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-1c4642-64f579d6bttr5r" Oct 30 21:09:17.781227 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:09:17.781049 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fba16a3f-2530-4780-be32-2f72358c7e47-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-1c4642-64f579d6bttr5r\" (UID: \"fba16a3f-2530-4780-be32-2f72358c7e47\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-1c4642-64f579d6bttr5r" Oct 30 21:09:17.781227 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:09:17.781149 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec926cab-0bca-4a28-bada-cc067d70d3b9-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-1c4642-7bf468k4cr\" (UID: \"ec926cab-0bca-4a28-bada-cc067d70d3b9\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-1c4642-7bf468k4cr" Oct 30 21:09:17.781899 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:09:17.781879 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec926cab-0bca-4a28-bada-cc067d70d3b9-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-1c4642-7bf468k4cr\" (UID: \"ec926cab-0bca-4a28-bada-cc067d70d3b9\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-1c4642-7bf468k4cr" Oct 30 21:09:17.781956 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:09:17.781895 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fba16a3f-2530-4780-be32-2f72358c7e47-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-1c4642-64f579d6bttr5r\" (UID: \"fba16a3f-2530-4780-be32-2f72358c7e47\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-1c4642-64f579d6bttr5r" Oct 30 21:09:17.793241 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:09:17.793219 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bwrzr\" (UniqueName: \"kubernetes.io/projected/fba16a3f-2530-4780-be32-2f72358c7e47-kube-api-access-bwrzr\") pod \"payload-processing-e2e-worker-single-main-1c4642-64f579d6bttr5r\" (UID: \"fba16a3f-2530-4780-be32-2f72358c7e47\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-1c4642-64f579d6bttr5r" Oct 30 21:09:17.793537 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:09:17.793493 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jsrkf\" (UniqueName: \"kubernetes.io/projected/ec926cab-0bca-4a28-bada-cc067d70d3b9-kube-api-access-jsrkf\") pod \"payload-pre-processing-e2e-worker-single-main-1c4642-7bf468k4cr\" (UID: \"ec926cab-0bca-4a28-bada-cc067d70d3b9\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-1c4642-7bf468k4cr" Oct 30 21:09:17.863501 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:09:17.863476 4611 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-1c4642-7bf468k4cr" Oct 30 21:09:17.894815 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:09:17.894778 4611 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-1c4642-64f579d6bttr5r" Oct 30 21:09:18.148780 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:09:18.148750 4611 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-1c4642-7bf468k4cr"] Oct 30 21:09:18.150092 ip-10-0-133-192 kubenswrapper[4611]: W1030 21:09:18.150069 4611 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec926cab_0bca_4a28_bada_cc067d70d3b9.slice/crio-288477348e86101a0da494e0eedff26c27fd647c8e8a5e3c83b9fb7584625866 WatchSource:0}: Error finding container 288477348e86101a0da494e0eedff26c27fd647c8e8a5e3c83b9fb7584625866: Status 404 returned error can't find the container with id 288477348e86101a0da494e0eedff26c27fd647c8e8a5e3c83b9fb7584625866 Oct 30 21:09:18.182623 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:09:18.182595 4611 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-1c4642-64f579d6bttr5r"] Oct 30 21:09:18.183567 ip-10-0-133-192 kubenswrapper[4611]: W1030 21:09:18.183539 4611 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfba16a3f_2530_4780_be32_2f72358c7e47.slice/crio-9370c8cc81101bd4b9f19d8964d23ce1026773e59284a910ea32ab371f7fe52b WatchSource:0}: Error finding container 9370c8cc81101bd4b9f19d8964d23ce1026773e59284a910ea32ab371f7fe52b: Status 404 returned error can't find the container with id 9370c8cc81101bd4b9f19d8964d23ce1026773e59284a910ea32ab371f7fe52b Oct 30 21:09:18.410938 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:09:18.410838 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:09:18.903671 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:09:18.903628 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-1c4642-64f579d6bttr5r" event={"ID":"fba16a3f-2530-4780-be32-2f72358c7e47","Type":"ContainerStarted","Data":"4a3ed81adc6fc5949da33cf3a449b845d63e35f7fa03ddbde33c3bbaab929c39"} Oct 30 21:09:18.904133 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:09:18.903697 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-1c4642-64f579d6bttr5r" event={"ID":"fba16a3f-2530-4780-be32-2f72358c7e47","Type":"ContainerStarted","Data":"9370c8cc81101bd4b9f19d8964d23ce1026773e59284a910ea32ab371f7fe52b"} Oct 30 21:09:18.904133 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:09:18.903729 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-1c4642-64f579d6bttr5r" Oct 30 21:09:18.905265 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:09:18.905239 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-1c4642-7bf468k4cr" event={"ID":"ec926cab-0bca-4a28-bada-cc067d70d3b9","Type":"ContainerStarted","Data":"dfc52b6e28060f61f85d4d94fd60651e6c4e19668c8791fed953be870fd9a82f"} Oct 30 21:09:18.905331 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:09:18.905289 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-1c4642-7bf468k4cr" Oct 30 21:09:18.905331 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:09:18.905311 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-1c4642-7bf468k4cr" event={"ID":"ec926cab-0bca-4a28-bada-cc067d70d3b9","Type":"ContainerStarted","Data":"288477348e86101a0da494e0eedff26c27fd647c8e8a5e3c83b9fb7584625866"} Oct 30 21:09:18.924824 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:09:18.924779 4611 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-1c4642-64f579d6bttr5r" podStartSLOduration=1.9247651879999998 podStartE2EDuration="1.924765188s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:09:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:09:18.92028233 +0000 UTC m=+1813.454308947" watchObservedRunningTime="2026-10-30 21:09:18.924765188 +0000 UTC m=+1813.458791805" Oct 30 21:09:18.937959 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:09:18.937912 4611 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-1c4642-7bf468k4cr" podStartSLOduration=1.937896063 podStartE2EDuration="1.937896063s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:09:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:09:18.937574889 +0000 UTC m=+1813.471601506" watchObservedRunningTime="2026-10-30 21:09:18.937896063 +0000 UTC m=+1813.471922684" Oct 30 21:09:19.203622 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:09:19.203530 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:09:19.932845 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:09:19.932807 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:09:20.615225 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:09:20.615184 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:09:21.415368 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:09:21.415332 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:09:22.095687 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:09:22.095647 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:09:22.723041 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:09:22.722983 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:09:23.413752 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:09:23.413703 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:09:24.089369 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:09:24.089335 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:09:24.986537 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:09:24.986296 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:09:25.882069 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:09:25.882027 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:09:26.236064 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:09:26.235985 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:09:26.639177 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:09:26.639138 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:09:26.890713 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:09:26.890621 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:09:27.036653 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:09:27.036614 4611 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-pxn6c"] Oct 30 21:09:27.044068 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:09:27.044034 4611 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-pxn6c"] Oct 30 21:09:27.047409 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:09:27.047379 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:09:27.429661 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:09:27.429537 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:09:27.605040 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:09:27.605002 4611 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="382d2eac-35bc-4c38-b87b-0140afd66f24" path="/var/lib/kubelet/pods/382d2eac-35bc-4c38-b87b-0140afd66f24/volumes" Oct 30 21:09:28.020995 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:09:28.020963 4611 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-single-main-1c4642/simulator-single-main-1c4642-kserve-554b47fb54-lsct9"] Oct 30 21:09:28.025922 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:09:28.025900 4611 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-1c4642/simulator-single-main-1c4642-kserve-554b47fb54-lsct9" Oct 30 21:09:28.035983 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:09:28.035949 4611 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-1c4642\"/\"default-dockercfg-jl8rr\"" Oct 30 21:09:28.036379 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:09:28.036358 4611 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-1c4642\"/\"simulator-single-main-1c4642-kserve-self-signed-certs\"" Oct 30 21:09:28.036501 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:09:28.036484 4611 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-1c4642\"/\"kube-root-ca.crt\"" Oct 30 21:09:28.036776 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:09:28.036641 4611 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-1c4642\"/\"openshift-service-ca.crt\"" Oct 30 21:09:28.040182 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:09:28.040159 4611 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-1c4642/simulator-single-main-1c4642-kserve-554b47fb54-lsct9"] Oct 30 21:09:28.112084 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:09:28.112036 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/06421a1e-3d03-416f-a3ca-389d48acc8d1-home\") pod \"simulator-single-main-1c4642-kserve-554b47fb54-lsct9\" (UID: \"06421a1e-3d03-416f-a3ca-389d48acc8d1\") " pod="e2e-models-e2e-worker-single-main-1c4642/simulator-single-main-1c4642-kserve-554b47fb54-lsct9" Oct 30 21:09:28.112242 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:09:28.112160 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/06421a1e-3d03-416f-a3ca-389d48acc8d1-dshm\") pod \"simulator-single-main-1c4642-kserve-554b47fb54-lsct9\" (UID: \"06421a1e-3d03-416f-a3ca-389d48acc8d1\") " pod="e2e-models-e2e-worker-single-main-1c4642/simulator-single-main-1c4642-kserve-554b47fb54-lsct9" Oct 30 21:09:28.112242 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:09:28.112227 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06421a1e-3d03-416f-a3ca-389d48acc8d1-tmp-dir\") pod \"simulator-single-main-1c4642-kserve-554b47fb54-lsct9\" (UID: \"06421a1e-3d03-416f-a3ca-389d48acc8d1\") " pod="e2e-models-e2e-worker-single-main-1c4642/simulator-single-main-1c4642-kserve-554b47fb54-lsct9" Oct 30 21:09:28.112334 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:09:28.112264 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hmx2g\" (UniqueName: \"kubernetes.io/projected/06421a1e-3d03-416f-a3ca-389d48acc8d1-kube-api-access-hmx2g\") pod \"simulator-single-main-1c4642-kserve-554b47fb54-lsct9\" (UID: \"06421a1e-3d03-416f-a3ca-389d48acc8d1\") " pod="e2e-models-e2e-worker-single-main-1c4642/simulator-single-main-1c4642-kserve-554b47fb54-lsct9" Oct 30 21:09:28.112334 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:09:28.112295 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/06421a1e-3d03-416f-a3ca-389d48acc8d1-tls-certs\") pod \"simulator-single-main-1c4642-kserve-554b47fb54-lsct9\" (UID: \"06421a1e-3d03-416f-a3ca-389d48acc8d1\") " pod="e2e-models-e2e-worker-single-main-1c4642/simulator-single-main-1c4642-kserve-554b47fb54-lsct9" Oct 30 21:09:28.112424 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:09:28.112391 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/06421a1e-3d03-416f-a3ca-389d48acc8d1-model-cache\") pod \"simulator-single-main-1c4642-kserve-554b47fb54-lsct9\" (UID: \"06421a1e-3d03-416f-a3ca-389d48acc8d1\") " pod="e2e-models-e2e-worker-single-main-1c4642/simulator-single-main-1c4642-kserve-554b47fb54-lsct9" Oct 30 21:09:28.129241 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:09:28.129214 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:09:28.213195 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:09:28.213164 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/06421a1e-3d03-416f-a3ca-389d48acc8d1-model-cache\") pod \"simulator-single-main-1c4642-kserve-554b47fb54-lsct9\" (UID: \"06421a1e-3d03-416f-a3ca-389d48acc8d1\") " pod="e2e-models-e2e-worker-single-main-1c4642/simulator-single-main-1c4642-kserve-554b47fb54-lsct9" Oct 30 21:09:28.213351 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:09:28.213210 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/06421a1e-3d03-416f-a3ca-389d48acc8d1-home\") pod \"simulator-single-main-1c4642-kserve-554b47fb54-lsct9\" (UID: \"06421a1e-3d03-416f-a3ca-389d48acc8d1\") " pod="e2e-models-e2e-worker-single-main-1c4642/simulator-single-main-1c4642-kserve-554b47fb54-lsct9" Oct 30 21:09:28.213351 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:09:28.213257 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/06421a1e-3d03-416f-a3ca-389d48acc8d1-dshm\") pod \"simulator-single-main-1c4642-kserve-554b47fb54-lsct9\" (UID: \"06421a1e-3d03-416f-a3ca-389d48acc8d1\") " pod="e2e-models-e2e-worker-single-main-1c4642/simulator-single-main-1c4642-kserve-554b47fb54-lsct9" Oct 30 21:09:28.213351 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:09:28.213278 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06421a1e-3d03-416f-a3ca-389d48acc8d1-tmp-dir\") pod \"simulator-single-main-1c4642-kserve-554b47fb54-lsct9\" (UID: \"06421a1e-3d03-416f-a3ca-389d48acc8d1\") " pod="e2e-models-e2e-worker-single-main-1c4642/simulator-single-main-1c4642-kserve-554b47fb54-lsct9" Oct 30 21:09:28.213351 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:09:28.213308 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hmx2g\" (UniqueName: \"kubernetes.io/projected/06421a1e-3d03-416f-a3ca-389d48acc8d1-kube-api-access-hmx2g\") pod \"simulator-single-main-1c4642-kserve-554b47fb54-lsct9\" (UID: \"06421a1e-3d03-416f-a3ca-389d48acc8d1\") " pod="e2e-models-e2e-worker-single-main-1c4642/simulator-single-main-1c4642-kserve-554b47fb54-lsct9" Oct 30 21:09:28.213351 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:09:28.213347 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/06421a1e-3d03-416f-a3ca-389d48acc8d1-tls-certs\") pod \"simulator-single-main-1c4642-kserve-554b47fb54-lsct9\" (UID: \"06421a1e-3d03-416f-a3ca-389d48acc8d1\") " pod="e2e-models-e2e-worker-single-main-1c4642/simulator-single-main-1c4642-kserve-554b47fb54-lsct9" Oct 30 21:09:28.213679 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:09:28.213632 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/06421a1e-3d03-416f-a3ca-389d48acc8d1-model-cache\") pod \"simulator-single-main-1c4642-kserve-554b47fb54-lsct9\" (UID: \"06421a1e-3d03-416f-a3ca-389d48acc8d1\") " pod="e2e-models-e2e-worker-single-main-1c4642/simulator-single-main-1c4642-kserve-554b47fb54-lsct9" Oct 30 21:09:28.213738 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:09:28.213683 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/06421a1e-3d03-416f-a3ca-389d48acc8d1-home\") pod \"simulator-single-main-1c4642-kserve-554b47fb54-lsct9\" (UID: \"06421a1e-3d03-416f-a3ca-389d48acc8d1\") " pod="e2e-models-e2e-worker-single-main-1c4642/simulator-single-main-1c4642-kserve-554b47fb54-lsct9" Oct 30 21:09:28.213738 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:09:28.213707 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06421a1e-3d03-416f-a3ca-389d48acc8d1-tmp-dir\") pod \"simulator-single-main-1c4642-kserve-554b47fb54-lsct9\" (UID: \"06421a1e-3d03-416f-a3ca-389d48acc8d1\") " pod="e2e-models-e2e-worker-single-main-1c4642/simulator-single-main-1c4642-kserve-554b47fb54-lsct9" Oct 30 21:09:28.215769 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:09:28.215742 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/06421a1e-3d03-416f-a3ca-389d48acc8d1-dshm\") pod \"simulator-single-main-1c4642-kserve-554b47fb54-lsct9\" (UID: \"06421a1e-3d03-416f-a3ca-389d48acc8d1\") " pod="e2e-models-e2e-worker-single-main-1c4642/simulator-single-main-1c4642-kserve-554b47fb54-lsct9" Oct 30 21:09:28.215901 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:09:28.215882 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/06421a1e-3d03-416f-a3ca-389d48acc8d1-tls-certs\") pod \"simulator-single-main-1c4642-kserve-554b47fb54-lsct9\" (UID: \"06421a1e-3d03-416f-a3ca-389d48acc8d1\") " pod="e2e-models-e2e-worker-single-main-1c4642/simulator-single-main-1c4642-kserve-554b47fb54-lsct9" Oct 30 21:09:28.224814 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:09:28.224783 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hmx2g\" (UniqueName: \"kubernetes.io/projected/06421a1e-3d03-416f-a3ca-389d48acc8d1-kube-api-access-hmx2g\") pod \"simulator-single-main-1c4642-kserve-554b47fb54-lsct9\" (UID: \"06421a1e-3d03-416f-a3ca-389d48acc8d1\") " pod="e2e-models-e2e-worker-single-main-1c4642/simulator-single-main-1c4642-kserve-554b47fb54-lsct9" Oct 30 21:09:28.311976 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:09:28.311729 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:09:28.338236 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:09:28.338201 4611 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-1c4642/simulator-single-main-1c4642-kserve-554b47fb54-lsct9" Oct 30 21:09:28.624412 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:09:28.624384 4611 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-1c4642/simulator-single-main-1c4642-kserve-554b47fb54-lsct9"] Oct 30 21:09:28.625920 ip-10-0-133-192 kubenswrapper[4611]: W1030 21:09:28.625888 4611 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod06421a1e_3d03_416f_a3ca_389d48acc8d1.slice/crio-ff03a97efb667bcc5a9222cd0693c772eb670671e24d676b0f922f955dec621f WatchSource:0}: Error finding container ff03a97efb667bcc5a9222cd0693c772eb670671e24d676b0f922f955dec621f: Status 404 returned error can't find the container with id ff03a97efb667bcc5a9222cd0693c772eb670671e24d676b0f922f955dec621f Oct 30 21:09:28.986207 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:09:28.986131 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-1c4642/simulator-single-main-1c4642-kserve-554b47fb54-lsct9" event={"ID":"06421a1e-3d03-416f-a3ca-389d48acc8d1","Type":"ContainerStarted","Data":"527eb80262d8066d3cca8fd0f89c8a47d1e7f0b7c55f287888f7234c25cd3d20"} Oct 30 21:09:28.986207 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:09:28.986181 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-1c4642/simulator-single-main-1c4642-kserve-554b47fb54-lsct9" event={"ID":"06421a1e-3d03-416f-a3ca-389d48acc8d1","Type":"ContainerStarted","Data":"ff03a97efb667bcc5a9222cd0693c772eb670671e24d676b0f922f955dec621f"} Oct 30 21:09:29.059006 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:09:29.058937 4611 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-single-main-1c4642/simulator-single-main-1c4642-kserve-554b47fb54-lsct9" podStartSLOduration=2.058920028 podStartE2EDuration="2.058920028s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:09:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:09:29.054994465 +0000 UTC m=+1823.589021084" watchObservedRunningTime="2026-10-30 21:09:29.058920028 +0000 UTC m=+1823.592946645" Oct 30 21:09:29.114146 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:09:29.114110 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:09:29.338584 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:09:29.338466 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-single-main-1c4642/simulator-single-main-1c4642-kserve-554b47fb54-lsct9" Oct 30 21:09:29.878983 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:09:29.878945 4611 patch_prober.go:29] interesting pod/e2e-worker-single-main-1c4642-openshift-default-6784bc96d9fswv6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" start-of-body= Oct 30 21:09:29.879146 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:09:29.878998 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-single-main-1c4642-openshift-default-6784bc96d9fswv6" podUID="4ef2c7e4-e9e7-4d0d-91b0-026b1f9cef76" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" Oct 30 21:09:29.914969 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:09:29.914935 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-1c4642-64f579d6bttr5r" Oct 30 21:09:29.915127 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:09:29.915023 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-1c4642-7bf468k4cr" Oct 30 21:09:29.995008 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:09:29.994978 4611 generic.go:356] "Generic (PLEG): container finished" podID="4ef2c7e4-e9e7-4d0d-91b0-026b1f9cef76" containerID="0df373cd1e01b6da9c8183c930636c153b5a820d1032218de833214526580bb1" exitCode=0 Oct 30 21:09:29.995202 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:09:29.995049 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-1c4642-openshift-default-6784bc96d9fswv6" event={"ID":"4ef2c7e4-e9e7-4d0d-91b0-026b1f9cef76","Type":"ContainerDied","Data":"0df373cd1e01b6da9c8183c930636c153b5a820d1032218de833214526580bb1"} Oct 30 21:09:29.995456 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:09:29.995428 4611 scope.go:122] "RemoveContainer" containerID="0df373cd1e01b6da9c8183c930636c153b5a820d1032218de833214526580bb1" Oct 30 21:09:29.996794 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:09:29.996760 4611 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:09:29.996916 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:09:29.996843 4611 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:09:29.996916 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:09:29.996897 4611 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:09:30.019480 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:09:30.019401 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:09:30.351684 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:09:30.351641 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:09:31.002470 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:09:31.002434 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-1c4642-openshift-default-6784bc96d9fswv6" event={"ID":"4ef2c7e4-e9e7-4d0d-91b0-026b1f9cef76","Type":"ContainerStarted","Data":"c16ef57d03d7bf68c683c491e197e9a87646dc42ba870562cc14671f25f32e68"} Oct 30 21:09:31.357198 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:09:31.357151 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:09:31.832817 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:09:31.832742 4611 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-1c4642-openshift-default-6784bc96d9fswv6" Oct 30 21:09:31.832969 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:09:31.832908 4611 patch_prober.go:29] interesting pod/e2e-worker-single-main-1c4642-openshift-default-6784bc96d9fswv6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" start-of-body= Oct 30 21:09:31.832969 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:09:31.832956 4611 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-1c4642-openshift-default-6784bc96d9fswv6" podUID="4ef2c7e4-e9e7-4d0d-91b0-026b1f9cef76" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" Oct 30 21:09:32.175124 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:09:32.175084 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:09:32.832795 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:09:32.832764 4611 patch_prober.go:29] interesting pod/e2e-worker-single-main-1c4642-openshift-default-6784bc96d9fswv6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" start-of-body= Oct 30 21:09:32.833189 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:09:32.832814 4611 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-1c4642-openshift-default-6784bc96d9fswv6" podUID="4ef2c7e4-e9e7-4d0d-91b0-026b1f9cef76" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" Oct 30 21:09:32.933144 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:09:32.933097 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:09:33.698254 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:09:33.698204 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:09:33.832896 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:09:33.832863 4611 patch_prober.go:29] interesting pod/e2e-worker-single-main-1c4642-openshift-default-6784bc96d9fswv6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" start-of-body= Oct 30 21:09:33.833324 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:09:33.832917 4611 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-1c4642-openshift-default-6784bc96d9fswv6" podUID="4ef2c7e4-e9e7-4d0d-91b0-026b1f9cef76" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" Oct 30 21:09:33.920642 ip-10-0-133-192 kubenswrapper[4611]: E1030 21:09:33.920602 4611 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hwzc6" containerName="extractor" Oct 30 21:09:34.387431 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:09:34.387392 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:09:34.832897 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:09:34.832818 4611 patch_prober.go:29] interesting pod/e2e-worker-single-main-1c4642-openshift-default-6784bc96d9fswv6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" start-of-body= Oct 30 21:09:34.832897 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:09:34.832873 4611 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-1c4642-openshift-default-6784bc96d9fswv6" podUID="4ef2c7e4-e9e7-4d0d-91b0-026b1f9cef76" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" Oct 30 21:09:35.197675 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:09:35.197633 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:09:35.832204 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:09:35.832167 4611 patch_prober.go:29] interesting pod/e2e-worker-single-main-1c4642-openshift-default-6784bc96d9fswv6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" start-of-body= Oct 30 21:09:35.832401 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:09:35.832223 4611 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-1c4642-openshift-default-6784bc96d9fswv6" podUID="4ef2c7e4-e9e7-4d0d-91b0-026b1f9cef76" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" Oct 30 21:09:35.906453 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:09:35.906413 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:09:36.605864 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:09:36.605822 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:09:36.836490 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:09:36.836462 4611 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-1c4642-openshift-default-6784bc96d9fswv6" Oct 30 21:09:36.836947 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:09:36.836918 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-1c4642-openshift-default-6784bc96d9fswv6" Oct 30 21:09:36.837535 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:09:36.837501 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-1c4642-openshift-default-6784bc96d9fswv6" Oct 30 21:09:37.309651 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:09:37.309591 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:09:37.990100 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:09:37.990060 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:09:38.339071 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:09:38.338985 4611 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-single-main-1c4642/simulator-single-main-1c4642-kserve-554b47fb54-lsct9" Oct 30 21:09:38.351466 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:09:38.351442 4611 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-single-main-1c4642/simulator-single-main-1c4642-kserve-554b47fb54-lsct9" Oct 30 21:09:38.615123 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:09:38.615078 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:09:39.057650 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:09:39.057569 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-single-main-1c4642/simulator-single-main-1c4642-kserve-554b47fb54-lsct9" Oct 30 21:09:39.303221 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:09:39.303164 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:09:40.104716 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:09:40.104679 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:09:40.931856 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:09:40.931818 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:09:41.511575 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:09:41.511537 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:09:42.210791 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:09:42.210759 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:09:43.037812 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:09:43.037779 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:09:43.801916 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:09:43.801884 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:09:44.733975 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:09:44.733938 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:09:45.438287 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:09:45.438255 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:09:46.032638 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:09:46.032608 4611 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-kvgmg"] Oct 30 21:09:46.037969 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:09:46.037943 4611 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-kvgmg"] Oct 30 21:09:46.227607 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:09:46.227564 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:09:47.131102 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:09:47.131062 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:09:47.605107 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:09:47.605074 4611 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3909d445-54f4-43ae-b2f4-1bae843188d8" path="/var/lib/kubelet/pods/3909d445-54f4-43ae-b2f4-1bae843188d8/volumes" Oct 30 21:09:47.843160 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:09:47.843123 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:09:48.514091 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:09:48.514058 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:09:49.360735 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:09:49.360699 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:09:49.514405 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:09:49.514363 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:09:50.103618 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:09:50.103580 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:09:50.796578 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:09:50.796538 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:09:51.739199 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:09:51.739158 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:09:52.439610 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:09:52.439574 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:09:53.161792 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:09:53.161713 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:09:53.955847 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:09:53.955814 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:09:54.739555 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:09:54.739498 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:09:55.428321 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:09:55.428286 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:09:56.098790 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:09:56.098743 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:09:56.778383 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:09:56.778349 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:09:57.447049 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:09:57.447013 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:09:58.026699 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:09:58.026667 4611 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-w5w78"] Oct 30 21:09:58.029204 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:09:58.029180 4611 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-w5w78"] Oct 30 21:09:58.116738 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:09:58.116698 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:09:58.361388 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:09:58.361357 4611 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a047fdad-c4gv6"] Oct 30 21:09:58.366635 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:09:58.366617 4611 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a047fdad-c4gv6" Oct 30 21:09:58.372347 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:09:58.372320 4611 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7hf8r\"" Oct 30 21:09:58.374639 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:09:58.374606 4611 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:09:58.376493 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:09:58.376471 4611 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a047fdad-c4gv6"] Oct 30 21:09:58.378320 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:09:58.378300 4611 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:09:58.378439 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:09:58.378316 4611 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:09:58.440937 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:09:58.440908 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xhqj5\" (UniqueName: \"kubernetes.io/projected/5567ed88-fc6d-417e-b01c-c079d380ecef-kube-api-access-xhqj5\") pod \"maas-api-revoke-sub-a047fdad-c4gv6\" (UID: \"5567ed88-fc6d-417e-b01c-c079d380ecef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a047fdad-c4gv6" Oct 30 21:09:58.441042 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:09:58.440989 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5567ed88-fc6d-417e-b01c-c079d380ecef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a047fdad-c4gv6\" (UID: \"5567ed88-fc6d-417e-b01c-c079d380ecef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a047fdad-c4gv6" Oct 30 21:09:58.441311 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:09:58.441294 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5567ed88-fc6d-417e-b01c-c079d380ecef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a047fdad-c4gv6\" (UID: \"5567ed88-fc6d-417e-b01c-c079d380ecef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a047fdad-c4gv6" Oct 30 21:09:58.541926 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:09:58.541901 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5567ed88-fc6d-417e-b01c-c079d380ecef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a047fdad-c4gv6\" (UID: \"5567ed88-fc6d-417e-b01c-c079d380ecef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a047fdad-c4gv6" Oct 30 21:09:58.542013 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:09:58.541938 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xhqj5\" (UniqueName: \"kubernetes.io/projected/5567ed88-fc6d-417e-b01c-c079d380ecef-kube-api-access-xhqj5\") pod \"maas-api-revoke-sub-a047fdad-c4gv6\" (UID: \"5567ed88-fc6d-417e-b01c-c079d380ecef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a047fdad-c4gv6" Oct 30 21:09:58.542144 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:09:58.542124 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5567ed88-fc6d-417e-b01c-c079d380ecef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a047fdad-c4gv6\" (UID: \"5567ed88-fc6d-417e-b01c-c079d380ecef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a047fdad-c4gv6" Oct 30 21:09:58.542801 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:09:58.542782 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5567ed88-fc6d-417e-b01c-c079d380ecef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a047fdad-c4gv6\" (UID: \"5567ed88-fc6d-417e-b01c-c079d380ecef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a047fdad-c4gv6" Oct 30 21:09:58.544287 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:09:58.544270 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5567ed88-fc6d-417e-b01c-c079d380ecef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a047fdad-c4gv6\" (UID: \"5567ed88-fc6d-417e-b01c-c079d380ecef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a047fdad-c4gv6" Oct 30 21:09:58.555584 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:09:58.555566 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xhqj5\" (UniqueName: \"kubernetes.io/projected/5567ed88-fc6d-417e-b01c-c079d380ecef-kube-api-access-xhqj5\") pod \"maas-api-revoke-sub-a047fdad-c4gv6\" (UID: \"5567ed88-fc6d-417e-b01c-c079d380ecef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a047fdad-c4gv6" Oct 30 21:09:58.682197 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:09:58.682141 4611 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a047fdad-c4gv6" Oct 30 21:09:58.938880 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:09:58.938802 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:09:58.961706 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:09:58.961683 4611 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a047fdad-c4gv6"] Oct 30 21:09:58.963452 ip-10-0-133-192 kubenswrapper[4611]: W1030 21:09:58.963417 4611 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5567ed88_fc6d_417e_b01c_c079d380ecef.slice/crio-a9104a7b473ed99bcdffe5ba23c2f74d172aeb92a9e09b20a15a01120f5b4c71 WatchSource:0}: Error finding container a9104a7b473ed99bcdffe5ba23c2f74d172aeb92a9e09b20a15a01120f5b4c71: Status 404 returned error can't find the container with id a9104a7b473ed99bcdffe5ba23c2f74d172aeb92a9e09b20a15a01120f5b4c71 Oct 30 21:09:59.160332 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:09:59.160305 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a047fdad-c4gv6" event={"ID":"5567ed88-fc6d-417e-b01c-c079d380ecef","Type":"ContainerStarted","Data":"a9104a7b473ed99bcdffe5ba23c2f74d172aeb92a9e09b20a15a01120f5b4c71"} Oct 30 21:09:59.608830 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:09:59.608793 4611 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="235db0c6-6afd-470d-8a49-3cea7316a82a" path="/var/lib/kubelet/pods/235db0c6-6afd-470d-8a49-3cea7316a82a/volumes" Oct 30 21:09:59.624025 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:09:59.623996 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:10:00.167457 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:10:00.167427 4611 generic.go:356] "Generic (PLEG): container finished" podID="5567ed88-fc6d-417e-b01c-c079d380ecef" containerID="b1e7867143d9a27875cda89ed6b1d62a34f3b2b9fce1b36c4d8936fa43fa7c1c" exitCode=0 Oct 30 21:10:00.167898 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:10:00.167536 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a047fdad-c4gv6" event={"ID":"5567ed88-fc6d-417e-b01c-c079d380ecef","Type":"ContainerDied","Data":"b1e7867143d9a27875cda89ed6b1d62a34f3b2b9fce1b36c4d8936fa43fa7c1c"} Oct 30 21:10:00.369925 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:10:00.369887 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:10:01.023466 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:10:01.023429 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:10:01.308762 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:10:01.308737 4611 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a047fdad-c4gv6" Oct 30 21:10:01.370450 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:10:01.370418 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5567ed88-fc6d-417e-b01c-c079d380ecef-kube-api-access-xhqj5\" (UniqueName: \"kubernetes.io/projected/5567ed88-fc6d-417e-b01c-c079d380ecef-kube-api-access-xhqj5\") pod \"5567ed88-fc6d-417e-b01c-c079d380ecef\" (UID: \"5567ed88-fc6d-417e-b01c-c079d380ecef\") " Oct 30 21:10:01.370634 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:10:01.370537 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5567ed88-fc6d-417e-b01c-c079d380ecef-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5567ed88-fc6d-417e-b01c-c079d380ecef-maas-api-openshift-ca\") pod \"5567ed88-fc6d-417e-b01c-c079d380ecef\" (UID: \"5567ed88-fc6d-417e-b01c-c079d380ecef\") " Oct 30 21:10:01.370634 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:10:01.370613 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5567ed88-fc6d-417e-b01c-c079d380ecef-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5567ed88-fc6d-417e-b01c-c079d380ecef-maas-api-xks-ca\") pod \"5567ed88-fc6d-417e-b01c-c079d380ecef\" (UID: \"5567ed88-fc6d-417e-b01c-c079d380ecef\") " Oct 30 21:10:01.370912 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:10:01.370885 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5567ed88-fc6d-417e-b01c-c079d380ecef-maas-api-openshift-ca" pod "5567ed88-fc6d-417e-b01c-c079d380ecef" (UID: "5567ed88-fc6d-417e-b01c-c079d380ecef"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:10:01.372643 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:10:01.372614 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5567ed88-fc6d-417e-b01c-c079d380ecef-maas-api-xks-ca" pod "5567ed88-fc6d-417e-b01c-c079d380ecef" (UID: "5567ed88-fc6d-417e-b01c-c079d380ecef"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:10:01.372781 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:10:01.372713 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5567ed88-fc6d-417e-b01c-c079d380ecef-kube-api-access-xhqj5" pod "5567ed88-fc6d-417e-b01c-c079d380ecef" (UID: "5567ed88-fc6d-417e-b01c-c079d380ecef"). InnerVolumeSpecName "kube-api-access-xhqj5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:10:01.472272 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:10:01.472245 4611 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5567ed88-fc6d-417e-b01c-c079d380ecef-maas-api-xks-ca\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:10:01.472272 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:10:01.472270 4611 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xhqj5\" (UniqueName: \"kubernetes.io/projected/5567ed88-fc6d-417e-b01c-c079d380ecef-kube-api-access-xhqj5\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:10:01.472423 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:10:01.472282 4611 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5567ed88-fc6d-417e-b01c-c079d380ecef-maas-api-openshift-ca\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:10:01.709628 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:10:01.709589 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:10:02.179565 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:10:02.179535 4611 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a047fdad-c4gv6" Oct 30 21:10:02.179763 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:10:02.179564 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a047fdad-c4gv6" event={"ID":"5567ed88-fc6d-417e-b01c-c079d380ecef","Type":"ContainerDied","Data":"a9104a7b473ed99bcdffe5ba23c2f74d172aeb92a9e09b20a15a01120f5b4c71"} Oct 30 21:10:02.179763 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:10:02.179614 4611 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a9104a7b473ed99bcdffe5ba23c2f74d172aeb92a9e09b20a15a01120f5b4c71" Oct 30 21:10:02.426344 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:10:02.426311 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:10:03.107803 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:10:03.107761 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:10:03.919242 ip-10-0-133-192 kubenswrapper[4611]: E1030 21:10:03.919199 4611 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hwzc6" containerName="extractor" Oct 30 21:10:03.930113 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:10:03.930080 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:10:04.594691 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:10:04.594648 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:10:05.304908 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:10:05.304870 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:10:06.011373 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:10:06.011326 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:10:06.706720 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:10:06.706680 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:10:07.391974 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:10:07.391937 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:10:08.011061 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:10:08.011001 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:10:08.138653 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:10:08.138624 4611 scope.go:122] "RemoveContainer" containerID="dcb78882b314ca558a12ad39710e0f92ac64daa36ce2e361e5af22c570a885dd" Oct 30 21:10:08.817921 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:10:08.817860 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:10:09.304895 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:10:09.304860 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:10:09.403692 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:10:09.403653 4611 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-1c4642-9bfjx"] Oct 30 21:10:09.404102 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:10:09.404088 4611 cpu_manager.go:414] "containerMap: removing container" podUID="5567ed88-fc6d-417e-b01c-c079d380ecef" containerName="revoke-keys" Oct 30 21:10:09.404159 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:10:09.404103 4611 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5567ed88-fc6d-417e-b01c-c079d380ecef" containerName="revoke-keys" Oct 30 21:10:09.404202 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:10:09.404181 4611 memory_manager.go:365] "RemoveStaleState removing state" podUID="5567ed88-fc6d-417e-b01c-c079d380ecef" containerName="revoke-keys" Oct 30 21:10:09.408745 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:10:09.408727 4611 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-1c4642-9bfjx" Oct 30 21:10:09.415896 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:10:09.415870 4611 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:10:09.416024 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:10:09.415936 4611 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7hf8r\"" Oct 30 21:10:09.419038 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:10:09.419017 4611 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-1c4642-9bfjx"] Oct 30 21:10:09.433887 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:10:09.433868 4611 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:10:09.433993 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:10:09.433928 4611 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:10:09.554656 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:10:09.554626 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d3aa89f2-0881-437c-9750-c159ee74ee34-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-1c4642-9bfjx\" (UID: \"d3aa89f2-0881-437c-9750-c159ee74ee34\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-1c4642-9bfjx" Oct 30 21:10:09.554805 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:10:09.554754 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gdqz5\" (UniqueName: \"kubernetes.io/projected/d3aa89f2-0881-437c-9750-c159ee74ee34-kube-api-access-gdqz5\") pod \"maas-api-revoke-keys-e2e-worker-single-main-1c4642-9bfjx\" (UID: \"d3aa89f2-0881-437c-9750-c159ee74ee34\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-1c4642-9bfjx" Oct 30 21:10:09.554854 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:10:09.554808 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d3aa89f2-0881-437c-9750-c159ee74ee34-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-1c4642-9bfjx\" (UID: \"d3aa89f2-0881-437c-9750-c159ee74ee34\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-1c4642-9bfjx" Oct 30 21:10:09.602408 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:10:09.602379 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:10:09.655338 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:10:09.655308 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gdqz5\" (UniqueName: \"kubernetes.io/projected/d3aa89f2-0881-437c-9750-c159ee74ee34-kube-api-access-gdqz5\") pod \"maas-api-revoke-keys-e2e-worker-single-main-1c4642-9bfjx\" (UID: \"d3aa89f2-0881-437c-9750-c159ee74ee34\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-1c4642-9bfjx" Oct 30 21:10:09.655493 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:10:09.655364 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d3aa89f2-0881-437c-9750-c159ee74ee34-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-1c4642-9bfjx\" (UID: \"d3aa89f2-0881-437c-9750-c159ee74ee34\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-1c4642-9bfjx" Oct 30 21:10:09.655493 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:10:09.655403 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d3aa89f2-0881-437c-9750-c159ee74ee34-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-1c4642-9bfjx\" (UID: \"d3aa89f2-0881-437c-9750-c159ee74ee34\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-1c4642-9bfjx" Oct 30 21:10:09.656054 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:10:09.656032 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d3aa89f2-0881-437c-9750-c159ee74ee34-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-1c4642-9bfjx\" (UID: \"d3aa89f2-0881-437c-9750-c159ee74ee34\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-1c4642-9bfjx" Oct 30 21:10:09.657735 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:10:09.657714 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d3aa89f2-0881-437c-9750-c159ee74ee34-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-1c4642-9bfjx\" (UID: \"d3aa89f2-0881-437c-9750-c159ee74ee34\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-1c4642-9bfjx" Oct 30 21:10:09.677273 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:10:09.677250 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gdqz5\" (UniqueName: \"kubernetes.io/projected/d3aa89f2-0881-437c-9750-c159ee74ee34-kube-api-access-gdqz5\") pod \"maas-api-revoke-keys-e2e-worker-single-main-1c4642-9bfjx\" (UID: \"d3aa89f2-0881-437c-9750-c159ee74ee34\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-1c4642-9bfjx" Oct 30 21:10:09.722045 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:10:09.722025 4611 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-1c4642-9bfjx" Oct 30 21:10:09.999927 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:10:09.999903 4611 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-1c4642-9bfjx"] Oct 30 21:10:10.001412 ip-10-0-133-192 kubenswrapper[4611]: W1030 21:10:10.001389 4611 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3aa89f2_0881_437c_9750_c159ee74ee34.slice/crio-d1afa77916e2eefd0831f9a51909fa7c94a637ded5800dceaf014f558529daff WatchSource:0}: Error finding container d1afa77916e2eefd0831f9a51909fa7c94a637ded5800dceaf014f558529daff: Status 404 returned error can't find the container with id d1afa77916e2eefd0831f9a51909fa7c94a637ded5800dceaf014f558529daff Oct 30 21:10:10.224860 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:10:10.224831 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-1c4642-9bfjx" event={"ID":"d3aa89f2-0881-437c-9750-c159ee74ee34","Type":"ContainerStarted","Data":"d1afa77916e2eefd0831f9a51909fa7c94a637ded5800dceaf014f558529daff"} Oct 30 21:10:10.299061 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:10:10.299025 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:10:11.090564 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:10:11.090446 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:10:11.233091 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:10:11.233042 4611 generic.go:356] "Generic (PLEG): container finished" podID="d3aa89f2-0881-437c-9750-c159ee74ee34" containerID="16fc47839a8a57922cb8ba073a2519624217f1461bdcc7ff91233fd52f6037cc" exitCode=0 Oct 30 21:10:11.233265 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:10:11.233113 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-1c4642-9bfjx" event={"ID":"d3aa89f2-0881-437c-9750-c159ee74ee34","Type":"ContainerDied","Data":"16fc47839a8a57922cb8ba073a2519624217f1461bdcc7ff91233fd52f6037cc"} Oct 30 21:10:11.885954 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:10:11.885913 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:10:12.373823 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:10:12.373800 4611 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-1c4642-9bfjx" Oct 30 21:10:12.488126 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:10:12.488006 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d3aa89f2-0881-437c-9750-c159ee74ee34-kube-api-access-gdqz5\" (UniqueName: \"kubernetes.io/projected/d3aa89f2-0881-437c-9750-c159ee74ee34-kube-api-access-gdqz5\") pod \"d3aa89f2-0881-437c-9750-c159ee74ee34\" (UID: \"d3aa89f2-0881-437c-9750-c159ee74ee34\") " Oct 30 21:10:12.488126 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:10:12.488078 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d3aa89f2-0881-437c-9750-c159ee74ee34-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d3aa89f2-0881-437c-9750-c159ee74ee34-maas-api-xks-ca\") pod \"d3aa89f2-0881-437c-9750-c159ee74ee34\" (UID: \"d3aa89f2-0881-437c-9750-c159ee74ee34\") " Oct 30 21:10:12.488126 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:10:12.488113 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d3aa89f2-0881-437c-9750-c159ee74ee34-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d3aa89f2-0881-437c-9750-c159ee74ee34-maas-api-openshift-ca\") pod \"d3aa89f2-0881-437c-9750-c159ee74ee34\" (UID: \"d3aa89f2-0881-437c-9750-c159ee74ee34\") " Oct 30 21:10:12.489022 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:10:12.488968 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3aa89f2-0881-437c-9750-c159ee74ee34-maas-api-openshift-ca" pod "d3aa89f2-0881-437c-9750-c159ee74ee34" (UID: "d3aa89f2-0881-437c-9750-c159ee74ee34"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:10:12.491405 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:10:12.491334 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d3aa89f2-0881-437c-9750-c159ee74ee34-maas-api-xks-ca" pod "d3aa89f2-0881-437c-9750-c159ee74ee34" (UID: "d3aa89f2-0881-437c-9750-c159ee74ee34"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:10:12.491905 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:10:12.491883 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3aa89f2-0881-437c-9750-c159ee74ee34-kube-api-access-gdqz5" pod "d3aa89f2-0881-437c-9750-c159ee74ee34" (UID: "d3aa89f2-0881-437c-9750-c159ee74ee34"). InnerVolumeSpecName "kube-api-access-gdqz5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:10:12.533544 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:10:12.533469 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:10:12.589455 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:10:12.589428 4611 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gdqz5\" (UniqueName: \"kubernetes.io/projected/d3aa89f2-0881-437c-9750-c159ee74ee34-kube-api-access-gdqz5\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:10:12.589455 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:10:12.589453 4611 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d3aa89f2-0881-437c-9750-c159ee74ee34-maas-api-xks-ca\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:10:12.589625 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:10:12.589465 4611 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d3aa89f2-0881-437c-9750-c159ee74ee34-maas-api-openshift-ca\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:10:13.110479 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:10:13.110445 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:10:13.246199 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:10:13.246165 4611 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-1c4642-9bfjx" Oct 30 21:10:13.246354 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:10:13.246198 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-1c4642-9bfjx" event={"ID":"d3aa89f2-0881-437c-9750-c159ee74ee34","Type":"ContainerDied","Data":"d1afa77916e2eefd0831f9a51909fa7c94a637ded5800dceaf014f558529daff"} Oct 30 21:10:13.246354 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:10:13.246253 4611 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d1afa77916e2eefd0831f9a51909fa7c94a637ded5800dceaf014f558529daff" Oct 30 21:10:13.800543 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:10:13.800387 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:10:14.211693 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:10:14.211652 4611 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-1c4642/simulator-single-main-1c4642-kserve-554b47fb54-lsct9"] Oct 30 21:10:14.353166 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:10:14.353116 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:10:14.500773 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:10:14.500652 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:10:15.208550 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:10:15.206933 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:10:15.885732 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:10:15.885678 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:10:16.594148 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:10:16.594100 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:10:17.299556 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:10:17.299504 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:10:17.919846 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:10:17.919810 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:10:18.730424 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:10:18.730387 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:10:19.392931 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:10:19.392525 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:10:19.553018 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:10:19.552986 4611 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-1c4642-64f579d6bttr5r"] Oct 30 21:10:19.563466 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:10:19.563429 4611 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-1c4642-7bf468k4cr"] Oct 30 21:10:19.674555 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:10:19.674447 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:10:19.824282 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:10:19.824242 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:10:19.973941 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:10:19.973852 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:10:20.122735 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:10:20.122686 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:10:20.164442 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:10:20.164401 4611 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-1c4642-9bfjx"] Oct 30 21:10:20.167246 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:10:20.167223 4611 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-1c4642-9bfjx"] Oct 30 21:10:20.267792 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:10:20.267595 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:10:20.752342 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:10:20.752309 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:10:21.604302 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:10:21.604268 4611 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d3aa89f2-0881-437c-9750-c159ee74ee34" path="/var/lib/kubelet/pods/d3aa89f2-0881-437c-9750-c159ee74ee34/volumes" Oct 30 21:10:23.864132 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:10:23.864090 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:10:24.027325 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:10:24.027288 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:10:24.553735 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:10:24.553689 4611 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-1c4642-64f579d6bttr5r" podUID="fba16a3f-2530-4780-be32-2f72358c7e47" containerName="payload-processing" containerID="cri-o://4a3ed81adc6fc5949da33cf3a449b845d63e35f7fa03ddbde33c3bbaab929c39" gracePeriod=25 Oct 30 21:10:24.564229 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:10:24.564196 4611 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-1c4642-7bf468k4cr" podUID="ec926cab-0bca-4a28-bada-cc067d70d3b9" containerName="payload-pre-processing" containerID="cri-o://dfc52b6e28060f61f85d4d94fd60651e6c4e19668c8791fed953be870fd9a82f" gracePeriod=25 Oct 30 21:10:26.811421 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:10:26.811384 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:10:26.963648 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:10:26.963611 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:10:27.112693 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:10:27.112659 4611 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-1c4642-openshift-default-6784bc96d9fswv6"] Oct 30 21:10:27.112888 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:10:27.112866 4611 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-single-main-1c4642-openshift-default-6784bc96d9fswv6" podUID="4ef2c7e4-e9e7-4d0d-91b0-026b1f9cef76" containerName="istio-proxy" containerID="cri-o://c16ef57d03d7bf68c683c491e197e9a87646dc42ba870562cc14671f25f32e68" gracePeriod=30 Oct 30 21:10:27.195193 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:10:27.195156 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:10:28.371682 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:10:28.371645 4611 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-hjrht"] Oct 30 21:10:28.378985 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:10:28.378960 4611 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-hjrht"] Oct 30 21:10:28.396048 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:10:28.396024 4611 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8ms7c"] Oct 30 21:10:28.396412 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:10:28.396400 4611 cpu_manager.go:414] "containerMap: removing container" podUID="d3aa89f2-0881-437c-9750-c159ee74ee34" containerName="revoke-keys" Oct 30 21:10:28.396462 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:10:28.396413 4611 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d3aa89f2-0881-437c-9750-c159ee74ee34" containerName="revoke-keys" Oct 30 21:10:28.396533 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:10:28.396523 4611 memory_manager.go:365] "RemoveStaleState removing state" podUID="d3aa89f2-0881-437c-9750-c159ee74ee34" containerName="revoke-keys" Oct 30 21:10:28.400090 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:10:28.400075 4611 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8ms7c" Oct 30 21:10:28.406288 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:10:28.406263 4611 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:10:28.407894 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:10:28.407702 4611 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:10:28.407894 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:10:28.407821 4611 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7hf8r\"" Oct 30 21:10:28.407894 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:10:28.407869 4611 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:10:28.410202 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:10:28.410179 4611 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8ms7c"] Oct 30 21:10:28.439234 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:10:28.439198 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2648a43f-15d6-4fa1-9111-8188e578075b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-8ms7c\" (UID: \"2648a43f-15d6-4fa1-9111-8188e578075b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8ms7c" Oct 30 21:10:28.439399 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:10:28.439381 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2648a43f-15d6-4fa1-9111-8188e578075b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-8ms7c\" (UID: \"2648a43f-15d6-4fa1-9111-8188e578075b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8ms7c" Oct 30 21:10:28.439447 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:10:28.439414 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zqd88\" (UniqueName: \"kubernetes.io/projected/2648a43f-15d6-4fa1-9111-8188e578075b-kube-api-access-zqd88\") pod \"maas-api-revoke-sub-3d4156f1-8ms7c\" (UID: \"2648a43f-15d6-4fa1-9111-8188e578075b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8ms7c" Oct 30 21:10:28.540493 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:10:28.540460 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2648a43f-15d6-4fa1-9111-8188e578075b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-8ms7c\" (UID: \"2648a43f-15d6-4fa1-9111-8188e578075b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8ms7c" Oct 30 21:10:28.540676 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:10:28.540605 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2648a43f-15d6-4fa1-9111-8188e578075b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-8ms7c\" (UID: \"2648a43f-15d6-4fa1-9111-8188e578075b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8ms7c" Oct 30 21:10:28.540676 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:10:28.540633 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zqd88\" (UniqueName: \"kubernetes.io/projected/2648a43f-15d6-4fa1-9111-8188e578075b-kube-api-access-zqd88\") pod \"maas-api-revoke-sub-3d4156f1-8ms7c\" (UID: \"2648a43f-15d6-4fa1-9111-8188e578075b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8ms7c" Oct 30 21:10:28.541095 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:10:28.541075 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2648a43f-15d6-4fa1-9111-8188e578075b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-8ms7c\" (UID: \"2648a43f-15d6-4fa1-9111-8188e578075b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8ms7c" Oct 30 21:10:28.542896 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:10:28.542872 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2648a43f-15d6-4fa1-9111-8188e578075b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-8ms7c\" (UID: \"2648a43f-15d6-4fa1-9111-8188e578075b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8ms7c" Oct 30 21:10:28.552366 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:10:28.552342 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zqd88\" (UniqueName: \"kubernetes.io/projected/2648a43f-15d6-4fa1-9111-8188e578075b-kube-api-access-zqd88\") pod \"maas-api-revoke-sub-3d4156f1-8ms7c\" (UID: \"2648a43f-15d6-4fa1-9111-8188e578075b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8ms7c" Oct 30 21:10:28.714429 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:10:28.714361 4611 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8ms7c" Oct 30 21:10:28.990797 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:10:28.990763 4611 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8ms7c"] Oct 30 21:10:28.992656 ip-10-0-133-192 kubenswrapper[4611]: W1030 21:10:28.992626 4611 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2648a43f_15d6_4fa1_9111_8188e578075b.slice/crio-07d65238d224ab07f455520887e877aad82a6ecae299b08f96a8ba85841c58ff WatchSource:0}: Error finding container 07d65238d224ab07f455520887e877aad82a6ecae299b08f96a8ba85841c58ff: Status 404 returned error can't find the container with id 07d65238d224ab07f455520887e877aad82a6ecae299b08f96a8ba85841c58ff Oct 30 21:10:29.231482 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:10:29.231431 4611 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-single-main-1c4642/simulator-single-main-1c4642-kserve-554b47fb54-lsct9" podUID="06421a1e-3d03-416f-a3ca-389d48acc8d1" containerName="main" containerID="cri-o://527eb80262d8066d3cca8fd0f89c8a47d1e7f0b7c55f287888f7234c25cd3d20" gracePeriod=45 Oct 30 21:10:29.343696 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:10:29.343659 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8ms7c" event={"ID":"2648a43f-15d6-4fa1-9111-8188e578075b","Type":"ContainerStarted","Data":"07d65238d224ab07f455520887e877aad82a6ecae299b08f96a8ba85841c58ff"} Oct 30 21:10:29.467307 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:10:29.467282 4611 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-1c4642/simulator-single-main-1c4642-kserve-554b47fb54-lsct9" Oct 30 21:10:29.553761 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:10:29.550783 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06421a1e-3d03-416f-a3ca-389d48acc8d1-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06421a1e-3d03-416f-a3ca-389d48acc8d1-tmp-dir\") pod \"06421a1e-3d03-416f-a3ca-389d48acc8d1\" (UID: \"06421a1e-3d03-416f-a3ca-389d48acc8d1\") " Oct 30 21:10:29.553761 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:10:29.550866 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06421a1e-3d03-416f-a3ca-389d48acc8d1-dshm\" (UniqueName: \"kubernetes.io/empty-dir/06421a1e-3d03-416f-a3ca-389d48acc8d1-dshm\") pod \"06421a1e-3d03-416f-a3ca-389d48acc8d1\" (UID: \"06421a1e-3d03-416f-a3ca-389d48acc8d1\") " Oct 30 21:10:29.553761 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:10:29.550902 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/06421a1e-3d03-416f-a3ca-389d48acc8d1-tls-certs\" (UniqueName: \"kubernetes.io/secret/06421a1e-3d03-416f-a3ca-389d48acc8d1-tls-certs\") pod \"06421a1e-3d03-416f-a3ca-389d48acc8d1\" (UID: \"06421a1e-3d03-416f-a3ca-389d48acc8d1\") " Oct 30 21:10:29.553761 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:10:29.551010 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06421a1e-3d03-416f-a3ca-389d48acc8d1-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/06421a1e-3d03-416f-a3ca-389d48acc8d1-model-cache\") pod \"06421a1e-3d03-416f-a3ca-389d48acc8d1\" (UID: \"06421a1e-3d03-416f-a3ca-389d48acc8d1\") " Oct 30 21:10:29.553761 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:10:29.551103 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06421a1e-3d03-416f-a3ca-389d48acc8d1-kube-api-access-hmx2g\" (UniqueName: \"kubernetes.io/projected/06421a1e-3d03-416f-a3ca-389d48acc8d1-kube-api-access-hmx2g\") pod \"06421a1e-3d03-416f-a3ca-389d48acc8d1\" (UID: \"06421a1e-3d03-416f-a3ca-389d48acc8d1\") " Oct 30 21:10:29.553761 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:10:29.551135 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06421a1e-3d03-416f-a3ca-389d48acc8d1-tmp-dir" pod "06421a1e-3d03-416f-a3ca-389d48acc8d1" (UID: "06421a1e-3d03-416f-a3ca-389d48acc8d1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:10:29.553761 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:10:29.551155 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06421a1e-3d03-416f-a3ca-389d48acc8d1-home\" (UniqueName: \"kubernetes.io/empty-dir/06421a1e-3d03-416f-a3ca-389d48acc8d1-home\") pod \"06421a1e-3d03-416f-a3ca-389d48acc8d1\" (UID: \"06421a1e-3d03-416f-a3ca-389d48acc8d1\") " Oct 30 21:10:29.553761 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:10:29.551557 4611 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06421a1e-3d03-416f-a3ca-389d48acc8d1-tmp-dir\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:10:29.553761 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:10:29.551789 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06421a1e-3d03-416f-a3ca-389d48acc8d1-home" pod "06421a1e-3d03-416f-a3ca-389d48acc8d1" (UID: "06421a1e-3d03-416f-a3ca-389d48acc8d1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:10:29.553761 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:10:29.551969 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06421a1e-3d03-416f-a3ca-389d48acc8d1-model-cache" pod "06421a1e-3d03-416f-a3ca-389d48acc8d1" (UID: "06421a1e-3d03-416f-a3ca-389d48acc8d1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:10:29.554398 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:10:29.553984 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/06421a1e-3d03-416f-a3ca-389d48acc8d1-tls-certs" pod "06421a1e-3d03-416f-a3ca-389d48acc8d1" (UID: "06421a1e-3d03-416f-a3ca-389d48acc8d1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:10:29.554398 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:10:29.554237 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06421a1e-3d03-416f-a3ca-389d48acc8d1-kube-api-access-hmx2g" pod "06421a1e-3d03-416f-a3ca-389d48acc8d1" (UID: "06421a1e-3d03-416f-a3ca-389d48acc8d1"). InnerVolumeSpecName "kube-api-access-hmx2g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:10:29.555458 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:10:29.555431 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06421a1e-3d03-416f-a3ca-389d48acc8d1-dshm" pod "06421a1e-3d03-416f-a3ca-389d48acc8d1" (UID: "06421a1e-3d03-416f-a3ca-389d48acc8d1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:10:29.605225 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:10:29.605198 4611 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="58003568-802b-479c-aa1e-04276bff15cc" path="/var/lib/kubelet/pods/58003568-802b-479c-aa1e-04276bff15cc/volumes" Oct 30 21:10:29.652497 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:10:29.652471 4611 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/06421a1e-3d03-416f-a3ca-389d48acc8d1-home\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:10:29.652497 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:10:29.652497 4611 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/06421a1e-3d03-416f-a3ca-389d48acc8d1-dshm\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:10:29.652659 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:10:29.652631 4611 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/06421a1e-3d03-416f-a3ca-389d48acc8d1-tls-certs\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:10:29.652756 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:10:29.652735 4611 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/06421a1e-3d03-416f-a3ca-389d48acc8d1-model-cache\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:10:29.652794 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:10:29.652766 4611 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hmx2g\" (UniqueName: \"kubernetes.io/projected/06421a1e-3d03-416f-a3ca-389d48acc8d1-kube-api-access-hmx2g\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:10:29.912640 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:10:29.912609 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-1c4642-64f579d6bttr5r" podUID="fba16a3f-2530-4780-be32-2f72358c7e47" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 21:10:29.912804 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:10:29.912609 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-1c4642-7bf468k4cr" podUID="ec926cab-0bca-4a28-bada-cc067d70d3b9" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 21:10:30.350790 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:10:30.350760 4611 generic.go:356] "Generic (PLEG): container finished" podID="06421a1e-3d03-416f-a3ca-389d48acc8d1" containerID="527eb80262d8066d3cca8fd0f89c8a47d1e7f0b7c55f287888f7234c25cd3d20" exitCode=0 Oct 30 21:10:30.350964 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:10:30.350812 4611 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-1c4642/simulator-single-main-1c4642-kserve-554b47fb54-lsct9" Oct 30 21:10:30.350964 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:10:30.350847 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-1c4642/simulator-single-main-1c4642-kserve-554b47fb54-lsct9" event={"ID":"06421a1e-3d03-416f-a3ca-389d48acc8d1","Type":"ContainerDied","Data":"527eb80262d8066d3cca8fd0f89c8a47d1e7f0b7c55f287888f7234c25cd3d20"} Oct 30 21:10:30.350964 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:10:30.350895 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-1c4642/simulator-single-main-1c4642-kserve-554b47fb54-lsct9" event={"ID":"06421a1e-3d03-416f-a3ca-389d48acc8d1","Type":"ContainerDied","Data":"ff03a97efb667bcc5a9222cd0693c772eb670671e24d676b0f922f955dec621f"} Oct 30 21:10:30.350964 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:10:30.350919 4611 scope.go:122] "RemoveContainer" containerID="527eb80262d8066d3cca8fd0f89c8a47d1e7f0b7c55f287888f7234c25cd3d20" Oct 30 21:10:30.352661 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:10:30.352630 4611 generic.go:356] "Generic (PLEG): container finished" podID="2648a43f-15d6-4fa1-9111-8188e578075b" containerID="da39c9b636ce8f54cc47aae80579cc940e02e1f2317015983c484e26eeb8df9b" exitCode=0 Oct 30 21:10:30.352790 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:10:30.352721 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8ms7c" event={"ID":"2648a43f-15d6-4fa1-9111-8188e578075b","Type":"ContainerDied","Data":"da39c9b636ce8f54cc47aae80579cc940e02e1f2317015983c484e26eeb8df9b"} Oct 30 21:10:30.360371 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:10:30.360355 4611 scope.go:122] "RemoveContainer" containerID="527eb80262d8066d3cca8fd0f89c8a47d1e7f0b7c55f287888f7234c25cd3d20" Oct 30 21:10:30.360657 ip-10-0-133-192 kubenswrapper[4611]: E1030 21:10:30.360636 4611 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"527eb80262d8066d3cca8fd0f89c8a47d1e7f0b7c55f287888f7234c25cd3d20\": container with ID starting with 527eb80262d8066d3cca8fd0f89c8a47d1e7f0b7c55f287888f7234c25cd3d20 not found: ID does not exist" containerID="527eb80262d8066d3cca8fd0f89c8a47d1e7f0b7c55f287888f7234c25cd3d20" Oct 30 21:10:30.360758 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:10:30.360661 4611 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"527eb80262d8066d3cca8fd0f89c8a47d1e7f0b7c55f287888f7234c25cd3d20"} err="failed to get container status \"527eb80262d8066d3cca8fd0f89c8a47d1e7f0b7c55f287888f7234c25cd3d20\": rpc error: code = NotFound desc = could not find container \"527eb80262d8066d3cca8fd0f89c8a47d1e7f0b7c55f287888f7234c25cd3d20\": container with ID starting with 527eb80262d8066d3cca8fd0f89c8a47d1e7f0b7c55f287888f7234c25cd3d20 not found: ID does not exist" Oct 30 21:10:30.388136 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:10:30.388110 4611 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-1c4642/simulator-single-main-1c4642-kserve-554b47fb54-lsct9"] Oct 30 21:10:30.391811 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:10:30.391791 4611 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-single-main-1c4642/simulator-single-main-1c4642-kserve-554b47fb54-lsct9"] Oct 30 21:10:31.023798 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:10:31.023765 4611 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-pbg7g"] Oct 30 21:10:31.027852 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:10:31.027826 4611 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-pbg7g"] Oct 30 21:10:31.499158 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:10:31.499131 4611 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8ms7c" Oct 30 21:10:31.572561 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:10:31.572536 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2648a43f-15d6-4fa1-9111-8188e578075b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2648a43f-15d6-4fa1-9111-8188e578075b-maas-api-xks-ca\") pod \"2648a43f-15d6-4fa1-9111-8188e578075b\" (UID: \"2648a43f-15d6-4fa1-9111-8188e578075b\") " Oct 30 21:10:31.572694 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:10:31.572621 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2648a43f-15d6-4fa1-9111-8188e578075b-kube-api-access-zqd88\" (UniqueName: \"kubernetes.io/projected/2648a43f-15d6-4fa1-9111-8188e578075b-kube-api-access-zqd88\") pod \"2648a43f-15d6-4fa1-9111-8188e578075b\" (UID: \"2648a43f-15d6-4fa1-9111-8188e578075b\") " Oct 30 21:10:31.572750 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:10:31.572724 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2648a43f-15d6-4fa1-9111-8188e578075b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2648a43f-15d6-4fa1-9111-8188e578075b-maas-api-openshift-ca\") pod \"2648a43f-15d6-4fa1-9111-8188e578075b\" (UID: \"2648a43f-15d6-4fa1-9111-8188e578075b\") " Oct 30 21:10:31.573014 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:10:31.572988 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2648a43f-15d6-4fa1-9111-8188e578075b-maas-api-openshift-ca" pod "2648a43f-15d6-4fa1-9111-8188e578075b" (UID: "2648a43f-15d6-4fa1-9111-8188e578075b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:10:31.574563 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:10:31.574541 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2648a43f-15d6-4fa1-9111-8188e578075b-maas-api-xks-ca" pod "2648a43f-15d6-4fa1-9111-8188e578075b" (UID: "2648a43f-15d6-4fa1-9111-8188e578075b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:10:31.574633 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:10:31.574621 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2648a43f-15d6-4fa1-9111-8188e578075b-kube-api-access-zqd88" pod "2648a43f-15d6-4fa1-9111-8188e578075b" (UID: "2648a43f-15d6-4fa1-9111-8188e578075b"). InnerVolumeSpecName "kube-api-access-zqd88". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:10:31.605046 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:10:31.605023 4611 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="06421a1e-3d03-416f-a3ca-389d48acc8d1" path="/var/lib/kubelet/pods/06421a1e-3d03-416f-a3ca-389d48acc8d1/volumes" Oct 30 21:10:31.605427 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:10:31.605414 4611 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9a6184dd-ccdd-4f76-8f61-02f92047efcf" path="/var/lib/kubelet/pods/9a6184dd-ccdd-4f76-8f61-02f92047efcf/volumes" Oct 30 21:10:31.673641 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:10:31.673617 4611 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2648a43f-15d6-4fa1-9111-8188e578075b-maas-api-xks-ca\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:10:31.673641 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:10:31.673640 4611 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zqd88\" (UniqueName: \"kubernetes.io/projected/2648a43f-15d6-4fa1-9111-8188e578075b-kube-api-access-zqd88\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:10:31.673771 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:10:31.673652 4611 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2648a43f-15d6-4fa1-9111-8188e578075b-maas-api-openshift-ca\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:10:32.348321 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:10:32.348292 4611 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-1c4642-openshift-default-6784bc96d9fswv6" Oct 30 21:10:32.365756 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:10:32.365733 4611 generic.go:356] "Generic (PLEG): container finished" podID="4ef2c7e4-e9e7-4d0d-91b0-026b1f9cef76" containerID="c16ef57d03d7bf68c683c491e197e9a87646dc42ba870562cc14671f25f32e68" exitCode=0 Oct 30 21:10:32.365876 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:10:32.365777 4611 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-1c4642-openshift-default-6784bc96d9fswv6" Oct 30 21:10:32.365876 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:10:32.365806 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-1c4642-openshift-default-6784bc96d9fswv6" event={"ID":"4ef2c7e4-e9e7-4d0d-91b0-026b1f9cef76","Type":"ContainerDied","Data":"c16ef57d03d7bf68c683c491e197e9a87646dc42ba870562cc14671f25f32e68"} Oct 30 21:10:32.365952 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:10:32.365888 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-1c4642-openshift-default-6784bc96d9fswv6" event={"ID":"4ef2c7e4-e9e7-4d0d-91b0-026b1f9cef76","Type":"ContainerDied","Data":"f8ade14db29be3eb12cd8c1527249283f06ec2b872b00a4f70f9b396b10b7fe8"} Oct 30 21:10:32.365952 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:10:32.365908 4611 scope.go:122] "RemoveContainer" containerID="c16ef57d03d7bf68c683c491e197e9a87646dc42ba870562cc14671f25f32e68" Oct 30 21:10:32.367155 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:10:32.367133 4611 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8ms7c" Oct 30 21:10:32.367266 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:10:32.367136 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8ms7c" event={"ID":"2648a43f-15d6-4fa1-9111-8188e578075b","Type":"ContainerDied","Data":"07d65238d224ab07f455520887e877aad82a6ecae299b08f96a8ba85841c58ff"} Oct 30 21:10:32.367266 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:10:32.367249 4611 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="07d65238d224ab07f455520887e877aad82a6ecae299b08f96a8ba85841c58ff" Oct 30 21:10:32.376096 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:10:32.376077 4611 scope.go:122] "RemoveContainer" containerID="0df373cd1e01b6da9c8183c930636c153b5a820d1032218de833214526580bb1" Oct 30 21:10:32.385370 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:10:32.385352 4611 scope.go:122] "RemoveContainer" containerID="c16ef57d03d7bf68c683c491e197e9a87646dc42ba870562cc14671f25f32e68" Oct 30 21:10:32.385685 ip-10-0-133-192 kubenswrapper[4611]: E1030 21:10:32.385664 4611 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c16ef57d03d7bf68c683c491e197e9a87646dc42ba870562cc14671f25f32e68\": container with ID starting with c16ef57d03d7bf68c683c491e197e9a87646dc42ba870562cc14671f25f32e68 not found: ID does not exist" containerID="c16ef57d03d7bf68c683c491e197e9a87646dc42ba870562cc14671f25f32e68" Oct 30 21:10:32.385768 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:10:32.385694 4611 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c16ef57d03d7bf68c683c491e197e9a87646dc42ba870562cc14671f25f32e68"} err="failed to get container status \"c16ef57d03d7bf68c683c491e197e9a87646dc42ba870562cc14671f25f32e68\": rpc error: code = NotFound desc = could not find container \"c16ef57d03d7bf68c683c491e197e9a87646dc42ba870562cc14671f25f32e68\": container with ID starting with c16ef57d03d7bf68c683c491e197e9a87646dc42ba870562cc14671f25f32e68 not found: ID does not exist" Oct 30 21:10:32.385768 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:10:32.385709 4611 scope.go:122] "RemoveContainer" containerID="0df373cd1e01b6da9c8183c930636c153b5a820d1032218de833214526580bb1" Oct 30 21:10:32.385956 ip-10-0-133-192 kubenswrapper[4611]: E1030 21:10:32.385936 4611 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0df373cd1e01b6da9c8183c930636c153b5a820d1032218de833214526580bb1\": container with ID starting with 0df373cd1e01b6da9c8183c930636c153b5a820d1032218de833214526580bb1 not found: ID does not exist" containerID="0df373cd1e01b6da9c8183c930636c153b5a820d1032218de833214526580bb1" Oct 30 21:10:32.386032 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:10:32.385960 4611 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0df373cd1e01b6da9c8183c930636c153b5a820d1032218de833214526580bb1"} err="failed to get container status \"0df373cd1e01b6da9c8183c930636c153b5a820d1032218de833214526580bb1\": rpc error: code = NotFound desc = could not find container \"0df373cd1e01b6da9c8183c930636c153b5a820d1032218de833214526580bb1\": container with ID starting with 0df373cd1e01b6da9c8183c930636c153b5a820d1032218de833214526580bb1 not found: ID does not exist" Oct 30 21:10:32.481154 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:10:32.480988 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4ef2c7e4-e9e7-4d0d-91b0-026b1f9cef76-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ef2c7e4-e9e7-4d0d-91b0-026b1f9cef76-istio-podinfo\") pod \"4ef2c7e4-e9e7-4d0d-91b0-026b1f9cef76\" (UID: \"4ef2c7e4-e9e7-4d0d-91b0-026b1f9cef76\") " Oct 30 21:10:32.481154 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:10:32.481055 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4ef2c7e4-e9e7-4d0d-91b0-026b1f9cef76-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ef2c7e4-e9e7-4d0d-91b0-026b1f9cef76-istiod-ca-cert\") pod \"4ef2c7e4-e9e7-4d0d-91b0-026b1f9cef76\" (UID: \"4ef2c7e4-e9e7-4d0d-91b0-026b1f9cef76\") " Oct 30 21:10:32.481154 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:10:32.481089 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ef2c7e4-e9e7-4d0d-91b0-026b1f9cef76-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ef2c7e4-e9e7-4d0d-91b0-026b1f9cef76-istio-data\") pod \"4ef2c7e4-e9e7-4d0d-91b0-026b1f9cef76\" (UID: \"4ef2c7e4-e9e7-4d0d-91b0-026b1f9cef76\") " Oct 30 21:10:32.481412 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:10:32.481191 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ef2c7e4-e9e7-4d0d-91b0-026b1f9cef76-istio-token\" (UniqueName: \"kubernetes.io/projected/4ef2c7e4-e9e7-4d0d-91b0-026b1f9cef76-istio-token\") pod \"4ef2c7e4-e9e7-4d0d-91b0-026b1f9cef76\" (UID: \"4ef2c7e4-e9e7-4d0d-91b0-026b1f9cef76\") " Oct 30 21:10:32.481412 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:10:32.481217 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ef2c7e4-e9e7-4d0d-91b0-026b1f9cef76-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ef2c7e4-e9e7-4d0d-91b0-026b1f9cef76-istio-envoy\") pod \"4ef2c7e4-e9e7-4d0d-91b0-026b1f9cef76\" (UID: \"4ef2c7e4-e9e7-4d0d-91b0-026b1f9cef76\") " Oct 30 21:10:32.481412 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:10:32.481263 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ef2c7e4-e9e7-4d0d-91b0-026b1f9cef76-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ef2c7e4-e9e7-4d0d-91b0-026b1f9cef76-credential-socket\") pod \"4ef2c7e4-e9e7-4d0d-91b0-026b1f9cef76\" (UID: \"4ef2c7e4-e9e7-4d0d-91b0-026b1f9cef76\") " Oct 30 21:10:32.481412 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:10:32.481319 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ef2c7e4-e9e7-4d0d-91b0-026b1f9cef76-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ef2c7e4-e9e7-4d0d-91b0-026b1f9cef76-workload-certs\") pod \"4ef2c7e4-e9e7-4d0d-91b0-026b1f9cef76\" (UID: \"4ef2c7e4-e9e7-4d0d-91b0-026b1f9cef76\") " Oct 30 21:10:32.481412 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:10:32.481358 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ef2c7e4-e9e7-4d0d-91b0-026b1f9cef76-kube-api-access-prsdq\" (UniqueName: \"kubernetes.io/projected/4ef2c7e4-e9e7-4d0d-91b0-026b1f9cef76-kube-api-access-prsdq\") pod \"4ef2c7e4-e9e7-4d0d-91b0-026b1f9cef76\" (UID: \"4ef2c7e4-e9e7-4d0d-91b0-026b1f9cef76\") " Oct 30 21:10:32.481412 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:10:32.481395 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ef2c7e4-e9e7-4d0d-91b0-026b1f9cef76-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ef2c7e4-e9e7-4d0d-91b0-026b1f9cef76-workload-socket\") pod \"4ef2c7e4-e9e7-4d0d-91b0-026b1f9cef76\" (UID: \"4ef2c7e4-e9e7-4d0d-91b0-026b1f9cef76\") " Oct 30 21:10:32.482037 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:10:32.481452 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ef2c7e4-e9e7-4d0d-91b0-026b1f9cef76-istiod-ca-cert" pod "4ef2c7e4-e9e7-4d0d-91b0-026b1f9cef76" (UID: "4ef2c7e4-e9e7-4d0d-91b0-026b1f9cef76"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:10:32.482037 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:10:32.481459 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ef2c7e4-e9e7-4d0d-91b0-026b1f9cef76-istio-data" pod "4ef2c7e4-e9e7-4d0d-91b0-026b1f9cef76" (UID: "4ef2c7e4-e9e7-4d0d-91b0-026b1f9cef76"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:10:32.482037 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:10:32.481638 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ef2c7e4-e9e7-4d0d-91b0-026b1f9cef76-workload-certs" pod "4ef2c7e4-e9e7-4d0d-91b0-026b1f9cef76" (UID: "4ef2c7e4-e9e7-4d0d-91b0-026b1f9cef76"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:10:32.482037 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:10:32.481767 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ef2c7e4-e9e7-4d0d-91b0-026b1f9cef76-credential-socket" pod "4ef2c7e4-e9e7-4d0d-91b0-026b1f9cef76" (UID: "4ef2c7e4-e9e7-4d0d-91b0-026b1f9cef76"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:10:32.482037 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:10:32.481834 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ef2c7e4-e9e7-4d0d-91b0-026b1f9cef76-workload-socket" pod "4ef2c7e4-e9e7-4d0d-91b0-026b1f9cef76" (UID: "4ef2c7e4-e9e7-4d0d-91b0-026b1f9cef76"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:10:32.482037 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:10:32.481914 4611 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ef2c7e4-e9e7-4d0d-91b0-026b1f9cef76-istiod-ca-cert\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:10:32.482037 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:10:32.481932 4611 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ef2c7e4-e9e7-4d0d-91b0-026b1f9cef76-istio-data\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:10:32.482037 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:10:32.481946 4611 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ef2c7e4-e9e7-4d0d-91b0-026b1f9cef76-workload-certs\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:10:32.483693 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:10:32.483664 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ef2c7e4-e9e7-4d0d-91b0-026b1f9cef76-istio-token" pod "4ef2c7e4-e9e7-4d0d-91b0-026b1f9cef76" (UID: "4ef2c7e4-e9e7-4d0d-91b0-026b1f9cef76"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:10:32.483807 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:10:32.483757 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ef2c7e4-e9e7-4d0d-91b0-026b1f9cef76-istio-envoy" pod "4ef2c7e4-e9e7-4d0d-91b0-026b1f9cef76" (UID: "4ef2c7e4-e9e7-4d0d-91b0-026b1f9cef76"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:10:32.484021 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:10:32.483933 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4ef2c7e4-e9e7-4d0d-91b0-026b1f9cef76-istio-podinfo" pod "4ef2c7e4-e9e7-4d0d-91b0-026b1f9cef76" (UID: "4ef2c7e4-e9e7-4d0d-91b0-026b1f9cef76"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:10:32.484241 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:10:32.484224 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ef2c7e4-e9e7-4d0d-91b0-026b1f9cef76-kube-api-access-prsdq" pod "4ef2c7e4-e9e7-4d0d-91b0-026b1f9cef76" (UID: "4ef2c7e4-e9e7-4d0d-91b0-026b1f9cef76"). InnerVolumeSpecName "kube-api-access-prsdq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:10:32.582361 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:10:32.582337 4611 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ef2c7e4-e9e7-4d0d-91b0-026b1f9cef76-istio-token\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:10:32.582361 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:10:32.582359 4611 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ef2c7e4-e9e7-4d0d-91b0-026b1f9cef76-istio-envoy\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:10:32.582477 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:10:32.582369 4611 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ef2c7e4-e9e7-4d0d-91b0-026b1f9cef76-credential-socket\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:10:32.582477 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:10:32.582379 4611 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-prsdq\" (UniqueName: \"kubernetes.io/projected/4ef2c7e4-e9e7-4d0d-91b0-026b1f9cef76-kube-api-access-prsdq\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:10:32.582477 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:10:32.582388 4611 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ef2c7e4-e9e7-4d0d-91b0-026b1f9cef76-workload-socket\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:10:32.582477 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:10:32.582398 4611 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ef2c7e4-e9e7-4d0d-91b0-026b1f9cef76-istio-podinfo\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:10:32.689052 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:10:32.689030 4611 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-1c4642-openshift-default-6784bc96d9fswv6"] Oct 30 21:10:32.696269 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:10:32.696249 4611 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-single-main-1c4642-openshift-default-6784bc96d9fswv6"] Oct 30 21:10:33.603841 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:10:33.603809 4611 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4ef2c7e4-e9e7-4d0d-91b0-026b1f9cef76" path="/var/lib/kubelet/pods/4ef2c7e4-e9e7-4d0d-91b0-026b1f9cef76/volumes" Oct 30 21:10:33.916310 ip-10-0-133-192 kubenswrapper[4611]: E1030 21:10:33.916231 4611 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hwzc6" containerName="extractor" Oct 30 21:10:39.912920 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:10:39.912876 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-1c4642-7bf468k4cr" podUID="ec926cab-0bca-4a28-bada-cc067d70d3b9" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 21:10:39.913300 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:10:39.912876 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-1c4642-64f579d6bttr5r" podUID="fba16a3f-2530-4780-be32-2f72358c7e47" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 21:10:42.029532 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:10:42.029059 4611 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-mtvnz"] Oct 30 21:10:42.048534 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:10:42.038353 4611 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-mtvnz"] Oct 30 21:10:43.604383 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:10:43.604343 4611 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4792453f-2774-409b-8dc4-a9b8ce0f7bf7" path="/var/lib/kubelet/pods/4792453f-2774-409b-8dc4-a9b8ce0f7bf7/volumes" Oct 30 21:10:44.808863 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:10:44.808839 4611 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-1c4642-64f579d6bttr5r" Oct 30 21:10:44.850260 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:10:44.850234 4611 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-1c4642-7bf468k4cr" Oct 30 21:10:44.943140 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:10:44.943110 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fba16a3f-2530-4780-be32-2f72358c7e47-kube-api-access-bwrzr\" (UniqueName: \"kubernetes.io/projected/fba16a3f-2530-4780-be32-2f72358c7e47-kube-api-access-bwrzr\") pod \"fba16a3f-2530-4780-be32-2f72358c7e47\" (UID: \"fba16a3f-2530-4780-be32-2f72358c7e47\") " Oct 30 21:10:44.943283 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:10:44.943192 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ec926cab-0bca-4a28-bada-cc067d70d3b9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec926cab-0bca-4a28-bada-cc067d70d3b9-plugins-config-volume\") pod \"ec926cab-0bca-4a28-bada-cc067d70d3b9\" (UID: \"ec926cab-0bca-4a28-bada-cc067d70d3b9\") " Oct 30 21:10:44.943283 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:10:44.943219 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fba16a3f-2530-4780-be32-2f72358c7e47-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fba16a3f-2530-4780-be32-2f72358c7e47-plugins-config-volume\") pod \"fba16a3f-2530-4780-be32-2f72358c7e47\" (UID: \"fba16a3f-2530-4780-be32-2f72358c7e47\") " Oct 30 21:10:44.943418 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:10:44.943299 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec926cab-0bca-4a28-bada-cc067d70d3b9-kube-api-access-jsrkf\" (UniqueName: \"kubernetes.io/projected/ec926cab-0bca-4a28-bada-cc067d70d3b9-kube-api-access-jsrkf\") pod \"ec926cab-0bca-4a28-bada-cc067d70d3b9\" (UID: \"ec926cab-0bca-4a28-bada-cc067d70d3b9\") " Oct 30 21:10:44.943610 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:10:44.943588 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec926cab-0bca-4a28-bada-cc067d70d3b9-plugins-config-volume" pod "ec926cab-0bca-4a28-bada-cc067d70d3b9" (UID: "ec926cab-0bca-4a28-bada-cc067d70d3b9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:10:44.943686 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:10:44.943627 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fba16a3f-2530-4780-be32-2f72358c7e47-plugins-config-volume" pod "fba16a3f-2530-4780-be32-2f72358c7e47" (UID: "fba16a3f-2530-4780-be32-2f72358c7e47"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:10:44.945260 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:10:44.945238 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fba16a3f-2530-4780-be32-2f72358c7e47-kube-api-access-bwrzr" pod "fba16a3f-2530-4780-be32-2f72358c7e47" (UID: "fba16a3f-2530-4780-be32-2f72358c7e47"). InnerVolumeSpecName "kube-api-access-bwrzr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:10:44.945328 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:10:44.945291 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec926cab-0bca-4a28-bada-cc067d70d3b9-kube-api-access-jsrkf" pod "ec926cab-0bca-4a28-bada-cc067d70d3b9" (UID: "ec926cab-0bca-4a28-bada-cc067d70d3b9"). InnerVolumeSpecName "kube-api-access-jsrkf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:10:45.044092 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:10:45.044069 4611 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec926cab-0bca-4a28-bada-cc067d70d3b9-plugins-config-volume\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:10:45.044092 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:10:45.044092 4611 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fba16a3f-2530-4780-be32-2f72358c7e47-plugins-config-volume\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:10:45.044278 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:10:45.044102 4611 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jsrkf\" (UniqueName: \"kubernetes.io/projected/ec926cab-0bca-4a28-bada-cc067d70d3b9-kube-api-access-jsrkf\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:10:45.044278 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:10:45.044110 4611 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bwrzr\" (UniqueName: \"kubernetes.io/projected/fba16a3f-2530-4780-be32-2f72358c7e47-kube-api-access-bwrzr\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:10:45.461589 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:10:45.461558 4611 generic.go:356] "Generic (PLEG): container finished" podID="fba16a3f-2530-4780-be32-2f72358c7e47" containerID="4a3ed81adc6fc5949da33cf3a449b845d63e35f7fa03ddbde33c3bbaab929c39" exitCode=0 Oct 30 21:10:45.461749 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:10:45.461621 4611 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-1c4642-64f579d6bttr5r" Oct 30 21:10:45.461749 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:10:45.461638 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-1c4642-64f579d6bttr5r" event={"ID":"fba16a3f-2530-4780-be32-2f72358c7e47","Type":"ContainerDied","Data":"4a3ed81adc6fc5949da33cf3a449b845d63e35f7fa03ddbde33c3bbaab929c39"} Oct 30 21:10:45.461749 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:10:45.461728 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-1c4642-64f579d6bttr5r" event={"ID":"fba16a3f-2530-4780-be32-2f72358c7e47","Type":"ContainerDied","Data":"9370c8cc81101bd4b9f19d8964d23ce1026773e59284a910ea32ab371f7fe52b"} Oct 30 21:10:45.461931 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:10:45.461760 4611 scope.go:122] "RemoveContainer" containerID="4a3ed81adc6fc5949da33cf3a449b845d63e35f7fa03ddbde33c3bbaab929c39" Oct 30 21:10:45.463189 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:10:45.463164 4611 generic.go:356] "Generic (PLEG): container finished" podID="ec926cab-0bca-4a28-bada-cc067d70d3b9" containerID="dfc52b6e28060f61f85d4d94fd60651e6c4e19668c8791fed953be870fd9a82f" exitCode=0 Oct 30 21:10:45.463308 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:10:45.463206 4611 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-1c4642-7bf468k4cr" Oct 30 21:10:45.463308 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:10:45.463251 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-1c4642-7bf468k4cr" event={"ID":"ec926cab-0bca-4a28-bada-cc067d70d3b9","Type":"ContainerDied","Data":"dfc52b6e28060f61f85d4d94fd60651e6c4e19668c8791fed953be870fd9a82f"} Oct 30 21:10:45.463308 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:10:45.463303 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-1c4642-7bf468k4cr" event={"ID":"ec926cab-0bca-4a28-bada-cc067d70d3b9","Type":"ContainerDied","Data":"288477348e86101a0da494e0eedff26c27fd647c8e8a5e3c83b9fb7584625866"} Oct 30 21:10:45.470925 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:10:45.470769 4611 scope.go:122] "RemoveContainer" containerID="4a3ed81adc6fc5949da33cf3a449b845d63e35f7fa03ddbde33c3bbaab929c39" Oct 30 21:10:45.471049 ip-10-0-133-192 kubenswrapper[4611]: E1030 21:10:45.471029 4611 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a3ed81adc6fc5949da33cf3a449b845d63e35f7fa03ddbde33c3bbaab929c39\": container with ID starting with 4a3ed81adc6fc5949da33cf3a449b845d63e35f7fa03ddbde33c3bbaab929c39 not found: ID does not exist" containerID="4a3ed81adc6fc5949da33cf3a449b845d63e35f7fa03ddbde33c3bbaab929c39" Oct 30 21:10:45.471113 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:10:45.471059 4611 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a3ed81adc6fc5949da33cf3a449b845d63e35f7fa03ddbde33c3bbaab929c39"} err="failed to get container status \"4a3ed81adc6fc5949da33cf3a449b845d63e35f7fa03ddbde33c3bbaab929c39\": rpc error: code = NotFound desc = could not find container \"4a3ed81adc6fc5949da33cf3a449b845d63e35f7fa03ddbde33c3bbaab929c39\": container with ID starting with 4a3ed81adc6fc5949da33cf3a449b845d63e35f7fa03ddbde33c3bbaab929c39 not found: ID does not exist" Oct 30 21:10:45.471113 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:10:45.471083 4611 scope.go:122] "RemoveContainer" containerID="dfc52b6e28060f61f85d4d94fd60651e6c4e19668c8791fed953be870fd9a82f" Oct 30 21:10:45.479203 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:10:45.479189 4611 scope.go:122] "RemoveContainer" containerID="dfc52b6e28060f61f85d4d94fd60651e6c4e19668c8791fed953be870fd9a82f" Oct 30 21:10:45.479452 ip-10-0-133-192 kubenswrapper[4611]: E1030 21:10:45.479432 4611 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dfc52b6e28060f61f85d4d94fd60651e6c4e19668c8791fed953be870fd9a82f\": container with ID starting with dfc52b6e28060f61f85d4d94fd60651e6c4e19668c8791fed953be870fd9a82f not found: ID does not exist" containerID="dfc52b6e28060f61f85d4d94fd60651e6c4e19668c8791fed953be870fd9a82f" Oct 30 21:10:45.479545 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:10:45.479462 4611 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dfc52b6e28060f61f85d4d94fd60651e6c4e19668c8791fed953be870fd9a82f"} err="failed to get container status \"dfc52b6e28060f61f85d4d94fd60651e6c4e19668c8791fed953be870fd9a82f\": rpc error: code = NotFound desc = could not find container \"dfc52b6e28060f61f85d4d94fd60651e6c4e19668c8791fed953be870fd9a82f\": container with ID starting with dfc52b6e28060f61f85d4d94fd60651e6c4e19668c8791fed953be870fd9a82f not found: ID does not exist" Oct 30 21:10:45.485874 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:10:45.485842 4611 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-1c4642-64f579d6bttr5r"] Oct 30 21:10:45.490540 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:10:45.490495 4611 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-1c4642-64f579d6bttr5r"] Oct 30 21:10:45.500965 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:10:45.500945 4611 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-1c4642-7bf468k4cr"] Oct 30 21:10:45.503980 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:10:45.503959 4611 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-1c4642-7bf468k4cr"] Oct 30 21:10:45.603911 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:10:45.603891 4611 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ec926cab-0bca-4a28-bada-cc067d70d3b9" path="/var/lib/kubelet/pods/ec926cab-0bca-4a28-bada-cc067d70d3b9/volumes" Oct 30 21:10:45.604251 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:10:45.604239 4611 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fba16a3f-2530-4780-be32-2f72358c7e47" path="/var/lib/kubelet/pods/fba16a3f-2530-4780-be32-2f72358c7e47/volumes" Oct 30 21:10:49.430747 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:10:49.430714 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:10:49.727917 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:10:49.727826 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:10:50.785800 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:10:50.785751 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:10:52.142726 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:10:52.142683 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:10:58.087554 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:10:58.087498 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:11:02.024144 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:11:02.024110 4611 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-nf5px"] Oct 30 21:11:02.027478 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:11:02.027452 4611 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-nf5px"] Oct 30 21:11:03.604796 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:11:03.604761 4611 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dcb51fc1-b1d9-4ca4-9ea4-32a8f21f2c93" path="/var/lib/kubelet/pods/dcb51fc1-b1d9-4ca4-9ea4-32a8f21f2c93/volumes" Oct 30 21:11:03.916431 ip-10-0-133-192 kubenswrapper[4611]: E1030 21:11:03.916352 4611 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hwzc6" containerName="extractor" Oct 30 21:11:08.162062 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:11:08.162032 4611 scope.go:122] "RemoveContainer" containerID="4ff0ac7b0038c429ee9d56e92d59cc563181405975e8706c7edcde3b84d4000d" Oct 30 21:11:08.170704 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:11:08.170684 4611 scope.go:122] "RemoveContainer" containerID="95f514df524b461231af54589071497892056ea6cf17e094a06fbc648a6befdf" Oct 30 21:11:08.178449 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:11:08.178425 4611 scope.go:122] "RemoveContainer" containerID="a2ac1015e9bd303fdcabf13a1419179ca3dbd77a4946cb0b23240dc410b6467c" Oct 30 21:11:20.025478 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:11:20.025382 4611 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-7t8l5"] Oct 30 21:11:20.027642 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:11:20.027620 4611 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-7t8l5"] Oct 30 21:11:21.610726 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:11:21.610682 4611 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="22f1a1fc-7bc9-42d6-9f36-ecbc3279cb06" path="/var/lib/kubelet/pods/22f1a1fc-7bc9-42d6-9f36-ecbc3279cb06/volumes" Oct 30 21:11:33.664356 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:11:33.664322 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:11:33.812478 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:11:33.812443 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:11:33.916465 ip-10-0-133-192 kubenswrapper[4611]: E1030 21:11:33.916380 4611 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hwzc6" containerName="extractor" Oct 30 21:11:38.057987 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:11:38.057945 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:11:39.024247 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:11:39.024212 4611 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-b26lj"] Oct 30 21:11:39.026798 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:11:39.026768 4611 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-b26lj"] Oct 30 21:11:39.605987 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:11:39.605945 4611 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eaa6301f-743d-46d5-b4b7-84d36896eec7" path="/var/lib/kubelet/pods/eaa6301f-743d-46d5-b4b7-84d36896eec7/volumes" Oct 30 21:11:49.518189 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:11:49.518147 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:12:03.915001 ip-10-0-133-192 kubenswrapper[4611]: E1030 21:12:03.914966 4611 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hwzc6" containerName="extractor" Oct 30 21:12:07.023944 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:12:07.023914 4611 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-6nhlh"] Oct 30 21:12:07.026794 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:12:07.026769 4611 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-6nhlh"] Oct 30 21:12:07.605236 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:12:07.605200 4611 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="43120b8c-224c-474e-858f-12ab24ef455a" path="/var/lib/kubelet/pods/43120b8c-224c-474e-858f-12ab24ef455a/volumes" Oct 30 21:12:08.221556 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:12:08.221526 4611 scope.go:122] "RemoveContainer" containerID="3cc02d14f136f926b51ddca3758f741f2e2b4616a5a0805f32b1c1c5d0060a60" Oct 30 21:12:08.229963 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:12:08.229933 4611 scope.go:122] "RemoveContainer" containerID="55c2edc295b367157a85b69b7e9679018546cabb9f4bbb7d116e94c7cda79786" Oct 30 21:12:08.237839 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:12:08.237812 4611 scope.go:122] "RemoveContainer" containerID="3b8879221448b0b50275051e14a71070d57b45857299705ec87f77acc89e6f2d" Oct 30 21:12:23.607112 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:12:23.607076 4611 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-5pb4b"] Oct 30 21:12:23.607492 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:12:23.607431 4611 cpu_manager.go:414] "containerMap: removing container" podUID="ec926cab-0bca-4a28-bada-cc067d70d3b9" containerName="payload-pre-processing" Oct 30 21:12:23.607492 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:12:23.607442 4611 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec926cab-0bca-4a28-bada-cc067d70d3b9" containerName="payload-pre-processing" Oct 30 21:12:23.607492 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:12:23.607455 4611 cpu_manager.go:414] "containerMap: removing container" podUID="fba16a3f-2530-4780-be32-2f72358c7e47" containerName="payload-processing" Oct 30 21:12:23.607492 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:12:23.607460 4611 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fba16a3f-2530-4780-be32-2f72358c7e47" containerName="payload-processing" Oct 30 21:12:23.607492 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:12:23.607467 4611 cpu_manager.go:414] "containerMap: removing container" podUID="06421a1e-3d03-416f-a3ca-389d48acc8d1" containerName="main" Oct 30 21:12:23.607492 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:12:23.607472 4611 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06421a1e-3d03-416f-a3ca-389d48acc8d1" containerName="main" Oct 30 21:12:23.607492 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:12:23.607494 4611 cpu_manager.go:414] "containerMap: removing container" podUID="4ef2c7e4-e9e7-4d0d-91b0-026b1f9cef76" containerName="istio-proxy" Oct 30 21:12:23.607745 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:12:23.607500 4611 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ef2c7e4-e9e7-4d0d-91b0-026b1f9cef76" containerName="istio-proxy" Oct 30 21:12:23.607745 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:12:23.607519 4611 cpu_manager.go:414] "containerMap: removing container" podUID="2648a43f-15d6-4fa1-9111-8188e578075b" containerName="revoke-keys" Oct 30 21:12:23.607745 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:12:23.607524 4611 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2648a43f-15d6-4fa1-9111-8188e578075b" containerName="revoke-keys" Oct 30 21:12:23.607745 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:12:23.607533 4611 cpu_manager.go:414] "containerMap: removing container" podUID="4ef2c7e4-e9e7-4d0d-91b0-026b1f9cef76" containerName="istio-proxy" Oct 30 21:12:23.607745 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:12:23.607538 4611 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ef2c7e4-e9e7-4d0d-91b0-026b1f9cef76" containerName="istio-proxy" Oct 30 21:12:23.607745 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:12:23.607603 4611 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec926cab-0bca-4a28-bada-cc067d70d3b9" containerName="payload-pre-processing" Oct 30 21:12:23.607745 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:12:23.607610 4611 memory_manager.go:365] "RemoveStaleState removing state" podUID="fba16a3f-2530-4780-be32-2f72358c7e47" containerName="payload-processing" Oct 30 21:12:23.607745 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:12:23.607617 4611 memory_manager.go:365] "RemoveStaleState removing state" podUID="06421a1e-3d03-416f-a3ca-389d48acc8d1" containerName="main" Oct 30 21:12:23.607745 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:12:23.607627 4611 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ef2c7e4-e9e7-4d0d-91b0-026b1f9cef76" containerName="istio-proxy" Oct 30 21:12:23.607745 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:12:23.607634 4611 memory_manager.go:365] "RemoveStaleState removing state" podUID="2648a43f-15d6-4fa1-9111-8188e578075b" containerName="revoke-keys" Oct 30 21:12:23.607745 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:12:23.607640 4611 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ef2c7e4-e9e7-4d0d-91b0-026b1f9cef76" containerName="istio-proxy" Oct 30 21:12:23.612679 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:12:23.612659 4611 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-5pb4b" Oct 30 21:12:23.617342 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:12:23.617301 4611 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:12:23.617870 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:12:23.617845 4611 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:12:23.618037 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:12:23.617899 4611 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:12:23.618144 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:12:23.618109 4611 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7hf8r\"" Oct 30 21:12:23.620850 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:12:23.620830 4611 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-5pb4b"] Oct 30 21:12:23.766777 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:12:23.766740 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/883ae4b7-4e14-4ad9-9d69-944966b15e98-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-5pb4b\" (UID: \"883ae4b7-4e14-4ad9-9d69-944966b15e98\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-5pb4b" Oct 30 21:12:23.766938 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:12:23.766861 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/883ae4b7-4e14-4ad9-9d69-944966b15e98-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-5pb4b\" (UID: \"883ae4b7-4e14-4ad9-9d69-944966b15e98\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-5pb4b" Oct 30 21:12:23.766938 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:12:23.766901 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qln4f\" (UniqueName: \"kubernetes.io/projected/883ae4b7-4e14-4ad9-9d69-944966b15e98-kube-api-access-qln4f\") pod \"maas-api-revoke-sub-d576e68b-5pb4b\" (UID: \"883ae4b7-4e14-4ad9-9d69-944966b15e98\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-5pb4b" Oct 30 21:12:23.868241 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:12:23.868169 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/883ae4b7-4e14-4ad9-9d69-944966b15e98-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-5pb4b\" (UID: \"883ae4b7-4e14-4ad9-9d69-944966b15e98\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-5pb4b" Oct 30 21:12:23.868241 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:12:23.868207 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qln4f\" (UniqueName: \"kubernetes.io/projected/883ae4b7-4e14-4ad9-9d69-944966b15e98-kube-api-access-qln4f\") pod \"maas-api-revoke-sub-d576e68b-5pb4b\" (UID: \"883ae4b7-4e14-4ad9-9d69-944966b15e98\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-5pb4b" Oct 30 21:12:23.868403 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:12:23.868261 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/883ae4b7-4e14-4ad9-9d69-944966b15e98-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-5pb4b\" (UID: \"883ae4b7-4e14-4ad9-9d69-944966b15e98\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-5pb4b" Oct 30 21:12:23.868839 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:12:23.868808 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/883ae4b7-4e14-4ad9-9d69-944966b15e98-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-5pb4b\" (UID: \"883ae4b7-4e14-4ad9-9d69-944966b15e98\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-5pb4b" Oct 30 21:12:23.870625 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:12:23.870606 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/883ae4b7-4e14-4ad9-9d69-944966b15e98-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-5pb4b\" (UID: \"883ae4b7-4e14-4ad9-9d69-944966b15e98\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-5pb4b" Oct 30 21:12:23.880252 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:12:23.880230 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qln4f\" (UniqueName: \"kubernetes.io/projected/883ae4b7-4e14-4ad9-9d69-944966b15e98-kube-api-access-qln4f\") pod \"maas-api-revoke-sub-d576e68b-5pb4b\" (UID: \"883ae4b7-4e14-4ad9-9d69-944966b15e98\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-5pb4b" Oct 30 21:12:23.927961 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:12:23.927937 4611 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-5pb4b" Oct 30 21:12:24.203966 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:12:24.203942 4611 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-5pb4b"] Oct 30 21:12:24.205962 ip-10-0-133-192 kubenswrapper[4611]: W1030 21:12:24.205932 4611 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod883ae4b7_4e14_4ad9_9d69_944966b15e98.slice/crio-2fefe9c759f39417d007b4cf264674bb12c9eb2925b5730760f282c3cf8b28a3 WatchSource:0}: Error finding container 2fefe9c759f39417d007b4cf264674bb12c9eb2925b5730760f282c3cf8b28a3: Status 404 returned error can't find the container with id 2fefe9c759f39417d007b4cf264674bb12c9eb2925b5730760f282c3cf8b28a3 Oct 30 21:12:25.007253 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:12:25.007223 4611 generic.go:356] "Generic (PLEG): container finished" podID="883ae4b7-4e14-4ad9-9d69-944966b15e98" containerID="ba5ca9aae42f3deec9e3952d9f66f051003122b45473b5390904ce2442b6a8cb" exitCode=0 Oct 30 21:12:25.007638 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:12:25.007313 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-5pb4b" event={"ID":"883ae4b7-4e14-4ad9-9d69-944966b15e98","Type":"ContainerDied","Data":"ba5ca9aae42f3deec9e3952d9f66f051003122b45473b5390904ce2442b6a8cb"} Oct 30 21:12:25.007638 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:12:25.007381 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-5pb4b" event={"ID":"883ae4b7-4e14-4ad9-9d69-944966b15e98","Type":"ContainerStarted","Data":"2fefe9c759f39417d007b4cf264674bb12c9eb2925b5730760f282c3cf8b28a3"} Oct 30 21:12:26.145343 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:12:26.145318 4611 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-5pb4b" Oct 30 21:12:26.291167 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:12:26.291094 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/883ae4b7-4e14-4ad9-9d69-944966b15e98-kube-api-access-qln4f\" (UniqueName: \"kubernetes.io/projected/883ae4b7-4e14-4ad9-9d69-944966b15e98-kube-api-access-qln4f\") pod \"883ae4b7-4e14-4ad9-9d69-944966b15e98\" (UID: \"883ae4b7-4e14-4ad9-9d69-944966b15e98\") " Oct 30 21:12:26.291167 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:12:26.291144 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/883ae4b7-4e14-4ad9-9d69-944966b15e98-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/883ae4b7-4e14-4ad9-9d69-944966b15e98-maas-api-xks-ca\") pod \"883ae4b7-4e14-4ad9-9d69-944966b15e98\" (UID: \"883ae4b7-4e14-4ad9-9d69-944966b15e98\") " Oct 30 21:12:26.291383 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:12:26.291185 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/883ae4b7-4e14-4ad9-9d69-944966b15e98-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/883ae4b7-4e14-4ad9-9d69-944966b15e98-maas-api-openshift-ca\") pod \"883ae4b7-4e14-4ad9-9d69-944966b15e98\" (UID: \"883ae4b7-4e14-4ad9-9d69-944966b15e98\") " Oct 30 21:12:26.291596 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:12:26.291574 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/883ae4b7-4e14-4ad9-9d69-944966b15e98-maas-api-openshift-ca" pod "883ae4b7-4e14-4ad9-9d69-944966b15e98" (UID: "883ae4b7-4e14-4ad9-9d69-944966b15e98"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:12:26.293102 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:12:26.293077 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/883ae4b7-4e14-4ad9-9d69-944966b15e98-maas-api-xks-ca" pod "883ae4b7-4e14-4ad9-9d69-944966b15e98" (UID: "883ae4b7-4e14-4ad9-9d69-944966b15e98"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:12:26.293198 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:12:26.293135 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/883ae4b7-4e14-4ad9-9d69-944966b15e98-kube-api-access-qln4f" pod "883ae4b7-4e14-4ad9-9d69-944966b15e98" (UID: "883ae4b7-4e14-4ad9-9d69-944966b15e98"). InnerVolumeSpecName "kube-api-access-qln4f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:12:26.392140 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:12:26.392101 4611 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qln4f\" (UniqueName: \"kubernetes.io/projected/883ae4b7-4e14-4ad9-9d69-944966b15e98-kube-api-access-qln4f\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:12:26.392140 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:12:26.392138 4611 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/883ae4b7-4e14-4ad9-9d69-944966b15e98-maas-api-xks-ca\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:12:26.392140 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:12:26.392149 4611 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/883ae4b7-4e14-4ad9-9d69-944966b15e98-maas-api-openshift-ca\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:12:27.018594 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:12:27.018554 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-5pb4b" event={"ID":"883ae4b7-4e14-4ad9-9d69-944966b15e98","Type":"ContainerDied","Data":"2fefe9c759f39417d007b4cf264674bb12c9eb2925b5730760f282c3cf8b28a3"} Oct 30 21:12:27.018766 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:12:27.018608 4611 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-5pb4b" Oct 30 21:12:27.018766 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:12:27.018613 4611 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2fefe9c759f39417d007b4cf264674bb12c9eb2925b5730760f282c3cf8b28a3" Oct 30 21:12:33.917113 ip-10-0-133-192 kubenswrapper[4611]: E1030 21:12:33.917077 4611 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hwzc6" containerName="extractor" Oct 30 21:12:48.171589 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:12:48.170611 4611 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-8tvvl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 21:12:48.171589 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:12:48.170679 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8tvvl" podUID="46ae03da-7a80-4f78-8abd-391ebc697f2f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:12:48.192651 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:12:48.192590 4611 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-w44qs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 21:12:48.192938 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:12:48.192889 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-w44qs" podUID="6d6fd4b2-ad0a-4ce3-b5b3-966de34bd8b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:12:49.526760 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:12:49.526717 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:12:49.688616 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:12:49.688587 4611 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-n7z25"] Oct 30 21:12:49.689034 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:12:49.689008 4611 cpu_manager.go:414] "containerMap: removing container" podUID="883ae4b7-4e14-4ad9-9d69-944966b15e98" containerName="revoke-keys" Oct 30 21:12:49.689034 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:12:49.689022 4611 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="883ae4b7-4e14-4ad9-9d69-944966b15e98" containerName="revoke-keys" Oct 30 21:12:49.689114 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:12:49.689079 4611 memory_manager.go:365] "RemoveStaleState removing state" podUID="883ae4b7-4e14-4ad9-9d69-944966b15e98" containerName="revoke-keys" Oct 30 21:12:49.692164 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:12:49.692148 4611 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-n7z25" Oct 30 21:12:49.701036 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:12:49.701013 4611 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:12:49.704472 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:12:49.704448 4611 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:12:49.704472 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:12:49.704469 4611 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:12:49.707570 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:12:49.707547 4611 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-n7z25"] Oct 30 21:12:49.756453 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:12:49.756405 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08204e6f-3e3e-4414-b3ea-da35d807ca4d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-n7z25\" (UID: \"08204e6f-3e3e-4414-b3ea-da35d807ca4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-n7z25" Oct 30 21:12:49.756640 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:12:49.756558 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qltt6\" (UniqueName: \"kubernetes.io/projected/08204e6f-3e3e-4414-b3ea-da35d807ca4d-kube-api-access-qltt6\") pod \"maas-api-revoke-sub-d2bc93bc-n7z25\" (UID: \"08204e6f-3e3e-4414-b3ea-da35d807ca4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-n7z25" Oct 30 21:12:49.756640 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:12:49.756602 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08204e6f-3e3e-4414-b3ea-da35d807ca4d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-n7z25\" (UID: \"08204e6f-3e3e-4414-b3ea-da35d807ca4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-n7z25" Oct 30 21:12:49.801398 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:12:49.801331 4611 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7hf8r\"" Oct 30 21:12:49.857856 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:12:49.857825 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08204e6f-3e3e-4414-b3ea-da35d807ca4d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-n7z25\" (UID: \"08204e6f-3e3e-4414-b3ea-da35d807ca4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-n7z25" Oct 30 21:12:49.857989 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:12:49.857901 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08204e6f-3e3e-4414-b3ea-da35d807ca4d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-n7z25\" (UID: \"08204e6f-3e3e-4414-b3ea-da35d807ca4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-n7z25" Oct 30 21:12:49.857989 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:12:49.857952 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qltt6\" (UniqueName: \"kubernetes.io/projected/08204e6f-3e3e-4414-b3ea-da35d807ca4d-kube-api-access-qltt6\") pod \"maas-api-revoke-sub-d2bc93bc-n7z25\" (UID: \"08204e6f-3e3e-4414-b3ea-da35d807ca4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-n7z25" Oct 30 21:12:49.858582 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:12:49.858564 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08204e6f-3e3e-4414-b3ea-da35d807ca4d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-n7z25\" (UID: \"08204e6f-3e3e-4414-b3ea-da35d807ca4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-n7z25" Oct 30 21:12:49.860142 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:12:49.860115 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08204e6f-3e3e-4414-b3ea-da35d807ca4d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-n7z25\" (UID: \"08204e6f-3e3e-4414-b3ea-da35d807ca4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-n7z25" Oct 30 21:12:49.869794 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:12:49.869769 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qltt6\" (UniqueName: \"kubernetes.io/projected/08204e6f-3e3e-4414-b3ea-da35d807ca4d-kube-api-access-qltt6\") pod \"maas-api-revoke-sub-d2bc93bc-n7z25\" (UID: \"08204e6f-3e3e-4414-b3ea-da35d807ca4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-n7z25" Oct 30 21:12:50.005564 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:12:50.005543 4611 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-n7z25" Oct 30 21:12:50.279858 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:12:50.279831 4611 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-n7z25"] Oct 30 21:12:50.281910 ip-10-0-133-192 kubenswrapper[4611]: W1030 21:12:50.281878 4611 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08204e6f_3e3e_4414_b3ea_da35d807ca4d.slice/crio-142d57646650cedba91bbdcf9968ee32596f700e6eb7779d98239beebc0c635d WatchSource:0}: Error finding container 142d57646650cedba91bbdcf9968ee32596f700e6eb7779d98239beebc0c635d: Status 404 returned error can't find the container with id 142d57646650cedba91bbdcf9968ee32596f700e6eb7779d98239beebc0c635d Oct 30 21:12:51.164578 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:12:51.164551 4611 generic.go:356] "Generic (PLEG): container finished" podID="08204e6f-3e3e-4414-b3ea-da35d807ca4d" containerID="8519f88de223a25ffe8432db977462b083edb4476715f146a539c8928afeb463" exitCode=0 Oct 30 21:12:51.164920 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:12:51.164634 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-n7z25" event={"ID":"08204e6f-3e3e-4414-b3ea-da35d807ca4d","Type":"ContainerDied","Data":"8519f88de223a25ffe8432db977462b083edb4476715f146a539c8928afeb463"} Oct 30 21:12:51.164920 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:12:51.164689 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-n7z25" event={"ID":"08204e6f-3e3e-4414-b3ea-da35d807ca4d","Type":"ContainerStarted","Data":"142d57646650cedba91bbdcf9968ee32596f700e6eb7779d98239beebc0c635d"} Oct 30 21:12:52.295885 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:12:52.295858 4611 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-n7z25" Oct 30 21:12:52.381678 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:12:52.381639 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/08204e6f-3e3e-4414-b3ea-da35d807ca4d-kube-api-access-qltt6\" (UniqueName: \"kubernetes.io/projected/08204e6f-3e3e-4414-b3ea-da35d807ca4d-kube-api-access-qltt6\") pod \"08204e6f-3e3e-4414-b3ea-da35d807ca4d\" (UID: \"08204e6f-3e3e-4414-b3ea-da35d807ca4d\") " Oct 30 21:12:52.381811 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:12:52.381695 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/08204e6f-3e3e-4414-b3ea-da35d807ca4d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08204e6f-3e3e-4414-b3ea-da35d807ca4d-maas-api-openshift-ca\") pod \"08204e6f-3e3e-4414-b3ea-da35d807ca4d\" (UID: \"08204e6f-3e3e-4414-b3ea-da35d807ca4d\") " Oct 30 21:12:52.381811 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:12:52.381753 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/08204e6f-3e3e-4414-b3ea-da35d807ca4d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08204e6f-3e3e-4414-b3ea-da35d807ca4d-maas-api-xks-ca\") pod \"08204e6f-3e3e-4414-b3ea-da35d807ca4d\" (UID: \"08204e6f-3e3e-4414-b3ea-da35d807ca4d\") " Oct 30 21:12:52.382039 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:12:52.382016 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08204e6f-3e3e-4414-b3ea-da35d807ca4d-maas-api-openshift-ca" pod "08204e6f-3e3e-4414-b3ea-da35d807ca4d" (UID: "08204e6f-3e3e-4414-b3ea-da35d807ca4d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:12:52.383778 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:12:52.383754 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/08204e6f-3e3e-4414-b3ea-da35d807ca4d-maas-api-xks-ca" pod "08204e6f-3e3e-4414-b3ea-da35d807ca4d" (UID: "08204e6f-3e3e-4414-b3ea-da35d807ca4d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:12:52.383860 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:12:52.383787 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08204e6f-3e3e-4414-b3ea-da35d807ca4d-kube-api-access-qltt6" pod "08204e6f-3e3e-4414-b3ea-da35d807ca4d" (UID: "08204e6f-3e3e-4414-b3ea-da35d807ca4d"). InnerVolumeSpecName "kube-api-access-qltt6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:12:52.482808 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:12:52.482741 4611 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qltt6\" (UniqueName: \"kubernetes.io/projected/08204e6f-3e3e-4414-b3ea-da35d807ca4d-kube-api-access-qltt6\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:12:52.482808 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:12:52.482776 4611 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08204e6f-3e3e-4414-b3ea-da35d807ca4d-maas-api-openshift-ca\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:12:52.482808 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:12:52.482793 4611 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08204e6f-3e3e-4414-b3ea-da35d807ca4d-maas-api-xks-ca\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:12:53.175225 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:12:53.175195 4611 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-n7z25" Oct 30 21:12:53.175400 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:12:53.175231 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-n7z25" event={"ID":"08204e6f-3e3e-4414-b3ea-da35d807ca4d","Type":"ContainerDied","Data":"142d57646650cedba91bbdcf9968ee32596f700e6eb7779d98239beebc0c635d"} Oct 30 21:12:53.175400 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:12:53.175277 4611 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="142d57646650cedba91bbdcf9968ee32596f700e6eb7779d98239beebc0c635d" Oct 30 21:13:03.916945 ip-10-0-133-192 kubenswrapper[4611]: E1030 21:13:03.916909 4611 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hwzc6" containerName="extractor" Oct 30 21:13:08.264673 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:13:08.264641 4611 scope.go:122] "RemoveContainer" containerID="ad481ae2fdcccb97669ce22f9621029d69c5219d3213a54487117958b2d140b7" Oct 30 21:13:08.273976 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:13:08.273954 4611 scope.go:122] "RemoveContainer" containerID="a1ed426187dc69dfc6a3d6d7eb2c6a9e9261674e9973d14f47134ca1ed0221a9" Oct 30 21:13:08.281888 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:13:08.281870 4611 scope.go:122] "RemoveContainer" containerID="3bf57584ca0cc4264d8e372e1b09ba939d1d43dc56d0117d081cc10f1b6cd69a" Oct 30 21:13:09.705258 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:13:09.705221 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:13:11.332942 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:13:11.332884 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:13:18.024588 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:13:18.024549 4611 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-hxr75"] Oct 30 21:13:18.027737 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:13:18.027708 4611 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-hxr75"] Oct 30 21:13:19.604918 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:13:19.604882 4611 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="395d5b9b-8e6a-4219-97dd-f550ed328d6a" path="/var/lib/kubelet/pods/395d5b9b-8e6a-4219-97dd-f550ed328d6a/volumes" Oct 30 21:13:21.702294 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:13:21.702260 4611 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-single-main-30d5ac-openshift-default-696b8c9db9mffng"] Oct 30 21:13:21.702786 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:13:21.702665 4611 cpu_manager.go:414] "containerMap: removing container" podUID="08204e6f-3e3e-4414-b3ea-da35d807ca4d" containerName="revoke-keys" Oct 30 21:13:21.702786 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:13:21.702681 4611 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08204e6f-3e3e-4414-b3ea-da35d807ca4d" containerName="revoke-keys" Oct 30 21:13:21.702786 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:13:21.702767 4611 memory_manager.go:365] "RemoveStaleState removing state" podUID="08204e6f-3e3e-4414-b3ea-da35d807ca4d" containerName="revoke-keys" Oct 30 21:13:21.707301 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:13:21.707280 4611 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-30d5ac-openshift-default-696b8c9db9mffng" Oct 30 21:13:21.710466 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:13:21.710440 4611 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-single-main-30d5ac-openshift-default-dockercfg-6h9fj\"" Oct 30 21:13:21.720122 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:13:21.720099 4611 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-30d5ac-openshift-default-696b8c9db9mffng"] Oct 30 21:13:21.756695 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:13:21.756668 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dcdd7301-b310-4518-ad18-02d055ab1f2a-istio-envoy\") pod \"e2e-worker-single-main-30d5ac-openshift-default-696b8c9db9mffng\" (UID: \"dcdd7301-b310-4518-ad18-02d055ab1f2a\") " pod="openshift-ingress/e2e-worker-single-main-30d5ac-openshift-default-696b8c9db9mffng" Oct 30 21:13:21.756855 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:13:21.756727 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dcdd7301-b310-4518-ad18-02d055ab1f2a-istio-token\") pod \"e2e-worker-single-main-30d5ac-openshift-default-696b8c9db9mffng\" (UID: \"dcdd7301-b310-4518-ad18-02d055ab1f2a\") " pod="openshift-ingress/e2e-worker-single-main-30d5ac-openshift-default-696b8c9db9mffng" Oct 30 21:13:21.756855 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:13:21.756808 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcdd7301-b310-4518-ad18-02d055ab1f2a-workload-socket\") pod \"e2e-worker-single-main-30d5ac-openshift-default-696b8c9db9mffng\" (UID: \"dcdd7301-b310-4518-ad18-02d055ab1f2a\") " pod="openshift-ingress/e2e-worker-single-main-30d5ac-openshift-default-696b8c9db9mffng" Oct 30 21:13:21.756967 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:13:21.756915 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dcdd7301-b310-4518-ad18-02d055ab1f2a-istio-data\") pod \"e2e-worker-single-main-30d5ac-openshift-default-696b8c9db9mffng\" (UID: \"dcdd7301-b310-4518-ad18-02d055ab1f2a\") " pod="openshift-ingress/e2e-worker-single-main-30d5ac-openshift-default-696b8c9db9mffng" Oct 30 21:13:21.756967 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:13:21.756955 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dcdd7301-b310-4518-ad18-02d055ab1f2a-istio-podinfo\") pod \"e2e-worker-single-main-30d5ac-openshift-default-696b8c9db9mffng\" (UID: \"dcdd7301-b310-4518-ad18-02d055ab1f2a\") " pod="openshift-ingress/e2e-worker-single-main-30d5ac-openshift-default-696b8c9db9mffng" Oct 30 21:13:21.757057 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:13:21.757014 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dcdd7301-b310-4518-ad18-02d055ab1f2a-workload-certs\") pod \"e2e-worker-single-main-30d5ac-openshift-default-696b8c9db9mffng\" (UID: \"dcdd7301-b310-4518-ad18-02d055ab1f2a\") " pod="openshift-ingress/e2e-worker-single-main-30d5ac-openshift-default-696b8c9db9mffng" Oct 30 21:13:21.757115 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:13:21.757066 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcdd7301-b310-4518-ad18-02d055ab1f2a-credential-socket\") pod \"e2e-worker-single-main-30d5ac-openshift-default-696b8c9db9mffng\" (UID: \"dcdd7301-b310-4518-ad18-02d055ab1f2a\") " pod="openshift-ingress/e2e-worker-single-main-30d5ac-openshift-default-696b8c9db9mffng" Oct 30 21:13:21.757115 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:13:21.757103 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lz7rj\" (UniqueName: \"kubernetes.io/projected/dcdd7301-b310-4518-ad18-02d055ab1f2a-kube-api-access-lz7rj\") pod \"e2e-worker-single-main-30d5ac-openshift-default-696b8c9db9mffng\" (UID: \"dcdd7301-b310-4518-ad18-02d055ab1f2a\") " pod="openshift-ingress/e2e-worker-single-main-30d5ac-openshift-default-696b8c9db9mffng" Oct 30 21:13:21.757204 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:13:21.757136 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dcdd7301-b310-4518-ad18-02d055ab1f2a-istiod-ca-cert\") pod \"e2e-worker-single-main-30d5ac-openshift-default-696b8c9db9mffng\" (UID: \"dcdd7301-b310-4518-ad18-02d055ab1f2a\") " pod="openshift-ingress/e2e-worker-single-main-30d5ac-openshift-default-696b8c9db9mffng" Oct 30 21:13:21.822485 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:13:21.822444 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:13:21.858481 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:13:21.858440 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dcdd7301-b310-4518-ad18-02d055ab1f2a-istio-data\") pod \"e2e-worker-single-main-30d5ac-openshift-default-696b8c9db9mffng\" (UID: \"dcdd7301-b310-4518-ad18-02d055ab1f2a\") " pod="openshift-ingress/e2e-worker-single-main-30d5ac-openshift-default-696b8c9db9mffng" Oct 30 21:13:21.858706 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:13:21.858494 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dcdd7301-b310-4518-ad18-02d055ab1f2a-istio-podinfo\") pod \"e2e-worker-single-main-30d5ac-openshift-default-696b8c9db9mffng\" (UID: \"dcdd7301-b310-4518-ad18-02d055ab1f2a\") " pod="openshift-ingress/e2e-worker-single-main-30d5ac-openshift-default-696b8c9db9mffng" Oct 30 21:13:21.858706 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:13:21.858587 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dcdd7301-b310-4518-ad18-02d055ab1f2a-workload-certs\") pod \"e2e-worker-single-main-30d5ac-openshift-default-696b8c9db9mffng\" (UID: \"dcdd7301-b310-4518-ad18-02d055ab1f2a\") " pod="openshift-ingress/e2e-worker-single-main-30d5ac-openshift-default-696b8c9db9mffng" Oct 30 21:13:21.858706 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:13:21.858619 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcdd7301-b310-4518-ad18-02d055ab1f2a-credential-socket\") pod \"e2e-worker-single-main-30d5ac-openshift-default-696b8c9db9mffng\" (UID: \"dcdd7301-b310-4518-ad18-02d055ab1f2a\") " pod="openshift-ingress/e2e-worker-single-main-30d5ac-openshift-default-696b8c9db9mffng" Oct 30 21:13:21.858706 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:13:21.858645 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lz7rj\" (UniqueName: \"kubernetes.io/projected/dcdd7301-b310-4518-ad18-02d055ab1f2a-kube-api-access-lz7rj\") pod \"e2e-worker-single-main-30d5ac-openshift-default-696b8c9db9mffng\" (UID: \"dcdd7301-b310-4518-ad18-02d055ab1f2a\") " pod="openshift-ingress/e2e-worker-single-main-30d5ac-openshift-default-696b8c9db9mffng" Oct 30 21:13:21.858706 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:13:21.858672 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dcdd7301-b310-4518-ad18-02d055ab1f2a-istiod-ca-cert\") pod \"e2e-worker-single-main-30d5ac-openshift-default-696b8c9db9mffng\" (UID: \"dcdd7301-b310-4518-ad18-02d055ab1f2a\") " pod="openshift-ingress/e2e-worker-single-main-30d5ac-openshift-default-696b8c9db9mffng" Oct 30 21:13:21.858706 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:13:21.858705 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dcdd7301-b310-4518-ad18-02d055ab1f2a-istio-envoy\") pod \"e2e-worker-single-main-30d5ac-openshift-default-696b8c9db9mffng\" (UID: \"dcdd7301-b310-4518-ad18-02d055ab1f2a\") " pod="openshift-ingress/e2e-worker-single-main-30d5ac-openshift-default-696b8c9db9mffng" Oct 30 21:13:21.859001 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:13:21.858789 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dcdd7301-b310-4518-ad18-02d055ab1f2a-istio-token\") pod \"e2e-worker-single-main-30d5ac-openshift-default-696b8c9db9mffng\" (UID: \"dcdd7301-b310-4518-ad18-02d055ab1f2a\") " pod="openshift-ingress/e2e-worker-single-main-30d5ac-openshift-default-696b8c9db9mffng" Oct 30 21:13:21.859001 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:13:21.858817 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcdd7301-b310-4518-ad18-02d055ab1f2a-workload-socket\") pod \"e2e-worker-single-main-30d5ac-openshift-default-696b8c9db9mffng\" (UID: \"dcdd7301-b310-4518-ad18-02d055ab1f2a\") " pod="openshift-ingress/e2e-worker-single-main-30d5ac-openshift-default-696b8c9db9mffng" Oct 30 21:13:21.859001 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:13:21.858944 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dcdd7301-b310-4518-ad18-02d055ab1f2a-istio-data\") pod \"e2e-worker-single-main-30d5ac-openshift-default-696b8c9db9mffng\" (UID: \"dcdd7301-b310-4518-ad18-02d055ab1f2a\") " pod="openshift-ingress/e2e-worker-single-main-30d5ac-openshift-default-696b8c9db9mffng" Oct 30 21:13:21.859160 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:13:21.859042 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcdd7301-b310-4518-ad18-02d055ab1f2a-credential-socket\") pod \"e2e-worker-single-main-30d5ac-openshift-default-696b8c9db9mffng\" (UID: \"dcdd7301-b310-4518-ad18-02d055ab1f2a\") " pod="openshift-ingress/e2e-worker-single-main-30d5ac-openshift-default-696b8c9db9mffng" Oct 30 21:13:21.859320 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:13:21.859289 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcdd7301-b310-4518-ad18-02d055ab1f2a-workload-socket\") pod \"e2e-worker-single-main-30d5ac-openshift-default-696b8c9db9mffng\" (UID: \"dcdd7301-b310-4518-ad18-02d055ab1f2a\") " pod="openshift-ingress/e2e-worker-single-main-30d5ac-openshift-default-696b8c9db9mffng" Oct 30 21:13:21.859320 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:13:21.859306 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dcdd7301-b310-4518-ad18-02d055ab1f2a-workload-certs\") pod \"e2e-worker-single-main-30d5ac-openshift-default-696b8c9db9mffng\" (UID: \"dcdd7301-b310-4518-ad18-02d055ab1f2a\") " pod="openshift-ingress/e2e-worker-single-main-30d5ac-openshift-default-696b8c9db9mffng" Oct 30 21:13:21.859475 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:13:21.859458 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dcdd7301-b310-4518-ad18-02d055ab1f2a-istiod-ca-cert\") pod \"e2e-worker-single-main-30d5ac-openshift-default-696b8c9db9mffng\" (UID: \"dcdd7301-b310-4518-ad18-02d055ab1f2a\") " pod="openshift-ingress/e2e-worker-single-main-30d5ac-openshift-default-696b8c9db9mffng" Oct 30 21:13:21.861140 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:13:21.861120 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dcdd7301-b310-4518-ad18-02d055ab1f2a-istio-envoy\") pod \"e2e-worker-single-main-30d5ac-openshift-default-696b8c9db9mffng\" (UID: \"dcdd7301-b310-4518-ad18-02d055ab1f2a\") " pod="openshift-ingress/e2e-worker-single-main-30d5ac-openshift-default-696b8c9db9mffng" Oct 30 21:13:21.861274 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:13:21.861254 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dcdd7301-b310-4518-ad18-02d055ab1f2a-istio-podinfo\") pod \"e2e-worker-single-main-30d5ac-openshift-default-696b8c9db9mffng\" (UID: \"dcdd7301-b310-4518-ad18-02d055ab1f2a\") " pod="openshift-ingress/e2e-worker-single-main-30d5ac-openshift-default-696b8c9db9mffng" Oct 30 21:13:21.870699 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:13:21.870679 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dcdd7301-b310-4518-ad18-02d055ab1f2a-istio-token\") pod \"e2e-worker-single-main-30d5ac-openshift-default-696b8c9db9mffng\" (UID: \"dcdd7301-b310-4518-ad18-02d055ab1f2a\") " pod="openshift-ingress/e2e-worker-single-main-30d5ac-openshift-default-696b8c9db9mffng" Oct 30 21:13:21.870919 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:13:21.870900 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lz7rj\" (UniqueName: \"kubernetes.io/projected/dcdd7301-b310-4518-ad18-02d055ab1f2a-kube-api-access-lz7rj\") pod \"e2e-worker-single-main-30d5ac-openshift-default-696b8c9db9mffng\" (UID: \"dcdd7301-b310-4518-ad18-02d055ab1f2a\") " pod="openshift-ingress/e2e-worker-single-main-30d5ac-openshift-default-696b8c9db9mffng" Oct 30 21:13:22.020559 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:13:22.020455 4611 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-30d5ac-openshift-default-696b8c9db9mffng" Oct 30 21:13:22.035533 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:13:22.035473 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:13:22.316004 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:13:22.315773 4611 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-30d5ac-openshift-default-696b8c9db9mffng"] Oct 30 21:13:22.318328 ip-10-0-133-192 kubenswrapper[4611]: W1030 21:13:22.318290 4611 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddcdd7301_b310_4518_ad18_02d055ab1f2a.slice/crio-0ce0a0392cd32539cc4d5eda5ab83aae760cacf75b6f6ba4f61f8324b1e1f876 WatchSource:0}: Error finding container 0ce0a0392cd32539cc4d5eda5ab83aae760cacf75b6f6ba4f61f8324b1e1f876: Status 404 returned error can't find the container with id 0ce0a0392cd32539cc4d5eda5ab83aae760cacf75b6f6ba4f61f8324b1e1f876 Oct 30 21:13:22.321081 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:13:22.321049 4611 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:13:22.321185 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:13:22.321124 4611 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:13:22.321185 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:13:22.321160 4611 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:13:23.326949 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:13:23.326904 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-30d5ac-openshift-default-696b8c9db9mffng" event={"ID":"dcdd7301-b310-4518-ad18-02d055ab1f2a","Type":"ContainerStarted","Data":"eb685aeea054fb9755d88ed1ff22d760aba38b09830ac99e5138982263be5a76"} Oct 30 21:13:23.327348 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:13:23.326967 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-30d5ac-openshift-default-696b8c9db9mffng" event={"ID":"dcdd7301-b310-4518-ad18-02d055ab1f2a","Type":"ContainerStarted","Data":"0ce0a0392cd32539cc4d5eda5ab83aae760cacf75b6f6ba4f61f8324b1e1f876"} Oct 30 21:13:23.357017 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:13:23.354791 4611 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-single-main-30d5ac-openshift-default-696b8c9db9mffng" podStartSLOduration=2.354773057 podStartE2EDuration="2.354773057s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:13:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:13:23.350035437 +0000 UTC m=+2057.884062055" watchObservedRunningTime="2026-10-30 21:13:23.354773057 +0000 UTC m=+2057.888799673" Oct 30 21:13:24.020678 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:13:24.020640 4611 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-30d5ac-openshift-default-696b8c9db9mffng" Oct 30 21:13:24.025697 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:13:24.025671 4611 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-30d5ac-openshift-default-696b8c9db9mffng" Oct 30 21:13:24.332115 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:13:24.332036 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-30d5ac-openshift-default-696b8c9db9mffng" Oct 30 21:13:24.332965 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:13:24.332944 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-30d5ac-openshift-default-696b8c9db9mffng" Oct 30 21:13:25.650135 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:13:25.650098 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:13:25.797325 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:13:25.797297 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:13:26.713017 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:13:26.712984 4611 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-30d5ac-6bcd5f86d6-br5ns"] Oct 30 21:13:26.716742 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:13:26.716720 4611 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-30d5ac-6bcd5f86d6-br5ns" Oct 30 21:13:26.723143 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:13:26.723122 4611 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:13:26.723806 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:13:26.723778 4611 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-k976f\"" Oct 30 21:13:26.723936 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:13:26.723902 4611 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Oct 30 21:13:26.725820 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:13:26.725801 4611 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-single-main-30d5ac-serving-cert\"" Oct 30 21:13:26.725992 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:13:26.725972 4611 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:13:26.728942 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:13:26.728920 4611 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-30d5ac-6bcd5f86d6-br5ns"] Oct 30 21:13:26.811129 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:13:26.811095 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6df48c84-6810-48a5-960f-98d6f5288994-metrics-certs\") pod \"maas-api-e2e-worker-single-main-30d5ac-6bcd5f86d6-br5ns\" (UID: \"6df48c84-6810-48a5-960f-98d6f5288994\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-30d5ac-6bcd5f86d6-br5ns" Oct 30 21:13:26.811129 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:13:26.811133 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6df48c84-6810-48a5-960f-98d6f5288994-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-30d5ac-6bcd5f86d6-br5ns\" (UID: \"6df48c84-6810-48a5-960f-98d6f5288994\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-30d5ac-6bcd5f86d6-br5ns" Oct 30 21:13:26.811377 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:13:26.811260 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2nq7c\" (UniqueName: \"kubernetes.io/projected/6df48c84-6810-48a5-960f-98d6f5288994-kube-api-access-2nq7c\") pod \"maas-api-e2e-worker-single-main-30d5ac-6bcd5f86d6-br5ns\" (UID: \"6df48c84-6810-48a5-960f-98d6f5288994\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-30d5ac-6bcd5f86d6-br5ns" Oct 30 21:13:26.811377 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:13:26.811362 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6df48c84-6810-48a5-960f-98d6f5288994-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-30d5ac-6bcd5f86d6-br5ns\" (UID: \"6df48c84-6810-48a5-960f-98d6f5288994\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-30d5ac-6bcd5f86d6-br5ns" Oct 30 21:13:26.829294 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:13:26.829265 4611 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-30d5ac-549c7w6hcp"] Oct 30 21:13:26.832897 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:13:26.832881 4611 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-30d5ac-549c7w6hcp" Oct 30 21:13:26.839503 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:13:26.839464 4611 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-8843dd91-dockercfg-fw72h\"" Oct 30 21:13:26.840577 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:13:26.840556 4611 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-30d5ac\"" Oct 30 21:13:26.843547 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:13:26.843527 4611 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-30d5ac-549c7w6hcp"] Oct 30 21:13:26.878740 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:13:26.878713 4611 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-30d5ac-55684b65b7z28r"] Oct 30 21:13:26.882078 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:13:26.882061 4611 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-30d5ac-55684b65b7z28r" Oct 30 21:13:26.893794 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:13:26.893771 4611 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-30d5ac-55684b65b7z28r"] Oct 30 21:13:26.912717 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:13:26.912690 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/259e63af-b601-4d70-bbf9-fc86fa181634-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-30d5ac-549c7w6hcp\" (UID: \"259e63af-b601-4d70-bbf9-fc86fa181634\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-30d5ac-549c7w6hcp" Oct 30 21:13:26.912858 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:13:26.912765 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rglt7\" (UniqueName: \"kubernetes.io/projected/6cecd88c-66fd-4f1a-93c3-c4c00cd0622c-kube-api-access-rglt7\") pod \"payload-processing-e2e-worker-single-main-30d5ac-55684b65b7z28r\" (UID: \"6cecd88c-66fd-4f1a-93c3-c4c00cd0622c\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-30d5ac-55684b65b7z28r" Oct 30 21:13:26.912904 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:13:26.912880 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6df48c84-6810-48a5-960f-98d6f5288994-metrics-certs\") pod \"maas-api-e2e-worker-single-main-30d5ac-6bcd5f86d6-br5ns\" (UID: \"6df48c84-6810-48a5-960f-98d6f5288994\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-30d5ac-6bcd5f86d6-br5ns" Oct 30 21:13:26.912947 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:13:26.912923 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wcwst\" (UniqueName: \"kubernetes.io/projected/259e63af-b601-4d70-bbf9-fc86fa181634-kube-api-access-wcwst\") pod \"payload-pre-processing-e2e-worker-single-main-30d5ac-549c7w6hcp\" (UID: \"259e63af-b601-4d70-bbf9-fc86fa181634\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-30d5ac-549c7w6hcp" Oct 30 21:13:26.912984 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:13:26.912966 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6df48c84-6810-48a5-960f-98d6f5288994-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-30d5ac-6bcd5f86d6-br5ns\" (UID: \"6df48c84-6810-48a5-960f-98d6f5288994\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-30d5ac-6bcd5f86d6-br5ns" Oct 30 21:13:26.913191 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:13:26.913172 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2nq7c\" (UniqueName: \"kubernetes.io/projected/6df48c84-6810-48a5-960f-98d6f5288994-kube-api-access-2nq7c\") pod \"maas-api-e2e-worker-single-main-30d5ac-6bcd5f86d6-br5ns\" (UID: \"6df48c84-6810-48a5-960f-98d6f5288994\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-30d5ac-6bcd5f86d6-br5ns" Oct 30 21:13:26.913251 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:13:26.913233 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6cecd88c-66fd-4f1a-93c3-c4c00cd0622c-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-30d5ac-55684b65b7z28r\" (UID: \"6cecd88c-66fd-4f1a-93c3-c4c00cd0622c\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-30d5ac-55684b65b7z28r" Oct 30 21:13:26.913308 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:13:26.913281 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6df48c84-6810-48a5-960f-98d6f5288994-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-30d5ac-6bcd5f86d6-br5ns\" (UID: \"6df48c84-6810-48a5-960f-98d6f5288994\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-30d5ac-6bcd5f86d6-br5ns" Oct 30 21:13:26.913688 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:13:26.913667 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6df48c84-6810-48a5-960f-98d6f5288994-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-30d5ac-6bcd5f86d6-br5ns\" (UID: \"6df48c84-6810-48a5-960f-98d6f5288994\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-30d5ac-6bcd5f86d6-br5ns" Oct 30 21:13:26.915567 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:13:26.915548 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6df48c84-6810-48a5-960f-98d6f5288994-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-30d5ac-6bcd5f86d6-br5ns\" (UID: \"6df48c84-6810-48a5-960f-98d6f5288994\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-30d5ac-6bcd5f86d6-br5ns" Oct 30 21:13:26.915969 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:13:26.915945 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6df48c84-6810-48a5-960f-98d6f5288994-metrics-certs\") pod \"maas-api-e2e-worker-single-main-30d5ac-6bcd5f86d6-br5ns\" (UID: \"6df48c84-6810-48a5-960f-98d6f5288994\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-30d5ac-6bcd5f86d6-br5ns" Oct 30 21:13:26.917941 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:13:26.917917 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:13:26.924279 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:13:26.924258 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2nq7c\" (UniqueName: \"kubernetes.io/projected/6df48c84-6810-48a5-960f-98d6f5288994-kube-api-access-2nq7c\") pod \"maas-api-e2e-worker-single-main-30d5ac-6bcd5f86d6-br5ns\" (UID: \"6df48c84-6810-48a5-960f-98d6f5288994\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-30d5ac-6bcd5f86d6-br5ns" Oct 30 21:13:27.014002 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:13:27.013926 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6cecd88c-66fd-4f1a-93c3-c4c00cd0622c-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-30d5ac-55684b65b7z28r\" (UID: \"6cecd88c-66fd-4f1a-93c3-c4c00cd0622c\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-30d5ac-55684b65b7z28r" Oct 30 21:13:27.014002 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:13:27.013981 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/259e63af-b601-4d70-bbf9-fc86fa181634-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-30d5ac-549c7w6hcp\" (UID: \"259e63af-b601-4d70-bbf9-fc86fa181634\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-30d5ac-549c7w6hcp" Oct 30 21:13:27.014184 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:13:27.014029 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rglt7\" (UniqueName: \"kubernetes.io/projected/6cecd88c-66fd-4f1a-93c3-c4c00cd0622c-kube-api-access-rglt7\") pod \"payload-processing-e2e-worker-single-main-30d5ac-55684b65b7z28r\" (UID: \"6cecd88c-66fd-4f1a-93c3-c4c00cd0622c\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-30d5ac-55684b65b7z28r" Oct 30 21:13:27.014184 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:13:27.014067 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wcwst\" (UniqueName: \"kubernetes.io/projected/259e63af-b601-4d70-bbf9-fc86fa181634-kube-api-access-wcwst\") pod \"payload-pre-processing-e2e-worker-single-main-30d5ac-549c7w6hcp\" (UID: \"259e63af-b601-4d70-bbf9-fc86fa181634\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-30d5ac-549c7w6hcp" Oct 30 21:13:27.014757 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:13:27.014735 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6cecd88c-66fd-4f1a-93c3-c4c00cd0622c-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-30d5ac-55684b65b7z28r\" (UID: \"6cecd88c-66fd-4f1a-93c3-c4c00cd0622c\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-30d5ac-55684b65b7z28r" Oct 30 21:13:27.014823 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:13:27.014761 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/259e63af-b601-4d70-bbf9-fc86fa181634-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-30d5ac-549c7w6hcp\" (UID: \"259e63af-b601-4d70-bbf9-fc86fa181634\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-30d5ac-549c7w6hcp" Oct 30 21:13:27.025963 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:13:27.025933 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rglt7\" (UniqueName: \"kubernetes.io/projected/6cecd88c-66fd-4f1a-93c3-c4c00cd0622c-kube-api-access-rglt7\") pod \"payload-processing-e2e-worker-single-main-30d5ac-55684b65b7z28r\" (UID: \"6cecd88c-66fd-4f1a-93c3-c4c00cd0622c\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-30d5ac-55684b65b7z28r" Oct 30 21:13:27.026084 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:13:27.026011 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wcwst\" (UniqueName: \"kubernetes.io/projected/259e63af-b601-4d70-bbf9-fc86fa181634-kube-api-access-wcwst\") pod \"payload-pre-processing-e2e-worker-single-main-30d5ac-549c7w6hcp\" (UID: \"259e63af-b601-4d70-bbf9-fc86fa181634\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-30d5ac-549c7w6hcp" Oct 30 21:13:27.033829 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:13:27.033809 4611 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-30d5ac-6bcd5f86d6-br5ns" Oct 30 21:13:27.074823 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:13:27.074786 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:13:27.147019 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:13:27.146989 4611 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-30d5ac-549c7w6hcp" Oct 30 21:13:27.195583 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:13:27.195543 4611 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-30d5ac-55684b65b7z28r" Oct 30 21:13:27.333937 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:13:27.333167 4611 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-30d5ac-6bcd5f86d6-br5ns"] Oct 30 21:13:27.336530 ip-10-0-133-192 kubenswrapper[4611]: W1030 21:13:27.336489 4611 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6df48c84_6810_48a5_960f_98d6f5288994.slice/crio-d2c3c1711d910e8b3c4ac49da44ed9d764ac902547709142782db5c340a9ae6a WatchSource:0}: Error finding container d2c3c1711d910e8b3c4ac49da44ed9d764ac902547709142782db5c340a9ae6a: Status 404 returned error can't find the container with id d2c3c1711d910e8b3c4ac49da44ed9d764ac902547709142782db5c340a9ae6a Oct 30 21:13:27.338267 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:13:27.338246 4611 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 21:13:27.348987 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:13:27.348960 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-30d5ac-6bcd5f86d6-br5ns" event={"ID":"6df48c84-6810-48a5-960f-98d6f5288994","Type":"ContainerStarted","Data":"d2c3c1711d910e8b3c4ac49da44ed9d764ac902547709142782db5c340a9ae6a"} Oct 30 21:13:27.440615 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:13:27.440587 4611 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-30d5ac-549c7w6hcp"] Oct 30 21:13:27.442458 ip-10-0-133-192 kubenswrapper[4611]: W1030 21:13:27.442431 4611 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod259e63af_b601_4d70_bbf9_fc86fa181634.slice/crio-f374885765512df2529c6a5b5d4ab01c9718050ee9b181bd523077cd45671583 WatchSource:0}: Error finding container f374885765512df2529c6a5b5d4ab01c9718050ee9b181bd523077cd45671583: Status 404 returned error can't find the container with id f374885765512df2529c6a5b5d4ab01c9718050ee9b181bd523077cd45671583 Oct 30 21:13:27.487684 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:13:27.487654 4611 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-30d5ac-55684b65b7z28r"] Oct 30 21:13:27.489684 ip-10-0-133-192 kubenswrapper[4611]: W1030 21:13:27.489660 4611 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6cecd88c_66fd_4f1a_93c3_c4c00cd0622c.slice/crio-b5e73eb7c73109c9df0d386828157322f17c7d02a19d3fd1863db667cb7f642b WatchSource:0}: Error finding container b5e73eb7c73109c9df0d386828157322f17c7d02a19d3fd1863db667cb7f642b: Status 404 returned error can't find the container with id b5e73eb7c73109c9df0d386828157322f17c7d02a19d3fd1863db667cb7f642b Oct 30 21:13:27.817963 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:13:27.817937 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:13:28.355871 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:13:28.355818 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-30d5ac-6bcd5f86d6-br5ns" event={"ID":"6df48c84-6810-48a5-960f-98d6f5288994","Type":"ContainerStarted","Data":"bfe0b9a0253aed42cf27bbd167af5a164cce1ccdbec9c515664522f4abf1d1bd"} Oct 30 21:13:28.356057 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:13:28.355929 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-30d5ac-6bcd5f86d6-br5ns" Oct 30 21:13:28.357447 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:13:28.357415 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-30d5ac-55684b65b7z28r" event={"ID":"6cecd88c-66fd-4f1a-93c3-c4c00cd0622c","Type":"ContainerStarted","Data":"5e4cd84cf5ea01b90903cb71b5ac6075f14cf71fcf512401a47144d70ac3f513"} Oct 30 21:13:28.357592 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:13:28.357467 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-30d5ac-55684b65b7z28r" event={"ID":"6cecd88c-66fd-4f1a-93c3-c4c00cd0622c","Type":"ContainerStarted","Data":"b5e73eb7c73109c9df0d386828157322f17c7d02a19d3fd1863db667cb7f642b"} Oct 30 21:13:28.357592 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:13:28.357543 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-30d5ac-55684b65b7z28r" Oct 30 21:13:28.359022 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:13:28.358999 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-30d5ac-549c7w6hcp" event={"ID":"259e63af-b601-4d70-bbf9-fc86fa181634","Type":"ContainerStarted","Data":"be6d15107d37a4ef06272580929e7c43afdaf72704add160193fc560de531737"} Oct 30 21:13:28.359112 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:13:28.359038 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-30d5ac-549c7w6hcp" event={"ID":"259e63af-b601-4d70-bbf9-fc86fa181634","Type":"ContainerStarted","Data":"f374885765512df2529c6a5b5d4ab01c9718050ee9b181bd523077cd45671583"} Oct 30 21:13:28.359112 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:13:28.359056 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-30d5ac-549c7w6hcp" Oct 30 21:13:28.443103 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:13:28.443054 4611 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-30d5ac-6bcd5f86d6-br5ns" podStartSLOduration=1.976471482 podStartE2EDuration="2.44303953s" totalImagesPullingTime="466.568048ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:13:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:13:28.440914688 +0000 UTC m=+2062.974941305" watchObservedRunningTime="2026-10-30 21:13:28.44303953 +0000 UTC m=+2062.977066475" Oct 30 21:13:28.500006 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:13:28.499959 4611 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-30d5ac-549c7w6hcp" podStartSLOduration=2.499946697 podStartE2EDuration="2.499946697s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:13:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:13:28.498395583 +0000 UTC m=+2063.032422199" watchObservedRunningTime="2026-10-30 21:13:28.499946697 +0000 UTC m=+2063.033973338" Oct 30 21:13:28.536771 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:13:28.536724 4611 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-30d5ac-55684b65b7z28r" podStartSLOduration=2.5367122589999997 podStartE2EDuration="2.536712259s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:13:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:13:28.534841966 +0000 UTC m=+2063.068868579" watchObservedRunningTime="2026-10-30 21:13:28.536712259 +0000 UTC m=+2063.070738875" Oct 30 21:13:28.537533 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:13:28.537480 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:13:29.023520 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:13:29.023470 4611 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-55fn8"] Oct 30 21:13:29.028805 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:13:29.028775 4611 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-55fn8"] Oct 30 21:13:29.200976 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:13:29.200944 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:13:29.619130 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:13:29.619090 4611 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="edb49403-8ee6-481d-b874-67c0d802f201" path="/var/lib/kubelet/pods/edb49403-8ee6-481d-b874-67c0d802f201/volumes" Oct 30 21:13:30.114690 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:13:30.114660 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:13:30.917373 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:13:30.917338 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:13:31.411969 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:13:31.411934 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:13:32.011093 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:13:32.010858 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:13:32.617074 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:13:32.617039 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:13:33.198292 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:13:33.198254 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:13:33.874532 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:13:33.872489 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:13:33.917262 ip-10-0-133-192 kubenswrapper[4611]: E1030 21:13:33.917223 4611 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hwzc6" containerName="extractor" Oct 30 21:13:34.374904 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:13:34.374878 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-30d5ac-6bcd5f86d6-br5ns" Oct 30 21:13:34.503681 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:13:34.503639 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:13:35.108558 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:13:35.108518 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:13:35.843624 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:13:35.843582 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:13:36.598140 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:13:36.598050 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:13:37.271798 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:13:37.271760 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:13:37.888620 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:13:37.888451 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:13:38.656287 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:13:38.656248 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:13:39.275892 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:13:39.275854 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:13:39.334102 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:13:39.334074 4611 patch_prober.go:29] interesting pod/e2e-worker-single-main-30d5ac-openshift-default-696b8c9db9mffng container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" start-of-body= Oct 30 21:13:39.334211 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:13:39.334154 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-single-main-30d5ac-openshift-default-696b8c9db9mffng" podUID="dcdd7301-b310-4518-ad18-02d055ab1f2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" Oct 30 21:13:39.367217 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:13:39.367195 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-30d5ac-55684b65b7z28r" Oct 30 21:13:39.367366 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:13:39.367354 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-30d5ac-549c7w6hcp" Oct 30 21:13:39.425601 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:13:39.425527 4611 generic.go:356] "Generic (PLEG): container finished" podID="dcdd7301-b310-4518-ad18-02d055ab1f2a" containerID="eb685aeea054fb9755d88ed1ff22d760aba38b09830ac99e5138982263be5a76" exitCode=0 Oct 30 21:13:39.425764 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:13:39.425612 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-30d5ac-openshift-default-696b8c9db9mffng" event={"ID":"dcdd7301-b310-4518-ad18-02d055ab1f2a","Type":"ContainerDied","Data":"eb685aeea054fb9755d88ed1ff22d760aba38b09830ac99e5138982263be5a76"} Oct 30 21:13:39.426023 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:13:39.426005 4611 scope.go:122] "RemoveContainer" containerID="eb685aeea054fb9755d88ed1ff22d760aba38b09830ac99e5138982263be5a76" Oct 30 21:13:39.427334 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:13:39.427306 4611 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:13:39.427441 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:13:39.427369 4611 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:13:39.427441 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:13:39.427401 4611 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:13:40.079119 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:13:40.079080 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:13:40.432722 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:13:40.432681 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-30d5ac-openshift-default-696b8c9db9mffng" event={"ID":"dcdd7301-b310-4518-ad18-02d055ab1f2a","Type":"ContainerStarted","Data":"79a8266c768c3ebdc816f424cab5847e114636e2bd6cc1add4ccf96495350af4"} Oct 30 21:13:40.689162 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:13:40.689056 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:13:40.830708 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:13:40.830641 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:13:41.020763 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:13:41.020690 4611 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-30d5ac-openshift-default-696b8c9db9mffng" Oct 30 21:13:41.020939 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:13:41.020918 4611 patch_prober.go:29] interesting pod/e2e-worker-single-main-30d5ac-openshift-default-696b8c9db9mffng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" start-of-body= Oct 30 21:13:41.020993 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:13:41.020968 4611 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-30d5ac-openshift-default-696b8c9db9mffng" podUID="dcdd7301-b310-4518-ad18-02d055ab1f2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" Oct 30 21:13:41.299582 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:13:41.299465 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:13:41.469533 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:13:41.469467 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:13:41.614464 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:13:41.614432 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:13:42.021015 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:13:42.020934 4611 patch_prober.go:29] interesting pod/e2e-worker-single-main-30d5ac-openshift-default-696b8c9db9mffng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" start-of-body= Oct 30 21:13:42.021015 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:13:42.020983 4611 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-30d5ac-openshift-default-696b8c9db9mffng" podUID="dcdd7301-b310-4518-ad18-02d055ab1f2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" Oct 30 21:13:42.266533 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:13:42.262435 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:13:42.500894 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:13:42.500855 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:13:43.014034 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:13:43.013993 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:13:43.021138 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:13:43.021104 4611 patch_prober.go:29] interesting pod/e2e-worker-single-main-30d5ac-openshift-default-696b8c9db9mffng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" start-of-body= Oct 30 21:13:43.021333 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:13:43.021171 4611 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-30d5ac-openshift-default-696b8c9db9mffng" podUID="dcdd7301-b310-4518-ad18-02d055ab1f2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" Oct 30 21:13:43.727360 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:13:43.727323 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:13:44.021372 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:13:44.021282 4611 patch_prober.go:29] interesting pod/e2e-worker-single-main-30d5ac-openshift-default-696b8c9db9mffng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" start-of-body= Oct 30 21:13:44.021560 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:13:44.021366 4611 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-30d5ac-openshift-default-696b8c9db9mffng" podUID="dcdd7301-b310-4518-ad18-02d055ab1f2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" Oct 30 21:13:44.678534 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:13:44.677172 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:13:45.021713 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:13:45.021632 4611 patch_prober.go:29] interesting pod/e2e-worker-single-main-30d5ac-openshift-default-696b8c9db9mffng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" start-of-body= Oct 30 21:13:45.022092 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:13:45.021705 4611 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-30d5ac-openshift-default-696b8c9db9mffng" podUID="dcdd7301-b310-4518-ad18-02d055ab1f2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" Oct 30 21:13:45.606754 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:13:45.606718 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:13:46.024716 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:13:46.024645 4611 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-30d5ac-openshift-default-696b8c9db9mffng" Oct 30 21:13:46.025129 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:13:46.024985 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-30d5ac-openshift-default-696b8c9db9mffng" Oct 30 21:13:46.025607 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:13:46.025590 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-30d5ac-openshift-default-696b8c9db9mffng" Oct 30 21:13:46.205146 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:13:46.205109 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:13:46.894826 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:13:46.894791 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:13:47.704048 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:13:47.703990 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:13:48.398669 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:13:48.398632 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:13:49.017974 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:13:49.017939 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:13:49.565330 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:13:49.565299 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:13:49.722178 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:13:49.722140 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:13:50.340042 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:13:50.340002 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:13:50.992718 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:13:50.992652 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:13:52.318306 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:13:52.318247 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:13:52.947599 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:13:52.947559 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:13:53.595334 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:13:53.595279 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:13:54.399754 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:13:54.399704 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:13:55.181399 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:13:55.181362 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:13:55.803215 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:13:55.803180 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:13:56.414674 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:13:56.414637 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:13:57.106268 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:13:57.106189 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:13:57.843930 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:13:57.843896 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:13:58.600236 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:13:58.600193 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:13:59.346584 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:13:59.346542 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:14:00.052578 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:14:00.052540 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:14:00.828785 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:14:00.828747 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:14:01.477436 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:14:01.477396 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:14:02.101455 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:14:02.101404 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:14:02.769156 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:14:02.769117 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:14:03.395906 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:14:03.395860 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:14:03.917201 ip-10-0-133-192 kubenswrapper[4611]: E1030 21:14:03.917164 4611 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hwzc6" containerName="extractor" Oct 30 21:14:04.089102 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:14:04.089067 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:14:04.816363 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:14:04.816322 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:14:05.493460 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:14:05.493420 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:14:05.901890 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:14:05.901865 4611 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7w5wz_7e2574c1-b59c-453c-9501-25ec13a1ccad/ovn-acl-logging/0.log" Oct 30 21:14:05.910566 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:14:05.910544 4611 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7w5wz_7e2574c1-b59c-453c-9501-25ec13a1ccad/ovn-acl-logging/0.log" Oct 30 21:14:06.182769 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:14:06.182565 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:14:06.813899 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:14:06.813868 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:14:07.502694 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:14:07.502655 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:14:08.087817 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:14:08.087781 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:14:08.312000 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:14:08.311970 4611 scope.go:122] "RemoveContainer" containerID="0603157d4809fe0dd66ded80eb78bf01c70638061cd19f7392a67c9e7dcd86b0" Oct 30 21:14:08.322840 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:14:08.322822 4611 scope.go:122] "RemoveContainer" containerID="c787c074b85f2ce8b6b686e0cc2bd9959d1c1008295ef063f64941f62fced449" Oct 30 21:14:08.740391 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:14:08.740358 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:14:09.408225 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:14:09.408171 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:14:10.012620 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:14:10.012580 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:14:10.830687 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:14:10.830646 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:14:11.576477 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:14:11.576362 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:14:12.183434 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:14:12.183400 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:14:12.812247 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:14:12.812210 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:14:13.481272 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:14:13.481234 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:14:14.093876 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:14:14.093827 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:14:14.808203 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:14:14.808161 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:14:15.500691 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:14:15.500655 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:14:16.189828 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:14:16.189789 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:14:16.813324 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:14:16.813195 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:14:17.503637 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:14:17.503585 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:14:18.101069 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:14:18.100869 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:14:18.707913 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:14:18.707879 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:14:19.421564 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:14:19.421499 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:14:20.099049 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:14:20.098961 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:14:20.703967 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:14:20.703929 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:14:21.379575 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:14:21.379537 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:14:22.096695 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:14:22.096659 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:14:22.822156 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:14:22.822114 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:14:23.614965 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:14:23.614927 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:14:24.357779 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:14:24.357738 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:14:24.991099 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:14:24.991051 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:14:25.600696 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:14:25.600626 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:14:26.344574 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:14:26.344535 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:14:26.889427 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:14:26.889393 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:14:27.494266 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:14:27.494231 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:14:28.091980 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:14:28.091948 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:14:28.722946 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:14:28.722911 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:14:29.496930 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:14:29.496892 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:14:30.133479 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:14:30.133445 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:14:30.801127 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:14:30.801062 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:14:31.496814 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:14:31.496724 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:14:32.155470 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:14:32.155439 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:14:32.322293 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:14:32.322256 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:14:32.815813 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:14:32.815749 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:14:33.131872 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:14:33.131837 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:14:33.508010 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:14:33.507926 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:14:33.916174 ip-10-0-133-192 kubenswrapper[4611]: E1030 21:14:33.916138 4611 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hwzc6" containerName="extractor" Oct 30 21:14:34.196198 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:14:34.196116 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:14:34.795826 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:14:34.795785 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:14:35.487315 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:14:35.487268 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:14:36.201950 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:14:36.201905 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:14:36.926289 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:14:36.926251 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:14:37.090413 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:14:37.090375 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:14:37.615094 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:14:37.615000 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:14:38.249266 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:14:38.249228 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:14:38.450911 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:14:38.450875 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:14:38.893718 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:14:38.893682 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:14:39.508150 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:14:39.508103 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:14:40.100386 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:14:40.100349 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:14:40.711524 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:14:40.711476 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:14:41.306092 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:14:41.306049 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:14:42.008972 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:14:42.008935 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:14:42.607948 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:14:42.607906 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:14:43.242556 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:14:43.242493 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:14:43.893479 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:14:43.893440 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:14:44.500252 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:14:44.500213 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:14:45.243422 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:14:45.243385 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:14:45.820479 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:14:45.820443 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:14:46.456211 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:14:46.456177 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:14:47.216024 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:14:47.215953 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:14:47.878341 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:14:47.878299 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:14:48.627911 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:14:48.627874 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:14:49.269041 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:14:49.269002 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:14:49.609207 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:14:49.609177 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:14:49.977865 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:14:49.977756 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:14:50.609568 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:14:50.609535 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:14:51.283149 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:14:51.283101 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:14:52.102529 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:14:52.102479 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:14:52.581375 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:14:52.581263 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:14:52.884348 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:14:52.884294 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:14:53.342773 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:14:53.342698 4611 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-30d5ac-55684b65b7z28r"] Oct 30 21:14:53.353219 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:14:53.353188 4611 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-30d5ac-549c7w6hcp"] Oct 30 21:14:53.462263 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:14:53.462228 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:14:53.617625 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:14:53.617585 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:14:53.944357 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:14:53.944274 4611 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-30d5ac-6bcd5f86d6-br5ns"] Oct 30 21:14:53.944560 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:14:53.944481 4611 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-30d5ac-6bcd5f86d6-br5ns" podUID="6df48c84-6810-48a5-960f-98d6f5288994" containerName="maas-api" containerID="cri-o://bfe0b9a0253aed42cf27bbd167af5a164cce1ccdbec9c515664522f4abf1d1bd" gracePeriod=30 Oct 30 21:14:54.021454 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:14:54.021423 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:14:54.168350 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:14:54.168321 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:14:54.224052 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:14:54.224030 4611 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-30d5ac-6bcd5f86d6-br5ns" Oct 30 21:14:54.313149 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:14:54.313111 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6df48c84-6810-48a5-960f-98d6f5288994-metrics-certs\" (UniqueName: \"kubernetes.io/secret/6df48c84-6810-48a5-960f-98d6f5288994-metrics-certs\") pod \"6df48c84-6810-48a5-960f-98d6f5288994\" (UID: \"6df48c84-6810-48a5-960f-98d6f5288994\") " Oct 30 21:14:54.313322 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:14:54.313225 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6df48c84-6810-48a5-960f-98d6f5288994-kube-api-access-2nq7c\" (UniqueName: \"kubernetes.io/projected/6df48c84-6810-48a5-960f-98d6f5288994-kube-api-access-2nq7c\") pod \"6df48c84-6810-48a5-960f-98d6f5288994\" (UID: \"6df48c84-6810-48a5-960f-98d6f5288994\") " Oct 30 21:14:54.313322 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:14:54.313269 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6df48c84-6810-48a5-960f-98d6f5288994-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6df48c84-6810-48a5-960f-98d6f5288994-maas-api-tls\") pod \"6df48c84-6810-48a5-960f-98d6f5288994\" (UID: \"6df48c84-6810-48a5-960f-98d6f5288994\") " Oct 30 21:14:54.313322 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:14:54.313316 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6df48c84-6810-48a5-960f-98d6f5288994-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6df48c84-6810-48a5-960f-98d6f5288994-openshift-service-ca\") pod \"6df48c84-6810-48a5-960f-98d6f5288994\" (UID: \"6df48c84-6810-48a5-960f-98d6f5288994\") " Oct 30 21:14:54.313780 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:14:54.313752 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6df48c84-6810-48a5-960f-98d6f5288994-openshift-service-ca" pod "6df48c84-6810-48a5-960f-98d6f5288994" (UID: "6df48c84-6810-48a5-960f-98d6f5288994"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:14:54.315197 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:14:54.315173 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6df48c84-6810-48a5-960f-98d6f5288994-metrics-certs" pod "6df48c84-6810-48a5-960f-98d6f5288994" (UID: "6df48c84-6810-48a5-960f-98d6f5288994"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:14:54.315824 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:14:54.315798 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6df48c84-6810-48a5-960f-98d6f5288994-maas-api-tls" pod "6df48c84-6810-48a5-960f-98d6f5288994" (UID: "6df48c84-6810-48a5-960f-98d6f5288994"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:14:54.315932 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:14:54.315825 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6df48c84-6810-48a5-960f-98d6f5288994-kube-api-access-2nq7c" pod "6df48c84-6810-48a5-960f-98d6f5288994" (UID: "6df48c84-6810-48a5-960f-98d6f5288994"). InnerVolumeSpecName "kube-api-access-2nq7c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:14:54.414895 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:14:54.414866 4611 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6df48c84-6810-48a5-960f-98d6f5288994-metrics-certs\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:14:54.414895 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:14:54.414895 4611 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2nq7c\" (UniqueName: \"kubernetes.io/projected/6df48c84-6810-48a5-960f-98d6f5288994-kube-api-access-2nq7c\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:14:54.415052 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:14:54.414905 4611 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6df48c84-6810-48a5-960f-98d6f5288994-maas-api-tls\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:14:54.415052 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:14:54.414914 4611 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6df48c84-6810-48a5-960f-98d6f5288994-openshift-service-ca\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:14:54.473501 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:14:54.473427 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:14:54.844123 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:14:54.844059 4611 generic.go:356] "Generic (PLEG): container finished" podID="6df48c84-6810-48a5-960f-98d6f5288994" containerID="bfe0b9a0253aed42cf27bbd167af5a164cce1ccdbec9c515664522f4abf1d1bd" exitCode=0 Oct 30 21:14:54.844123 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:14:54.844107 4611 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-30d5ac-6bcd5f86d6-br5ns" Oct 30 21:14:54.844527 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:14:54.844132 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-30d5ac-6bcd5f86d6-br5ns" event={"ID":"6df48c84-6810-48a5-960f-98d6f5288994","Type":"ContainerDied","Data":"bfe0b9a0253aed42cf27bbd167af5a164cce1ccdbec9c515664522f4abf1d1bd"} Oct 30 21:14:54.844527 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:14:54.844172 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-30d5ac-6bcd5f86d6-br5ns" event={"ID":"6df48c84-6810-48a5-960f-98d6f5288994","Type":"ContainerDied","Data":"d2c3c1711d910e8b3c4ac49da44ed9d764ac902547709142782db5c340a9ae6a"} Oct 30 21:14:54.844527 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:14:54.844193 4611 scope.go:122] "RemoveContainer" containerID="bfe0b9a0253aed42cf27bbd167af5a164cce1ccdbec9c515664522f4abf1d1bd" Oct 30 21:14:54.854411 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:14:54.854394 4611 scope.go:122] "RemoveContainer" containerID="bfe0b9a0253aed42cf27bbd167af5a164cce1ccdbec9c515664522f4abf1d1bd" Oct 30 21:14:54.854780 ip-10-0-133-192 kubenswrapper[4611]: E1030 21:14:54.854759 4611 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bfe0b9a0253aed42cf27bbd167af5a164cce1ccdbec9c515664522f4abf1d1bd\": container with ID starting with bfe0b9a0253aed42cf27bbd167af5a164cce1ccdbec9c515664522f4abf1d1bd not found: ID does not exist" containerID="bfe0b9a0253aed42cf27bbd167af5a164cce1ccdbec9c515664522f4abf1d1bd" Oct 30 21:14:54.854850 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:14:54.854786 4611 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bfe0b9a0253aed42cf27bbd167af5a164cce1ccdbec9c515664522f4abf1d1bd"} err="failed to get container status \"bfe0b9a0253aed42cf27bbd167af5a164cce1ccdbec9c515664522f4abf1d1bd\": rpc error: code = NotFound desc = could not find container \"bfe0b9a0253aed42cf27bbd167af5a164cce1ccdbec9c515664522f4abf1d1bd\": container with ID starting with bfe0b9a0253aed42cf27bbd167af5a164cce1ccdbec9c515664522f4abf1d1bd not found: ID does not exist" Oct 30 21:14:54.865447 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:14:54.865421 4611 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-30d5ac-6bcd5f86d6-br5ns"] Oct 30 21:14:54.868893 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:14:54.868874 4611 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-30d5ac-6bcd5f86d6-br5ns"] Oct 30 21:14:54.937354 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:14:54.937335 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:14:55.606027 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:14:55.605998 4611 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6df48c84-6810-48a5-960f-98d6f5288994" path="/var/lib/kubelet/pods/6df48c84-6810-48a5-960f-98d6f5288994/volumes" Oct 30 21:14:58.343302 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:14:58.343247 4611 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-30d5ac-55684b65b7z28r" podUID="6cecd88c-66fd-4f1a-93c3-c4c00cd0622c" containerName="payload-processing" containerID="cri-o://5e4cd84cf5ea01b90903cb71b5ac6075f14cf71fcf512401a47144d70ac3f513" gracePeriod=25 Oct 30 21:14:58.353916 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:14:58.353869 4611 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-30d5ac-549c7w6hcp" podUID="259e63af-b601-4d70-bbf9-fc86fa181634" containerName="payload-pre-processing" containerID="cri-o://be6d15107d37a4ef06272580929e7c43afdaf72704add160193fc560de531737" gracePeriod=25 Oct 30 21:14:59.365807 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:14:59.365762 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-30d5ac-549c7w6hcp" podUID="259e63af-b601-4d70-bbf9-fc86fa181634" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 21:14:59.366244 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:14:59.365763 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-30d5ac-55684b65b7z28r" podUID="6cecd88c-66fd-4f1a-93c3-c4c00cd0622c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 21:15:00.141909 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:15:00.141875 4611 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889915-crf2j"] Oct 30 21:15:00.142290 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:15:00.142278 4611 cpu_manager.go:414] "containerMap: removing container" podUID="6df48c84-6810-48a5-960f-98d6f5288994" containerName="maas-api" Oct 30 21:15:00.142341 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:15:00.142291 4611 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6df48c84-6810-48a5-960f-98d6f5288994" containerName="maas-api" Oct 30 21:15:00.142382 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:15:00.142362 4611 memory_manager.go:365] "RemoveStaleState removing state" podUID="6df48c84-6810-48a5-960f-98d6f5288994" containerName="maas-api" Oct 30 21:15:00.146763 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:15:00.146744 4611 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889915-crf2j" Oct 30 21:15:00.152390 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:15:00.152365 4611 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7hf8r\"" Oct 30 21:15:00.152526 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:15:00.152448 4611 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:15:00.152596 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:15:00.152532 4611 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:15:00.155164 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:15:00.155142 4611 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889915-crf2j"] Oct 30 21:15:00.267483 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:15:00.267453 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mjh72\" (UniqueName: \"kubernetes.io/projected/38ec024a-2a22-4664-9688-e08cd1ae1189-kube-api-access-mjh72\") pod \"maas-api-key-cleanup-29889915-crf2j\" (UID: \"38ec024a-2a22-4664-9688-e08cd1ae1189\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889915-crf2j" Oct 30 21:15:00.267632 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:15:00.267504 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/38ec024a-2a22-4664-9688-e08cd1ae1189-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889915-crf2j\" (UID: \"38ec024a-2a22-4664-9688-e08cd1ae1189\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889915-crf2j" Oct 30 21:15:00.368026 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:15:00.368002 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mjh72\" (UniqueName: \"kubernetes.io/projected/38ec024a-2a22-4664-9688-e08cd1ae1189-kube-api-access-mjh72\") pod \"maas-api-key-cleanup-29889915-crf2j\" (UID: \"38ec024a-2a22-4664-9688-e08cd1ae1189\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889915-crf2j" Oct 30 21:15:00.368328 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:15:00.368053 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/38ec024a-2a22-4664-9688-e08cd1ae1189-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889915-crf2j\" (UID: \"38ec024a-2a22-4664-9688-e08cd1ae1189\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889915-crf2j" Oct 30 21:15:00.368717 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:15:00.368692 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/38ec024a-2a22-4664-9688-e08cd1ae1189-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889915-crf2j\" (UID: \"38ec024a-2a22-4664-9688-e08cd1ae1189\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889915-crf2j" Oct 30 21:15:00.379232 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:15:00.379209 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mjh72\" (UniqueName: \"kubernetes.io/projected/38ec024a-2a22-4664-9688-e08cd1ae1189-kube-api-access-mjh72\") pod \"maas-api-key-cleanup-29889915-crf2j\" (UID: \"38ec024a-2a22-4664-9688-e08cd1ae1189\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889915-crf2j" Oct 30 21:15:00.461768 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:15:00.461716 4611 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889915-crf2j" Oct 30 21:15:00.736123 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:15:00.736082 4611 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889915-crf2j"] Oct 30 21:15:00.738668 ip-10-0-133-192 kubenswrapper[4611]: W1030 21:15:00.738633 4611 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38ec024a_2a22_4664_9688_e08cd1ae1189.slice/crio-dc619a9b2b1c62d30111432dc27d7106f2ae0ad0eb0fca06fb817cd25c9ce574 WatchSource:0}: Error finding container dc619a9b2b1c62d30111432dc27d7106f2ae0ad0eb0fca06fb817cd25c9ce574: Status 404 returned error can't find the container with id dc619a9b2b1c62d30111432dc27d7106f2ae0ad0eb0fca06fb817cd25c9ce574 Oct 30 21:15:00.842195 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:15:00.842165 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:15:00.886789 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:15:00.886755 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889915-crf2j" event={"ID":"38ec024a-2a22-4664-9688-e08cd1ae1189","Type":"ContainerStarted","Data":"dc619a9b2b1c62d30111432dc27d7106f2ae0ad0eb0fca06fb817cd25c9ce574"} Oct 30 21:15:00.996135 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:15:00.996074 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:15:01.126592 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:15:01.126562 4611 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-30d5ac-openshift-default-696b8c9db9mffng"] Oct 30 21:15:01.126866 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:15:01.126815 4611 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-single-main-30d5ac-openshift-default-696b8c9db9mffng" podUID="dcdd7301-b310-4518-ad18-02d055ab1f2a" containerName="istio-proxy" containerID="cri-o://79a8266c768c3ebdc816f424cab5847e114636e2bd6cc1add4ccf96495350af4" gracePeriod=30 Oct 30 21:15:01.203151 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:15:01.203107 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:15:01.893461 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:15:01.893424 4611 generic.go:356] "Generic (PLEG): container finished" podID="38ec024a-2a22-4664-9688-e08cd1ae1189" containerID="0b4a3fa0e4581ea0b22ec6c338036eafd0451a776acc59ef7e738bfad1300ae7" exitCode=60 Oct 30 21:15:01.893949 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:15:01.893527 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889915-crf2j" event={"ID":"38ec024a-2a22-4664-9688-e08cd1ae1189","Type":"ContainerDied","Data":"0b4a3fa0e4581ea0b22ec6c338036eafd0451a776acc59ef7e738bfad1300ae7"} Oct 30 21:15:01.893949 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:15:01.893699 4611 scope.go:122] "RemoveContainer" containerID="0b4a3fa0e4581ea0b22ec6c338036eafd0451a776acc59ef7e738bfad1300ae7" Oct 30 21:15:02.027994 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:15:02.027963 4611 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a047fdad-c4gv6"] Oct 30 21:15:02.032854 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:15:02.032832 4611 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a047fdad-c4gv6"] Oct 30 21:15:02.901049 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:15:02.901018 4611 generic.go:356] "Generic (PLEG): container finished" podID="38ec024a-2a22-4664-9688-e08cd1ae1189" containerID="667d4e8a2b90731e307669e378ae3cf6f9c46821dcffdcd6713ae68a20723300" exitCode=60 Oct 30 21:15:02.901486 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:15:02.901106 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889915-crf2j" event={"ID":"38ec024a-2a22-4664-9688-e08cd1ae1189","Type":"ContainerDied","Data":"667d4e8a2b90731e307669e378ae3cf6f9c46821dcffdcd6713ae68a20723300"} Oct 30 21:15:02.901486 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:15:02.901160 4611 scope.go:122] "RemoveContainer" containerID="0b4a3fa0e4581ea0b22ec6c338036eafd0451a776acc59ef7e738bfad1300ae7" Oct 30 21:15:02.901486 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:15:02.901290 4611 scope.go:122] "RemoveContainer" containerID="667d4e8a2b90731e307669e378ae3cf6f9c46821dcffdcd6713ae68a20723300" Oct 30 21:15:02.901715 ip-10-0-133-192 kubenswrapper[4611]: E1030 21:15:02.901541 4611 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889915-crf2j_odh-ai-gateway-infra(38ec024a-2a22-4664-9688-e08cd1ae1189)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889915-crf2j" podUID="38ec024a-2a22-4664-9688-e08cd1ae1189" Oct 30 21:15:03.604466 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:15:03.604433 4611 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5567ed88-fc6d-417e-b01c-c079d380ecef" path="/var/lib/kubelet/pods/5567ed88-fc6d-417e-b01c-c079d380ecef/volumes" Oct 30 21:15:03.912594 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:15:03.912483 4611 scope.go:122] "RemoveContainer" containerID="667d4e8a2b90731e307669e378ae3cf6f9c46821dcffdcd6713ae68a20723300" Oct 30 21:15:03.913081 ip-10-0-133-192 kubenswrapper[4611]: E1030 21:15:03.912740 4611 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889915-crf2j_odh-ai-gateway-infra(38ec024a-2a22-4664-9688-e08cd1ae1189)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889915-crf2j" podUID="38ec024a-2a22-4664-9688-e08cd1ae1189" Oct 30 21:15:03.931667 ip-10-0-133-192 kubenswrapper[4611]: E1030 21:15:03.931635 4611 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hwzc6" containerName="extractor" Oct 30 21:15:04.604157 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:15:04.604120 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:15:05.368792 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:15:05.368726 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:15:05.732404 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:15:05.732309 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:15:06.375331 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:15:06.375308 4611 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-30d5ac-openshift-default-696b8c9db9mffng" Oct 30 21:15:06.528250 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:15:06.528166 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dcdd7301-b310-4518-ad18-02d055ab1f2a-kube-api-access-lz7rj\" (UniqueName: \"kubernetes.io/projected/dcdd7301-b310-4518-ad18-02d055ab1f2a-kube-api-access-lz7rj\") pod \"dcdd7301-b310-4518-ad18-02d055ab1f2a\" (UID: \"dcdd7301-b310-4518-ad18-02d055ab1f2a\") " Oct 30 21:15:06.528250 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:15:06.528217 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dcdd7301-b310-4518-ad18-02d055ab1f2a-istio-token\" (UniqueName: \"kubernetes.io/projected/dcdd7301-b310-4518-ad18-02d055ab1f2a-istio-token\") pod \"dcdd7301-b310-4518-ad18-02d055ab1f2a\" (UID: \"dcdd7301-b310-4518-ad18-02d055ab1f2a\") " Oct 30 21:15:06.528479 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:15:06.528257 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dcdd7301-b310-4518-ad18-02d055ab1f2a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcdd7301-b310-4518-ad18-02d055ab1f2a-workload-socket\") pod \"dcdd7301-b310-4518-ad18-02d055ab1f2a\" (UID: \"dcdd7301-b310-4518-ad18-02d055ab1f2a\") " Oct 30 21:15:06.528479 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:15:06.528290 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dcdd7301-b310-4518-ad18-02d055ab1f2a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dcdd7301-b310-4518-ad18-02d055ab1f2a-istio-envoy\") pod \"dcdd7301-b310-4518-ad18-02d055ab1f2a\" (UID: \"dcdd7301-b310-4518-ad18-02d055ab1f2a\") " Oct 30 21:15:06.528479 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:15:06.528308 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dcdd7301-b310-4518-ad18-02d055ab1f2a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcdd7301-b310-4518-ad18-02d055ab1f2a-credential-socket\") pod \"dcdd7301-b310-4518-ad18-02d055ab1f2a\" (UID: \"dcdd7301-b310-4518-ad18-02d055ab1f2a\") " Oct 30 21:15:06.528479 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:15:06.528329 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dcdd7301-b310-4518-ad18-02d055ab1f2a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dcdd7301-b310-4518-ad18-02d055ab1f2a-workload-certs\") pod \"dcdd7301-b310-4518-ad18-02d055ab1f2a\" (UID: \"dcdd7301-b310-4518-ad18-02d055ab1f2a\") " Oct 30 21:15:06.528479 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:15:06.528359 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dcdd7301-b310-4518-ad18-02d055ab1f2a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dcdd7301-b310-4518-ad18-02d055ab1f2a-istiod-ca-cert\") pod \"dcdd7301-b310-4518-ad18-02d055ab1f2a\" (UID: \"dcdd7301-b310-4518-ad18-02d055ab1f2a\") " Oct 30 21:15:06.528479 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:15:06.528400 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dcdd7301-b310-4518-ad18-02d055ab1f2a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dcdd7301-b310-4518-ad18-02d055ab1f2a-istio-podinfo\") pod \"dcdd7301-b310-4518-ad18-02d055ab1f2a\" (UID: \"dcdd7301-b310-4518-ad18-02d055ab1f2a\") " Oct 30 21:15:06.528479 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:15:06.528450 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dcdd7301-b310-4518-ad18-02d055ab1f2a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dcdd7301-b310-4518-ad18-02d055ab1f2a-istio-data\") pod \"dcdd7301-b310-4518-ad18-02d055ab1f2a\" (UID: \"dcdd7301-b310-4518-ad18-02d055ab1f2a\") " Oct 30 21:15:06.528920 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:15:06.528652 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcdd7301-b310-4518-ad18-02d055ab1f2a-workload-certs" pod "dcdd7301-b310-4518-ad18-02d055ab1f2a" (UID: "dcdd7301-b310-4518-ad18-02d055ab1f2a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:15:06.528920 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:15:06.528672 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcdd7301-b310-4518-ad18-02d055ab1f2a-credential-socket" pod "dcdd7301-b310-4518-ad18-02d055ab1f2a" (UID: "dcdd7301-b310-4518-ad18-02d055ab1f2a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:15:06.528920 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:15:06.528658 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcdd7301-b310-4518-ad18-02d055ab1f2a-workload-socket" pod "dcdd7301-b310-4518-ad18-02d055ab1f2a" (UID: "dcdd7301-b310-4518-ad18-02d055ab1f2a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:15:06.528920 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:15:06.528879 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcdd7301-b310-4518-ad18-02d055ab1f2a-istio-data" pod "dcdd7301-b310-4518-ad18-02d055ab1f2a" (UID: "dcdd7301-b310-4518-ad18-02d055ab1f2a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:15:06.529143 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:15:06.528957 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dcdd7301-b310-4518-ad18-02d055ab1f2a-istiod-ca-cert" pod "dcdd7301-b310-4518-ad18-02d055ab1f2a" (UID: "dcdd7301-b310-4518-ad18-02d055ab1f2a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:15:06.529143 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:15:06.528967 4611 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcdd7301-b310-4518-ad18-02d055ab1f2a-credential-socket\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:15:06.529143 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:15:06.528987 4611 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dcdd7301-b310-4518-ad18-02d055ab1f2a-workload-certs\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:15:06.529143 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:15:06.529002 4611 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dcdd7301-b310-4518-ad18-02d055ab1f2a-istio-data\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:15:06.529143 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:15:06.529016 4611 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcdd7301-b310-4518-ad18-02d055ab1f2a-workload-socket\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:15:06.530494 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:15:06.530468 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dcdd7301-b310-4518-ad18-02d055ab1f2a-kube-api-access-lz7rj" pod "dcdd7301-b310-4518-ad18-02d055ab1f2a" (UID: "dcdd7301-b310-4518-ad18-02d055ab1f2a"). InnerVolumeSpecName "kube-api-access-lz7rj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:15:06.530757 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:15:06.530731 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dcdd7301-b310-4518-ad18-02d055ab1f2a-istio-podinfo" pod "dcdd7301-b310-4518-ad18-02d055ab1f2a" (UID: "dcdd7301-b310-4518-ad18-02d055ab1f2a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:15:06.530825 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:15:06.530797 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dcdd7301-b310-4518-ad18-02d055ab1f2a-istio-token" pod "dcdd7301-b310-4518-ad18-02d055ab1f2a" (UID: "dcdd7301-b310-4518-ad18-02d055ab1f2a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:15:06.530884 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:15:06.530871 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcdd7301-b310-4518-ad18-02d055ab1f2a-istio-envoy" pod "dcdd7301-b310-4518-ad18-02d055ab1f2a" (UID: "dcdd7301-b310-4518-ad18-02d055ab1f2a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:15:06.630099 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:15:06.630069 4611 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dcdd7301-b310-4518-ad18-02d055ab1f2a-istio-podinfo\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:15:06.630099 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:15:06.630094 4611 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lz7rj\" (UniqueName: \"kubernetes.io/projected/dcdd7301-b310-4518-ad18-02d055ab1f2a-kube-api-access-lz7rj\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:15:06.630229 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:15:06.630106 4611 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dcdd7301-b310-4518-ad18-02d055ab1f2a-istio-token\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:15:06.630229 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:15:06.630114 4611 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dcdd7301-b310-4518-ad18-02d055ab1f2a-istio-envoy\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:15:06.630229 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:15:06.630122 4611 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dcdd7301-b310-4518-ad18-02d055ab1f2a-istiod-ca-cert\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:15:06.930132 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:15:06.930092 4611 generic.go:356] "Generic (PLEG): container finished" podID="dcdd7301-b310-4518-ad18-02d055ab1f2a" containerID="79a8266c768c3ebdc816f424cab5847e114636e2bd6cc1add4ccf96495350af4" exitCode=0 Oct 30 21:15:06.930314 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:15:06.930155 4611 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-30d5ac-openshift-default-696b8c9db9mffng" Oct 30 21:15:06.930314 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:15:06.930183 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-30d5ac-openshift-default-696b8c9db9mffng" event={"ID":"dcdd7301-b310-4518-ad18-02d055ab1f2a","Type":"ContainerDied","Data":"79a8266c768c3ebdc816f424cab5847e114636e2bd6cc1add4ccf96495350af4"} Oct 30 21:15:06.930314 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:15:06.930254 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-30d5ac-openshift-default-696b8c9db9mffng" event={"ID":"dcdd7301-b310-4518-ad18-02d055ab1f2a","Type":"ContainerDied","Data":"0ce0a0392cd32539cc4d5eda5ab83aae760cacf75b6f6ba4f61f8324b1e1f876"} Oct 30 21:15:06.930314 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:15:06.930288 4611 scope.go:122] "RemoveContainer" containerID="79a8266c768c3ebdc816f424cab5847e114636e2bd6cc1add4ccf96495350af4" Oct 30 21:15:06.944671 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:15:06.944650 4611 scope.go:122] "RemoveContainer" containerID="eb685aeea054fb9755d88ed1ff22d760aba38b09830ac99e5138982263be5a76" Oct 30 21:15:06.953756 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:15:06.953736 4611 scope.go:122] "RemoveContainer" containerID="79a8266c768c3ebdc816f424cab5847e114636e2bd6cc1add4ccf96495350af4" Oct 30 21:15:06.954038 ip-10-0-133-192 kubenswrapper[4611]: E1030 21:15:06.954014 4611 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"79a8266c768c3ebdc816f424cab5847e114636e2bd6cc1add4ccf96495350af4\": container with ID starting with 79a8266c768c3ebdc816f424cab5847e114636e2bd6cc1add4ccf96495350af4 not found: ID does not exist" containerID="79a8266c768c3ebdc816f424cab5847e114636e2bd6cc1add4ccf96495350af4" Oct 30 21:15:06.954097 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:15:06.954048 4611 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"79a8266c768c3ebdc816f424cab5847e114636e2bd6cc1add4ccf96495350af4"} err="failed to get container status \"79a8266c768c3ebdc816f424cab5847e114636e2bd6cc1add4ccf96495350af4\": rpc error: code = NotFound desc = could not find container \"79a8266c768c3ebdc816f424cab5847e114636e2bd6cc1add4ccf96495350af4\": container with ID starting with 79a8266c768c3ebdc816f424cab5847e114636e2bd6cc1add4ccf96495350af4 not found: ID does not exist" Oct 30 21:15:06.954097 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:15:06.954068 4611 scope.go:122] "RemoveContainer" containerID="eb685aeea054fb9755d88ed1ff22d760aba38b09830ac99e5138982263be5a76" Oct 30 21:15:06.954363 ip-10-0-133-192 kubenswrapper[4611]: E1030 21:15:06.954332 4611 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb685aeea054fb9755d88ed1ff22d760aba38b09830ac99e5138982263be5a76\": container with ID starting with eb685aeea054fb9755d88ed1ff22d760aba38b09830ac99e5138982263be5a76 not found: ID does not exist" containerID="eb685aeea054fb9755d88ed1ff22d760aba38b09830ac99e5138982263be5a76" Oct 30 21:15:06.954455 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:15:06.954368 4611 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb685aeea054fb9755d88ed1ff22d760aba38b09830ac99e5138982263be5a76"} err="failed to get container status \"eb685aeea054fb9755d88ed1ff22d760aba38b09830ac99e5138982263be5a76\": rpc error: code = NotFound desc = could not find container \"eb685aeea054fb9755d88ed1ff22d760aba38b09830ac99e5138982263be5a76\": container with ID starting with eb685aeea054fb9755d88ed1ff22d760aba38b09830ac99e5138982263be5a76 not found: ID does not exist" Oct 30 21:15:06.961633 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:15:06.961611 4611 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-30d5ac-openshift-default-696b8c9db9mffng"] Oct 30 21:15:06.965418 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:15:06.965398 4611 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-single-main-30d5ac-openshift-default-696b8c9db9mffng"] Oct 30 21:15:07.609047 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:15:07.608997 4611 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dcdd7301-b310-4518-ad18-02d055ab1f2a" path="/var/lib/kubelet/pods/dcdd7301-b310-4518-ad18-02d055ab1f2a/volumes" Oct 30 21:15:08.363533 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:15:08.352048 4611 scope.go:122] "RemoveContainer" containerID="d8ba9ad13d7d452bc0b44e598e8b7567a25cac49c340d022a8f98dff62cc035a" Oct 30 21:15:08.419756 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:15:08.399262 4611 scope.go:122] "RemoveContainer" containerID="fa1ad561e8cf07be219281b5506d7ca076056d0434d66b545d16ef4f37a3e7ed" Oct 30 21:15:08.440540 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:15:08.437692 4611 scope.go:122] "RemoveContainer" containerID="15bb295c7703f215a1f7a45271e993fbf813050b4e84a5628f92f22f105441ee" Oct 30 21:15:09.393634 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:15:09.393592 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-30d5ac-549c7w6hcp" podUID="259e63af-b601-4d70-bbf9-fc86fa181634" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 21:15:09.398634 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:15:09.398589 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-30d5ac-55684b65b7z28r" podUID="6cecd88c-66fd-4f1a-93c3-c4c00cd0622c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 21:15:14.725888 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:15:14.725826 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:15:15.602689 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:15:15.602657 4611 scope.go:122] "RemoveContainer" containerID="667d4e8a2b90731e307669e378ae3cf6f9c46821dcffdcd6713ae68a20723300" Oct 30 21:15:15.948707 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:15:15.948675 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:15:16.026100 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:15:16.026072 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889915-crf2j" event={"ID":"38ec024a-2a22-4664-9688-e08cd1ae1189","Type":"ContainerStarted","Data":"48f33af82c39b7eefadb6ab568ad97e33b5f1a8126a600d386e9e40b614ece41"} Oct 30 21:15:16.044229 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:15:16.044171 4611 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889915-crf2j" podStartSLOduration=16.044159231 podStartE2EDuration="16.044159231s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:15:16.042928292 +0000 UTC m=+2170.576954910" watchObservedRunningTime="2026-10-30 21:15:16.044159231 +0000 UTC m=+2170.578185846" Oct 30 21:15:17.033341 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:15:17.033312 4611 generic.go:356] "Generic (PLEG): container finished" podID="38ec024a-2a22-4664-9688-e08cd1ae1189" containerID="48f33af82c39b7eefadb6ab568ad97e33b5f1a8126a600d386e9e40b614ece41" exitCode=60 Oct 30 21:15:17.033802 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:15:17.033378 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889915-crf2j" event={"ID":"38ec024a-2a22-4664-9688-e08cd1ae1189","Type":"ContainerDied","Data":"48f33af82c39b7eefadb6ab568ad97e33b5f1a8126a600d386e9e40b614ece41"} Oct 30 21:15:17.033802 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:15:17.033435 4611 scope.go:122] "RemoveContainer" containerID="667d4e8a2b90731e307669e378ae3cf6f9c46821dcffdcd6713ae68a20723300" Oct 30 21:15:17.033802 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:15:17.033681 4611 scope.go:122] "RemoveContainer" containerID="48f33af82c39b7eefadb6ab568ad97e33b5f1a8126a600d386e9e40b614ece41" Oct 30 21:15:17.033933 ip-10-0-133-192 kubenswrapper[4611]: E1030 21:15:17.033893 4611 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29889915-crf2j_odh-ai-gateway-infra(38ec024a-2a22-4664-9688-e08cd1ae1189)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889915-crf2j" podUID="38ec024a-2a22-4664-9688-e08cd1ae1189" Oct 30 21:15:17.061985 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:15:17.061963 4611 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889915-crf2j"] Oct 30 21:15:18.174241 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:15:18.174219 4611 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889915-crf2j" Oct 30 21:15:18.288026 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:15:18.287985 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/38ec024a-2a22-4664-9688-e08cd1ae1189-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/38ec024a-2a22-4664-9688-e08cd1ae1189-maas-api-service-ca\") pod \"38ec024a-2a22-4664-9688-e08cd1ae1189\" (UID: \"38ec024a-2a22-4664-9688-e08cd1ae1189\") " Oct 30 21:15:18.288225 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:15:18.288072 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/38ec024a-2a22-4664-9688-e08cd1ae1189-kube-api-access-mjh72\" (UniqueName: \"kubernetes.io/projected/38ec024a-2a22-4664-9688-e08cd1ae1189-kube-api-access-mjh72\") pod \"38ec024a-2a22-4664-9688-e08cd1ae1189\" (UID: \"38ec024a-2a22-4664-9688-e08cd1ae1189\") " Oct 30 21:15:18.288326 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:15:18.288303 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38ec024a-2a22-4664-9688-e08cd1ae1189-maas-api-service-ca" pod "38ec024a-2a22-4664-9688-e08cd1ae1189" (UID: "38ec024a-2a22-4664-9688-e08cd1ae1189"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:15:18.290067 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:15:18.290046 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38ec024a-2a22-4664-9688-e08cd1ae1189-kube-api-access-mjh72" pod "38ec024a-2a22-4664-9688-e08cd1ae1189" (UID: "38ec024a-2a22-4664-9688-e08cd1ae1189"). InnerVolumeSpecName "kube-api-access-mjh72". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:15:18.389145 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:15:18.389122 4611 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/38ec024a-2a22-4664-9688-e08cd1ae1189-maas-api-service-ca\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:15:18.389145 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:15:18.389145 4611 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mjh72\" (UniqueName: \"kubernetes.io/projected/38ec024a-2a22-4664-9688-e08cd1ae1189-kube-api-access-mjh72\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:15:18.593151 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:15:18.593127 4611 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-30d5ac-55684b65b7z28r" Oct 30 21:15:18.629083 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:15:18.629058 4611 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-30d5ac-549c7w6hcp" Oct 30 21:15:18.784811 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:15:18.784768 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:15:18.791763 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:15:18.791734 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6cecd88c-66fd-4f1a-93c3-c4c00cd0622c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6cecd88c-66fd-4f1a-93c3-c4c00cd0622c-plugins-config-volume\") pod \"6cecd88c-66fd-4f1a-93c3-c4c00cd0622c\" (UID: \"6cecd88c-66fd-4f1a-93c3-c4c00cd0622c\") " Oct 30 21:15:18.791890 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:15:18.791829 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/259e63af-b601-4d70-bbf9-fc86fa181634-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/259e63af-b601-4d70-bbf9-fc86fa181634-plugins-config-volume\") pod \"259e63af-b601-4d70-bbf9-fc86fa181634\" (UID: \"259e63af-b601-4d70-bbf9-fc86fa181634\") " Oct 30 21:15:18.791935 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:15:18.791926 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/259e63af-b601-4d70-bbf9-fc86fa181634-kube-api-access-wcwst\" (UniqueName: \"kubernetes.io/projected/259e63af-b601-4d70-bbf9-fc86fa181634-kube-api-access-wcwst\") pod \"259e63af-b601-4d70-bbf9-fc86fa181634\" (UID: \"259e63af-b601-4d70-bbf9-fc86fa181634\") " Oct 30 21:15:18.791970 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:15:18.791951 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6cecd88c-66fd-4f1a-93c3-c4c00cd0622c-kube-api-access-rglt7\" (UniqueName: \"kubernetes.io/projected/6cecd88c-66fd-4f1a-93c3-c4c00cd0622c-kube-api-access-rglt7\") pod \"6cecd88c-66fd-4f1a-93c3-c4c00cd0622c\" (UID: \"6cecd88c-66fd-4f1a-93c3-c4c00cd0622c\") " Oct 30 21:15:18.792097 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:15:18.792073 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6cecd88c-66fd-4f1a-93c3-c4c00cd0622c-plugins-config-volume" pod "6cecd88c-66fd-4f1a-93c3-c4c00cd0622c" (UID: "6cecd88c-66fd-4f1a-93c3-c4c00cd0622c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:15:18.792168 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:15:18.792154 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/259e63af-b601-4d70-bbf9-fc86fa181634-plugins-config-volume" pod "259e63af-b601-4d70-bbf9-fc86fa181634" (UID: "259e63af-b601-4d70-bbf9-fc86fa181634"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:15:18.792229 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:15:18.792209 4611 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6cecd88c-66fd-4f1a-93c3-c4c00cd0622c-plugins-config-volume\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:15:18.793924 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:15:18.793901 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6cecd88c-66fd-4f1a-93c3-c4c00cd0622c-kube-api-access-rglt7" pod "6cecd88c-66fd-4f1a-93c3-c4c00cd0622c" (UID: "6cecd88c-66fd-4f1a-93c3-c4c00cd0622c"). InnerVolumeSpecName "kube-api-access-rglt7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:15:18.794021 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:15:18.793934 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/259e63af-b601-4d70-bbf9-fc86fa181634-kube-api-access-wcwst" pod "259e63af-b601-4d70-bbf9-fc86fa181634" (UID: "259e63af-b601-4d70-bbf9-fc86fa181634"). InnerVolumeSpecName "kube-api-access-wcwst". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:15:18.893072 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:15:18.893052 4611 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/259e63af-b601-4d70-bbf9-fc86fa181634-plugins-config-volume\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:15:18.893150 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:15:18.893072 4611 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wcwst\" (UniqueName: \"kubernetes.io/projected/259e63af-b601-4d70-bbf9-fc86fa181634-kube-api-access-wcwst\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:15:18.893150 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:15:18.893084 4611 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rglt7\" (UniqueName: \"kubernetes.io/projected/6cecd88c-66fd-4f1a-93c3-c4c00cd0622c-kube-api-access-rglt7\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:15:18.925095 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:15:18.925066 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:15:19.046594 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:15:19.046569 4611 generic.go:356] "Generic (PLEG): container finished" podID="6cecd88c-66fd-4f1a-93c3-c4c00cd0622c" containerID="5e4cd84cf5ea01b90903cb71b5ac6075f14cf71fcf512401a47144d70ac3f513" exitCode=0 Oct 30 21:15:19.046686 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:15:19.046619 4611 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-30d5ac-55684b65b7z28r" Oct 30 21:15:19.046686 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:15:19.046662 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-30d5ac-55684b65b7z28r" event={"ID":"6cecd88c-66fd-4f1a-93c3-c4c00cd0622c","Type":"ContainerDied","Data":"5e4cd84cf5ea01b90903cb71b5ac6075f14cf71fcf512401a47144d70ac3f513"} Oct 30 21:15:19.046793 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:15:19.046721 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-30d5ac-55684b65b7z28r" event={"ID":"6cecd88c-66fd-4f1a-93c3-c4c00cd0622c","Type":"ContainerDied","Data":"b5e73eb7c73109c9df0d386828157322f17c7d02a19d3fd1863db667cb7f642b"} Oct 30 21:15:19.046793 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:15:19.046754 4611 scope.go:122] "RemoveContainer" containerID="5e4cd84cf5ea01b90903cb71b5ac6075f14cf71fcf512401a47144d70ac3f513" Oct 30 21:15:19.048059 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:15:19.048037 4611 generic.go:356] "Generic (PLEG): container finished" podID="259e63af-b601-4d70-bbf9-fc86fa181634" containerID="be6d15107d37a4ef06272580929e7c43afdaf72704add160193fc560de531737" exitCode=0 Oct 30 21:15:19.048138 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:15:19.048077 4611 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-30d5ac-549c7w6hcp" Oct 30 21:15:19.048195 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:15:19.048131 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-30d5ac-549c7w6hcp" event={"ID":"259e63af-b601-4d70-bbf9-fc86fa181634","Type":"ContainerDied","Data":"be6d15107d37a4ef06272580929e7c43afdaf72704add160193fc560de531737"} Oct 30 21:15:19.048195 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:15:19.048189 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-30d5ac-549c7w6hcp" event={"ID":"259e63af-b601-4d70-bbf9-fc86fa181634","Type":"ContainerDied","Data":"f374885765512df2529c6a5b5d4ab01c9718050ee9b181bd523077cd45671583"} Oct 30 21:15:19.049675 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:15:19.049648 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889915-crf2j" event={"ID":"38ec024a-2a22-4664-9688-e08cd1ae1189","Type":"ContainerDied","Data":"dc619a9b2b1c62d30111432dc27d7106f2ae0ad0eb0fca06fb817cd25c9ce574"} Oct 30 21:15:19.049762 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:15:19.049738 4611 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889915-crf2j" Oct 30 21:15:19.059994 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:15:19.059978 4611 scope.go:122] "RemoveContainer" containerID="5e4cd84cf5ea01b90903cb71b5ac6075f14cf71fcf512401a47144d70ac3f513" Oct 30 21:15:19.060246 ip-10-0-133-192 kubenswrapper[4611]: E1030 21:15:19.060226 4611 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e4cd84cf5ea01b90903cb71b5ac6075f14cf71fcf512401a47144d70ac3f513\": container with ID starting with 5e4cd84cf5ea01b90903cb71b5ac6075f14cf71fcf512401a47144d70ac3f513 not found: ID does not exist" containerID="5e4cd84cf5ea01b90903cb71b5ac6075f14cf71fcf512401a47144d70ac3f513" Oct 30 21:15:19.060331 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:15:19.060253 4611 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e4cd84cf5ea01b90903cb71b5ac6075f14cf71fcf512401a47144d70ac3f513"} err="failed to get container status \"5e4cd84cf5ea01b90903cb71b5ac6075f14cf71fcf512401a47144d70ac3f513\": rpc error: code = NotFound desc = could not find container \"5e4cd84cf5ea01b90903cb71b5ac6075f14cf71fcf512401a47144d70ac3f513\": container with ID starting with 5e4cd84cf5ea01b90903cb71b5ac6075f14cf71fcf512401a47144d70ac3f513 not found: ID does not exist" Oct 30 21:15:19.060331 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:15:19.060270 4611 scope.go:122] "RemoveContainer" containerID="be6d15107d37a4ef06272580929e7c43afdaf72704add160193fc560de531737" Oct 30 21:15:19.068592 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:15:19.068575 4611 scope.go:122] "RemoveContainer" containerID="be6d15107d37a4ef06272580929e7c43afdaf72704add160193fc560de531737" Oct 30 21:15:19.068837 ip-10-0-133-192 kubenswrapper[4611]: E1030 21:15:19.068821 4611 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be6d15107d37a4ef06272580929e7c43afdaf72704add160193fc560de531737\": container with ID starting with be6d15107d37a4ef06272580929e7c43afdaf72704add160193fc560de531737 not found: ID does not exist" containerID="be6d15107d37a4ef06272580929e7c43afdaf72704add160193fc560de531737" Oct 30 21:15:19.068922 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:15:19.068869 4611 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be6d15107d37a4ef06272580929e7c43afdaf72704add160193fc560de531737"} err="failed to get container status \"be6d15107d37a4ef06272580929e7c43afdaf72704add160193fc560de531737\": rpc error: code = NotFound desc = could not find container \"be6d15107d37a4ef06272580929e7c43afdaf72704add160193fc560de531737\": container with ID starting with be6d15107d37a4ef06272580929e7c43afdaf72704add160193fc560de531737 not found: ID does not exist" Oct 30 21:15:19.068922 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:15:19.068884 4611 scope.go:122] "RemoveContainer" containerID="48f33af82c39b7eefadb6ab568ad97e33b5f1a8126a600d386e9e40b614ece41" Oct 30 21:15:19.071936 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:15:19.071912 4611 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-30d5ac-55684b65b7z28r"] Oct 30 21:15:19.078744 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:15:19.078726 4611 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-30d5ac-55684b65b7z28r"] Oct 30 21:15:19.087577 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:15:19.087555 4611 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889915-crf2j"] Oct 30 21:15:19.092705 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:15:19.092684 4611 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889915-crf2j"] Oct 30 21:15:19.103012 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:15:19.102990 4611 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-30d5ac-549c7w6hcp"] Oct 30 21:15:19.108283 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:15:19.108263 4611 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-30d5ac-549c7w6hcp"] Oct 30 21:15:19.610501 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:15:19.610466 4611 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="259e63af-b601-4d70-bbf9-fc86fa181634" path="/var/lib/kubelet/pods/259e63af-b601-4d70-bbf9-fc86fa181634/volumes" Oct 30 21:15:19.610971 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:15:19.610845 4611 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="38ec024a-2a22-4664-9688-e08cd1ae1189" path="/var/lib/kubelet/pods/38ec024a-2a22-4664-9688-e08cd1ae1189/volumes" Oct 30 21:15:19.611186 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:15:19.611172 4611 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6cecd88c-66fd-4f1a-93c3-c4c00cd0622c" path="/var/lib/kubelet/pods/6cecd88c-66fd-4f1a-93c3-c4c00cd0622c/volumes" Oct 30 21:15:22.068830 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:15:22.068796 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:15:22.628166 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:15:22.628131 4611 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-0cbb4221-kserve-5cb4b7bf-hpxng"] Oct 30 21:15:22.628739 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:15:22.628710 4611 cpu_manager.go:414] "containerMap: removing container" podUID="259e63af-b601-4d70-bbf9-fc86fa181634" containerName="payload-pre-processing" Oct 30 21:15:22.628739 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:15:22.628737 4611 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="259e63af-b601-4d70-bbf9-fc86fa181634" containerName="payload-pre-processing" Oct 30 21:15:22.628956 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:15:22.628752 4611 cpu_manager.go:414] "containerMap: removing container" podUID="dcdd7301-b310-4518-ad18-02d055ab1f2a" containerName="istio-proxy" Oct 30 21:15:22.628956 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:15:22.628760 4611 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dcdd7301-b310-4518-ad18-02d055ab1f2a" containerName="istio-proxy" Oct 30 21:15:22.628956 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:15:22.628772 4611 cpu_manager.go:414] "containerMap: removing container" podUID="38ec024a-2a22-4664-9688-e08cd1ae1189" containerName="cleanup" Oct 30 21:15:22.628956 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:15:22.628780 4611 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38ec024a-2a22-4664-9688-e08cd1ae1189" containerName="cleanup" Oct 30 21:15:22.628956 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:15:22.628802 4611 cpu_manager.go:414] "containerMap: removing container" podUID="6cecd88c-66fd-4f1a-93c3-c4c00cd0622c" containerName="payload-processing" Oct 30 21:15:22.628956 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:15:22.628809 4611 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6cecd88c-66fd-4f1a-93c3-c4c00cd0622c" containerName="payload-processing" Oct 30 21:15:22.628956 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:15:22.628840 4611 cpu_manager.go:414] "containerMap: removing container" podUID="38ec024a-2a22-4664-9688-e08cd1ae1189" containerName="cleanup" Oct 30 21:15:22.628956 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:15:22.628847 4611 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38ec024a-2a22-4664-9688-e08cd1ae1189" containerName="cleanup" Oct 30 21:15:22.628956 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:15:22.628860 4611 cpu_manager.go:414] "containerMap: removing container" podUID="38ec024a-2a22-4664-9688-e08cd1ae1189" containerName="cleanup" Oct 30 21:15:22.628956 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:15:22.628869 4611 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38ec024a-2a22-4664-9688-e08cd1ae1189" containerName="cleanup" Oct 30 21:15:22.629593 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:15:22.629001 4611 memory_manager.go:365] "RemoveStaleState removing state" podUID="259e63af-b601-4d70-bbf9-fc86fa181634" containerName="payload-pre-processing" Oct 30 21:15:22.629593 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:15:22.629011 4611 memory_manager.go:365] "RemoveStaleState removing state" podUID="dcdd7301-b310-4518-ad18-02d055ab1f2a" containerName="istio-proxy" Oct 30 21:15:22.629593 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:15:22.629021 4611 memory_manager.go:365] "RemoveStaleState removing state" podUID="38ec024a-2a22-4664-9688-e08cd1ae1189" containerName="cleanup" Oct 30 21:15:22.629593 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:15:22.629034 4611 memory_manager.go:365] "RemoveStaleState removing state" podUID="6cecd88c-66fd-4f1a-93c3-c4c00cd0622c" containerName="payload-processing" Oct 30 21:15:22.629593 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:15:22.629047 4611 memory_manager.go:365] "RemoveStaleState removing state" podUID="38ec024a-2a22-4664-9688-e08cd1ae1189" containerName="cleanup" Oct 30 21:15:22.629593 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:15:22.629057 4611 memory_manager.go:365] "RemoveStaleState removing state" podUID="38ec024a-2a22-4664-9688-e08cd1ae1189" containerName="cleanup" Oct 30 21:15:22.633941 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:15:22.633917 4611 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-0cbb4221-kserve-5cb4b7bf-hpxng" Oct 30 21:15:22.640077 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:15:22.640049 4611 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-0cbb4221-kserve-self-signed-certs\"" Oct 30 21:15:22.644339 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:15:22.641957 4611 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-0cbb4221-kserve-5cb4b7bf-hpxng"] Oct 30 21:15:22.726425 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:15:22.726393 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-stvpr\" (UniqueName: \"kubernetes.io/projected/daa7078a-68b8-449a-be10-5a87bc9e5eb5-kube-api-access-stvpr\") pod \"e2e-same-id-b-0cbb4221-kserve-5cb4b7bf-hpxng\" (UID: \"daa7078a-68b8-449a-be10-5a87bc9e5eb5\") " pod="llm/e2e-same-id-b-0cbb4221-kserve-5cb4b7bf-hpxng" Oct 30 21:15:22.726425 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:15:22.726426 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/daa7078a-68b8-449a-be10-5a87bc9e5eb5-dshm\") pod \"e2e-same-id-b-0cbb4221-kserve-5cb4b7bf-hpxng\" (UID: \"daa7078a-68b8-449a-be10-5a87bc9e5eb5\") " pod="llm/e2e-same-id-b-0cbb4221-kserve-5cb4b7bf-hpxng" Oct 30 21:15:22.726632 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:15:22.726541 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/daa7078a-68b8-449a-be10-5a87bc9e5eb5-tmp-dir\") pod \"e2e-same-id-b-0cbb4221-kserve-5cb4b7bf-hpxng\" (UID: \"daa7078a-68b8-449a-be10-5a87bc9e5eb5\") " pod="llm/e2e-same-id-b-0cbb4221-kserve-5cb4b7bf-hpxng" Oct 30 21:15:22.726632 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:15:22.726597 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/daa7078a-68b8-449a-be10-5a87bc9e5eb5-home\") pod \"e2e-same-id-b-0cbb4221-kserve-5cb4b7bf-hpxng\" (UID: \"daa7078a-68b8-449a-be10-5a87bc9e5eb5\") " pod="llm/e2e-same-id-b-0cbb4221-kserve-5cb4b7bf-hpxng" Oct 30 21:15:22.726701 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:15:22.726658 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/daa7078a-68b8-449a-be10-5a87bc9e5eb5-tls-certs\") pod \"e2e-same-id-b-0cbb4221-kserve-5cb4b7bf-hpxng\" (UID: \"daa7078a-68b8-449a-be10-5a87bc9e5eb5\") " pod="llm/e2e-same-id-b-0cbb4221-kserve-5cb4b7bf-hpxng" Oct 30 21:15:22.726735 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:15:22.726723 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/daa7078a-68b8-449a-be10-5a87bc9e5eb5-model-cache\") pod \"e2e-same-id-b-0cbb4221-kserve-5cb4b7bf-hpxng\" (UID: \"daa7078a-68b8-449a-be10-5a87bc9e5eb5\") " pod="llm/e2e-same-id-b-0cbb4221-kserve-5cb4b7bf-hpxng" Oct 30 21:15:22.739343 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:15:22.739313 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:15:22.827669 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:15:22.827634 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/daa7078a-68b8-449a-be10-5a87bc9e5eb5-model-cache\") pod \"e2e-same-id-b-0cbb4221-kserve-5cb4b7bf-hpxng\" (UID: \"daa7078a-68b8-449a-be10-5a87bc9e5eb5\") " pod="llm/e2e-same-id-b-0cbb4221-kserve-5cb4b7bf-hpxng" Oct 30 21:15:22.827859 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:15:22.827700 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-stvpr\" (UniqueName: \"kubernetes.io/projected/daa7078a-68b8-449a-be10-5a87bc9e5eb5-kube-api-access-stvpr\") pod \"e2e-same-id-b-0cbb4221-kserve-5cb4b7bf-hpxng\" (UID: \"daa7078a-68b8-449a-be10-5a87bc9e5eb5\") " pod="llm/e2e-same-id-b-0cbb4221-kserve-5cb4b7bf-hpxng" Oct 30 21:15:22.827859 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:15:22.827720 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/daa7078a-68b8-449a-be10-5a87bc9e5eb5-dshm\") pod \"e2e-same-id-b-0cbb4221-kserve-5cb4b7bf-hpxng\" (UID: \"daa7078a-68b8-449a-be10-5a87bc9e5eb5\") " pod="llm/e2e-same-id-b-0cbb4221-kserve-5cb4b7bf-hpxng" Oct 30 21:15:22.827979 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:15:22.827875 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/daa7078a-68b8-449a-be10-5a87bc9e5eb5-tmp-dir\") pod \"e2e-same-id-b-0cbb4221-kserve-5cb4b7bf-hpxng\" (UID: \"daa7078a-68b8-449a-be10-5a87bc9e5eb5\") " pod="llm/e2e-same-id-b-0cbb4221-kserve-5cb4b7bf-hpxng" Oct 30 21:15:22.828041 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:15:22.827981 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/daa7078a-68b8-449a-be10-5a87bc9e5eb5-home\") pod \"e2e-same-id-b-0cbb4221-kserve-5cb4b7bf-hpxng\" (UID: \"daa7078a-68b8-449a-be10-5a87bc9e5eb5\") " pod="llm/e2e-same-id-b-0cbb4221-kserve-5cb4b7bf-hpxng" Oct 30 21:15:22.828041 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:15:22.828033 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/daa7078a-68b8-449a-be10-5a87bc9e5eb5-tls-certs\") pod \"e2e-same-id-b-0cbb4221-kserve-5cb4b7bf-hpxng\" (UID: \"daa7078a-68b8-449a-be10-5a87bc9e5eb5\") " pod="llm/e2e-same-id-b-0cbb4221-kserve-5cb4b7bf-hpxng" Oct 30 21:15:22.828146 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:15:22.828054 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/daa7078a-68b8-449a-be10-5a87bc9e5eb5-model-cache\") pod \"e2e-same-id-b-0cbb4221-kserve-5cb4b7bf-hpxng\" (UID: \"daa7078a-68b8-449a-be10-5a87bc9e5eb5\") " pod="llm/e2e-same-id-b-0cbb4221-kserve-5cb4b7bf-hpxng" Oct 30 21:15:22.828232 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:15:22.828212 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/daa7078a-68b8-449a-be10-5a87bc9e5eb5-tmp-dir\") pod \"e2e-same-id-b-0cbb4221-kserve-5cb4b7bf-hpxng\" (UID: \"daa7078a-68b8-449a-be10-5a87bc9e5eb5\") " pod="llm/e2e-same-id-b-0cbb4221-kserve-5cb4b7bf-hpxng" Oct 30 21:15:22.828318 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:15:22.828298 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/daa7078a-68b8-449a-be10-5a87bc9e5eb5-home\") pod \"e2e-same-id-b-0cbb4221-kserve-5cb4b7bf-hpxng\" (UID: \"daa7078a-68b8-449a-be10-5a87bc9e5eb5\") " pod="llm/e2e-same-id-b-0cbb4221-kserve-5cb4b7bf-hpxng" Oct 30 21:15:22.829976 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:15:22.829955 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/daa7078a-68b8-449a-be10-5a87bc9e5eb5-dshm\") pod \"e2e-same-id-b-0cbb4221-kserve-5cb4b7bf-hpxng\" (UID: \"daa7078a-68b8-449a-be10-5a87bc9e5eb5\") " pod="llm/e2e-same-id-b-0cbb4221-kserve-5cb4b7bf-hpxng" Oct 30 21:15:22.830294 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:15:22.830277 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/daa7078a-68b8-449a-be10-5a87bc9e5eb5-tls-certs\") pod \"e2e-same-id-b-0cbb4221-kserve-5cb4b7bf-hpxng\" (UID: \"daa7078a-68b8-449a-be10-5a87bc9e5eb5\") " pod="llm/e2e-same-id-b-0cbb4221-kserve-5cb4b7bf-hpxng" Oct 30 21:15:22.839082 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:15:22.839063 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-stvpr\" (UniqueName: \"kubernetes.io/projected/daa7078a-68b8-449a-be10-5a87bc9e5eb5-kube-api-access-stvpr\") pod \"e2e-same-id-b-0cbb4221-kserve-5cb4b7bf-hpxng\" (UID: \"daa7078a-68b8-449a-be10-5a87bc9e5eb5\") " pod="llm/e2e-same-id-b-0cbb4221-kserve-5cb4b7bf-hpxng" Oct 30 21:15:22.951238 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:15:22.951183 4611 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-0cbb4221-kserve-5cb4b7bf-hpxng" Oct 30 21:15:23.225360 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:15:23.225333 4611 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-0cbb4221-kserve-5cb4b7bf-hpxng"] Oct 30 21:15:23.227708 ip-10-0-133-192 kubenswrapper[4611]: W1030 21:15:23.227665 4611 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddaa7078a_68b8_449a_be10_5a87bc9e5eb5.slice/crio-4a10601cbdd7c37508bde7632e633c47f311083e1716fc8e5a56ad09b4dfc78e WatchSource:0}: Error finding container 4a10601cbdd7c37508bde7632e633c47f311083e1716fc8e5a56ad09b4dfc78e: Status 404 returned error can't find the container with id 4a10601cbdd7c37508bde7632e633c47f311083e1716fc8e5a56ad09b4dfc78e Oct 30 21:15:24.081995 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:15:24.081962 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-0cbb4221-kserve-5cb4b7bf-hpxng" event={"ID":"daa7078a-68b8-449a-be10-5a87bc9e5eb5","Type":"ContainerStarted","Data":"fc6c4f0f6b5d925088d8e719ce43a7bbe480b76c19447d41caa9bf85d90b30aa"} Oct 30 21:15:24.082163 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:15:24.082003 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-0cbb4221-kserve-5cb4b7bf-hpxng" event={"ID":"daa7078a-68b8-449a-be10-5a87bc9e5eb5","Type":"ContainerStarted","Data":"4a10601cbdd7c37508bde7632e633c47f311083e1716fc8e5a56ad09b4dfc78e"} Oct 30 21:15:24.100815 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:15:24.100744 4611 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-same-id-b-0cbb4221-kserve-5cb4b7bf-hpxng" podStartSLOduration=2.100731322 podStartE2EDuration="2.100731322s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:15:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:15:24.100263278 +0000 UTC m=+2178.634289935" watchObservedRunningTime="2026-10-30 21:15:24.100731322 +0000 UTC m=+2178.634757937" Oct 30 21:15:24.953350 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:15:24.953311 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-0cbb4221-kserve-5cb4b7bf-hpxng" Oct 30 21:15:32.025469 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:15:32.025436 4611 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8ms7c"] Oct 30 21:15:32.028969 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:15:32.028942 4611 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8ms7c"] Oct 30 21:15:32.951381 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:15:32.951349 4611 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-0cbb4221-kserve-5cb4b7bf-hpxng" Oct 30 21:15:32.963916 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:15:32.963892 4611 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-0cbb4221-kserve-5cb4b7bf-hpxng" Oct 30 21:15:33.178704 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:15:33.178673 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-0cbb4221-kserve-5cb4b7bf-hpxng" Oct 30 21:15:33.605024 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:15:33.604980 4611 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2648a43f-15d6-4fa1-9111-8188e578075b" path="/var/lib/kubelet/pods/2648a43f-15d6-4fa1-9111-8188e578075b/volumes" Oct 30 21:15:33.916688 ip-10-0-133-192 kubenswrapper[4611]: E1030 21:15:33.916611 4611 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hwzc6" containerName="extractor" Oct 30 21:15:48.958493 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:15:48.958415 4611 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-grm5k"] Oct 30 21:15:48.958977 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:15:48.958818 4611 cpu_manager.go:414] "containerMap: removing container" podUID="dcdd7301-b310-4518-ad18-02d055ab1f2a" containerName="istio-proxy" Oct 30 21:15:48.958977 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:15:48.958832 4611 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dcdd7301-b310-4518-ad18-02d055ab1f2a" containerName="istio-proxy" Oct 30 21:15:48.958977 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:15:48.958948 4611 memory_manager.go:365] "RemoveStaleState removing state" podUID="dcdd7301-b310-4518-ad18-02d055ab1f2a" containerName="istio-proxy" Oct 30 21:15:48.962241 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:15:48.962221 4611 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-grm5k" Oct 30 21:15:48.965741 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:15:48.965719 4611 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:15:48.966277 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:15:48.966258 4611 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7hf8r\"" Oct 30 21:15:48.966339 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:15:48.966311 4611 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:15:48.966378 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:15:48.966335 4611 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:15:48.974666 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:15:48.974644 4611 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-grm5k"] Oct 30 21:15:49.120922 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:15:49.120887 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rsqfg\" (UniqueName: \"kubernetes.io/projected/0f9b7600-8024-469c-af45-f6ee6ff5ae78-kube-api-access-rsqfg\") pod \"maas-api-revoke-sub-452b6d7b-grm5k\" (UID: \"0f9b7600-8024-469c-af45-f6ee6ff5ae78\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-grm5k" Oct 30 21:15:49.120922 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:15:49.120928 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f9b7600-8024-469c-af45-f6ee6ff5ae78-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-grm5k\" (UID: \"0f9b7600-8024-469c-af45-f6ee6ff5ae78\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-grm5k" Oct 30 21:15:49.121112 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:15:49.121014 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f9b7600-8024-469c-af45-f6ee6ff5ae78-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-grm5k\" (UID: \"0f9b7600-8024-469c-af45-f6ee6ff5ae78\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-grm5k" Oct 30 21:15:49.221699 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:15:49.221637 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rsqfg\" (UniqueName: \"kubernetes.io/projected/0f9b7600-8024-469c-af45-f6ee6ff5ae78-kube-api-access-rsqfg\") pod \"maas-api-revoke-sub-452b6d7b-grm5k\" (UID: \"0f9b7600-8024-469c-af45-f6ee6ff5ae78\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-grm5k" Oct 30 21:15:49.221699 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:15:49.221673 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f9b7600-8024-469c-af45-f6ee6ff5ae78-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-grm5k\" (UID: \"0f9b7600-8024-469c-af45-f6ee6ff5ae78\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-grm5k" Oct 30 21:15:49.221866 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:15:49.221701 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f9b7600-8024-469c-af45-f6ee6ff5ae78-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-grm5k\" (UID: \"0f9b7600-8024-469c-af45-f6ee6ff5ae78\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-grm5k" Oct 30 21:15:49.222264 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:15:49.222240 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f9b7600-8024-469c-af45-f6ee6ff5ae78-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-grm5k\" (UID: \"0f9b7600-8024-469c-af45-f6ee6ff5ae78\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-grm5k" Oct 30 21:15:49.223994 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:15:49.223971 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f9b7600-8024-469c-af45-f6ee6ff5ae78-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-grm5k\" (UID: \"0f9b7600-8024-469c-af45-f6ee6ff5ae78\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-grm5k" Oct 30 21:15:49.233357 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:15:49.233336 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rsqfg\" (UniqueName: \"kubernetes.io/projected/0f9b7600-8024-469c-af45-f6ee6ff5ae78-kube-api-access-rsqfg\") pod \"maas-api-revoke-sub-452b6d7b-grm5k\" (UID: \"0f9b7600-8024-469c-af45-f6ee6ff5ae78\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-grm5k" Oct 30 21:15:49.275456 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:15:49.275425 4611 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-grm5k" Oct 30 21:15:49.552799 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:15:49.552772 4611 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-grm5k"] Oct 30 21:15:49.555276 ip-10-0-133-192 kubenswrapper[4611]: W1030 21:15:49.555246 4611 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f9b7600_8024_469c_af45_f6ee6ff5ae78.slice/crio-c5c0ff7a0e0d7e0a67653d53646beb7991cb071fc90996760c009f98a188361b WatchSource:0}: Error finding container c5c0ff7a0e0d7e0a67653d53646beb7991cb071fc90996760c009f98a188361b: Status 404 returned error can't find the container with id c5c0ff7a0e0d7e0a67653d53646beb7991cb071fc90996760c009f98a188361b Oct 30 21:15:49.671164 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:15:49.671137 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:15:50.291610 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:15:50.291522 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-grm5k" event={"ID":"0f9b7600-8024-469c-af45-f6ee6ff5ae78","Type":"ContainerStarted","Data":"d19560c35a519a4cd8fcf6ec353ffb8ce8e03c5ddb1841486d068e6f56aa90ec"} Oct 30 21:15:50.291610 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:15:50.291586 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-grm5k" event={"ID":"0f9b7600-8024-469c-af45-f6ee6ff5ae78","Type":"ContainerStarted","Data":"c5c0ff7a0e0d7e0a67653d53646beb7991cb071fc90996760c009f98a188361b"} Oct 30 21:15:50.309732 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:15:50.309674 4611 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-grm5k" podStartSLOduration=2.309660116 podStartE2EDuration="2.309660116s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:15:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:15:50.307611364 +0000 UTC m=+2204.841637981" watchObservedRunningTime="2026-10-30 21:15:50.309660116 +0000 UTC m=+2204.843686731" Oct 30 21:15:51.297735 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:15:51.297699 4611 generic.go:356] "Generic (PLEG): container finished" podID="0f9b7600-8024-469c-af45-f6ee6ff5ae78" containerID="d19560c35a519a4cd8fcf6ec353ffb8ce8e03c5ddb1841486d068e6f56aa90ec" exitCode=0 Oct 30 21:15:51.298107 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:15:51.297781 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-grm5k" event={"ID":"0f9b7600-8024-469c-af45-f6ee6ff5ae78","Type":"ContainerDied","Data":"d19560c35a519a4cd8fcf6ec353ffb8ce8e03c5ddb1841486d068e6f56aa90ec"} Oct 30 21:15:52.441910 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:15:52.441883 4611 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-grm5k" Oct 30 21:15:52.449074 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:15:52.449051 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0f9b7600-8024-469c-af45-f6ee6ff5ae78-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f9b7600-8024-469c-af45-f6ee6ff5ae78-maas-api-xks-ca\") pod \"0f9b7600-8024-469c-af45-f6ee6ff5ae78\" (UID: \"0f9b7600-8024-469c-af45-f6ee6ff5ae78\") " Oct 30 21:15:52.449184 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:15:52.449168 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f9b7600-8024-469c-af45-f6ee6ff5ae78-kube-api-access-rsqfg\" (UniqueName: \"kubernetes.io/projected/0f9b7600-8024-469c-af45-f6ee6ff5ae78-kube-api-access-rsqfg\") pod \"0f9b7600-8024-469c-af45-f6ee6ff5ae78\" (UID: \"0f9b7600-8024-469c-af45-f6ee6ff5ae78\") " Oct 30 21:15:52.449247 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:15:52.449195 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0f9b7600-8024-469c-af45-f6ee6ff5ae78-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f9b7600-8024-469c-af45-f6ee6ff5ae78-maas-api-openshift-ca\") pod \"0f9b7600-8024-469c-af45-f6ee6ff5ae78\" (UID: \"0f9b7600-8024-469c-af45-f6ee6ff5ae78\") " Oct 30 21:15:52.449595 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:15:52.449574 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f9b7600-8024-469c-af45-f6ee6ff5ae78-maas-api-openshift-ca" pod "0f9b7600-8024-469c-af45-f6ee6ff5ae78" (UID: "0f9b7600-8024-469c-af45-f6ee6ff5ae78"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:15:52.451009 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:15:52.450981 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f9b7600-8024-469c-af45-f6ee6ff5ae78-maas-api-xks-ca" pod "0f9b7600-8024-469c-af45-f6ee6ff5ae78" (UID: "0f9b7600-8024-469c-af45-f6ee6ff5ae78"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:15:52.451088 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:15:52.451074 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f9b7600-8024-469c-af45-f6ee6ff5ae78-kube-api-access-rsqfg" pod "0f9b7600-8024-469c-af45-f6ee6ff5ae78" (UID: "0f9b7600-8024-469c-af45-f6ee6ff5ae78"). InnerVolumeSpecName "kube-api-access-rsqfg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:15:52.550083 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:15:52.550053 4611 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f9b7600-8024-469c-af45-f6ee6ff5ae78-maas-api-openshift-ca\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:15:52.550083 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:15:52.550082 4611 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f9b7600-8024-469c-af45-f6ee6ff5ae78-maas-api-xks-ca\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:15:52.550238 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:15:52.550092 4611 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rsqfg\" (UniqueName: \"kubernetes.io/projected/0f9b7600-8024-469c-af45-f6ee6ff5ae78-kube-api-access-rsqfg\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:15:53.309893 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:15:53.309866 4611 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-grm5k" Oct 30 21:15:53.310098 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:15:53.309908 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-grm5k" event={"ID":"0f9b7600-8024-469c-af45-f6ee6ff5ae78","Type":"ContainerDied","Data":"c5c0ff7a0e0d7e0a67653d53646beb7991cb071fc90996760c009f98a188361b"} Oct 30 21:15:53.310098 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:15:53.309953 4611 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c5c0ff7a0e0d7e0a67653d53646beb7991cb071fc90996760c009f98a188361b" Oct 30 21:16:00.074885 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:16:00.074851 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:16:02.562572 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:16:02.562527 4611 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-0cbb4221-kserve-5cb4b7bf-hpxng"] Oct 30 21:16:02.646424 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:16:02.646334 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:16:04.070532 ip-10-0-133-192 kubenswrapper[4611]: E1030 21:16:04.067677 4611 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hwzc6" containerName="extractor" Oct 30 21:16:06.443943 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:16:06.443909 4611 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-nfcvw"] Oct 30 21:16:06.444934 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:16:06.444908 4611 cpu_manager.go:414] "containerMap: removing container" podUID="0f9b7600-8024-469c-af45-f6ee6ff5ae78" containerName="revoke-keys" Oct 30 21:16:06.460848 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:16:06.460816 4611 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f9b7600-8024-469c-af45-f6ee6ff5ae78" containerName="revoke-keys" Oct 30 21:16:06.472531 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:16:06.461209 4611 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f9b7600-8024-469c-af45-f6ee6ff5ae78" containerName="revoke-keys" Oct 30 21:16:06.527813 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:16:06.527778 4611 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-nfcvw"] Oct 30 21:16:06.542535 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:16:06.528598 4611 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-nfcvw" Oct 30 21:16:06.555698 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:16:06.543185 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tjv7m\" (UniqueName: \"kubernetes.io/projected/3f8ebac8-5c1b-4335-ad73-89d000d9d36d-kube-api-access-tjv7m\") pod \"maas-api-revoke-sub-3e31a506-nfcvw\" (UID: \"3f8ebac8-5c1b-4335-ad73-89d000d9d36d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-nfcvw" Oct 30 21:16:06.555698 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:16:06.543862 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f8ebac8-5c1b-4335-ad73-89d000d9d36d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-nfcvw\" (UID: \"3f8ebac8-5c1b-4335-ad73-89d000d9d36d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-nfcvw" Oct 30 21:16:06.555698 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:16:06.543914 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f8ebac8-5c1b-4335-ad73-89d000d9d36d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-nfcvw\" (UID: \"3f8ebac8-5c1b-4335-ad73-89d000d9d36d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-nfcvw" Oct 30 21:16:06.555698 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:16:06.548952 4611 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:16:06.555698 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:16:06.549160 4611 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:16:06.567903 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:16:06.567739 4611 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:16:06.567903 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:16:06.567850 4611 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7hf8r\"" Oct 30 21:16:06.648529 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:16:06.644659 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f8ebac8-5c1b-4335-ad73-89d000d9d36d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-nfcvw\" (UID: \"3f8ebac8-5c1b-4335-ad73-89d000d9d36d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-nfcvw" Oct 30 21:16:06.656528 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:16:06.652708 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tjv7m\" (UniqueName: \"kubernetes.io/projected/3f8ebac8-5c1b-4335-ad73-89d000d9d36d-kube-api-access-tjv7m\") pod \"maas-api-revoke-sub-3e31a506-nfcvw\" (UID: \"3f8ebac8-5c1b-4335-ad73-89d000d9d36d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-nfcvw" Oct 30 21:16:06.660597 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:16:06.660571 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f8ebac8-5c1b-4335-ad73-89d000d9d36d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-nfcvw\" (UID: \"3f8ebac8-5c1b-4335-ad73-89d000d9d36d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-nfcvw" Oct 30 21:16:06.664528 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:16:06.661670 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f8ebac8-5c1b-4335-ad73-89d000d9d36d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-nfcvw\" (UID: \"3f8ebac8-5c1b-4335-ad73-89d000d9d36d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-nfcvw" Oct 30 21:16:06.704662 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:16:06.704207 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tjv7m\" (UniqueName: \"kubernetes.io/projected/3f8ebac8-5c1b-4335-ad73-89d000d9d36d-kube-api-access-tjv7m\") pod \"maas-api-revoke-sub-3e31a506-nfcvw\" (UID: \"3f8ebac8-5c1b-4335-ad73-89d000d9d36d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-nfcvw" Oct 30 21:16:06.704662 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:16:06.704582 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f8ebac8-5c1b-4335-ad73-89d000d9d36d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-nfcvw\" (UID: \"3f8ebac8-5c1b-4335-ad73-89d000d9d36d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-nfcvw" Oct 30 21:16:06.987580 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:16:06.986006 4611 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-nfcvw" Oct 30 21:16:07.952423 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:16:07.952391 4611 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-nfcvw"] Oct 30 21:16:08.479626 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:16:08.479597 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-nfcvw" event={"ID":"3f8ebac8-5c1b-4335-ad73-89d000d9d36d","Type":"ContainerStarted","Data":"b9a5f81fc9ae4042c38007c3ca2e7ad332188ef72e2ed6cd42ffc85979e60fbc"} Oct 30 21:16:08.479753 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:16:08.479651 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-nfcvw" event={"ID":"3f8ebac8-5c1b-4335-ad73-89d000d9d36d","Type":"ContainerStarted","Data":"bc71f27dd96031733fc71b10bc51770badabf9e4a25fc87d3e68b40399f56b20"} Oct 30 21:16:08.498389 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:16:08.498343 4611 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-nfcvw" podStartSLOduration=2.498325869 podStartE2EDuration="2.498325869s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:16:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:16:08.495750719 +0000 UTC m=+2223.029777336" watchObservedRunningTime="2026-10-30 21:16:08.498325869 +0000 UTC m=+2223.032352484" Oct 30 21:16:08.665715 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:16:08.665691 4611 scope.go:122] "RemoveContainer" containerID="b1e7867143d9a27875cda89ed6b1d62a34f3b2b9fce1b36c4d8936fa43fa7c1c" Oct 30 21:16:09.485495 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:16:09.485456 4611 generic.go:356] "Generic (PLEG): container finished" podID="3f8ebac8-5c1b-4335-ad73-89d000d9d36d" containerID="b9a5f81fc9ae4042c38007c3ca2e7ad332188ef72e2ed6cd42ffc85979e60fbc" exitCode=0 Oct 30 21:16:09.485922 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:16:09.485538 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-nfcvw" event={"ID":"3f8ebac8-5c1b-4335-ad73-89d000d9d36d","Type":"ContainerDied","Data":"b9a5f81fc9ae4042c38007c3ca2e7ad332188ef72e2ed6cd42ffc85979e60fbc"} Oct 30 21:16:10.621931 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:16:10.621909 4611 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-nfcvw" Oct 30 21:16:10.661164 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:16:10.661127 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3f8ebac8-5c1b-4335-ad73-89d000d9d36d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f8ebac8-5c1b-4335-ad73-89d000d9d36d-maas-api-xks-ca\") pod \"3f8ebac8-5c1b-4335-ad73-89d000d9d36d\" (UID: \"3f8ebac8-5c1b-4335-ad73-89d000d9d36d\") " Oct 30 21:16:10.661290 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:16:10.661182 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f8ebac8-5c1b-4335-ad73-89d000d9d36d-kube-api-access-tjv7m\" (UniqueName: \"kubernetes.io/projected/3f8ebac8-5c1b-4335-ad73-89d000d9d36d-kube-api-access-tjv7m\") pod \"3f8ebac8-5c1b-4335-ad73-89d000d9d36d\" (UID: \"3f8ebac8-5c1b-4335-ad73-89d000d9d36d\") " Oct 30 21:16:10.661372 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:16:10.661306 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3f8ebac8-5c1b-4335-ad73-89d000d9d36d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f8ebac8-5c1b-4335-ad73-89d000d9d36d-maas-api-openshift-ca\") pod \"3f8ebac8-5c1b-4335-ad73-89d000d9d36d\" (UID: \"3f8ebac8-5c1b-4335-ad73-89d000d9d36d\") " Oct 30 21:16:10.661669 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:16:10.661641 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f8ebac8-5c1b-4335-ad73-89d000d9d36d-maas-api-openshift-ca" pod "3f8ebac8-5c1b-4335-ad73-89d000d9d36d" (UID: "3f8ebac8-5c1b-4335-ad73-89d000d9d36d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:16:10.663160 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:16:10.663132 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f8ebac8-5c1b-4335-ad73-89d000d9d36d-maas-api-xks-ca" pod "3f8ebac8-5c1b-4335-ad73-89d000d9d36d" (UID: "3f8ebac8-5c1b-4335-ad73-89d000d9d36d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:16:10.663627 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:16:10.663604 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f8ebac8-5c1b-4335-ad73-89d000d9d36d-kube-api-access-tjv7m" pod "3f8ebac8-5c1b-4335-ad73-89d000d9d36d" (UID: "3f8ebac8-5c1b-4335-ad73-89d000d9d36d"). InnerVolumeSpecName "kube-api-access-tjv7m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:16:10.762488 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:16:10.762429 4611 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f8ebac8-5c1b-4335-ad73-89d000d9d36d-maas-api-openshift-ca\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:16:10.762488 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:16:10.762453 4611 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f8ebac8-5c1b-4335-ad73-89d000d9d36d-maas-api-xks-ca\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:16:10.762488 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:16:10.762463 4611 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tjv7m\" (UniqueName: \"kubernetes.io/projected/3f8ebac8-5c1b-4335-ad73-89d000d9d36d-kube-api-access-tjv7m\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:16:11.496074 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:16:11.496039 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-nfcvw" event={"ID":"3f8ebac8-5c1b-4335-ad73-89d000d9d36d","Type":"ContainerDied","Data":"bc71f27dd96031733fc71b10bc51770badabf9e4a25fc87d3e68b40399f56b20"} Oct 30 21:16:11.496292 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:16:11.496088 4611 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bc71f27dd96031733fc71b10bc51770badabf9e4a25fc87d3e68b40399f56b20" Oct 30 21:16:11.496292 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:16:11.496053 4611 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-nfcvw" Oct 30 21:16:17.595989 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:16:17.595944 4611 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-same-id-b-0cbb4221-kserve-5cb4b7bf-hpxng" podUID="daa7078a-68b8-449a-be10-5a87bc9e5eb5" containerName="main" containerID="cri-o://fc6c4f0f6b5d925088d8e719ce43a7bbe480b76c19447d41caa9bf85d90b30aa" gracePeriod=45 Oct 30 21:16:18.241206 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:16:18.240942 4611 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-0cbb4221-kserve-5cb4b7bf-hpxng" Oct 30 21:16:18.316888 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:16:18.312802 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/daa7078a-68b8-449a-be10-5a87bc9e5eb5-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/daa7078a-68b8-449a-be10-5a87bc9e5eb5-model-cache\") pod \"daa7078a-68b8-449a-be10-5a87bc9e5eb5\" (UID: \"daa7078a-68b8-449a-be10-5a87bc9e5eb5\") " Oct 30 21:16:18.316888 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:16:18.312884 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/daa7078a-68b8-449a-be10-5a87bc9e5eb5-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/daa7078a-68b8-449a-be10-5a87bc9e5eb5-tmp-dir\") pod \"daa7078a-68b8-449a-be10-5a87bc9e5eb5\" (UID: \"daa7078a-68b8-449a-be10-5a87bc9e5eb5\") " Oct 30 21:16:18.316888 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:16:18.312962 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/daa7078a-68b8-449a-be10-5a87bc9e5eb5-tls-certs\" (UniqueName: \"kubernetes.io/secret/daa7078a-68b8-449a-be10-5a87bc9e5eb5-tls-certs\") pod \"daa7078a-68b8-449a-be10-5a87bc9e5eb5\" (UID: \"daa7078a-68b8-449a-be10-5a87bc9e5eb5\") " Oct 30 21:16:18.316888 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:16:18.313081 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/daa7078a-68b8-449a-be10-5a87bc9e5eb5-dshm\" (UniqueName: \"kubernetes.io/empty-dir/daa7078a-68b8-449a-be10-5a87bc9e5eb5-dshm\") pod \"daa7078a-68b8-449a-be10-5a87bc9e5eb5\" (UID: \"daa7078a-68b8-449a-be10-5a87bc9e5eb5\") " Oct 30 21:16:18.316888 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:16:18.313125 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/daa7078a-68b8-449a-be10-5a87bc9e5eb5-kube-api-access-stvpr\" (UniqueName: \"kubernetes.io/projected/daa7078a-68b8-449a-be10-5a87bc9e5eb5-kube-api-access-stvpr\") pod \"daa7078a-68b8-449a-be10-5a87bc9e5eb5\" (UID: \"daa7078a-68b8-449a-be10-5a87bc9e5eb5\") " Oct 30 21:16:18.316888 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:16:18.313154 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/daa7078a-68b8-449a-be10-5a87bc9e5eb5-home\" (UniqueName: \"kubernetes.io/empty-dir/daa7078a-68b8-449a-be10-5a87bc9e5eb5-home\") pod \"daa7078a-68b8-449a-be10-5a87bc9e5eb5\" (UID: \"daa7078a-68b8-449a-be10-5a87bc9e5eb5\") " Oct 30 21:16:18.317779 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:16:18.317736 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/daa7078a-68b8-449a-be10-5a87bc9e5eb5-home" pod "daa7078a-68b8-449a-be10-5a87bc9e5eb5" (UID: "daa7078a-68b8-449a-be10-5a87bc9e5eb5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:16:18.325584 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:16:18.318741 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/daa7078a-68b8-449a-be10-5a87bc9e5eb5-tmp-dir" pod "daa7078a-68b8-449a-be10-5a87bc9e5eb5" (UID: "daa7078a-68b8-449a-be10-5a87bc9e5eb5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:16:18.330619 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:16:18.330578 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/daa7078a-68b8-449a-be10-5a87bc9e5eb5-model-cache" pod "daa7078a-68b8-449a-be10-5a87bc9e5eb5" (UID: "daa7078a-68b8-449a-be10-5a87bc9e5eb5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:16:18.342918 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:16:18.342895 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/daa7078a-68b8-449a-be10-5a87bc9e5eb5-dshm" pod "daa7078a-68b8-449a-be10-5a87bc9e5eb5" (UID: "daa7078a-68b8-449a-be10-5a87bc9e5eb5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:16:18.344829 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:16:18.344792 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/daa7078a-68b8-449a-be10-5a87bc9e5eb5-tls-certs" pod "daa7078a-68b8-449a-be10-5a87bc9e5eb5" (UID: "daa7078a-68b8-449a-be10-5a87bc9e5eb5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:16:18.438291 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:16:18.438169 4611 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/daa7078a-68b8-449a-be10-5a87bc9e5eb5-tmp-dir\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:16:18.438291 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:16:18.438205 4611 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/daa7078a-68b8-449a-be10-5a87bc9e5eb5-tls-certs\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:16:18.438291 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:16:18.438220 4611 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/daa7078a-68b8-449a-be10-5a87bc9e5eb5-dshm\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:16:18.438291 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:16:18.438234 4611 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/daa7078a-68b8-449a-be10-5a87bc9e5eb5-home\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:16:18.438291 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:16:18.438249 4611 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/daa7078a-68b8-449a-be10-5a87bc9e5eb5-model-cache\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:16:18.444114 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:16:18.444033 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/daa7078a-68b8-449a-be10-5a87bc9e5eb5-kube-api-access-stvpr" pod "daa7078a-68b8-449a-be10-5a87bc9e5eb5" (UID: "daa7078a-68b8-449a-be10-5a87bc9e5eb5"). InnerVolumeSpecName "kube-api-access-stvpr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:16:18.536327 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:16:18.536276 4611 generic.go:356] "Generic (PLEG): container finished" podID="daa7078a-68b8-449a-be10-5a87bc9e5eb5" containerID="fc6c4f0f6b5d925088d8e719ce43a7bbe480b76c19447d41caa9bf85d90b30aa" exitCode=0 Oct 30 21:16:18.536731 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:16:18.536714 4611 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-0cbb4221-kserve-5cb4b7bf-hpxng" Oct 30 21:16:18.537941 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:16:18.537716 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-0cbb4221-kserve-5cb4b7bf-hpxng" event={"ID":"daa7078a-68b8-449a-be10-5a87bc9e5eb5","Type":"ContainerDied","Data":"fc6c4f0f6b5d925088d8e719ce43a7bbe480b76c19447d41caa9bf85d90b30aa"} Oct 30 21:16:18.537941 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:16:18.537763 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-0cbb4221-kserve-5cb4b7bf-hpxng" event={"ID":"daa7078a-68b8-449a-be10-5a87bc9e5eb5","Type":"ContainerDied","Data":"4a10601cbdd7c37508bde7632e633c47f311083e1716fc8e5a56ad09b4dfc78e"} Oct 30 21:16:18.537941 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:16:18.537786 4611 scope.go:122] "RemoveContainer" containerID="fc6c4f0f6b5d925088d8e719ce43a7bbe480b76c19447d41caa9bf85d90b30aa" Oct 30 21:16:18.543105 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:16:18.543063 4611 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-stvpr\" (UniqueName: \"kubernetes.io/projected/daa7078a-68b8-449a-be10-5a87bc9e5eb5-kube-api-access-stvpr\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:16:18.642799 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:16:18.632142 4611 scope.go:122] "RemoveContainer" containerID="fc6c4f0f6b5d925088d8e719ce43a7bbe480b76c19447d41caa9bf85d90b30aa" Oct 30 21:16:18.646571 ip-10-0-133-192 kubenswrapper[4611]: E1030 21:16:18.643417 4611 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc6c4f0f6b5d925088d8e719ce43a7bbe480b76c19447d41caa9bf85d90b30aa\": container with ID starting with fc6c4f0f6b5d925088d8e719ce43a7bbe480b76c19447d41caa9bf85d90b30aa not found: ID does not exist" containerID="fc6c4f0f6b5d925088d8e719ce43a7bbe480b76c19447d41caa9bf85d90b30aa" Oct 30 21:16:18.646571 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:16:18.643461 4611 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc6c4f0f6b5d925088d8e719ce43a7bbe480b76c19447d41caa9bf85d90b30aa"} err="failed to get container status \"fc6c4f0f6b5d925088d8e719ce43a7bbe480b76c19447d41caa9bf85d90b30aa\": rpc error: code = NotFound desc = could not find container \"fc6c4f0f6b5d925088d8e719ce43a7bbe480b76c19447d41caa9bf85d90b30aa\": container with ID starting with fc6c4f0f6b5d925088d8e719ce43a7bbe480b76c19447d41caa9bf85d90b30aa not found: ID does not exist" Oct 30 21:16:18.684596 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:16:18.684567 4611 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-0cbb4221-kserve-5cb4b7bf-hpxng"] Oct 30 21:16:18.699589 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:16:18.699499 4611 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-0cbb4221-kserve-5cb4b7bf-hpxng"] Oct 30 21:16:19.191318 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:16:19.190542 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-same-id-b-0cbb4221-kserve-5cb4b7bf-hpxng" podUID="daa7078a-68b8-449a-be10-5a87bc9e5eb5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.133:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:16:19.608669 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:16:19.608631 4611 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="daa7078a-68b8-449a-be10-5a87bc9e5eb5" path="/var/lib/kubelet/pods/daa7078a-68b8-449a-be10-5a87bc9e5eb5/volumes" Oct 30 21:16:33.916284 ip-10-0-133-192 kubenswrapper[4611]: E1030 21:16:33.916248 4611 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hwzc6" containerName="extractor" Oct 30 21:16:34.746741 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:16:34.746699 4611 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-360dbc5f-9n2dv"] Oct 30 21:16:34.747291 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:16:34.747269 4611 cpu_manager.go:414] "containerMap: removing container" podUID="daa7078a-68b8-449a-be10-5a87bc9e5eb5" containerName="main" Oct 30 21:16:34.747337 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:16:34.747295 4611 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="daa7078a-68b8-449a-be10-5a87bc9e5eb5" containerName="main" Oct 30 21:16:34.747337 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:16:34.747331 4611 cpu_manager.go:414] "containerMap: removing container" podUID="3f8ebac8-5c1b-4335-ad73-89d000d9d36d" containerName="revoke-keys" Oct 30 21:16:34.747409 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:16:34.747338 4611 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f8ebac8-5c1b-4335-ad73-89d000d9d36d" containerName="revoke-keys" Oct 30 21:16:34.747470 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:16:34.747457 4611 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f8ebac8-5c1b-4335-ad73-89d000d9d36d" containerName="revoke-keys" Oct 30 21:16:34.747517 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:16:34.747483 4611 memory_manager.go:365] "RemoveStaleState removing state" podUID="daa7078a-68b8-449a-be10-5a87bc9e5eb5" containerName="main" Oct 30 21:16:34.750459 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:16:34.750444 4611 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-360dbc5f-9n2dv" Oct 30 21:16:34.757129 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:16:34.757107 4611 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:16:34.757305 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:16:34.757133 4611 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:16:34.757395 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:16:34.757173 4611 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7hf8r\"" Oct 30 21:16:34.757395 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:16:34.757205 4611 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:16:34.760487 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:16:34.760461 4611 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-360dbc5f-9n2dv"] Oct 30 21:16:34.839803 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:16:34.839776 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38e51b01-a6a7-46fa-bf03-8dd55c611586-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-360dbc5f-9n2dv\" (UID: \"38e51b01-a6a7-46fa-bf03-8dd55c611586\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-360dbc5f-9n2dv" Oct 30 21:16:34.839952 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:16:34.839829 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38e51b01-a6a7-46fa-bf03-8dd55c611586-maas-api-xks-ca\") pod \"maas-api-revoke-sub-360dbc5f-9n2dv\" (UID: \"38e51b01-a6a7-46fa-bf03-8dd55c611586\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-360dbc5f-9n2dv" Oct 30 21:16:34.839952 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:16:34.839908 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h95q9\" (UniqueName: \"kubernetes.io/projected/38e51b01-a6a7-46fa-bf03-8dd55c611586-kube-api-access-h95q9\") pod \"maas-api-revoke-sub-360dbc5f-9n2dv\" (UID: \"38e51b01-a6a7-46fa-bf03-8dd55c611586\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-360dbc5f-9n2dv" Oct 30 21:16:34.940562 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:16:34.940528 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38e51b01-a6a7-46fa-bf03-8dd55c611586-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-360dbc5f-9n2dv\" (UID: \"38e51b01-a6a7-46fa-bf03-8dd55c611586\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-360dbc5f-9n2dv" Oct 30 21:16:34.940935 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:16:34.940580 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38e51b01-a6a7-46fa-bf03-8dd55c611586-maas-api-xks-ca\") pod \"maas-api-revoke-sub-360dbc5f-9n2dv\" (UID: \"38e51b01-a6a7-46fa-bf03-8dd55c611586\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-360dbc5f-9n2dv" Oct 30 21:16:34.940935 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:16:34.940618 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h95q9\" (UniqueName: \"kubernetes.io/projected/38e51b01-a6a7-46fa-bf03-8dd55c611586-kube-api-access-h95q9\") pod \"maas-api-revoke-sub-360dbc5f-9n2dv\" (UID: \"38e51b01-a6a7-46fa-bf03-8dd55c611586\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-360dbc5f-9n2dv" Oct 30 21:16:34.941168 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:16:34.941140 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38e51b01-a6a7-46fa-bf03-8dd55c611586-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-360dbc5f-9n2dv\" (UID: \"38e51b01-a6a7-46fa-bf03-8dd55c611586\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-360dbc5f-9n2dv" Oct 30 21:16:34.942858 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:16:34.942836 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38e51b01-a6a7-46fa-bf03-8dd55c611586-maas-api-xks-ca\") pod \"maas-api-revoke-sub-360dbc5f-9n2dv\" (UID: \"38e51b01-a6a7-46fa-bf03-8dd55c611586\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-360dbc5f-9n2dv" Oct 30 21:16:34.952219 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:16:34.952196 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h95q9\" (UniqueName: \"kubernetes.io/projected/38e51b01-a6a7-46fa-bf03-8dd55c611586-kube-api-access-h95q9\") pod \"maas-api-revoke-sub-360dbc5f-9n2dv\" (UID: \"38e51b01-a6a7-46fa-bf03-8dd55c611586\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-360dbc5f-9n2dv" Oct 30 21:16:35.066655 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:16:35.066601 4611 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-360dbc5f-9n2dv" Oct 30 21:16:35.341039 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:16:35.340938 4611 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-360dbc5f-9n2dv"] Oct 30 21:16:35.343760 ip-10-0-133-192 kubenswrapper[4611]: W1030 21:16:35.343713 4611 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38e51b01_a6a7_46fa_bf03_8dd55c611586.slice/crio-d8106932e88d8ca93626d4271a13d512df4e9cfde945eaaa75826ad251aba3e3 WatchSource:0}: Error finding container d8106932e88d8ca93626d4271a13d512df4e9cfde945eaaa75826ad251aba3e3: Status 404 returned error can't find the container with id d8106932e88d8ca93626d4271a13d512df4e9cfde945eaaa75826ad251aba3e3 Oct 30 21:16:35.633071 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:16:35.633040 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-360dbc5f-9n2dv" event={"ID":"38e51b01-a6a7-46fa-bf03-8dd55c611586","Type":"ContainerStarted","Data":"d8106932e88d8ca93626d4271a13d512df4e9cfde945eaaa75826ad251aba3e3"} Oct 30 21:16:36.639642 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:16:36.639608 4611 generic.go:356] "Generic (PLEG): container finished" podID="38e51b01-a6a7-46fa-bf03-8dd55c611586" containerID="af711a701d4897b2c025bc8b9f9d363b34d88cc732a7c7f11d5c27ae52952765" exitCode=0 Oct 30 21:16:36.640026 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:16:36.639697 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-360dbc5f-9n2dv" event={"ID":"38e51b01-a6a7-46fa-bf03-8dd55c611586","Type":"ContainerDied","Data":"af711a701d4897b2c025bc8b9f9d363b34d88cc732a7c7f11d5c27ae52952765"} Oct 30 21:16:37.778652 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:16:37.778627 4611 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-360dbc5f-9n2dv" Oct 30 21:16:37.864327 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:16:37.864294 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/38e51b01-a6a7-46fa-bf03-8dd55c611586-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38e51b01-a6a7-46fa-bf03-8dd55c611586-maas-api-xks-ca\") pod \"38e51b01-a6a7-46fa-bf03-8dd55c611586\" (UID: \"38e51b01-a6a7-46fa-bf03-8dd55c611586\") " Oct 30 21:16:37.864522 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:16:37.864380 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/38e51b01-a6a7-46fa-bf03-8dd55c611586-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38e51b01-a6a7-46fa-bf03-8dd55c611586-maas-api-openshift-ca\") pod \"38e51b01-a6a7-46fa-bf03-8dd55c611586\" (UID: \"38e51b01-a6a7-46fa-bf03-8dd55c611586\") " Oct 30 21:16:37.864522 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:16:37.864426 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/38e51b01-a6a7-46fa-bf03-8dd55c611586-kube-api-access-h95q9\" (UniqueName: \"kubernetes.io/projected/38e51b01-a6a7-46fa-bf03-8dd55c611586-kube-api-access-h95q9\") pod \"38e51b01-a6a7-46fa-bf03-8dd55c611586\" (UID: \"38e51b01-a6a7-46fa-bf03-8dd55c611586\") " Oct 30 21:16:37.864853 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:16:37.864825 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38e51b01-a6a7-46fa-bf03-8dd55c611586-maas-api-openshift-ca" pod "38e51b01-a6a7-46fa-bf03-8dd55c611586" (UID: "38e51b01-a6a7-46fa-bf03-8dd55c611586"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:16:37.866406 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:16:37.866376 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/38e51b01-a6a7-46fa-bf03-8dd55c611586-maas-api-xks-ca" pod "38e51b01-a6a7-46fa-bf03-8dd55c611586" (UID: "38e51b01-a6a7-46fa-bf03-8dd55c611586"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:16:37.866561 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:16:37.866539 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38e51b01-a6a7-46fa-bf03-8dd55c611586-kube-api-access-h95q9" pod "38e51b01-a6a7-46fa-bf03-8dd55c611586" (UID: "38e51b01-a6a7-46fa-bf03-8dd55c611586"). InnerVolumeSpecName "kube-api-access-h95q9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:16:37.965859 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:16:37.965789 4611 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38e51b01-a6a7-46fa-bf03-8dd55c611586-maas-api-openshift-ca\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:16:37.965859 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:16:37.965815 4611 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h95q9\" (UniqueName: \"kubernetes.io/projected/38e51b01-a6a7-46fa-bf03-8dd55c611586-kube-api-access-h95q9\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:16:37.965859 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:16:37.965825 4611 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38e51b01-a6a7-46fa-bf03-8dd55c611586-maas-api-xks-ca\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:16:38.651662 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:16:38.651632 4611 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-360dbc5f-9n2dv" Oct 30 21:16:38.651662 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:16:38.651653 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-360dbc5f-9n2dv" event={"ID":"38e51b01-a6a7-46fa-bf03-8dd55c611586","Type":"ContainerDied","Data":"d8106932e88d8ca93626d4271a13d512df4e9cfde945eaaa75826ad251aba3e3"} Oct 30 21:16:38.651873 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:16:38.651701 4611 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d8106932e88d8ca93626d4271a13d512df4e9cfde945eaaa75826ad251aba3e3" Oct 30 21:16:43.311044 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:16:43.310996 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:16:43.452028 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:16:43.451985 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:16:46.888771 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:16:46.888736 4611 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-5db654b954-cw7v6"] Oct 30 21:16:46.889164 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:16:46.889141 4611 cpu_manager.go:414] "containerMap: removing container" podUID="38e51b01-a6a7-46fa-bf03-8dd55c611586" containerName="revoke-keys" Oct 30 21:16:46.889164 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:16:46.889154 4611 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38e51b01-a6a7-46fa-bf03-8dd55c611586" containerName="revoke-keys" Oct 30 21:16:46.889243 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:16:46.889239 4611 memory_manager.go:365] "RemoveStaleState removing state" podUID="38e51b01-a6a7-46fa-bf03-8dd55c611586" containerName="revoke-keys" Oct 30 21:16:46.892354 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:16:46.892336 4611 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5db654b954-cw7v6" Oct 30 21:16:46.898335 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:16:46.898309 4611 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Oct 30 21:16:46.898739 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:16:46.898662 4611 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Oct 30 21:16:46.898739 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:16:46.898707 4611 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-k976f\"" Oct 30 21:16:46.903022 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:16:46.902992 4611 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5db654b954-cw7v6"] Oct 30 21:16:46.926849 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:16:46.926819 4611 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:16:46.927022 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:16:46.926879 4611 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:16:47.046137 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:16:47.046109 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5416f003-ed23-4af7-a278-b68184d0a3d9-openshift-service-ca\") pod \"maas-api-5db654b954-cw7v6\" (UID: \"5416f003-ed23-4af7-a278-b68184d0a3d9\") " pod="odh-ai-gateway-infra/maas-api-5db654b954-cw7v6" Oct 30 21:16:47.046291 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:16:47.046158 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5msbl\" (UniqueName: \"kubernetes.io/projected/5416f003-ed23-4af7-a278-b68184d0a3d9-kube-api-access-5msbl\") pod \"maas-api-5db654b954-cw7v6\" (UID: \"5416f003-ed23-4af7-a278-b68184d0a3d9\") " pod="odh-ai-gateway-infra/maas-api-5db654b954-cw7v6" Oct 30 21:16:47.046334 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:16:47.046285 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5416f003-ed23-4af7-a278-b68184d0a3d9-maas-api-tls\") pod \"maas-api-5db654b954-cw7v6\" (UID: \"5416f003-ed23-4af7-a278-b68184d0a3d9\") " pod="odh-ai-gateway-infra/maas-api-5db654b954-cw7v6" Oct 30 21:16:47.046334 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:16:47.046322 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5416f003-ed23-4af7-a278-b68184d0a3d9-metrics-certs\") pod \"maas-api-5db654b954-cw7v6\" (UID: \"5416f003-ed23-4af7-a278-b68184d0a3d9\") " pod="odh-ai-gateway-infra/maas-api-5db654b954-cw7v6" Oct 30 21:16:47.146996 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:16:47.146927 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5416f003-ed23-4af7-a278-b68184d0a3d9-maas-api-tls\") pod \"maas-api-5db654b954-cw7v6\" (UID: \"5416f003-ed23-4af7-a278-b68184d0a3d9\") " pod="odh-ai-gateway-infra/maas-api-5db654b954-cw7v6" Oct 30 21:16:47.146996 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:16:47.146968 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5416f003-ed23-4af7-a278-b68184d0a3d9-metrics-certs\") pod \"maas-api-5db654b954-cw7v6\" (UID: \"5416f003-ed23-4af7-a278-b68184d0a3d9\") " pod="odh-ai-gateway-infra/maas-api-5db654b954-cw7v6" Oct 30 21:16:47.146996 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:16:47.146990 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5416f003-ed23-4af7-a278-b68184d0a3d9-openshift-service-ca\") pod \"maas-api-5db654b954-cw7v6\" (UID: \"5416f003-ed23-4af7-a278-b68184d0a3d9\") " pod="odh-ai-gateway-infra/maas-api-5db654b954-cw7v6" Oct 30 21:16:47.147251 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:16:47.147032 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5msbl\" (UniqueName: \"kubernetes.io/projected/5416f003-ed23-4af7-a278-b68184d0a3d9-kube-api-access-5msbl\") pod \"maas-api-5db654b954-cw7v6\" (UID: \"5416f003-ed23-4af7-a278-b68184d0a3d9\") " pod="odh-ai-gateway-infra/maas-api-5db654b954-cw7v6" Oct 30 21:16:47.147867 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:16:47.147835 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5416f003-ed23-4af7-a278-b68184d0a3d9-openshift-service-ca\") pod \"maas-api-5db654b954-cw7v6\" (UID: \"5416f003-ed23-4af7-a278-b68184d0a3d9\") " pod="odh-ai-gateway-infra/maas-api-5db654b954-cw7v6" Oct 30 21:16:47.149417 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:16:47.149394 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5416f003-ed23-4af7-a278-b68184d0a3d9-metrics-certs\") pod \"maas-api-5db654b954-cw7v6\" (UID: \"5416f003-ed23-4af7-a278-b68184d0a3d9\") " pod="odh-ai-gateway-infra/maas-api-5db654b954-cw7v6" Oct 30 21:16:47.149546 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:16:47.149529 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5416f003-ed23-4af7-a278-b68184d0a3d9-maas-api-tls\") pod \"maas-api-5db654b954-cw7v6\" (UID: \"5416f003-ed23-4af7-a278-b68184d0a3d9\") " pod="odh-ai-gateway-infra/maas-api-5db654b954-cw7v6" Oct 30 21:16:47.158400 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:16:47.158362 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5msbl\" (UniqueName: \"kubernetes.io/projected/5416f003-ed23-4af7-a278-b68184d0a3d9-kube-api-access-5msbl\") pod \"maas-api-5db654b954-cw7v6\" (UID: \"5416f003-ed23-4af7-a278-b68184d0a3d9\") " pod="odh-ai-gateway-infra/maas-api-5db654b954-cw7v6" Oct 30 21:16:47.210038 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:16:47.210013 4611 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5db654b954-cw7v6" Oct 30 21:16:47.492798 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:16:47.492728 4611 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5db654b954-cw7v6"] Oct 30 21:16:47.496495 ip-10-0-133-192 kubenswrapper[4611]: W1030 21:16:47.496467 4611 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5416f003_ed23_4af7_a278_b68184d0a3d9.slice/crio-9d409f5940ec2754e3ead13a289e2099c5f633ab787e9f53d377aa06339e365c WatchSource:0}: Error finding container 9d409f5940ec2754e3ead13a289e2099c5f633ab787e9f53d377aa06339e365c: Status 404 returned error can't find the container with id 9d409f5940ec2754e3ead13a289e2099c5f633ab787e9f53d377aa06339e365c Oct 30 21:16:47.696267 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:16:47.696228 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5db654b954-cw7v6" event={"ID":"5416f003-ed23-4af7-a278-b68184d0a3d9","Type":"ContainerStarted","Data":"9d409f5940ec2754e3ead13a289e2099c5f633ab787e9f53d377aa06339e365c"} Oct 30 21:16:48.703011 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:16:48.702974 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5db654b954-cw7v6" event={"ID":"5416f003-ed23-4af7-a278-b68184d0a3d9","Type":"ContainerStarted","Data":"a71104c5a93ccf7e6d4ca5310c147496286f2af4462412c48984bdf994f59975"} Oct 30 21:16:48.703412 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:16:48.703041 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-5db654b954-cw7v6" Oct 30 21:16:48.721168 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:16:48.721115 4611 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-5db654b954-cw7v6" podStartSLOduration=2.325176389 podStartE2EDuration="2.721102501s" totalImagesPullingTime="395.926112ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:16:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:16:48.720104023 +0000 UTC m=+2263.254130639" watchObservedRunningTime="2026-10-30 21:16:48.721102501 +0000 UTC m=+2263.255129117" Oct 30 21:16:49.735981 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:16:49.735945 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:16:53.942877 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:16:53.942847 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:16:54.433162 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:16:54.433130 4611 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-c3e6d92c-openshift-default-749f7547c5-g8q6k"] Oct 30 21:16:54.436924 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:16:54.436906 4611 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-c3e6d92c-openshift-default-749f7547c5-g8q6k" Oct 30 21:16:54.445229 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:16:54.445208 4611 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-c3e6d92c-openshift-default-dockercfg-pdqpl\"" Oct 30 21:16:54.453279 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:16:54.453258 4611 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-c3e6d92c-openshift-default-749f7547c5-g8q6k"] Oct 30 21:16:54.555636 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:16:54.555597 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:16:54.613665 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:16:54.613638 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87e039ce-c6f8-40d2-934d-55ef50b088cc-credential-socket\") pod \"e2e-ait-c3e6d92c-openshift-default-749f7547c5-g8q6k\" (UID: \"87e039ce-c6f8-40d2-934d-55ef50b088cc\") " pod="openshift-ingress/e2e-ait-c3e6d92c-openshift-default-749f7547c5-g8q6k" Oct 30 21:16:54.613808 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:16:54.613688 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/87e039ce-c6f8-40d2-934d-55ef50b088cc-istio-token\") pod \"e2e-ait-c3e6d92c-openshift-default-749f7547c5-g8q6k\" (UID: \"87e039ce-c6f8-40d2-934d-55ef50b088cc\") " pod="openshift-ingress/e2e-ait-c3e6d92c-openshift-default-749f7547c5-g8q6k" Oct 30 21:16:54.613874 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:16:54.613799 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2wd75\" (UniqueName: \"kubernetes.io/projected/87e039ce-c6f8-40d2-934d-55ef50b088cc-kube-api-access-2wd75\") pod \"e2e-ait-c3e6d92c-openshift-default-749f7547c5-g8q6k\" (UID: \"87e039ce-c6f8-40d2-934d-55ef50b088cc\") " pod="openshift-ingress/e2e-ait-c3e6d92c-openshift-default-749f7547c5-g8q6k" Oct 30 21:16:54.613929 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:16:54.613889 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87e039ce-c6f8-40d2-934d-55ef50b088cc-workload-socket\") pod \"e2e-ait-c3e6d92c-openshift-default-749f7547c5-g8q6k\" (UID: \"87e039ce-c6f8-40d2-934d-55ef50b088cc\") " pod="openshift-ingress/e2e-ait-c3e6d92c-openshift-default-749f7547c5-g8q6k" Oct 30 21:16:54.613978 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:16:54.613951 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87e039ce-c6f8-40d2-934d-55ef50b088cc-workload-certs\") pod \"e2e-ait-c3e6d92c-openshift-default-749f7547c5-g8q6k\" (UID: \"87e039ce-c6f8-40d2-934d-55ef50b088cc\") " pod="openshift-ingress/e2e-ait-c3e6d92c-openshift-default-749f7547c5-g8q6k" Oct 30 21:16:54.613978 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:16:54.613970 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87e039ce-c6f8-40d2-934d-55ef50b088cc-istio-podinfo\") pod \"e2e-ait-c3e6d92c-openshift-default-749f7547c5-g8q6k\" (UID: \"87e039ce-c6f8-40d2-934d-55ef50b088cc\") " pod="openshift-ingress/e2e-ait-c3e6d92c-openshift-default-749f7547c5-g8q6k" Oct 30 21:16:54.614075 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:16:54.613986 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87e039ce-c6f8-40d2-934d-55ef50b088cc-istio-data\") pod \"e2e-ait-c3e6d92c-openshift-default-749f7547c5-g8q6k\" (UID: \"87e039ce-c6f8-40d2-934d-55ef50b088cc\") " pod="openshift-ingress/e2e-ait-c3e6d92c-openshift-default-749f7547c5-g8q6k" Oct 30 21:16:54.614075 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:16:54.614010 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87e039ce-c6f8-40d2-934d-55ef50b088cc-istio-envoy\") pod \"e2e-ait-c3e6d92c-openshift-default-749f7547c5-g8q6k\" (UID: \"87e039ce-c6f8-40d2-934d-55ef50b088cc\") " pod="openshift-ingress/e2e-ait-c3e6d92c-openshift-default-749f7547c5-g8q6k" Oct 30 21:16:54.614185 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:16:54.614153 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87e039ce-c6f8-40d2-934d-55ef50b088cc-istiod-ca-cert\") pod \"e2e-ait-c3e6d92c-openshift-default-749f7547c5-g8q6k\" (UID: \"87e039ce-c6f8-40d2-934d-55ef50b088cc\") " pod="openshift-ingress/e2e-ait-c3e6d92c-openshift-default-749f7547c5-g8q6k" Oct 30 21:16:54.706336 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:16:54.706260 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:16:54.712743 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:16:54.712722 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-5db654b954-cw7v6" Oct 30 21:16:54.715524 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:16:54.715488 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87e039ce-c6f8-40d2-934d-55ef50b088cc-istiod-ca-cert\") pod \"e2e-ait-c3e6d92c-openshift-default-749f7547c5-g8q6k\" (UID: \"87e039ce-c6f8-40d2-934d-55ef50b088cc\") " pod="openshift-ingress/e2e-ait-c3e6d92c-openshift-default-749f7547c5-g8q6k" Oct 30 21:16:54.715592 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:16:54.715567 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87e039ce-c6f8-40d2-934d-55ef50b088cc-credential-socket\") pod \"e2e-ait-c3e6d92c-openshift-default-749f7547c5-g8q6k\" (UID: \"87e039ce-c6f8-40d2-934d-55ef50b088cc\") " pod="openshift-ingress/e2e-ait-c3e6d92c-openshift-default-749f7547c5-g8q6k" Oct 30 21:16:54.715633 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:16:54.715612 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/87e039ce-c6f8-40d2-934d-55ef50b088cc-istio-token\") pod \"e2e-ait-c3e6d92c-openshift-default-749f7547c5-g8q6k\" (UID: \"87e039ce-c6f8-40d2-934d-55ef50b088cc\") " pod="openshift-ingress/e2e-ait-c3e6d92c-openshift-default-749f7547c5-g8q6k" Oct 30 21:16:54.715692 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:16:54.715656 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2wd75\" (UniqueName: \"kubernetes.io/projected/87e039ce-c6f8-40d2-934d-55ef50b088cc-kube-api-access-2wd75\") pod \"e2e-ait-c3e6d92c-openshift-default-749f7547c5-g8q6k\" (UID: \"87e039ce-c6f8-40d2-934d-55ef50b088cc\") " pod="openshift-ingress/e2e-ait-c3e6d92c-openshift-default-749f7547c5-g8q6k" Oct 30 21:16:54.715727 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:16:54.715706 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87e039ce-c6f8-40d2-934d-55ef50b088cc-workload-socket\") pod \"e2e-ait-c3e6d92c-openshift-default-749f7547c5-g8q6k\" (UID: \"87e039ce-c6f8-40d2-934d-55ef50b088cc\") " pod="openshift-ingress/e2e-ait-c3e6d92c-openshift-default-749f7547c5-g8q6k" Oct 30 21:16:54.715778 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:16:54.715752 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87e039ce-c6f8-40d2-934d-55ef50b088cc-workload-certs\") pod \"e2e-ait-c3e6d92c-openshift-default-749f7547c5-g8q6k\" (UID: \"87e039ce-c6f8-40d2-934d-55ef50b088cc\") " pod="openshift-ingress/e2e-ait-c3e6d92c-openshift-default-749f7547c5-g8q6k" Oct 30 21:16:54.715831 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:16:54.715779 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87e039ce-c6f8-40d2-934d-55ef50b088cc-istio-podinfo\") pod \"e2e-ait-c3e6d92c-openshift-default-749f7547c5-g8q6k\" (UID: \"87e039ce-c6f8-40d2-934d-55ef50b088cc\") " pod="openshift-ingress/e2e-ait-c3e6d92c-openshift-default-749f7547c5-g8q6k" Oct 30 21:16:54.715831 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:16:54.715805 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87e039ce-c6f8-40d2-934d-55ef50b088cc-istio-data\") pod \"e2e-ait-c3e6d92c-openshift-default-749f7547c5-g8q6k\" (UID: \"87e039ce-c6f8-40d2-934d-55ef50b088cc\") " pod="openshift-ingress/e2e-ait-c3e6d92c-openshift-default-749f7547c5-g8q6k" Oct 30 21:16:54.715940 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:16:54.715841 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87e039ce-c6f8-40d2-934d-55ef50b088cc-istio-envoy\") pod \"e2e-ait-c3e6d92c-openshift-default-749f7547c5-g8q6k\" (UID: \"87e039ce-c6f8-40d2-934d-55ef50b088cc\") " pod="openshift-ingress/e2e-ait-c3e6d92c-openshift-default-749f7547c5-g8q6k" Oct 30 21:16:54.716166 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:16:54.716140 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87e039ce-c6f8-40d2-934d-55ef50b088cc-istio-data\") pod \"e2e-ait-c3e6d92c-openshift-default-749f7547c5-g8q6k\" (UID: \"87e039ce-c6f8-40d2-934d-55ef50b088cc\") " pod="openshift-ingress/e2e-ait-c3e6d92c-openshift-default-749f7547c5-g8q6k" Oct 30 21:16:54.716233 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:16:54.716163 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87e039ce-c6f8-40d2-934d-55ef50b088cc-workload-socket\") pod \"e2e-ait-c3e6d92c-openshift-default-749f7547c5-g8q6k\" (UID: \"87e039ce-c6f8-40d2-934d-55ef50b088cc\") " pod="openshift-ingress/e2e-ait-c3e6d92c-openshift-default-749f7547c5-g8q6k" Oct 30 21:16:54.716336 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:16:54.716317 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87e039ce-c6f8-40d2-934d-55ef50b088cc-credential-socket\") pod \"e2e-ait-c3e6d92c-openshift-default-749f7547c5-g8q6k\" (UID: \"87e039ce-c6f8-40d2-934d-55ef50b088cc\") " pod="openshift-ingress/e2e-ait-c3e6d92c-openshift-default-749f7547c5-g8q6k" Oct 30 21:16:54.716395 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:16:54.716335 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87e039ce-c6f8-40d2-934d-55ef50b088cc-workload-certs\") pod \"e2e-ait-c3e6d92c-openshift-default-749f7547c5-g8q6k\" (UID: \"87e039ce-c6f8-40d2-934d-55ef50b088cc\") " pod="openshift-ingress/e2e-ait-c3e6d92c-openshift-default-749f7547c5-g8q6k" Oct 30 21:16:54.716449 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:16:54.716403 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87e039ce-c6f8-40d2-934d-55ef50b088cc-istiod-ca-cert\") pod \"e2e-ait-c3e6d92c-openshift-default-749f7547c5-g8q6k\" (UID: \"87e039ce-c6f8-40d2-934d-55ef50b088cc\") " pod="openshift-ingress/e2e-ait-c3e6d92c-openshift-default-749f7547c5-g8q6k" Oct 30 21:16:54.717952 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:16:54.717929 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87e039ce-c6f8-40d2-934d-55ef50b088cc-istio-envoy\") pod \"e2e-ait-c3e6d92c-openshift-default-749f7547c5-g8q6k\" (UID: \"87e039ce-c6f8-40d2-934d-55ef50b088cc\") " pod="openshift-ingress/e2e-ait-c3e6d92c-openshift-default-749f7547c5-g8q6k" Oct 30 21:16:54.718357 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:16:54.718336 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87e039ce-c6f8-40d2-934d-55ef50b088cc-istio-podinfo\") pod \"e2e-ait-c3e6d92c-openshift-default-749f7547c5-g8q6k\" (UID: \"87e039ce-c6f8-40d2-934d-55ef50b088cc\") " pod="openshift-ingress/e2e-ait-c3e6d92c-openshift-default-749f7547c5-g8q6k" Oct 30 21:16:54.727428 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:16:54.727400 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/87e039ce-c6f8-40d2-934d-55ef50b088cc-istio-token\") pod \"e2e-ait-c3e6d92c-openshift-default-749f7547c5-g8q6k\" (UID: \"87e039ce-c6f8-40d2-934d-55ef50b088cc\") " pod="openshift-ingress/e2e-ait-c3e6d92c-openshift-default-749f7547c5-g8q6k" Oct 30 21:16:54.727735 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:16:54.727718 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2wd75\" (UniqueName: \"kubernetes.io/projected/87e039ce-c6f8-40d2-934d-55ef50b088cc-kube-api-access-2wd75\") pod \"e2e-ait-c3e6d92c-openshift-default-749f7547c5-g8q6k\" (UID: \"87e039ce-c6f8-40d2-934d-55ef50b088cc\") " pod="openshift-ingress/e2e-ait-c3e6d92c-openshift-default-749f7547c5-g8q6k" Oct 30 21:16:54.750114 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:16:54.750088 4611 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-c3e6d92c-openshift-default-749f7547c5-g8q6k" Oct 30 21:16:55.041944 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:16:55.041916 4611 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-c3e6d92c-openshift-default-749f7547c5-g8q6k"] Oct 30 21:16:55.044139 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:16:55.044112 4611 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-6f84c677c8-bn6s4"] Oct 30 21:16:55.044533 ip-10-0-133-192 kubenswrapper[4611]: W1030 21:16:55.044499 4611 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87e039ce_c6f8_40d2_934d_55ef50b088cc.slice/crio-466227d09a2d0f7833060e25dd05175cd490ac0a1d247f640dac99e237fbc24b WatchSource:0}: Error finding container 466227d09a2d0f7833060e25dd05175cd490ac0a1d247f640dac99e237fbc24b: Status 404 returned error can't find the container with id 466227d09a2d0f7833060e25dd05175cd490ac0a1d247f640dac99e237fbc24b Oct 30 21:16:55.046683 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:16:55.046657 4611 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:16:55.046752 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:16:55.046713 4611 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:16:55.046752 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:16:55.046739 4611 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:16:55.049194 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:16:55.049179 4611 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6f84c677c8-bn6s4" Oct 30 21:16:55.056696 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:16:55.056671 4611 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6f84c677c8-bn6s4"] Oct 30 21:16:55.223124 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:16:55.223087 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/05acfee1-1b41-4ee1-a122-4551a125c23e-openshift-service-ca\") pod \"maas-api-6f84c677c8-bn6s4\" (UID: \"05acfee1-1b41-4ee1-a122-4551a125c23e\") " pod="odh-ai-gateway-infra/maas-api-6f84c677c8-bn6s4" Oct 30 21:16:55.223314 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:16:55.223201 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/05acfee1-1b41-4ee1-a122-4551a125c23e-metrics-certs\") pod \"maas-api-6f84c677c8-bn6s4\" (UID: \"05acfee1-1b41-4ee1-a122-4551a125c23e\") " pod="odh-ai-gateway-infra/maas-api-6f84c677c8-bn6s4" Oct 30 21:16:55.223314 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:16:55.223277 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bctwg\" (UniqueName: \"kubernetes.io/projected/05acfee1-1b41-4ee1-a122-4551a125c23e-kube-api-access-bctwg\") pod \"maas-api-6f84c677c8-bn6s4\" (UID: \"05acfee1-1b41-4ee1-a122-4551a125c23e\") " pod="odh-ai-gateway-infra/maas-api-6f84c677c8-bn6s4" Oct 30 21:16:55.223436 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:16:55.223417 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/05acfee1-1b41-4ee1-a122-4551a125c23e-maas-api-tls\") pod \"maas-api-6f84c677c8-bn6s4\" (UID: \"05acfee1-1b41-4ee1-a122-4551a125c23e\") " pod="odh-ai-gateway-infra/maas-api-6f84c677c8-bn6s4" Oct 30 21:16:55.324341 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:16:55.324256 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/05acfee1-1b41-4ee1-a122-4551a125c23e-maas-api-tls\") pod \"maas-api-6f84c677c8-bn6s4\" (UID: \"05acfee1-1b41-4ee1-a122-4551a125c23e\") " pod="odh-ai-gateway-infra/maas-api-6f84c677c8-bn6s4" Oct 30 21:16:55.324477 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:16:55.324346 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/05acfee1-1b41-4ee1-a122-4551a125c23e-openshift-service-ca\") pod \"maas-api-6f84c677c8-bn6s4\" (UID: \"05acfee1-1b41-4ee1-a122-4551a125c23e\") " pod="odh-ai-gateway-infra/maas-api-6f84c677c8-bn6s4" Oct 30 21:16:55.324477 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:16:55.324402 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/05acfee1-1b41-4ee1-a122-4551a125c23e-metrics-certs\") pod \"maas-api-6f84c677c8-bn6s4\" (UID: \"05acfee1-1b41-4ee1-a122-4551a125c23e\") " pod="odh-ai-gateway-infra/maas-api-6f84c677c8-bn6s4" Oct 30 21:16:55.324477 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:16:55.324443 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bctwg\" (UniqueName: \"kubernetes.io/projected/05acfee1-1b41-4ee1-a122-4551a125c23e-kube-api-access-bctwg\") pod \"maas-api-6f84c677c8-bn6s4\" (UID: \"05acfee1-1b41-4ee1-a122-4551a125c23e\") " pod="odh-ai-gateway-infra/maas-api-6f84c677c8-bn6s4" Oct 30 21:16:55.325278 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:16:55.325246 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/05acfee1-1b41-4ee1-a122-4551a125c23e-openshift-service-ca\") pod \"maas-api-6f84c677c8-bn6s4\" (UID: \"05acfee1-1b41-4ee1-a122-4551a125c23e\") " pod="odh-ai-gateway-infra/maas-api-6f84c677c8-bn6s4" Oct 30 21:16:55.326851 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:16:55.326829 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/05acfee1-1b41-4ee1-a122-4551a125c23e-metrics-certs\") pod \"maas-api-6f84c677c8-bn6s4\" (UID: \"05acfee1-1b41-4ee1-a122-4551a125c23e\") " pod="odh-ai-gateway-infra/maas-api-6f84c677c8-bn6s4" Oct 30 21:16:55.326930 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:16:55.326890 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/05acfee1-1b41-4ee1-a122-4551a125c23e-maas-api-tls\") pod \"maas-api-6f84c677c8-bn6s4\" (UID: \"05acfee1-1b41-4ee1-a122-4551a125c23e\") " pod="odh-ai-gateway-infra/maas-api-6f84c677c8-bn6s4" Oct 30 21:16:55.336094 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:16:55.336075 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bctwg\" (UniqueName: \"kubernetes.io/projected/05acfee1-1b41-4ee1-a122-4551a125c23e-kube-api-access-bctwg\") pod \"maas-api-6f84c677c8-bn6s4\" (UID: \"05acfee1-1b41-4ee1-a122-4551a125c23e\") " pod="odh-ai-gateway-infra/maas-api-6f84c677c8-bn6s4" Oct 30 21:16:55.368983 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:16:55.368953 4611 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6f84c677c8-bn6s4" Oct 30 21:16:55.645426 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:16:55.645398 4611 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6f84c677c8-bn6s4"] Oct 30 21:16:55.647287 ip-10-0-133-192 kubenswrapper[4611]: W1030 21:16:55.647262 4611 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05acfee1_1b41_4ee1_a122_4551a125c23e.slice/crio-70d4676a02c89baa45554a7c3ed2d6b2179451100588182982a1fff9214b7aed WatchSource:0}: Error finding container 70d4676a02c89baa45554a7c3ed2d6b2179451100588182982a1fff9214b7aed: Status 404 returned error can't find the container with id 70d4676a02c89baa45554a7c3ed2d6b2179451100588182982a1fff9214b7aed Oct 30 21:16:55.738862 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:16:55.738829 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6f84c677c8-bn6s4" event={"ID":"05acfee1-1b41-4ee1-a122-4551a125c23e","Type":"ContainerStarted","Data":"70d4676a02c89baa45554a7c3ed2d6b2179451100588182982a1fff9214b7aed"} Oct 30 21:16:55.740447 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:16:55.740419 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-c3e6d92c-openshift-default-749f7547c5-g8q6k" event={"ID":"87e039ce-c6f8-40d2-934d-55ef50b088cc","Type":"ContainerStarted","Data":"d0f4ab510a55bc55bd33684e92ad1aae734089d1f9aacd1fe71f6b5db0f08efd"} Oct 30 21:16:55.740574 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:16:55.740468 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-c3e6d92c-openshift-default-749f7547c5-g8q6k" event={"ID":"87e039ce-c6f8-40d2-934d-55ef50b088cc","Type":"ContainerStarted","Data":"466227d09a2d0f7833060e25dd05175cd490ac0a1d247f640dac99e237fbc24b"} Oct 30 21:16:55.750572 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:16:55.750554 4611 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-c3e6d92c-openshift-default-749f7547c5-g8q6k" Oct 30 21:16:55.762373 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:16:55.762328 4611 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-c3e6d92c-openshift-default-749f7547c5-g8q6k" podStartSLOduration=1.762314401 podStartE2EDuration="1.762314401s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:16:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:16:55.761595168 +0000 UTC m=+2270.295621783" watchObservedRunningTime="2026-10-30 21:16:55.762314401 +0000 UTC m=+2270.296341016" Oct 30 21:16:56.746641 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:16:56.746605 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6f84c677c8-bn6s4" event={"ID":"05acfee1-1b41-4ee1-a122-4551a125c23e","Type":"ContainerStarted","Data":"15581150b22c17308b7869fff17540229aef97facaf95f8497c1886c336cfff2"} Oct 30 21:16:56.747047 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:16:56.746921 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-6f84c677c8-bn6s4" Oct 30 21:16:56.755286 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:16:56.755263 4611 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-c3e6d92c-openshift-default-749f7547c5-g8q6k" Oct 30 21:16:56.766889 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:16:56.766844 4611 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-6f84c677c8-bn6s4" podStartSLOduration=1.366151171 podStartE2EDuration="1.766831159s" totalImagesPullingTime="400.679988ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:16:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:16:56.763976822 +0000 UTC m=+2271.298003438" watchObservedRunningTime="2026-10-30 21:16:56.766831159 +0000 UTC m=+2271.300857813" Oct 30 21:16:57.400457 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:16:57.400422 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:16:57.644779 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:16:57.644744 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:16:57.751856 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:16:57.751781 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-c3e6d92c-openshift-default-749f7547c5-g8q6k" Oct 30 21:16:57.752579 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:16:57.752560 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-c3e6d92c-openshift-default-749f7547c5-g8q6k" Oct 30 21:16:58.573219 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:16:58.573165 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:16:58.726245 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:16:58.726191 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:16:58.932723 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:16:58.932678 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:16:59.505648 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:16:59.505611 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:17:00.320841 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:17:00.320803 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:17:00.959301 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:17:00.959260 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:17:01.687824 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:17:01.687787 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:17:02.315968 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:17:02.315928 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:17:02.771670 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:17:02.771637 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-6f84c677c8-bn6s4" Oct 30 21:17:02.822773 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:17:02.822743 4611 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-5db654b954-cw7v6"] Oct 30 21:17:02.823005 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:17:02.822963 4611 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-5db654b954-cw7v6" podUID="5416f003-ed23-4af7-a278-b68184d0a3d9" containerName="maas-api" containerID="cri-o://a71104c5a93ccf7e6d4ca5310c147496286f2af4462412c48984bdf994f59975" gracePeriod=30 Oct 30 21:17:02.991499 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:17:02.991433 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:17:03.127341 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:17:03.127306 4611 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5db654b954-cw7v6" Oct 30 21:17:03.211832 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:17:03.211797 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5416f003-ed23-4af7-a278-b68184d0a3d9-kube-api-access-5msbl\" (UniqueName: \"kubernetes.io/projected/5416f003-ed23-4af7-a278-b68184d0a3d9-kube-api-access-5msbl\") pod \"5416f003-ed23-4af7-a278-b68184d0a3d9\" (UID: \"5416f003-ed23-4af7-a278-b68184d0a3d9\") " Oct 30 21:17:03.212021 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:17:03.211850 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5416f003-ed23-4af7-a278-b68184d0a3d9-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5416f003-ed23-4af7-a278-b68184d0a3d9-maas-api-tls\") pod \"5416f003-ed23-4af7-a278-b68184d0a3d9\" (UID: \"5416f003-ed23-4af7-a278-b68184d0a3d9\") " Oct 30 21:17:03.212021 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:17:03.211887 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5416f003-ed23-4af7-a278-b68184d0a3d9-metrics-certs\" (UniqueName: \"kubernetes.io/secret/5416f003-ed23-4af7-a278-b68184d0a3d9-metrics-certs\") pod \"5416f003-ed23-4af7-a278-b68184d0a3d9\" (UID: \"5416f003-ed23-4af7-a278-b68184d0a3d9\") " Oct 30 21:17:03.212021 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:17:03.211944 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5416f003-ed23-4af7-a278-b68184d0a3d9-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5416f003-ed23-4af7-a278-b68184d0a3d9-openshift-service-ca\") pod \"5416f003-ed23-4af7-a278-b68184d0a3d9\" (UID: \"5416f003-ed23-4af7-a278-b68184d0a3d9\") " Oct 30 21:17:03.212362 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:17:03.212336 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5416f003-ed23-4af7-a278-b68184d0a3d9-openshift-service-ca" pod "5416f003-ed23-4af7-a278-b68184d0a3d9" (UID: "5416f003-ed23-4af7-a278-b68184d0a3d9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:17:03.214086 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:17:03.214055 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5416f003-ed23-4af7-a278-b68184d0a3d9-metrics-certs" pod "5416f003-ed23-4af7-a278-b68184d0a3d9" (UID: "5416f003-ed23-4af7-a278-b68184d0a3d9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:17:03.214086 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:17:03.214074 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5416f003-ed23-4af7-a278-b68184d0a3d9-maas-api-tls" pod "5416f003-ed23-4af7-a278-b68184d0a3d9" (UID: "5416f003-ed23-4af7-a278-b68184d0a3d9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:17:03.214222 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:17:03.214081 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5416f003-ed23-4af7-a278-b68184d0a3d9-kube-api-access-5msbl" pod "5416f003-ed23-4af7-a278-b68184d0a3d9" (UID: "5416f003-ed23-4af7-a278-b68184d0a3d9"). InnerVolumeSpecName "kube-api-access-5msbl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:17:03.312922 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:17:03.312891 4611 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5msbl\" (UniqueName: \"kubernetes.io/projected/5416f003-ed23-4af7-a278-b68184d0a3d9-kube-api-access-5msbl\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:17:03.312922 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:17:03.312921 4611 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5416f003-ed23-4af7-a278-b68184d0a3d9-maas-api-tls\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:17:03.313130 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:17:03.312930 4611 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5416f003-ed23-4af7-a278-b68184d0a3d9-metrics-certs\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:17:03.313130 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:17:03.312940 4611 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5416f003-ed23-4af7-a278-b68184d0a3d9-openshift-service-ca\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:17:03.681135 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:17:03.681094 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:17:03.823819 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:17:03.823782 4611 generic.go:356] "Generic (PLEG): container finished" podID="5416f003-ed23-4af7-a278-b68184d0a3d9" containerID="a71104c5a93ccf7e6d4ca5310c147496286f2af4462412c48984bdf994f59975" exitCode=0 Oct 30 21:17:03.824207 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:17:03.823833 4611 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5db654b954-cw7v6" Oct 30 21:17:03.824207 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:17:03.823871 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5db654b954-cw7v6" event={"ID":"5416f003-ed23-4af7-a278-b68184d0a3d9","Type":"ContainerDied","Data":"a71104c5a93ccf7e6d4ca5310c147496286f2af4462412c48984bdf994f59975"} Oct 30 21:17:03.824207 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:17:03.823929 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5db654b954-cw7v6" event={"ID":"5416f003-ed23-4af7-a278-b68184d0a3d9","Type":"ContainerDied","Data":"9d409f5940ec2754e3ead13a289e2099c5f633ab787e9f53d377aa06339e365c"} Oct 30 21:17:03.824207 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:17:03.823953 4611 scope.go:122] "RemoveContainer" containerID="a71104c5a93ccf7e6d4ca5310c147496286f2af4462412c48984bdf994f59975" Oct 30 21:17:03.833358 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:17:03.833338 4611 scope.go:122] "RemoveContainer" containerID="a71104c5a93ccf7e6d4ca5310c147496286f2af4462412c48984bdf994f59975" Oct 30 21:17:03.833624 ip-10-0-133-192 kubenswrapper[4611]: E1030 21:17:03.833604 4611 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a71104c5a93ccf7e6d4ca5310c147496286f2af4462412c48984bdf994f59975\": container with ID starting with a71104c5a93ccf7e6d4ca5310c147496286f2af4462412c48984bdf994f59975 not found: ID does not exist" containerID="a71104c5a93ccf7e6d4ca5310c147496286f2af4462412c48984bdf994f59975" Oct 30 21:17:03.833669 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:17:03.833633 4611 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a71104c5a93ccf7e6d4ca5310c147496286f2af4462412c48984bdf994f59975"} err="failed to get container status \"a71104c5a93ccf7e6d4ca5310c147496286f2af4462412c48984bdf994f59975\": rpc error: code = NotFound desc = could not find container \"a71104c5a93ccf7e6d4ca5310c147496286f2af4462412c48984bdf994f59975\": container with ID starting with a71104c5a93ccf7e6d4ca5310c147496286f2af4462412c48984bdf994f59975 not found: ID does not exist" Oct 30 21:17:03.840484 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:17:03.840461 4611 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-5db654b954-cw7v6"] Oct 30 21:17:03.842982 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:17:03.842960 4611 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-5db654b954-cw7v6"] Oct 30 21:17:03.916429 ip-10-0-133-192 kubenswrapper[4611]: E1030 21:17:03.916344 4611 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hwzc6" containerName="extractor" Oct 30 21:17:04.312363 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:17:04.312261 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:17:04.896272 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:17:04.896238 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:17:05.597274 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:17:05.597236 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:17:05.606018 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:17:05.605987 4611 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5416f003-ed23-4af7-a278-b68184d0a3d9" path="/var/lib/kubelet/pods/5416f003-ed23-4af7-a278-b68184d0a3d9/volumes" Oct 30 21:17:06.200253 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:17:06.200196 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:17:06.877781 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:17:06.877727 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:17:07.657346 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:17:07.657301 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:17:08.401061 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:17:08.400965 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:17:08.696437 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:17:08.696357 4611 scope.go:122] "RemoveContainer" containerID="da39c9b636ce8f54cc47aae80579cc940e02e1f2317015983c484e26eeb8df9b" Oct 30 21:17:08.705204 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:17:08.705183 4611 scope.go:122] "RemoveContainer" containerID="16fc47839a8a57922cb8ba073a2519624217f1461bdcc7ff91233fd52f6037cc" Oct 30 21:17:09.106223 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:17:09.106187 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:17:09.802984 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:17:09.802950 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:17:10.450166 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:17:10.450079 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:17:10.862136 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:17:10.862104 4611 generic.go:356] "Generic (PLEG): container finished" podID="87e039ce-c6f8-40d2-934d-55ef50b088cc" containerID="d0f4ab510a55bc55bd33684e92ad1aae734089d1f9aacd1fe71f6b5db0f08efd" exitCode=0 Oct 30 21:17:10.862550 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:17:10.862147 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-c3e6d92c-openshift-default-749f7547c5-g8q6k" event={"ID":"87e039ce-c6f8-40d2-934d-55ef50b088cc","Type":"ContainerDied","Data":"d0f4ab510a55bc55bd33684e92ad1aae734089d1f9aacd1fe71f6b5db0f08efd"} Oct 30 21:17:10.862550 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:17:10.862537 4611 scope.go:122] "RemoveContainer" containerID="d0f4ab510a55bc55bd33684e92ad1aae734089d1f9aacd1fe71f6b5db0f08efd" Oct 30 21:17:10.863649 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:17:10.863621 4611 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:17:10.863712 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:17:10.863694 4611 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:17:10.863752 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:17:10.863727 4611 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:17:11.112893 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:17:11.112655 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:17:11.783259 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:17:11.783223 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:17:11.876489 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:17:11.876455 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-c3e6d92c-openshift-default-749f7547c5-g8q6k" event={"ID":"87e039ce-c6f8-40d2-934d-55ef50b088cc","Type":"ContainerStarted","Data":"3e393dc1d1f183ecb986c96ee648854adb4860ebef09f7b4e269a1ac65eedea0"} Oct 30 21:17:12.393437 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:17:12.393402 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:17:12.759586 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:17:12.752886 4611 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-c3e6d92c-openshift-default-749f7547c5-g8q6k" Oct 30 21:17:12.759586 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:17:12.753003 4611 patch_prober.go:29] interesting pod/e2e-ait-c3e6d92c-openshift-default-749f7547c5-g8q6k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" start-of-body= Oct 30 21:17:12.759586 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:17:12.753047 4611 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-c3e6d92c-openshift-default-749f7547c5-g8q6k" podUID="87e039ce-c6f8-40d2-934d-55ef50b088cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" Oct 30 21:17:12.759586 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:17:12.753310 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-c3e6d92c-openshift-default-749f7547c5-g8q6k" Oct 30 21:17:13.100902 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:17:13.100875 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:17:13.765532 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:17:13.764470 4611 patch_prober.go:29] interesting pod/e2e-ait-c3e6d92c-openshift-default-749f7547c5-g8q6k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" start-of-body= Oct 30 21:17:13.765532 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:17:13.764550 4611 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-c3e6d92c-openshift-default-749f7547c5-g8q6k" podUID="87e039ce-c6f8-40d2-934d-55ef50b088cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" Oct 30 21:17:13.877088 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:17:13.877054 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:17:14.613022 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:17:14.612988 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:17:14.764531 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:17:14.758867 4611 patch_prober.go:29] interesting pod/e2e-ait-c3e6d92c-openshift-default-749f7547c5-g8q6k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" start-of-body= Oct 30 21:17:14.764531 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:17:14.758924 4611 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-c3e6d92c-openshift-default-749f7547c5-g8q6k" podUID="87e039ce-c6f8-40d2-934d-55ef50b088cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" Oct 30 21:17:15.294134 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:17:15.294095 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:17:15.759533 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:17:15.752819 4611 patch_prober.go:29] interesting pod/e2e-ait-c3e6d92c-openshift-default-749f7547c5-g8q6k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" start-of-body= Oct 30 21:17:15.759533 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:17:15.752878 4611 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-c3e6d92c-openshift-default-749f7547c5-g8q6k" podUID="87e039ce-c6f8-40d2-934d-55ef50b088cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" Oct 30 21:17:15.891380 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:17:15.891339 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:17:16.601875 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:17:16.601839 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:17:16.751315 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:17:16.751281 4611 patch_prober.go:29] interesting pod/e2e-ait-c3e6d92c-openshift-default-749f7547c5-g8q6k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" start-of-body= Oct 30 21:17:16.751547 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:17:16.751335 4611 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-c3e6d92c-openshift-default-749f7547c5-g8q6k" podUID="87e039ce-c6f8-40d2-934d-55ef50b088cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" Oct 30 21:17:17.268420 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:17:17.268384 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:17:17.754457 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:17:17.754426 4611 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-c3e6d92c-openshift-default-749f7547c5-g8q6k" Oct 30 21:17:17.755583 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:17:17.755557 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-c3e6d92c-openshift-default-749f7547c5-g8q6k" Oct 30 21:17:17.902068 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:17:17.902031 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:17:18.754060 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:17:18.754028 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:17:19.393730 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:17:19.393697 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:17:20.021676 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:17:20.021639 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:17:20.625166 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:17:20.625125 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:17:21.288582 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:17:21.288545 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:17:21.901848 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:17:21.901809 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:17:22.590081 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:17:22.590010 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:17:23.200547 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:17:23.200486 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:17:23.490597 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:17:23.490499 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:17:23.639898 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:17:23.639861 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:17:23.885977 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:17:23.885923 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:17:24.411263 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:17:24.411228 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:17:24.549967 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:17:24.549932 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:17:24.696198 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:17:24.696121 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:17:27.021775 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:17:27.021738 4611 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-5pb4b"] Oct 30 21:17:27.024002 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:17:27.023978 4611 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-5pb4b"] Oct 30 21:17:27.604405 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:17:27.604374 4611 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="883ae4b7-4e14-4ad9-9d69-944966b15e98" path="/var/lib/kubelet/pods/883ae4b7-4e14-4ad9-9d69-944966b15e98/volumes" Oct 30 21:17:29.658616 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:17:29.658575 4611 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-c3e6d92c-openshift-default-749f7547c5-g8q6k"] Oct 30 21:17:29.659146 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:17:29.658824 4611 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-c3e6d92c-openshift-default-749f7547c5-g8q6k" podUID="87e039ce-c6f8-40d2-934d-55ef50b088cc" containerName="istio-proxy" containerID="cri-o://3e393dc1d1f183ecb986c96ee648854adb4860ebef09f7b4e269a1ac65eedea0" gracePeriod=30 Oct 30 21:17:29.734287 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:17:29.734254 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:17:32.756766 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:17:32.756716 4611 patch_prober.go:29] interesting pod/e2e-ait-c3e6d92c-openshift-default-749f7547c5-g8q6k container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 21:17:32.757144 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:17:32.756789 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-c3e6d92c-openshift-default-749f7547c5-g8q6k" podUID="87e039ce-c6f8-40d2-934d-55ef50b088cc" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 21:17:33.916793 ip-10-0-133-192 kubenswrapper[4611]: E1030 21:17:33.916755 4611 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hwzc6" containerName="extractor" Oct 30 21:17:34.904563 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:17:34.904535 4611 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-c3e6d92c-openshift-default-749f7547c5-g8q6k" Oct 30 21:17:34.996467 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:17:34.996383 4611 generic.go:356] "Generic (PLEG): container finished" podID="87e039ce-c6f8-40d2-934d-55ef50b088cc" containerID="3e393dc1d1f183ecb986c96ee648854adb4860ebef09f7b4e269a1ac65eedea0" exitCode=0 Oct 30 21:17:34.996467 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:17:34.996422 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-c3e6d92c-openshift-default-749f7547c5-g8q6k" event={"ID":"87e039ce-c6f8-40d2-934d-55ef50b088cc","Type":"ContainerDied","Data":"3e393dc1d1f183ecb986c96ee648854adb4860ebef09f7b4e269a1ac65eedea0"} Oct 30 21:17:34.996467 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:17:34.996441 4611 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-c3e6d92c-openshift-default-749f7547c5-g8q6k" Oct 30 21:17:34.996960 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:17:34.996495 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-c3e6d92c-openshift-default-749f7547c5-g8q6k" event={"ID":"87e039ce-c6f8-40d2-934d-55ef50b088cc","Type":"ContainerDied","Data":"466227d09a2d0f7833060e25dd05175cd490ac0a1d247f640dac99e237fbc24b"} Oct 30 21:17:34.996960 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:17:34.996537 4611 scope.go:122] "RemoveContainer" containerID="3e393dc1d1f183ecb986c96ee648854adb4860ebef09f7b4e269a1ac65eedea0" Oct 30 21:17:35.005320 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:17:35.005304 4611 scope.go:122] "RemoveContainer" containerID="d0f4ab510a55bc55bd33684e92ad1aae734089d1f9aacd1fe71f6b5db0f08efd" Oct 30 21:17:35.013119 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:17:35.013101 4611 scope.go:122] "RemoveContainer" containerID="3e393dc1d1f183ecb986c96ee648854adb4860ebef09f7b4e269a1ac65eedea0" Oct 30 21:17:35.013392 ip-10-0-133-192 kubenswrapper[4611]: E1030 21:17:35.013368 4611 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e393dc1d1f183ecb986c96ee648854adb4860ebef09f7b4e269a1ac65eedea0\": container with ID starting with 3e393dc1d1f183ecb986c96ee648854adb4860ebef09f7b4e269a1ac65eedea0 not found: ID does not exist" containerID="3e393dc1d1f183ecb986c96ee648854adb4860ebef09f7b4e269a1ac65eedea0" Oct 30 21:17:35.013474 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:17:35.013404 4611 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e393dc1d1f183ecb986c96ee648854adb4860ebef09f7b4e269a1ac65eedea0"} err="failed to get container status \"3e393dc1d1f183ecb986c96ee648854adb4860ebef09f7b4e269a1ac65eedea0\": rpc error: code = NotFound desc = could not find container \"3e393dc1d1f183ecb986c96ee648854adb4860ebef09f7b4e269a1ac65eedea0\": container with ID starting with 3e393dc1d1f183ecb986c96ee648854adb4860ebef09f7b4e269a1ac65eedea0 not found: ID does not exist" Oct 30 21:17:35.013474 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:17:35.013426 4611 scope.go:122] "RemoveContainer" containerID="d0f4ab510a55bc55bd33684e92ad1aae734089d1f9aacd1fe71f6b5db0f08efd" Oct 30 21:17:35.013735 ip-10-0-133-192 kubenswrapper[4611]: E1030 21:17:35.013716 4611 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d0f4ab510a55bc55bd33684e92ad1aae734089d1f9aacd1fe71f6b5db0f08efd\": container with ID starting with d0f4ab510a55bc55bd33684e92ad1aae734089d1f9aacd1fe71f6b5db0f08efd not found: ID does not exist" containerID="d0f4ab510a55bc55bd33684e92ad1aae734089d1f9aacd1fe71f6b5db0f08efd" Oct 30 21:17:35.013788 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:17:35.013741 4611 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d0f4ab510a55bc55bd33684e92ad1aae734089d1f9aacd1fe71f6b5db0f08efd"} err="failed to get container status \"d0f4ab510a55bc55bd33684e92ad1aae734089d1f9aacd1fe71f6b5db0f08efd\": rpc error: code = NotFound desc = could not find container \"d0f4ab510a55bc55bd33684e92ad1aae734089d1f9aacd1fe71f6b5db0f08efd\": container with ID starting with d0f4ab510a55bc55bd33684e92ad1aae734089d1f9aacd1fe71f6b5db0f08efd not found: ID does not exist" Oct 30 21:17:35.050372 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:17:35.050335 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87e039ce-c6f8-40d2-934d-55ef50b088cc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87e039ce-c6f8-40d2-934d-55ef50b088cc-istio-envoy\") pod \"87e039ce-c6f8-40d2-934d-55ef50b088cc\" (UID: \"87e039ce-c6f8-40d2-934d-55ef50b088cc\") " Oct 30 21:17:35.050534 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:17:35.050396 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/87e039ce-c6f8-40d2-934d-55ef50b088cc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87e039ce-c6f8-40d2-934d-55ef50b088cc-istiod-ca-cert\") pod \"87e039ce-c6f8-40d2-934d-55ef50b088cc\" (UID: \"87e039ce-c6f8-40d2-934d-55ef50b088cc\") " Oct 30 21:17:35.050534 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:17:35.050429 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87e039ce-c6f8-40d2-934d-55ef50b088cc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87e039ce-c6f8-40d2-934d-55ef50b088cc-istio-data\") pod \"87e039ce-c6f8-40d2-934d-55ef50b088cc\" (UID: \"87e039ce-c6f8-40d2-934d-55ef50b088cc\") " Oct 30 21:17:35.050534 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:17:35.050481 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87e039ce-c6f8-40d2-934d-55ef50b088cc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87e039ce-c6f8-40d2-934d-55ef50b088cc-workload-socket\") pod \"87e039ce-c6f8-40d2-934d-55ef50b088cc\" (UID: \"87e039ce-c6f8-40d2-934d-55ef50b088cc\") " Oct 30 21:17:35.050722 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:17:35.050538 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87e039ce-c6f8-40d2-934d-55ef50b088cc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87e039ce-c6f8-40d2-934d-55ef50b088cc-credential-socket\") pod \"87e039ce-c6f8-40d2-934d-55ef50b088cc\" (UID: \"87e039ce-c6f8-40d2-934d-55ef50b088cc\") " Oct 30 21:17:35.050722 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:17:35.050591 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/87e039ce-c6f8-40d2-934d-55ef50b088cc-kube-api-access-2wd75\" (UniqueName: \"kubernetes.io/projected/87e039ce-c6f8-40d2-934d-55ef50b088cc-kube-api-access-2wd75\") pod \"87e039ce-c6f8-40d2-934d-55ef50b088cc\" (UID: \"87e039ce-c6f8-40d2-934d-55ef50b088cc\") " Oct 30 21:17:35.050722 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:17:35.050646 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/87e039ce-c6f8-40d2-934d-55ef50b088cc-istio-token\" (UniqueName: \"kubernetes.io/projected/87e039ce-c6f8-40d2-934d-55ef50b088cc-istio-token\") pod \"87e039ce-c6f8-40d2-934d-55ef50b088cc\" (UID: \"87e039ce-c6f8-40d2-934d-55ef50b088cc\") " Oct 30 21:17:35.050722 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:17:35.050698 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87e039ce-c6f8-40d2-934d-55ef50b088cc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87e039ce-c6f8-40d2-934d-55ef50b088cc-workload-certs\") pod \"87e039ce-c6f8-40d2-934d-55ef50b088cc\" (UID: \"87e039ce-c6f8-40d2-934d-55ef50b088cc\") " Oct 30 21:17:35.050939 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:17:35.050730 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/87e039ce-c6f8-40d2-934d-55ef50b088cc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87e039ce-c6f8-40d2-934d-55ef50b088cc-istio-podinfo\") pod \"87e039ce-c6f8-40d2-934d-55ef50b088cc\" (UID: \"87e039ce-c6f8-40d2-934d-55ef50b088cc\") " Oct 30 21:17:35.050939 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:17:35.050760 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87e039ce-c6f8-40d2-934d-55ef50b088cc-istio-data" pod "87e039ce-c6f8-40d2-934d-55ef50b088cc" (UID: "87e039ce-c6f8-40d2-934d-55ef50b088cc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:17:35.050939 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:17:35.050779 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87e039ce-c6f8-40d2-934d-55ef50b088cc-workload-socket" pod "87e039ce-c6f8-40d2-934d-55ef50b088cc" (UID: "87e039ce-c6f8-40d2-934d-55ef50b088cc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:17:35.050939 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:17:35.050855 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87e039ce-c6f8-40d2-934d-55ef50b088cc-credential-socket" pod "87e039ce-c6f8-40d2-934d-55ef50b088cc" (UID: "87e039ce-c6f8-40d2-934d-55ef50b088cc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:17:35.050939 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:17:35.050868 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87e039ce-c6f8-40d2-934d-55ef50b088cc-istiod-ca-cert" pod "87e039ce-c6f8-40d2-934d-55ef50b088cc" (UID: "87e039ce-c6f8-40d2-934d-55ef50b088cc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:17:35.051205 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:17:35.051035 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87e039ce-c6f8-40d2-934d-55ef50b088cc-workload-certs" pod "87e039ce-c6f8-40d2-934d-55ef50b088cc" (UID: "87e039ce-c6f8-40d2-934d-55ef50b088cc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:17:35.051205 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:17:35.051165 4611 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87e039ce-c6f8-40d2-934d-55ef50b088cc-istio-data\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:17:35.051205 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:17:35.051184 4611 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87e039ce-c6f8-40d2-934d-55ef50b088cc-workload-socket\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:17:35.051205 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:17:35.051199 4611 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87e039ce-c6f8-40d2-934d-55ef50b088cc-credential-socket\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:17:35.051370 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:17:35.051212 4611 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87e039ce-c6f8-40d2-934d-55ef50b088cc-workload-certs\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:17:35.051370 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:17:35.051226 4611 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87e039ce-c6f8-40d2-934d-55ef50b088cc-istiod-ca-cert\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:17:35.052766 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:17:35.052741 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87e039ce-c6f8-40d2-934d-55ef50b088cc-istio-token" pod "87e039ce-c6f8-40d2-934d-55ef50b088cc" (UID: "87e039ce-c6f8-40d2-934d-55ef50b088cc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:17:35.052766 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:17:35.052736 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87e039ce-c6f8-40d2-934d-55ef50b088cc-kube-api-access-2wd75" pod "87e039ce-c6f8-40d2-934d-55ef50b088cc" (UID: "87e039ce-c6f8-40d2-934d-55ef50b088cc"). InnerVolumeSpecName "kube-api-access-2wd75". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:17:35.052921 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:17:35.052908 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87e039ce-c6f8-40d2-934d-55ef50b088cc-istio-envoy" pod "87e039ce-c6f8-40d2-934d-55ef50b088cc" (UID: "87e039ce-c6f8-40d2-934d-55ef50b088cc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:17:35.053110 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:17:35.053091 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/87e039ce-c6f8-40d2-934d-55ef50b088cc-istio-podinfo" pod "87e039ce-c6f8-40d2-934d-55ef50b088cc" (UID: "87e039ce-c6f8-40d2-934d-55ef50b088cc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:17:35.152575 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:17:35.152550 4611 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87e039ce-c6f8-40d2-934d-55ef50b088cc-istio-envoy\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:17:35.152575 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:17:35.152573 4611 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2wd75\" (UniqueName: \"kubernetes.io/projected/87e039ce-c6f8-40d2-934d-55ef50b088cc-kube-api-access-2wd75\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:17:35.152712 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:17:35.152583 4611 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/87e039ce-c6f8-40d2-934d-55ef50b088cc-istio-token\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:17:35.152712 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:17:35.152592 4611 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87e039ce-c6f8-40d2-934d-55ef50b088cc-istio-podinfo\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:17:35.319591 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:17:35.319562 4611 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-c3e6d92c-openshift-default-749f7547c5-g8q6k"] Oct 30 21:17:35.327719 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:17:35.327685 4611 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-c3e6d92c-openshift-default-749f7547c5-g8q6k"] Oct 30 21:17:35.605441 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:17:35.605403 4611 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="87e039ce-c6f8-40d2-934d-55ef50b088cc" path="/var/lib/kubelet/pods/87e039ce-c6f8-40d2-934d-55ef50b088cc/volumes" Oct 30 21:17:41.104381 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:17:41.104346 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:17:41.446399 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:17:41.446328 4611 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-authcfg-http-dd6b21-openshift-default-6b78fd9989-vlz98"] Oct 30 21:17:41.446776 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:17:41.446759 4611 cpu_manager.go:414] "containerMap: removing container" podUID="5416f003-ed23-4af7-a278-b68184d0a3d9" containerName="maas-api" Oct 30 21:17:41.446825 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:17:41.446778 4611 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5416f003-ed23-4af7-a278-b68184d0a3d9" containerName="maas-api" Oct 30 21:17:41.446825 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:17:41.446797 4611 cpu_manager.go:414] "containerMap: removing container" podUID="87e039ce-c6f8-40d2-934d-55ef50b088cc" containerName="istio-proxy" Oct 30 21:17:41.446825 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:17:41.446803 4611 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="87e039ce-c6f8-40d2-934d-55ef50b088cc" containerName="istio-proxy" Oct 30 21:17:41.446923 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:17:41.446831 4611 cpu_manager.go:414] "containerMap: removing container" podUID="87e039ce-c6f8-40d2-934d-55ef50b088cc" containerName="istio-proxy" Oct 30 21:17:41.446923 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:17:41.446837 4611 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="87e039ce-c6f8-40d2-934d-55ef50b088cc" containerName="istio-proxy" Oct 30 21:17:41.446923 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:17:41.446896 4611 memory_manager.go:365] "RemoveStaleState removing state" podUID="87e039ce-c6f8-40d2-934d-55ef50b088cc" containerName="istio-proxy" Oct 30 21:17:41.446923 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:17:41.446905 4611 memory_manager.go:365] "RemoveStaleState removing state" podUID="5416f003-ed23-4af7-a278-b68184d0a3d9" containerName="maas-api" Oct 30 21:17:41.451675 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:17:41.451649 4611 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-authcfg-http-dd6b21-openshift-default-6b78fd9989-vlz98" Oct 30 21:17:41.458296 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:17:41.458271 4611 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-authcfg-http-dd6b21-openshift-default-dockercfg-xlsp4\"" Oct 30 21:17:41.465149 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:17:41.465125 4611 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-authcfg-http-dd6b21-openshift-default-6b78fd9989-vlz98"] Oct 30 21:17:41.519208 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:17:41.519181 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35be0c3e-0dce-47c2-8bb8-d0d82d75ddb3-workload-socket\") pod \"e2e-authcfg-http-dd6b21-openshift-default-6b78fd9989-vlz98\" (UID: \"35be0c3e-0dce-47c2-8bb8-d0d82d75ddb3\") " pod="openshift-ingress/e2e-authcfg-http-dd6b21-openshift-default-6b78fd9989-vlz98" Oct 30 21:17:41.519326 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:17:41.519216 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35be0c3e-0dce-47c2-8bb8-d0d82d75ddb3-istiod-ca-cert\") pod \"e2e-authcfg-http-dd6b21-openshift-default-6b78fd9989-vlz98\" (UID: \"35be0c3e-0dce-47c2-8bb8-d0d82d75ddb3\") " pod="openshift-ingress/e2e-authcfg-http-dd6b21-openshift-default-6b78fd9989-vlz98" Oct 30 21:17:41.519326 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:17:41.519232 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pmjwt\" (UniqueName: \"kubernetes.io/projected/35be0c3e-0dce-47c2-8bb8-d0d82d75ddb3-kube-api-access-pmjwt\") pod \"e2e-authcfg-http-dd6b21-openshift-default-6b78fd9989-vlz98\" (UID: \"35be0c3e-0dce-47c2-8bb8-d0d82d75ddb3\") " pod="openshift-ingress/e2e-authcfg-http-dd6b21-openshift-default-6b78fd9989-vlz98" Oct 30 21:17:41.519326 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:17:41.519295 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35be0c3e-0dce-47c2-8bb8-d0d82d75ddb3-istio-podinfo\") pod \"e2e-authcfg-http-dd6b21-openshift-default-6b78fd9989-vlz98\" (UID: \"35be0c3e-0dce-47c2-8bb8-d0d82d75ddb3\") " pod="openshift-ingress/e2e-authcfg-http-dd6b21-openshift-default-6b78fd9989-vlz98" Oct 30 21:17:41.519435 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:17:41.519363 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35be0c3e-0dce-47c2-8bb8-d0d82d75ddb3-credential-socket\") pod \"e2e-authcfg-http-dd6b21-openshift-default-6b78fd9989-vlz98\" (UID: \"35be0c3e-0dce-47c2-8bb8-d0d82d75ddb3\") " pod="openshift-ingress/e2e-authcfg-http-dd6b21-openshift-default-6b78fd9989-vlz98" Oct 30 21:17:41.519481 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:17:41.519463 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35be0c3e-0dce-47c2-8bb8-d0d82d75ddb3-istio-envoy\") pod \"e2e-authcfg-http-dd6b21-openshift-default-6b78fd9989-vlz98\" (UID: \"35be0c3e-0dce-47c2-8bb8-d0d82d75ddb3\") " pod="openshift-ingress/e2e-authcfg-http-dd6b21-openshift-default-6b78fd9989-vlz98" Oct 30 21:17:41.519553 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:17:41.519496 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35be0c3e-0dce-47c2-8bb8-d0d82d75ddb3-istio-token\") pod \"e2e-authcfg-http-dd6b21-openshift-default-6b78fd9989-vlz98\" (UID: \"35be0c3e-0dce-47c2-8bb8-d0d82d75ddb3\") " pod="openshift-ingress/e2e-authcfg-http-dd6b21-openshift-default-6b78fd9989-vlz98" Oct 30 21:17:41.519594 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:17:41.519561 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35be0c3e-0dce-47c2-8bb8-d0d82d75ddb3-istio-data\") pod \"e2e-authcfg-http-dd6b21-openshift-default-6b78fd9989-vlz98\" (UID: \"35be0c3e-0dce-47c2-8bb8-d0d82d75ddb3\") " pod="openshift-ingress/e2e-authcfg-http-dd6b21-openshift-default-6b78fd9989-vlz98" Oct 30 21:17:41.519645 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:17:41.519633 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35be0c3e-0dce-47c2-8bb8-d0d82d75ddb3-workload-certs\") pod \"e2e-authcfg-http-dd6b21-openshift-default-6b78fd9989-vlz98\" (UID: \"35be0c3e-0dce-47c2-8bb8-d0d82d75ddb3\") " pod="openshift-ingress/e2e-authcfg-http-dd6b21-openshift-default-6b78fd9989-vlz98" Oct 30 21:17:41.548090 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:17:41.548058 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:17:41.620708 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:17:41.620676 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35be0c3e-0dce-47c2-8bb8-d0d82d75ddb3-istio-envoy\") pod \"e2e-authcfg-http-dd6b21-openshift-default-6b78fd9989-vlz98\" (UID: \"35be0c3e-0dce-47c2-8bb8-d0d82d75ddb3\") " pod="openshift-ingress/e2e-authcfg-http-dd6b21-openshift-default-6b78fd9989-vlz98" Oct 30 21:17:41.620708 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:17:41.620710 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35be0c3e-0dce-47c2-8bb8-d0d82d75ddb3-istio-token\") pod \"e2e-authcfg-http-dd6b21-openshift-default-6b78fd9989-vlz98\" (UID: \"35be0c3e-0dce-47c2-8bb8-d0d82d75ddb3\") " pod="openshift-ingress/e2e-authcfg-http-dd6b21-openshift-default-6b78fd9989-vlz98" Oct 30 21:17:41.620981 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:17:41.620733 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35be0c3e-0dce-47c2-8bb8-d0d82d75ddb3-istio-data\") pod \"e2e-authcfg-http-dd6b21-openshift-default-6b78fd9989-vlz98\" (UID: \"35be0c3e-0dce-47c2-8bb8-d0d82d75ddb3\") " pod="openshift-ingress/e2e-authcfg-http-dd6b21-openshift-default-6b78fd9989-vlz98" Oct 30 21:17:41.620981 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:17:41.620779 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35be0c3e-0dce-47c2-8bb8-d0d82d75ddb3-workload-certs\") pod \"e2e-authcfg-http-dd6b21-openshift-default-6b78fd9989-vlz98\" (UID: \"35be0c3e-0dce-47c2-8bb8-d0d82d75ddb3\") " pod="openshift-ingress/e2e-authcfg-http-dd6b21-openshift-default-6b78fd9989-vlz98" Oct 30 21:17:41.620981 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:17:41.620814 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35be0c3e-0dce-47c2-8bb8-d0d82d75ddb3-workload-socket\") pod \"e2e-authcfg-http-dd6b21-openshift-default-6b78fd9989-vlz98\" (UID: \"35be0c3e-0dce-47c2-8bb8-d0d82d75ddb3\") " pod="openshift-ingress/e2e-authcfg-http-dd6b21-openshift-default-6b78fd9989-vlz98" Oct 30 21:17:41.620981 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:17:41.620836 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35be0c3e-0dce-47c2-8bb8-d0d82d75ddb3-istiod-ca-cert\") pod \"e2e-authcfg-http-dd6b21-openshift-default-6b78fd9989-vlz98\" (UID: \"35be0c3e-0dce-47c2-8bb8-d0d82d75ddb3\") " pod="openshift-ingress/e2e-authcfg-http-dd6b21-openshift-default-6b78fd9989-vlz98" Oct 30 21:17:41.620981 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:17:41.620857 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pmjwt\" (UniqueName: \"kubernetes.io/projected/35be0c3e-0dce-47c2-8bb8-d0d82d75ddb3-kube-api-access-pmjwt\") pod \"e2e-authcfg-http-dd6b21-openshift-default-6b78fd9989-vlz98\" (UID: \"35be0c3e-0dce-47c2-8bb8-d0d82d75ddb3\") " pod="openshift-ingress/e2e-authcfg-http-dd6b21-openshift-default-6b78fd9989-vlz98" Oct 30 21:17:41.620981 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:17:41.620893 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35be0c3e-0dce-47c2-8bb8-d0d82d75ddb3-istio-podinfo\") pod \"e2e-authcfg-http-dd6b21-openshift-default-6b78fd9989-vlz98\" (UID: \"35be0c3e-0dce-47c2-8bb8-d0d82d75ddb3\") " pod="openshift-ingress/e2e-authcfg-http-dd6b21-openshift-default-6b78fd9989-vlz98" Oct 30 21:17:41.620981 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:17:41.620922 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35be0c3e-0dce-47c2-8bb8-d0d82d75ddb3-credential-socket\") pod \"e2e-authcfg-http-dd6b21-openshift-default-6b78fd9989-vlz98\" (UID: \"35be0c3e-0dce-47c2-8bb8-d0d82d75ddb3\") " pod="openshift-ingress/e2e-authcfg-http-dd6b21-openshift-default-6b78fd9989-vlz98" Oct 30 21:17:41.621610 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:17:41.621581 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35be0c3e-0dce-47c2-8bb8-d0d82d75ddb3-istio-data\") pod \"e2e-authcfg-http-dd6b21-openshift-default-6b78fd9989-vlz98\" (UID: \"35be0c3e-0dce-47c2-8bb8-d0d82d75ddb3\") " pod="openshift-ingress/e2e-authcfg-http-dd6b21-openshift-default-6b78fd9989-vlz98" Oct 30 21:17:41.621831 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:17:41.621747 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35be0c3e-0dce-47c2-8bb8-d0d82d75ddb3-workload-socket\") pod \"e2e-authcfg-http-dd6b21-openshift-default-6b78fd9989-vlz98\" (UID: \"35be0c3e-0dce-47c2-8bb8-d0d82d75ddb3\") " pod="openshift-ingress/e2e-authcfg-http-dd6b21-openshift-default-6b78fd9989-vlz98" Oct 30 21:17:41.621915 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:17:41.621860 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35be0c3e-0dce-47c2-8bb8-d0d82d75ddb3-credential-socket\") pod \"e2e-authcfg-http-dd6b21-openshift-default-6b78fd9989-vlz98\" (UID: \"35be0c3e-0dce-47c2-8bb8-d0d82d75ddb3\") " pod="openshift-ingress/e2e-authcfg-http-dd6b21-openshift-default-6b78fd9989-vlz98" Oct 30 21:17:41.622021 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:17:41.621994 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35be0c3e-0dce-47c2-8bb8-d0d82d75ddb3-istiod-ca-cert\") pod \"e2e-authcfg-http-dd6b21-openshift-default-6b78fd9989-vlz98\" (UID: \"35be0c3e-0dce-47c2-8bb8-d0d82d75ddb3\") " pod="openshift-ingress/e2e-authcfg-http-dd6b21-openshift-default-6b78fd9989-vlz98" Oct 30 21:17:41.622135 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:17:41.622071 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35be0c3e-0dce-47c2-8bb8-d0d82d75ddb3-workload-certs\") pod \"e2e-authcfg-http-dd6b21-openshift-default-6b78fd9989-vlz98\" (UID: \"35be0c3e-0dce-47c2-8bb8-d0d82d75ddb3\") " pod="openshift-ingress/e2e-authcfg-http-dd6b21-openshift-default-6b78fd9989-vlz98" Oct 30 21:17:41.623336 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:17:41.623314 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35be0c3e-0dce-47c2-8bb8-d0d82d75ddb3-istio-envoy\") pod \"e2e-authcfg-http-dd6b21-openshift-default-6b78fd9989-vlz98\" (UID: \"35be0c3e-0dce-47c2-8bb8-d0d82d75ddb3\") " pod="openshift-ingress/e2e-authcfg-http-dd6b21-openshift-default-6b78fd9989-vlz98" Oct 30 21:17:41.623816 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:17:41.623799 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35be0c3e-0dce-47c2-8bb8-d0d82d75ddb3-istio-podinfo\") pod \"e2e-authcfg-http-dd6b21-openshift-default-6b78fd9989-vlz98\" (UID: \"35be0c3e-0dce-47c2-8bb8-d0d82d75ddb3\") " pod="openshift-ingress/e2e-authcfg-http-dd6b21-openshift-default-6b78fd9989-vlz98" Oct 30 21:17:41.632280 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:17:41.632255 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35be0c3e-0dce-47c2-8bb8-d0d82d75ddb3-istio-token\") pod \"e2e-authcfg-http-dd6b21-openshift-default-6b78fd9989-vlz98\" (UID: \"35be0c3e-0dce-47c2-8bb8-d0d82d75ddb3\") " pod="openshift-ingress/e2e-authcfg-http-dd6b21-openshift-default-6b78fd9989-vlz98" Oct 30 21:17:41.632404 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:17:41.632385 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pmjwt\" (UniqueName: \"kubernetes.io/projected/35be0c3e-0dce-47c2-8bb8-d0d82d75ddb3-kube-api-access-pmjwt\") pod \"e2e-authcfg-http-dd6b21-openshift-default-6b78fd9989-vlz98\" (UID: \"35be0c3e-0dce-47c2-8bb8-d0d82d75ddb3\") " pod="openshift-ingress/e2e-authcfg-http-dd6b21-openshift-default-6b78fd9989-vlz98" Oct 30 21:17:41.700449 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:17:41.700356 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:17:41.770700 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:17:41.770669 4611 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-authcfg-http-dd6b21-openshift-default-6b78fd9989-vlz98" Oct 30 21:17:42.056457 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:17:42.056433 4611 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-authcfg-http-dd6b21-openshift-default-6b78fd9989-vlz98"] Oct 30 21:17:42.058258 ip-10-0-133-192 kubenswrapper[4611]: W1030 21:17:42.058228 4611 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35be0c3e_0dce_47c2_8bb8_d0d82d75ddb3.slice/crio-93ca80e3a764c9424bdfde35c8971d0d412d9847450ecb268ee0cb6290965941 WatchSource:0}: Error finding container 93ca80e3a764c9424bdfde35c8971d0d412d9847450ecb268ee0cb6290965941: Status 404 returned error can't find the container with id 93ca80e3a764c9424bdfde35c8971d0d412d9847450ecb268ee0cb6290965941 Oct 30 21:17:42.060336 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:17:42.060307 4611 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:17:42.060408 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:17:42.060369 4611 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:17:42.060451 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:17:42.060403 4611 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:17:43.039544 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:17:43.039483 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-dd6b21-openshift-default-6b78fd9989-vlz98" event={"ID":"35be0c3e-0dce-47c2-8bb8-d0d82d75ddb3","Type":"ContainerStarted","Data":"531848549a2da72b87201c5fb34deb132e76014ee47ff9566e915548ceab4894"} Oct 30 21:17:43.039947 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:17:43.039571 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-dd6b21-openshift-default-6b78fd9989-vlz98" event={"ID":"35be0c3e-0dce-47c2-8bb8-d0d82d75ddb3","Type":"ContainerStarted","Data":"93ca80e3a764c9424bdfde35c8971d0d412d9847450ecb268ee0cb6290965941"} Oct 30 21:17:43.062036 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:17:43.061981 4611 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-authcfg-http-dd6b21-openshift-default-6b78fd9989-vlz98" podStartSLOduration=2.061963049 podStartE2EDuration="2.061963049s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:17:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:17:43.060665828 +0000 UTC m=+2317.594692444" watchObservedRunningTime="2026-10-30 21:17:43.061963049 +0000 UTC m=+2317.595989667" Oct 30 21:17:43.771781 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:17:43.771746 4611 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-authcfg-http-dd6b21-openshift-default-6b78fd9989-vlz98" Oct 30 21:17:43.776886 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:17:43.776857 4611 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-authcfg-http-dd6b21-openshift-default-6b78fd9989-vlz98" Oct 30 21:17:44.044406 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:17:44.044327 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-authcfg-http-dd6b21-openshift-default-6b78fd9989-vlz98" Oct 30 21:17:44.045356 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:17:44.045333 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-authcfg-http-dd6b21-openshift-default-6b78fd9989-vlz98" Oct 30 21:17:45.402625 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:17:45.402596 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:17:45.629280 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:17:45.629233 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:17:46.636964 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:17:46.636926 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:17:46.637339 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:17:46.637137 4611 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-dd6b21-678576ccc9-mptbn"] Oct 30 21:17:46.637639 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:17:46.637621 4611 memory_manager.go:365] "RemoveStaleState removing state" podUID="87e039ce-c6f8-40d2-934d-55ef50b088cc" containerName="istio-proxy" Oct 30 21:17:46.640778 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:17:46.640764 4611 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-dd6b21-678576ccc9-mptbn" Oct 30 21:17:46.651851 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:17:46.651828 4611 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-authcfg-http-dd6b21-dockercfg-lzsfb\"" Oct 30 21:17:46.652210 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:17:46.652189 4611 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-authcfg-http-dd6b21\"" Oct 30 21:17:46.653569 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:17:46.653548 4611 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-dd6b21-678576ccc9-mptbn"] Oct 30 21:17:46.669324 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:17:46.669286 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4155e16-c4c3-48c9-b05f-85fe3a094805-plugins-config-volume\") pod \"payload-pre-processing-e2e-authcfg-http-dd6b21-678576ccc9-mptbn\" (UID: \"f4155e16-c4c3-48c9-b05f-85fe3a094805\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-dd6b21-678576ccc9-mptbn" Oct 30 21:17:46.669472 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:17:46.669423 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vfcfz\" (UniqueName: \"kubernetes.io/projected/f4155e16-c4c3-48c9-b05f-85fe3a094805-kube-api-access-vfcfz\") pod \"payload-pre-processing-e2e-authcfg-http-dd6b21-678576ccc9-mptbn\" (UID: \"f4155e16-c4c3-48c9-b05f-85fe3a094805\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-dd6b21-678576ccc9-mptbn" Oct 30 21:17:46.693207 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:17:46.693181 4611 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-dd6b21-6d7978649d-nzcxp"] Oct 30 21:17:46.696633 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:17:46.696616 4611 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-dd6b21-6d7978649d-nzcxp" Oct 30 21:17:46.709899 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:17:46.709875 4611 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-dd6b21-6d7978649d-nzcxp"] Oct 30 21:17:46.770744 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:17:46.770648 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4155e16-c4c3-48c9-b05f-85fe3a094805-plugins-config-volume\") pod \"payload-pre-processing-e2e-authcfg-http-dd6b21-678576ccc9-mptbn\" (UID: \"f4155e16-c4c3-48c9-b05f-85fe3a094805\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-dd6b21-678576ccc9-mptbn" Oct 30 21:17:46.771096 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:17:46.771059 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tg79m\" (UniqueName: \"kubernetes.io/projected/25df30b0-86a7-4259-b29f-fc9c9d01c268-kube-api-access-tg79m\") pod \"payload-processing-e2e-authcfg-http-dd6b21-6d7978649d-nzcxp\" (UID: \"25df30b0-86a7-4259-b29f-fc9c9d01c268\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-dd6b21-6d7978649d-nzcxp" Oct 30 21:17:46.771312 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:17:46.771295 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25df30b0-86a7-4259-b29f-fc9c9d01c268-plugins-config-volume\") pod \"payload-processing-e2e-authcfg-http-dd6b21-6d7978649d-nzcxp\" (UID: \"25df30b0-86a7-4259-b29f-fc9c9d01c268\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-dd6b21-6d7978649d-nzcxp" Oct 30 21:17:46.771489 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:17:46.771474 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vfcfz\" (UniqueName: \"kubernetes.io/projected/f4155e16-c4c3-48c9-b05f-85fe3a094805-kube-api-access-vfcfz\") pod \"payload-pre-processing-e2e-authcfg-http-dd6b21-678576ccc9-mptbn\" (UID: \"f4155e16-c4c3-48c9-b05f-85fe3a094805\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-dd6b21-678576ccc9-mptbn" Oct 30 21:17:46.773111 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:17:46.771674 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4155e16-c4c3-48c9-b05f-85fe3a094805-plugins-config-volume\") pod \"payload-pre-processing-e2e-authcfg-http-dd6b21-678576ccc9-mptbn\" (UID: \"f4155e16-c4c3-48c9-b05f-85fe3a094805\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-dd6b21-678576ccc9-mptbn" Oct 30 21:17:46.785317 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:17:46.785278 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vfcfz\" (UniqueName: \"kubernetes.io/projected/f4155e16-c4c3-48c9-b05f-85fe3a094805-kube-api-access-vfcfz\") pod \"payload-pre-processing-e2e-authcfg-http-dd6b21-678576ccc9-mptbn\" (UID: \"f4155e16-c4c3-48c9-b05f-85fe3a094805\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-dd6b21-678576ccc9-mptbn" Oct 30 21:17:46.835999 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:17:46.835966 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:17:46.873837 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:17:46.873809 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tg79m\" (UniqueName: \"kubernetes.io/projected/25df30b0-86a7-4259-b29f-fc9c9d01c268-kube-api-access-tg79m\") pod \"payload-processing-e2e-authcfg-http-dd6b21-6d7978649d-nzcxp\" (UID: \"25df30b0-86a7-4259-b29f-fc9c9d01c268\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-dd6b21-6d7978649d-nzcxp" Oct 30 21:17:46.873954 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:17:46.873850 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25df30b0-86a7-4259-b29f-fc9c9d01c268-plugins-config-volume\") pod \"payload-processing-e2e-authcfg-http-dd6b21-6d7978649d-nzcxp\" (UID: \"25df30b0-86a7-4259-b29f-fc9c9d01c268\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-dd6b21-6d7978649d-nzcxp" Oct 30 21:17:46.874569 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:17:46.874547 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25df30b0-86a7-4259-b29f-fc9c9d01c268-plugins-config-volume\") pod \"payload-processing-e2e-authcfg-http-dd6b21-6d7978649d-nzcxp\" (UID: \"25df30b0-86a7-4259-b29f-fc9c9d01c268\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-dd6b21-6d7978649d-nzcxp" Oct 30 21:17:46.885447 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:17:46.885424 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tg79m\" (UniqueName: \"kubernetes.io/projected/25df30b0-86a7-4259-b29f-fc9c9d01c268-kube-api-access-tg79m\") pod \"payload-processing-e2e-authcfg-http-dd6b21-6d7978649d-nzcxp\" (UID: \"25df30b0-86a7-4259-b29f-fc9c9d01c268\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-dd6b21-6d7978649d-nzcxp" Oct 30 21:17:46.954211 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:17:46.954151 4611 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-dd6b21-678576ccc9-mptbn" Oct 30 21:17:47.010745 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:17:47.010722 4611 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-dd6b21-6d7978649d-nzcxp" Oct 30 21:17:47.231734 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:17:47.231705 4611 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-dd6b21-678576ccc9-mptbn"] Oct 30 21:17:47.233904 ip-10-0-133-192 kubenswrapper[4611]: W1030 21:17:47.233863 4611 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4155e16_c4c3_48c9_b05f_85fe3a094805.slice/crio-9192466ebbefdb2c64452b5b03d0a7dea525e3e07e15544bdc0365092bd969fe WatchSource:0}: Error finding container 9192466ebbefdb2c64452b5b03d0a7dea525e3e07e15544bdc0365092bd969fe: Status 404 returned error can't find the container with id 9192466ebbefdb2c64452b5b03d0a7dea525e3e07e15544bdc0365092bd969fe Oct 30 21:17:47.306641 ip-10-0-133-192 kubenswrapper[4611]: W1030 21:17:47.306608 4611 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25df30b0_86a7_4259_b29f_fc9c9d01c268.slice/crio-203bed807088077c2df35221a0a7be6cefa2f39872f846d15ecc405ddf2d1418 WatchSource:0}: Error finding container 203bed807088077c2df35221a0a7be6cefa2f39872f846d15ecc405ddf2d1418: Status 404 returned error can't find the container with id 203bed807088077c2df35221a0a7be6cefa2f39872f846d15ecc405ddf2d1418 Oct 30 21:17:47.306792 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:17:47.306651 4611 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-dd6b21-6d7978649d-nzcxp"] Oct 30 21:17:47.505449 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:17:47.505357 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:17:48.067221 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:17:48.067180 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-dd6b21-6d7978649d-nzcxp" event={"ID":"25df30b0-86a7-4259-b29f-fc9c9d01c268","Type":"ContainerStarted","Data":"770b1093f3eccf71777b1f3228aa1b42e404526aa895bd68b969ce3c886a106c"} Oct 30 21:17:48.067726 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:17:48.067258 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-authcfg-http-dd6b21-6d7978649d-nzcxp" Oct 30 21:17:48.067726 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:17:48.067284 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-dd6b21-6d7978649d-nzcxp" event={"ID":"25df30b0-86a7-4259-b29f-fc9c9d01c268","Type":"ContainerStarted","Data":"203bed807088077c2df35221a0a7be6cefa2f39872f846d15ecc405ddf2d1418"} Oct 30 21:17:48.068823 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:17:48.068795 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-dd6b21-678576ccc9-mptbn" event={"ID":"f4155e16-c4c3-48c9-b05f-85fe3a094805","Type":"ContainerStarted","Data":"6b2ecf0337265d2c267c9f01ea5456eadc3f450014ea5a7638996381865116c1"} Oct 30 21:17:48.068949 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:17:48.068835 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-dd6b21-678576ccc9-mptbn" event={"ID":"f4155e16-c4c3-48c9-b05f-85fe3a094805","Type":"ContainerStarted","Data":"9192466ebbefdb2c64452b5b03d0a7dea525e3e07e15544bdc0365092bd969fe"} Oct 30 21:17:48.069022 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:17:48.068946 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-dd6b21-678576ccc9-mptbn" Oct 30 21:17:48.084481 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:17:48.084429 4611 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-authcfg-http-dd6b21-6d7978649d-nzcxp" podStartSLOduration=2.084415441 podStartE2EDuration="2.084415441s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:17:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:17:48.083265188 +0000 UTC m=+2322.617291806" watchObservedRunningTime="2026-10-30 21:17:48.084415441 +0000 UTC m=+2322.618442055" Oct 30 21:17:48.101036 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:17:48.100991 4611 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-dd6b21-678576ccc9-mptbn" podStartSLOduration=2.100979797 podStartE2EDuration="2.100979797s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:17:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:17:48.099118218 +0000 UTC m=+2322.633144866" watchObservedRunningTime="2026-10-30 21:17:48.100979797 +0000 UTC m=+2322.635006437" Oct 30 21:17:48.300542 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:17:48.300493 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:17:49.153304 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:17:49.146997 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:17:49.773766 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:17:49.773723 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:17:49.935390 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:17:49.935340 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:17:50.625820 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:17:50.625782 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:17:51.387952 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:17:51.387918 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:17:52.089632 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:17:52.089578 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:17:52.731600 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:17:52.731560 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:17:53.021617 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:17:53.021534 4611 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-n7z25"] Oct 30 21:17:53.026045 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:17:53.026017 4611 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-n7z25"] Oct 30 21:17:53.299289 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:17:53.299178 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:17:53.604307 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:17:53.604276 4611 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="08204e6f-3e3e-4414-b3ea-da35d807ca4d" path="/var/lib/kubelet/pods/08204e6f-3e3e-4414-b3ea-da35d807ca4d/volumes" Oct 30 21:17:53.914177 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:17:53.914051 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:17:54.752952 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:17:54.752912 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:17:55.273403 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:17:55.273364 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:17:55.486558 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:17:55.486438 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:17:56.202086 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:17:56.202045 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:17:57.088702 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:17:57.088661 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:17:57.864969 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:17:57.864926 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:17:58.508154 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:17:58.508122 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:17:59.045680 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:17:59.045644 4611 patch_prober.go:29] interesting pod/e2e-authcfg-http-dd6b21-openshift-default-6b78fd9989-vlz98 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" start-of-body= Oct 30 21:17:59.046061 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:17:59.045696 4611 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-authcfg-http-dd6b21-openshift-default-6b78fd9989-vlz98" podUID="35be0c3e-0dce-47c2-8bb8-d0d82d75ddb3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" Oct 30 21:17:59.078357 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:17:59.078330 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-dd6b21-678576ccc9-mptbn" Oct 30 21:17:59.078521 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:17:59.078491 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-authcfg-http-dd6b21-6d7978649d-nzcxp" Oct 30 21:17:59.135875 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:17:59.135838 4611 generic.go:356] "Generic (PLEG): container finished" podID="35be0c3e-0dce-47c2-8bb8-d0d82d75ddb3" containerID="531848549a2da72b87201c5fb34deb132e76014ee47ff9566e915548ceab4894" exitCode=0 Oct 30 21:17:59.136056 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:17:59.135913 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-dd6b21-openshift-default-6b78fd9989-vlz98" event={"ID":"35be0c3e-0dce-47c2-8bb8-d0d82d75ddb3","Type":"ContainerDied","Data":"531848549a2da72b87201c5fb34deb132e76014ee47ff9566e915548ceab4894"} Oct 30 21:17:59.136377 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:17:59.136312 4611 scope.go:122] "RemoveContainer" containerID="531848549a2da72b87201c5fb34deb132e76014ee47ff9566e915548ceab4894" Oct 30 21:17:59.137579 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:17:59.137551 4611 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:17:59.137652 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:17:59.137621 4611 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:17:59.137693 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:17:59.137668 4611 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:17:59.245676 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:17:59.245633 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:17:59.997269 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:17:59.997236 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:18:00.144075 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:18:00.144037 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-dd6b21-openshift-default-6b78fd9989-vlz98" event={"ID":"35be0c3e-0dce-47c2-8bb8-d0d82d75ddb3","Type":"ContainerStarted","Data":"790f8e22f0d64fc218a57ee248677c8e3c35594641852e4b0a494c5220e7f141"} Oct 30 21:18:00.771761 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:18:00.771729 4611 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-authcfg-http-dd6b21-openshift-default-6b78fd9989-vlz98" Oct 30 21:18:00.771956 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:18:00.771934 4611 patch_prober.go:29] interesting pod/e2e-authcfg-http-dd6b21-openshift-default-6b78fd9989-vlz98 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" start-of-body= Oct 30 21:18:00.772004 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:18:00.771985 4611 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-dd6b21-openshift-default-6b78fd9989-vlz98" podUID="35be0c3e-0dce-47c2-8bb8-d0d82d75ddb3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" Oct 30 21:18:00.802675 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:18:00.802643 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:18:01.513246 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:18:01.513206 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:18:01.772015 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:18:01.771926 4611 patch_prober.go:29] interesting pod/e2e-authcfg-http-dd6b21-openshift-default-6b78fd9989-vlz98 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" start-of-body= Oct 30 21:18:01.772015 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:18:01.771985 4611 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-dd6b21-openshift-default-6b78fd9989-vlz98" podUID="35be0c3e-0dce-47c2-8bb8-d0d82d75ddb3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" Oct 30 21:18:02.100342 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:18:02.100243 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:18:02.772537 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:18:02.771914 4611 patch_prober.go:29] interesting pod/e2e-authcfg-http-dd6b21-openshift-default-6b78fd9989-vlz98 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" start-of-body= Oct 30 21:18:02.772537 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:18:02.771971 4611 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-dd6b21-openshift-default-6b78fd9989-vlz98" podUID="35be0c3e-0dce-47c2-8bb8-d0d82d75ddb3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" Oct 30 21:18:02.772537 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:18:02.772007 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:18:03.387829 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:18:03.387786 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:18:03.772122 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:18:03.772029 4611 patch_prober.go:29] interesting pod/e2e-authcfg-http-dd6b21-openshift-default-6b78fd9989-vlz98 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" start-of-body= Oct 30 21:18:03.772284 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:18:03.772117 4611 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-dd6b21-openshift-default-6b78fd9989-vlz98" podUID="35be0c3e-0dce-47c2-8bb8-d0d82d75ddb3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" Oct 30 21:18:03.916648 ip-10-0-133-192 kubenswrapper[4611]: E1030 21:18:03.916613 4611 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hwzc6" containerName="extractor" Oct 30 21:18:03.987813 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:18:03.987778 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:18:04.590599 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:18:04.590561 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:18:04.772005 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:18:04.771971 4611 patch_prober.go:29] interesting pod/e2e-authcfg-http-dd6b21-openshift-default-6b78fd9989-vlz98 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" start-of-body= Oct 30 21:18:04.772183 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:18:04.772028 4611 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-dd6b21-openshift-default-6b78fd9989-vlz98" podUID="35be0c3e-0dce-47c2-8bb8-d0d82d75ddb3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" Oct 30 21:18:05.190911 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:18:05.190875 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:18:05.775708 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:18:05.775679 4611 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-authcfg-http-dd6b21-openshift-default-6b78fd9989-vlz98" Oct 30 21:18:05.776025 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:18:05.776002 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-authcfg-http-dd6b21-openshift-default-6b78fd9989-vlz98" Oct 30 21:18:05.776672 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:18:05.776651 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-authcfg-http-dd6b21-openshift-default-6b78fd9989-vlz98" Oct 30 21:18:05.868181 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:18:05.868149 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:18:06.500348 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:18:06.500293 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:18:07.137791 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:18:07.137759 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:18:08.005672 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:18:08.005638 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:18:08.586901 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:18:08.586859 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:18:09.193318 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:18:09.193275 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:18:09.799163 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:18:09.798962 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:18:10.407032 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:18:10.406983 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:18:11.205949 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:18:11.205913 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:18:11.889663 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:18:11.889620 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:18:12.590394 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:18:12.590339 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:18:13.199462 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:18:13.199418 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:18:13.906774 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:18:13.906732 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:18:14.519167 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:18:14.519128 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:18:15.107686 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:18:15.107645 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:18:15.776283 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:18:15.776245 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:18:16.492989 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:18:16.492951 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:18:17.113119 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:18:17.113086 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:18:17.706300 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:18:17.706262 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:18:18.340405 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:18:18.340358 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:18:19.044598 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:18:19.044530 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:18:19.680837 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:18:19.680796 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:18:20.295132 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:18:20.295097 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:18:20.895967 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:18:20.895919 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:18:21.622601 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:18:21.622550 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:18:22.377874 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:18:22.377825 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:18:22.995791 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:18:22.995751 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:18:23.600455 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:18:23.600418 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:18:24.218328 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:18:24.218293 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:18:24.801798 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:18:24.801062 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:18:25.395959 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:18:25.395922 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:18:26.119692 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:18:26.119656 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:18:26.805348 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:18:26.805280 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:18:27.478944 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:18:27.478906 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:18:28.111918 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:18:28.111870 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:18:28.884667 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:18:28.884632 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:18:29.593088 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:18:29.593048 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:18:30.200316 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:18:30.200278 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:18:30.799490 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:18:30.797315 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:18:31.478953 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:18:31.478921 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:18:32.201193 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:18:32.201157 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:18:32.954053 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:18:32.954020 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:18:33.594165 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:18:33.594115 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:18:33.917143 ip-10-0-133-192 kubenswrapper[4611]: E1030 21:18:33.917103 4611 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hwzc6" containerName="extractor" Oct 30 21:18:34.297333 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:18:34.297237 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:18:34.887285 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:18:34.887249 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:18:35.503175 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:18:35.503132 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:18:36.176408 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:18:36.176369 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:18:36.916309 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:18:36.916270 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:18:37.596784 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:18:37.596733 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:18:38.378230 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:18:38.378190 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:18:38.989685 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:18:38.989650 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:18:39.599402 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:18:39.599365 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:18:40.260050 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:18:40.260015 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:18:40.799261 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:18:40.799226 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:18:41.416194 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:18:41.413759 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:18:42.092367 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:18:42.092327 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:18:42.779586 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:18:42.779466 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:18:43.400550 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:18:43.400495 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:18:44.011078 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:18:44.011044 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:18:44.697429 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:18:44.697392 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:18:45.303703 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:18:45.303656 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:18:46.103042 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:18:46.103001 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:18:46.688157 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:18:46.688078 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:18:47.461491 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:18:47.461454 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:18:48.097831 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:18:48.097724 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:18:48.703819 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:18:48.703784 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:18:49.301969 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:18:49.301936 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:18:49.813985 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:18:49.813948 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:18:49.957038 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:18:49.956981 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:18:50.591371 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:18:50.591327 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:18:51.214687 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:18:51.214631 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:18:51.824807 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:18:51.824766 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:18:52.478697 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:18:52.478657 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:18:53.444169 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:18:53.444124 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:18:53.779947 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:18:53.779682 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:18:54.384202 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:18:54.384161 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:18:54.995589 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:18:54.995554 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:18:55.603307 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:18:55.603232 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:18:56.290810 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:18:56.290775 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:18:56.894986 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:18:56.894942 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:18:57.503874 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:18:57.503834 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:18:58.198320 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:18:58.198277 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:18:59.015377 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:18:59.015327 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:18:59.618815 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:18:59.618770 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:19:00.287678 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:19:00.287635 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:19:00.902900 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:19:00.902860 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:19:01.068002 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:19:01.067971 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:19:01.211165 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:19:01.211086 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:19:01.515331 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:19:01.515236 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:19:02.110496 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:19:02.110455 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:19:02.738610 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:19:02.738575 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:19:03.377175 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:19:03.377134 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:19:03.916199 ip-10-0-133-192 kubenswrapper[4611]: E1030 21:19:03.916164 4611 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hwzc6" containerName="extractor" Oct 30 21:19:03.991313 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:19:03.991281 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:19:04.600228 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:19:04.600179 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:19:05.275552 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:19:05.275518 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:19:05.944290 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:19:05.944258 4611 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7w5wz_7e2574c1-b59c-453c-9501-25ec13a1ccad/ovn-acl-logging/0.log" Oct 30 21:19:05.955304 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:19:05.955280 4611 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7w5wz_7e2574c1-b59c-453c-9501-25ec13a1ccad/ovn-acl-logging/0.log" Oct 30 21:19:06.027756 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:19:06.027718 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:19:06.670698 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:19:06.670654 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:19:07.304926 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:19:07.304886 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:19:08.002005 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:19:08.001969 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:19:08.686357 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:19:08.686319 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:19:08.754713 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:19:08.754684 4611 scope.go:122] "RemoveContainer" containerID="8519f88de223a25ffe8432db977462b083edb4476715f146a539c8928afeb463" Oct 30 21:19:08.763555 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:19:08.763537 4611 scope.go:122] "RemoveContainer" containerID="ba5ca9aae42f3deec9e3952d9f66f051003122b45473b5390904ce2442b6a8cb" Oct 30 21:19:09.318263 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:19:09.318229 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:19:09.979287 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:19:09.979235 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:19:10.598864 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:19:10.598829 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:19:11.266106 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:19:11.266071 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:19:11.924743 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:19:11.924704 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:19:12.499674 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:19:12.499632 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:19:13.220996 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:19:13.220951 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:19:13.968190 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:19:13.968163 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:19:14.605314 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:19:14.605277 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:19:15.262734 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:19:15.262685 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:19:16.000136 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:19:16.000101 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:19:16.699495 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:19:16.699456 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:19:17.303296 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:19:17.303257 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:19:18.005912 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:19:18.005869 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:19:18.693886 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:19:18.693854 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:19:19.416452 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:19:19.416420 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:19:20.263570 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:19:20.263532 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:19:21.178417 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:19:21.178385 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:19:21.872114 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:19:21.872075 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:19:22.547115 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:19:22.547079 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:19:23.178855 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:19:23.178823 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:19:23.798926 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:19:23.798887 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:19:24.392836 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:19:24.392801 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:19:24.992052 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:19:24.992011 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:19:25.536387 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:19:25.536350 4611 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-dd6b21-6d7978649d-nzcxp"] Oct 30 21:19:25.550062 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:19:25.550024 4611 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-dd6b21-678576ccc9-mptbn"] Oct 30 21:19:25.704611 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:19:25.704583 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:19:25.914979 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:19:25.914942 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:19:30.537583 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:19:30.537544 4611 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-authcfg-http-dd6b21-6d7978649d-nzcxp" podUID="25df30b0-86a7-4259-b29f-fc9c9d01c268" containerName="payload-processing" containerID="cri-o://770b1093f3eccf71777b1f3228aa1b42e404526aa895bd68b969ce3c886a106c" gracePeriod=25 Oct 30 21:19:30.551204 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:19:30.551172 4611 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-dd6b21-678576ccc9-mptbn" podUID="f4155e16-c4c3-48c9-b05f-85fe3a094805" containerName="payload-pre-processing" containerID="cri-o://6b2ecf0337265d2c267c9f01ea5456eadc3f450014ea5a7638996381865116c1" gracePeriod=25 Oct 30 21:19:30.676088 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:19:30.676061 4611 generic.go:356] "Generic (PLEG): container finished" podID="f4155e16-c4c3-48c9-b05f-85fe3a094805" containerID="6b2ecf0337265d2c267c9f01ea5456eadc3f450014ea5a7638996381865116c1" exitCode=0 Oct 30 21:19:30.676206 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:19:30.676151 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-dd6b21-678576ccc9-mptbn" event={"ID":"f4155e16-c4c3-48c9-b05f-85fe3a094805","Type":"ContainerDied","Data":"6b2ecf0337265d2c267c9f01ea5456eadc3f450014ea5a7638996381865116c1"} Oct 30 21:19:30.678014 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:19:30.677988 4611 generic.go:356] "Generic (PLEG): container finished" podID="25df30b0-86a7-4259-b29f-fc9c9d01c268" containerID="770b1093f3eccf71777b1f3228aa1b42e404526aa895bd68b969ce3c886a106c" exitCode=0 Oct 30 21:19:30.678125 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:19:30.678051 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-dd6b21-6d7978649d-nzcxp" event={"ID":"25df30b0-86a7-4259-b29f-fc9c9d01c268","Type":"ContainerDied","Data":"770b1093f3eccf71777b1f3228aa1b42e404526aa895bd68b969ce3c886a106c"} Oct 30 21:19:30.712012 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:19:30.710810 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:19:30.794087 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:19:30.794029 4611 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-dd6b21-6d7978649d-nzcxp" Oct 30 21:19:30.826806 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:19:30.826785 4611 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-dd6b21-678576ccc9-mptbn" Oct 30 21:19:30.842611 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:19:30.842586 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f4155e16-c4c3-48c9-b05f-85fe3a094805-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4155e16-c4c3-48c9-b05f-85fe3a094805-plugins-config-volume\") pod \"f4155e16-c4c3-48c9-b05f-85fe3a094805\" (UID: \"f4155e16-c4c3-48c9-b05f-85fe3a094805\") " Oct 30 21:19:30.842734 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:19:30.842642 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/25df30b0-86a7-4259-b29f-fc9c9d01c268-kube-api-access-tg79m\" (UniqueName: \"kubernetes.io/projected/25df30b0-86a7-4259-b29f-fc9c9d01c268-kube-api-access-tg79m\") pod \"25df30b0-86a7-4259-b29f-fc9c9d01c268\" (UID: \"25df30b0-86a7-4259-b29f-fc9c9d01c268\") " Oct 30 21:19:30.842777 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:19:30.842763 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/25df30b0-86a7-4259-b29f-fc9c9d01c268-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25df30b0-86a7-4259-b29f-fc9c9d01c268-plugins-config-volume\") pod \"25df30b0-86a7-4259-b29f-fc9c9d01c268\" (UID: \"25df30b0-86a7-4259-b29f-fc9c9d01c268\") " Oct 30 21:19:30.842863 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:19:30.842846 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f4155e16-c4c3-48c9-b05f-85fe3a094805-kube-api-access-vfcfz\" (UniqueName: \"kubernetes.io/projected/f4155e16-c4c3-48c9-b05f-85fe3a094805-kube-api-access-vfcfz\") pod \"f4155e16-c4c3-48c9-b05f-85fe3a094805\" (UID: \"f4155e16-c4c3-48c9-b05f-85fe3a094805\") " Oct 30 21:19:30.843137 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:19:30.842935 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4155e16-c4c3-48c9-b05f-85fe3a094805-plugins-config-volume" pod "f4155e16-c4c3-48c9-b05f-85fe3a094805" (UID: "f4155e16-c4c3-48c9-b05f-85fe3a094805"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:19:30.843137 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:19:30.843130 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25df30b0-86a7-4259-b29f-fc9c9d01c268-plugins-config-volume" pod "25df30b0-86a7-4259-b29f-fc9c9d01c268" (UID: "25df30b0-86a7-4259-b29f-fc9c9d01c268"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:19:30.843326 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:19:30.843242 4611 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25df30b0-86a7-4259-b29f-fc9c9d01c268-plugins-config-volume\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:19:30.843326 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:19:30.843260 4611 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4155e16-c4c3-48c9-b05f-85fe3a094805-plugins-config-volume\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:19:30.845153 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:19:30.845107 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25df30b0-86a7-4259-b29f-fc9c9d01c268-kube-api-access-tg79m" pod "25df30b0-86a7-4259-b29f-fc9c9d01c268" (UID: "25df30b0-86a7-4259-b29f-fc9c9d01c268"). InnerVolumeSpecName "kube-api-access-tg79m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:19:30.845946 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:19:30.845910 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4155e16-c4c3-48c9-b05f-85fe3a094805-kube-api-access-vfcfz" pod "f4155e16-c4c3-48c9-b05f-85fe3a094805" (UID: "f4155e16-c4c3-48c9-b05f-85fe3a094805"). InnerVolumeSpecName "kube-api-access-vfcfz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:19:30.857321 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:19:30.857277 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:19:30.944342 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:19:30.944308 4611 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vfcfz\" (UniqueName: \"kubernetes.io/projected/f4155e16-c4c3-48c9-b05f-85fe3a094805-kube-api-access-vfcfz\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:19:30.944342 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:19:30.944337 4611 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tg79m\" (UniqueName: \"kubernetes.io/projected/25df30b0-86a7-4259-b29f-fc9c9d01c268-kube-api-access-tg79m\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:19:31.398754 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:19:31.398717 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:19:31.684027 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:19:31.683954 4611 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-dd6b21-678576ccc9-mptbn" Oct 30 21:19:31.684446 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:19:31.683954 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-dd6b21-678576ccc9-mptbn" event={"ID":"f4155e16-c4c3-48c9-b05f-85fe3a094805","Type":"ContainerDied","Data":"9192466ebbefdb2c64452b5b03d0a7dea525e3e07e15544bdc0365092bd969fe"} Oct 30 21:19:31.684446 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:19:31.684076 4611 scope.go:122] "RemoveContainer" containerID="6b2ecf0337265d2c267c9f01ea5456eadc3f450014ea5a7638996381865116c1" Oct 30 21:19:31.685488 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:19:31.685469 4611 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-dd6b21-6d7978649d-nzcxp" Oct 30 21:19:31.685591 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:19:31.685483 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-dd6b21-6d7978649d-nzcxp" event={"ID":"25df30b0-86a7-4259-b29f-fc9c9d01c268","Type":"ContainerDied","Data":"203bed807088077c2df35221a0a7be6cefa2f39872f846d15ecc405ddf2d1418"} Oct 30 21:19:31.693184 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:19:31.693167 4611 scope.go:122] "RemoveContainer" containerID="770b1093f3eccf71777b1f3228aa1b42e404526aa895bd68b969ce3c886a106c" Oct 30 21:19:31.702956 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:19:31.702929 4611 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-dd6b21-6d7978649d-nzcxp"] Oct 30 21:19:31.708668 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:19:31.708648 4611 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-dd6b21-6d7978649d-nzcxp"] Oct 30 21:19:31.717983 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:19:31.717960 4611 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-dd6b21-678576ccc9-mptbn"] Oct 30 21:19:31.723937 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:19:31.723915 4611 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-dd6b21-678576ccc9-mptbn"] Oct 30 21:19:33.604766 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:19:33.604731 4611 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="25df30b0-86a7-4259-b29f-fc9c9d01c268" path="/var/lib/kubelet/pods/25df30b0-86a7-4259-b29f-fc9c9d01c268/volumes" Oct 30 21:19:33.605270 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:19:33.605234 4611 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f4155e16-c4c3-48c9-b05f-85fe3a094805" path="/var/lib/kubelet/pods/f4155e16-c4c3-48c9-b05f-85fe3a094805/volumes" Oct 30 21:19:33.916267 ip-10-0-133-192 kubenswrapper[4611]: E1030 21:19:33.916183 4611 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hwzc6" containerName="extractor" Oct 30 21:19:37.470939 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:19:37.470883 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:19:37.619610 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:19:37.619575 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:19:37.749893 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:19:37.749786 4611 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-authcfg-http-dd6b21-openshift-default-6b78fd9989-vlz98"] Oct 30 21:19:37.750080 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:19:37.750032 4611 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-authcfg-http-dd6b21-openshift-default-6b78fd9989-vlz98" podUID="35be0c3e-0dce-47c2-8bb8-d0d82d75ddb3" containerName="istio-proxy" containerID="cri-o://790f8e22f0d64fc218a57ee248677c8e3c35594641852e4b0a494c5220e7f141" gracePeriod=30 Oct 30 21:19:37.839144 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:19:37.839107 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:19:42.994208 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:19:42.994181 4611 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-authcfg-http-dd6b21-openshift-default-6b78fd9989-vlz98" Oct 30 21:19:43.050637 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:19:43.050564 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35be0c3e-0dce-47c2-8bb8-d0d82d75ddb3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35be0c3e-0dce-47c2-8bb8-d0d82d75ddb3-workload-socket\") pod \"35be0c3e-0dce-47c2-8bb8-d0d82d75ddb3\" (UID: \"35be0c3e-0dce-47c2-8bb8-d0d82d75ddb3\") " Oct 30 21:19:43.050637 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:19:43.050610 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/35be0c3e-0dce-47c2-8bb8-d0d82d75ddb3-kube-api-access-pmjwt\" (UniqueName: \"kubernetes.io/projected/35be0c3e-0dce-47c2-8bb8-d0d82d75ddb3-kube-api-access-pmjwt\") pod \"35be0c3e-0dce-47c2-8bb8-d0d82d75ddb3\" (UID: \"35be0c3e-0dce-47c2-8bb8-d0d82d75ddb3\") " Oct 30 21:19:43.050637 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:19:43.050631 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/35be0c3e-0dce-47c2-8bb8-d0d82d75ddb3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35be0c3e-0dce-47c2-8bb8-d0d82d75ddb3-istio-podinfo\") pod \"35be0c3e-0dce-47c2-8bb8-d0d82d75ddb3\" (UID: \"35be0c3e-0dce-47c2-8bb8-d0d82d75ddb3\") " Oct 30 21:19:43.050910 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:19:43.050660 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35be0c3e-0dce-47c2-8bb8-d0d82d75ddb3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35be0c3e-0dce-47c2-8bb8-d0d82d75ddb3-istio-data\") pod \"35be0c3e-0dce-47c2-8bb8-d0d82d75ddb3\" (UID: \"35be0c3e-0dce-47c2-8bb8-d0d82d75ddb3\") " Oct 30 21:19:43.050910 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:19:43.050693 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35be0c3e-0dce-47c2-8bb8-d0d82d75ddb3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35be0c3e-0dce-47c2-8bb8-d0d82d75ddb3-credential-socket\") pod \"35be0c3e-0dce-47c2-8bb8-d0d82d75ddb3\" (UID: \"35be0c3e-0dce-47c2-8bb8-d0d82d75ddb3\") " Oct 30 21:19:43.050910 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:19:43.050765 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/35be0c3e-0dce-47c2-8bb8-d0d82d75ddb3-istio-token\" (UniqueName: \"kubernetes.io/projected/35be0c3e-0dce-47c2-8bb8-d0d82d75ddb3-istio-token\") pod \"35be0c3e-0dce-47c2-8bb8-d0d82d75ddb3\" (UID: \"35be0c3e-0dce-47c2-8bb8-d0d82d75ddb3\") " Oct 30 21:19:43.050910 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:19:43.050816 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35be0c3e-0dce-47c2-8bb8-d0d82d75ddb3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35be0c3e-0dce-47c2-8bb8-d0d82d75ddb3-istio-envoy\") pod \"35be0c3e-0dce-47c2-8bb8-d0d82d75ddb3\" (UID: \"35be0c3e-0dce-47c2-8bb8-d0d82d75ddb3\") " Oct 30 21:19:43.050910 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:19:43.050851 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35be0c3e-0dce-47c2-8bb8-d0d82d75ddb3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35be0c3e-0dce-47c2-8bb8-d0d82d75ddb3-workload-certs\") pod \"35be0c3e-0dce-47c2-8bb8-d0d82d75ddb3\" (UID: \"35be0c3e-0dce-47c2-8bb8-d0d82d75ddb3\") " Oct 30 21:19:43.050910 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:19:43.050852 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35be0c3e-0dce-47c2-8bb8-d0d82d75ddb3-workload-socket" pod "35be0c3e-0dce-47c2-8bb8-d0d82d75ddb3" (UID: "35be0c3e-0dce-47c2-8bb8-d0d82d75ddb3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:19:43.051248 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:19:43.050954 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/35be0c3e-0dce-47c2-8bb8-d0d82d75ddb3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35be0c3e-0dce-47c2-8bb8-d0d82d75ddb3-istiod-ca-cert\") pod \"35be0c3e-0dce-47c2-8bb8-d0d82d75ddb3\" (UID: \"35be0c3e-0dce-47c2-8bb8-d0d82d75ddb3\") " Oct 30 21:19:43.051248 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:19:43.050955 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35be0c3e-0dce-47c2-8bb8-d0d82d75ddb3-istio-data" pod "35be0c3e-0dce-47c2-8bb8-d0d82d75ddb3" (UID: "35be0c3e-0dce-47c2-8bb8-d0d82d75ddb3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:19:43.051248 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:19:43.051139 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35be0c3e-0dce-47c2-8bb8-d0d82d75ddb3-workload-certs" pod "35be0c3e-0dce-47c2-8bb8-d0d82d75ddb3" (UID: "35be0c3e-0dce-47c2-8bb8-d0d82d75ddb3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:19:43.051427 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:19:43.051332 4611 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35be0c3e-0dce-47c2-8bb8-d0d82d75ddb3-istio-data\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:19:43.051427 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:19:43.051351 4611 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35be0c3e-0dce-47c2-8bb8-d0d82d75ddb3-workload-certs\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:19:43.051427 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:19:43.051365 4611 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35be0c3e-0dce-47c2-8bb8-d0d82d75ddb3-workload-socket\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:19:43.051618 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:19:43.051472 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35be0c3e-0dce-47c2-8bb8-d0d82d75ddb3-istiod-ca-cert" pod "35be0c3e-0dce-47c2-8bb8-d0d82d75ddb3" (UID: "35be0c3e-0dce-47c2-8bb8-d0d82d75ddb3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:19:43.051618 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:19:43.051555 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35be0c3e-0dce-47c2-8bb8-d0d82d75ddb3-credential-socket" pod "35be0c3e-0dce-47c2-8bb8-d0d82d75ddb3" (UID: "35be0c3e-0dce-47c2-8bb8-d0d82d75ddb3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:19:43.052837 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:19:43.052816 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35be0c3e-0dce-47c2-8bb8-d0d82d75ddb3-kube-api-access-pmjwt" pod "35be0c3e-0dce-47c2-8bb8-d0d82d75ddb3" (UID: "35be0c3e-0dce-47c2-8bb8-d0d82d75ddb3"). InnerVolumeSpecName "kube-api-access-pmjwt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:19:43.053223 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:19:43.053204 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35be0c3e-0dce-47c2-8bb8-d0d82d75ddb3-istio-envoy" pod "35be0c3e-0dce-47c2-8bb8-d0d82d75ddb3" (UID: "35be0c3e-0dce-47c2-8bb8-d0d82d75ddb3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:19:43.053223 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:19:43.053211 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/35be0c3e-0dce-47c2-8bb8-d0d82d75ddb3-istio-podinfo" pod "35be0c3e-0dce-47c2-8bb8-d0d82d75ddb3" (UID: "35be0c3e-0dce-47c2-8bb8-d0d82d75ddb3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:19:43.053417 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:19:43.053400 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35be0c3e-0dce-47c2-8bb8-d0d82d75ddb3-istio-token" pod "35be0c3e-0dce-47c2-8bb8-d0d82d75ddb3" (UID: "35be0c3e-0dce-47c2-8bb8-d0d82d75ddb3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:19:43.152402 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:19:43.152378 4611 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35be0c3e-0dce-47c2-8bb8-d0d82d75ddb3-istio-token\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:19:43.152402 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:19:43.152400 4611 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35be0c3e-0dce-47c2-8bb8-d0d82d75ddb3-istio-envoy\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:19:43.152558 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:19:43.152409 4611 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35be0c3e-0dce-47c2-8bb8-d0d82d75ddb3-istiod-ca-cert\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:19:43.152558 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:19:43.152419 4611 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pmjwt\" (UniqueName: \"kubernetes.io/projected/35be0c3e-0dce-47c2-8bb8-d0d82d75ddb3-kube-api-access-pmjwt\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:19:43.152558 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:19:43.152428 4611 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35be0c3e-0dce-47c2-8bb8-d0d82d75ddb3-istio-podinfo\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:19:43.152558 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:19:43.152437 4611 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35be0c3e-0dce-47c2-8bb8-d0d82d75ddb3-credential-socket\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:19:43.747657 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:19:43.747621 4611 generic.go:356] "Generic (PLEG): container finished" podID="35be0c3e-0dce-47c2-8bb8-d0d82d75ddb3" containerID="790f8e22f0d64fc218a57ee248677c8e3c35594641852e4b0a494c5220e7f141" exitCode=0 Oct 30 21:19:43.747821 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:19:43.747662 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-dd6b21-openshift-default-6b78fd9989-vlz98" event={"ID":"35be0c3e-0dce-47c2-8bb8-d0d82d75ddb3","Type":"ContainerDied","Data":"790f8e22f0d64fc218a57ee248677c8e3c35594641852e4b0a494c5220e7f141"} Oct 30 21:19:43.747821 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:19:43.747677 4611 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-authcfg-http-dd6b21-openshift-default-6b78fd9989-vlz98" Oct 30 21:19:43.747821 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:19:43.747717 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-dd6b21-openshift-default-6b78fd9989-vlz98" event={"ID":"35be0c3e-0dce-47c2-8bb8-d0d82d75ddb3","Type":"ContainerDied","Data":"93ca80e3a764c9424bdfde35c8971d0d412d9847450ecb268ee0cb6290965941"} Oct 30 21:19:43.747821 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:19:43.747740 4611 scope.go:122] "RemoveContainer" containerID="790f8e22f0d64fc218a57ee248677c8e3c35594641852e4b0a494c5220e7f141" Oct 30 21:19:43.756611 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:19:43.756590 4611 scope.go:122] "RemoveContainer" containerID="531848549a2da72b87201c5fb34deb132e76014ee47ff9566e915548ceab4894" Oct 30 21:19:43.763938 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:19:43.763914 4611 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-authcfg-http-dd6b21-openshift-default-6b78fd9989-vlz98"] Oct 30 21:19:43.764889 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:19:43.764872 4611 scope.go:122] "RemoveContainer" containerID="790f8e22f0d64fc218a57ee248677c8e3c35594641852e4b0a494c5220e7f141" Oct 30 21:19:43.765490 ip-10-0-133-192 kubenswrapper[4611]: E1030 21:19:43.765463 4611 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"790f8e22f0d64fc218a57ee248677c8e3c35594641852e4b0a494c5220e7f141\": container with ID starting with 790f8e22f0d64fc218a57ee248677c8e3c35594641852e4b0a494c5220e7f141 not found: ID does not exist" containerID="790f8e22f0d64fc218a57ee248677c8e3c35594641852e4b0a494c5220e7f141" Oct 30 21:19:43.765653 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:19:43.765504 4611 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"790f8e22f0d64fc218a57ee248677c8e3c35594641852e4b0a494c5220e7f141"} err="failed to get container status \"790f8e22f0d64fc218a57ee248677c8e3c35594641852e4b0a494c5220e7f141\": rpc error: code = NotFound desc = could not find container \"790f8e22f0d64fc218a57ee248677c8e3c35594641852e4b0a494c5220e7f141\": container with ID starting with 790f8e22f0d64fc218a57ee248677c8e3c35594641852e4b0a494c5220e7f141 not found: ID does not exist" Oct 30 21:19:43.765736 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:19:43.765659 4611 scope.go:122] "RemoveContainer" containerID="531848549a2da72b87201c5fb34deb132e76014ee47ff9566e915548ceab4894" Oct 30 21:19:43.765936 ip-10-0-133-192 kubenswrapper[4611]: E1030 21:19:43.765919 4611 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"531848549a2da72b87201c5fb34deb132e76014ee47ff9566e915548ceab4894\": container with ID starting with 531848549a2da72b87201c5fb34deb132e76014ee47ff9566e915548ceab4894 not found: ID does not exist" containerID="531848549a2da72b87201c5fb34deb132e76014ee47ff9566e915548ceab4894" Oct 30 21:19:43.765999 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:19:43.765950 4611 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"531848549a2da72b87201c5fb34deb132e76014ee47ff9566e915548ceab4894"} err="failed to get container status \"531848549a2da72b87201c5fb34deb132e76014ee47ff9566e915548ceab4894\": rpc error: code = NotFound desc = could not find container \"531848549a2da72b87201c5fb34deb132e76014ee47ff9566e915548ceab4894\": container with ID starting with 531848549a2da72b87201c5fb34deb132e76014ee47ff9566e915548ceab4894 not found: ID does not exist" Oct 30 21:19:43.767877 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:19:43.767856 4611 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-authcfg-http-dd6b21-openshift-default-6b78fd9989-vlz98"] Oct 30 21:19:45.605380 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:19:45.605340 4611 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="35be0c3e-0dce-47c2-8bb8-d0d82d75ddb3" path="/var/lib/kubelet/pods/35be0c3e-0dce-47c2-8bb8-d0d82d75ddb3/volumes" Oct 30 21:19:49.870888 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:19:49.870845 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:20:03.916284 ip-10-0-133-192 kubenswrapper[4611]: E1030 21:20:03.916247 4611 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hwzc6" containerName="extractor" Oct 30 21:20:33.916357 ip-10-0-133-192 kubenswrapper[4611]: E1030 21:20:33.916275 4611 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hwzc6" containerName="extractor" Oct 30 21:20:37.209190 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:20:37.209161 4611 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7b8677f665-47n2d_6394d876-8759-4272-9351-9abed42ee4c8/authorino/0.log" Oct 30 21:20:46.461957 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:20:46.461920 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:20:47.461582 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:20:47.461541 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:20:49.943018 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:20:49.942976 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:20:50.192909 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:20:50.192874 4611 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-6f84c677c8-bn6s4_05acfee1-1b41-4ee1-a122-4551a125c23e/maas-api/0.log" Oct 30 21:20:50.747482 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:20:50.747456 4611 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-360dbc5f-9n2dv_38e51b01-a6a7-46fa-bf03-8dd55c611586/revoke-keys/0.log" Oct 30 21:20:50.932396 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:20:50.932370 4611 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-3e31a506-nfcvw_3f8ebac8-5c1b-4335-ad73-89d000d9d36d/revoke-keys/0.log" Oct 30 21:20:51.119935 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:20:51.119914 4611 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-452b6d7b-grm5k_0f9b7600-8024-469c-af45-f6ee6ff5ae78/revoke-keys/0.log" Oct 30 21:20:53.025606 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:20:53.025569 4611 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-grm5k"] Oct 30 21:20:53.027985 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:20:53.027955 4611 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-grm5k"] Oct 30 21:20:53.178646 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:20:53.178614 4611 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7b8677f665-47n2d_6394d876-8759-4272-9351-9abed42ee4c8/authorino/0.log" Oct 30 21:20:53.604137 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:20:53.604098 4611 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0f9b7600-8024-469c-af45-f6ee6ff5ae78" path="/var/lib/kubelet/pods/0f9b7600-8024-469c-af45-f6ee6ff5ae78/volumes" Oct 30 21:20:53.604347 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:20:53.604168 4611 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-zmvxk_2755a243-2222-407a-bf25-a13b54bcc5f4/manager/0.log" Oct 30 21:20:53.781885 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:20:53.781857 4611 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-vp92r_860bdddb-14bc-4ba4-bf61-45ef8ffe5ed7/kuadrant-console-plugin/0.log" Oct 30 21:20:55.539435 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:20:55.539390 4611 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6c6fcc99c4-flnqs_022d8287-926f-4445-8931-98ab50429582/kube-auth-proxy/0.log" Oct 30 21:20:56.072362 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:20:56.072322 4611 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6db586dd6c-8tvvl_46ae03da-7a80-4f78-8abd-391ebc697f2f/istio-proxy/0.log" Oct 30 21:20:56.263222 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:20:56.263188 4611 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6db586dd6c-cwrqt_5da54d6d-30b1-4c0c-9824-e06f7a4ae0ee/istio-proxy/0.log" Oct 30 21:20:57.371145 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:20:57.371110 4611 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6db586dd6c-tg9jv_ee56d291-6c05-4674-895a-88f339e18e44/istio-proxy/0.log" Oct 30 21:20:57.554235 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:20:57.554200 4611 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6db586dd6c-w44qs_6d6fd4b2-ad0a-4ce3-b5b3-966de34bd8b0/istio-proxy/0.log" Oct 30 21:20:57.735605 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:20:57.735530 4611 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-69694c4c9-r6d8g_23e3feb4-2305-4e6e-8d6b-ab240a27cd91/istio-proxy/0.log" Oct 30 21:20:58.356038 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:20:58.355999 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:20:59.354328 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:20:59.354292 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:20:59.489438 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:20:59.489408 4611 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-7b998b79d9-dq6vs_f23cb637-ada7-4e6d-897f-83978d56ac7d/main/0.log" Oct 30 21:20:59.845421 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:20:59.845316 4611 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-69b588897d-qpdpp_aa85e194-38f7-4dfe-8be9-6df7df92f0a1/main/0.log" Oct 30 21:21:00.379305 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:21:00.379272 4611 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-84b4ff656c-s9l7p_a86251b2-e065-4aaa-9b7b-b8b7bee34bc2/main/0.log" Oct 30 21:21:01.054853 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:21:01.054817 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:21:01.235381 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:21:01.235342 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:21:01.521056 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:21:01.521019 4611 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7b8677f665-47n2d_6394d876-8759-4272-9351-9abed42ee4c8/authorino/0.log" Oct 30 21:21:03.916704 ip-10-0-133-192 kubenswrapper[4611]: E1030 21:21:03.916663 4611 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hwzc6" containerName="extractor" Oct 30 21:21:10.021794 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:21:10.021752 4611 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-nfcvw"] Oct 30 21:21:10.026481 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:21:10.026452 4611 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-nfcvw"] Oct 30 21:21:11.604303 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:21:11.604268 4611 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3f8ebac8-5c1b-4335-ad73-89d000d9d36d" path="/var/lib/kubelet/pods/3f8ebac8-5c1b-4335-ad73-89d000d9d36d/volumes" Oct 30 21:21:12.551021 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:21:12.550982 4611 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-6f84c677c8-bn6s4_05acfee1-1b41-4ee1-a122-4551a125c23e/maas-api/0.log" Oct 30 21:21:13.115446 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:21:13.115419 4611 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-360dbc5f-9n2dv_38e51b01-a6a7-46fa-bf03-8dd55c611586/revoke-keys/0.log" Oct 30 21:21:24.805893 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:21:24.805859 4611 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7b8677f665-47n2d_6394d876-8759-4272-9351-9abed42ee4c8/authorino/0.log" Oct 30 21:21:25.230968 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:21:25.230935 4611 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-zmvxk_2755a243-2222-407a-bf25-a13b54bcc5f4/manager/0.log" Oct 30 21:21:25.420384 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:21:25.420359 4611 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-vp92r_860bdddb-14bc-4ba4-bf61-45ef8ffe5ed7/kuadrant-console-plugin/0.log" Oct 30 21:21:26.963120 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:21:26.963081 4611 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6c6fcc99c4-flnqs_022d8287-926f-4445-8931-98ab50429582/kube-auth-proxy/0.log" Oct 30 21:21:27.512945 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:21:27.512906 4611 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6db586dd6c-8tvvl_46ae03da-7a80-4f78-8abd-391ebc697f2f/istio-proxy/0.log" Oct 30 21:21:27.707830 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:21:27.707797 4611 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6db586dd6c-cwrqt_5da54d6d-30b1-4c0c-9824-e06f7a4ae0ee/istio-proxy/0.log" Oct 30 21:21:28.817846 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:21:28.817817 4611 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6db586dd6c-tg9jv_ee56d291-6c05-4674-895a-88f339e18e44/istio-proxy/0.log" Oct 30 21:21:28.996095 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:21:28.996068 4611 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6db586dd6c-w44qs_6d6fd4b2-ad0a-4ce3-b5b3-966de34bd8b0/istio-proxy/0.log" Oct 30 21:21:29.178276 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:21:29.178249 4611 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-69694c4c9-r6d8g_23e3feb4-2305-4e6e-8d6b-ab240a27cd91/istio-proxy/0.log" Oct 30 21:21:30.427983 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:21:30.427947 4611 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-7b998b79d9-dq6vs_f23cb637-ada7-4e6d-897f-83978d56ac7d/main/0.log" Oct 30 21:21:30.801158 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:21:30.801081 4611 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-69b588897d-qpdpp_aa85e194-38f7-4dfe-8be9-6df7df92f0a1/main/0.log" Oct 30 21:21:31.338480 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:21:31.338441 4611 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-84b4ff656c-s9l7p_a86251b2-e065-4aaa-9b7b-b8b7bee34bc2/main/0.log" Oct 30 21:21:33.916326 ip-10-0-133-192 kubenswrapper[4611]: E1030 21:21:33.916281 4611 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hwzc6" containerName="extractor" Oct 30 21:21:36.180920 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:21:36.180887 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:21:38.023731 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:21:38.023699 4611 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-360dbc5f-9n2dv"] Oct 30 21:21:38.026273 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:21:38.026246 4611 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-360dbc5f-9n2dv"] Oct 30 21:21:39.604890 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:21:39.604855 4611 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="38e51b01-a6a7-46fa-bf03-8dd55c611586" path="/var/lib/kubelet/pods/38e51b01-a6a7-46fa-bf03-8dd55c611586/volumes" Oct 30 21:21:49.975129 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:21:49.975039 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:22:03.916463 ip-10-0-133-192 kubenswrapper[4611]: E1030 21:22:03.916427 4611 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hwzc6" containerName="extractor" Oct 30 21:22:04.289562 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:22:04.289449 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:22:08.823128 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:22:08.823094 4611 scope.go:122] "RemoveContainer" containerID="d19560c35a519a4cd8fcf6ec353ffb8ce8e03c5ddb1841486d068e6f56aa90ec" Oct 30 21:22:08.831479 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:22:08.831456 4611 scope.go:122] "RemoveContainer" containerID="b9a5f81fc9ae4042c38007c3ca2e7ad332188ef72e2ed6cd42ffc85979e60fbc" Oct 30 21:22:33.918068 ip-10-0-133-192 kubenswrapper[4611]: E1030 21:22:33.918030 4611 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hwzc6" containerName="extractor" Oct 30 21:22:50.025722 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:22:50.025678 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:23:02.443267 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:23:02.443229 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:23:02.597924 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:23:02.597886 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:23:03.915764 ip-10-0-133-192 kubenswrapper[4611]: E1030 21:23:03.915729 4611 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hwzc6" containerName="extractor" Oct 30 21:23:08.853639 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:23:08.853607 4611 scope.go:122] "RemoveContainer" containerID="af711a701d4897b2c025bc8b9f9d363b34d88cc732a7c7f11d5c27ae52952765" Oct 30 21:23:33.916917 ip-10-0-133-192 kubenswrapper[4611]: E1030 21:23:33.916835 4611 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hwzc6" containerName="extractor" Oct 30 21:23:50.079196 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:23:50.079154 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:24:03.916005 ip-10-0-133-192 kubenswrapper[4611]: E1030 21:24:03.915965 4611 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hwzc6" containerName="extractor" Oct 30 21:24:05.978461 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:24:05.978431 4611 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7w5wz_7e2574c1-b59c-453c-9501-25ec13a1ccad/ovn-acl-logging/0.log" Oct 30 21:24:05.989893 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:24:05.989872 4611 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7w5wz_7e2574c1-b59c-453c-9501-25ec13a1ccad/ovn-acl-logging/0.log" Oct 30 21:24:33.916721 ip-10-0-133-192 kubenswrapper[4611]: E1030 21:24:33.916673 4611 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hwzc6" containerName="extractor" Oct 30 21:24:50.137659 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:24:50.137571 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:24:53.352358 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:24:53.352324 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:25:03.916303 ip-10-0-133-192 kubenswrapper[4611]: E1030 21:25:03.916266 4611 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hwzc6" containerName="extractor" Oct 30 21:25:11.261826 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:25:11.261789 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:25:11.414649 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:25:11.414614 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:25:13.415765 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:25:13.415731 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:25:13.560044 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:25:13.560009 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:25:14.613525 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:25:14.613482 4611 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-857r2_bf21cf19-4475-41c3-9153-44d8fb329cdb/global-pull-secret-syncer/0.log" Oct 30 21:25:14.975875 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:25:14.975789 4611 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" Oct 30 21:25:15.098622 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:25:15.098594 4611 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-133-192.us-east-2.compute.internal_10e325ce488448e0259da3eb69cd60ba/haproxy/0.log" Oct 30 21:25:33.916409 ip-10-0-133-192 kubenswrapper[4611]: E1030 21:25:33.916374 4611 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hwzc6" containerName="extractor" Oct 30 21:25:50.186319 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:25:50.186279 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:25:53.842864 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:25:53.842826 4611 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-tg9jv"] Oct 30 21:25:53.843388 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:25:53.843078 4611 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-tg9jv" podUID="ee56d291-6c05-4674-895a-88f339e18e44" containerName="istio-proxy" containerID="cri-o://289f6b0c2f22dcd6e6d93f4870c65c6516738680641d94232273824ef00a4158" gracePeriod=30 Oct 30 21:25:53.852364 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:25:53.852333 4611 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8tvvl"] Oct 30 21:25:53.852556 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:25:53.852391 4611 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-w44qs"] Oct 30 21:25:53.852556 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:25:53.852548 4611 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-w44qs" podUID="6d6fd4b2-ad0a-4ce3-b5b3-966de34bd8b0" containerName="istio-proxy" containerID="cri-o://294c6631ef8999b7e72bd3c483e1a97ecc657e94a3d9ccc2c0234f5fbc212c1e" gracePeriod=30 Oct 30 21:25:53.852759 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:25:53.852739 4611 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8tvvl" podUID="46ae03da-7a80-4f78-8abd-391ebc697f2f" containerName="istio-proxy" containerID="cri-o://052ae2e95eed9bc6ddfcf31320811ce5a003836a7d00de45f65a7973d480d9bc" gracePeriod=30 Oct 30 21:25:53.854993 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:25:53.854965 4611 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-cwrqt"] Oct 30 21:25:53.855162 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:25:53.855135 4611 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-cwrqt" podUID="5da54d6d-30b1-4c0c-9824-e06f7a4ae0ee" containerName="istio-proxy" containerID="cri-o://9c714c781d32fade35666567542cb168264cfee21b36c83a7f19bf26970f441b" gracePeriod=30 Oct 30 21:25:59.121605 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:25:59.121576 4611 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-w44qs" Oct 30 21:25:59.140451 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:25:59.139930 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d6fd4b2-ad0a-4ce3-b5b3-966de34bd8b0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d6fd4b2-ad0a-4ce3-b5b3-966de34bd8b0-credential-socket\") pod \"6d6fd4b2-ad0a-4ce3-b5b3-966de34bd8b0\" (UID: \"6d6fd4b2-ad0a-4ce3-b5b3-966de34bd8b0\") " Oct 30 21:25:59.140451 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:25:59.140000 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d6fd4b2-ad0a-4ce3-b5b3-966de34bd8b0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d6fd4b2-ad0a-4ce3-b5b3-966de34bd8b0-istio-envoy\") pod \"6d6fd4b2-ad0a-4ce3-b5b3-966de34bd8b0\" (UID: \"6d6fd4b2-ad0a-4ce3-b5b3-966de34bd8b0\") " Oct 30 21:25:59.140451 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:25:59.140056 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d6fd4b2-ad0a-4ce3-b5b3-966de34bd8b0-kube-api-access-qdj6x\" (UniqueName: \"kubernetes.io/projected/6d6fd4b2-ad0a-4ce3-b5b3-966de34bd8b0-kube-api-access-qdj6x\") pod \"6d6fd4b2-ad0a-4ce3-b5b3-966de34bd8b0\" (UID: \"6d6fd4b2-ad0a-4ce3-b5b3-966de34bd8b0\") " Oct 30 21:25:59.140451 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:25:59.140109 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d6fd4b2-ad0a-4ce3-b5b3-966de34bd8b0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d6fd4b2-ad0a-4ce3-b5b3-966de34bd8b0-workload-certs\") pod \"6d6fd4b2-ad0a-4ce3-b5b3-966de34bd8b0\" (UID: \"6d6fd4b2-ad0a-4ce3-b5b3-966de34bd8b0\") " Oct 30 21:25:59.140451 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:25:59.140152 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d6fd4b2-ad0a-4ce3-b5b3-966de34bd8b0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d6fd4b2-ad0a-4ce3-b5b3-966de34bd8b0-workload-socket\") pod \"6d6fd4b2-ad0a-4ce3-b5b3-966de34bd8b0\" (UID: \"6d6fd4b2-ad0a-4ce3-b5b3-966de34bd8b0\") " Oct 30 21:25:59.140451 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:25:59.140194 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6d6fd4b2-ad0a-4ce3-b5b3-966de34bd8b0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d6fd4b2-ad0a-4ce3-b5b3-966de34bd8b0-istiod-ca-cert\") pod \"6d6fd4b2-ad0a-4ce3-b5b3-966de34bd8b0\" (UID: \"6d6fd4b2-ad0a-4ce3-b5b3-966de34bd8b0\") " Oct 30 21:25:59.140451 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:25:59.140225 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d6fd4b2-ad0a-4ce3-b5b3-966de34bd8b0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d6fd4b2-ad0a-4ce3-b5b3-966de34bd8b0-istio-data\") pod \"6d6fd4b2-ad0a-4ce3-b5b3-966de34bd8b0\" (UID: \"6d6fd4b2-ad0a-4ce3-b5b3-966de34bd8b0\") " Oct 30 21:25:59.140451 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:25:59.140270 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d6fd4b2-ad0a-4ce3-b5b3-966de34bd8b0-istio-token\" (UniqueName: \"kubernetes.io/projected/6d6fd4b2-ad0a-4ce3-b5b3-966de34bd8b0-istio-token\") pod \"6d6fd4b2-ad0a-4ce3-b5b3-966de34bd8b0\" (UID: \"6d6fd4b2-ad0a-4ce3-b5b3-966de34bd8b0\") " Oct 30 21:25:59.140451 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:25:59.140303 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6d6fd4b2-ad0a-4ce3-b5b3-966de34bd8b0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d6fd4b2-ad0a-4ce3-b5b3-966de34bd8b0-istio-podinfo\") pod \"6d6fd4b2-ad0a-4ce3-b5b3-966de34bd8b0\" (UID: \"6d6fd4b2-ad0a-4ce3-b5b3-966de34bd8b0\") " Oct 30 21:25:59.143843 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:25:59.140216 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d6fd4b2-ad0a-4ce3-b5b3-966de34bd8b0-credential-socket" pod "6d6fd4b2-ad0a-4ce3-b5b3-966de34bd8b0" (UID: "6d6fd4b2-ad0a-4ce3-b5b3-966de34bd8b0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:25:59.143843 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:25:59.140429 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d6fd4b2-ad0a-4ce3-b5b3-966de34bd8b0-workload-certs" pod "6d6fd4b2-ad0a-4ce3-b5b3-966de34bd8b0" (UID: "6d6fd4b2-ad0a-4ce3-b5b3-966de34bd8b0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:25:59.143843 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:25:59.141592 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d6fd4b2-ad0a-4ce3-b5b3-966de34bd8b0-workload-socket" pod "6d6fd4b2-ad0a-4ce3-b5b3-966de34bd8b0" (UID: "6d6fd4b2-ad0a-4ce3-b5b3-966de34bd8b0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:25:59.143843 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:25:59.141869 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d6fd4b2-ad0a-4ce3-b5b3-966de34bd8b0-istiod-ca-cert" pod "6d6fd4b2-ad0a-4ce3-b5b3-966de34bd8b0" (UID: "6d6fd4b2-ad0a-4ce3-b5b3-966de34bd8b0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:25:59.143843 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:25:59.142092 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d6fd4b2-ad0a-4ce3-b5b3-966de34bd8b0-istio-data" pod "6d6fd4b2-ad0a-4ce3-b5b3-966de34bd8b0" (UID: "6d6fd4b2-ad0a-4ce3-b5b3-966de34bd8b0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:25:59.145385 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:25:59.144343 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6d6fd4b2-ad0a-4ce3-b5b3-966de34bd8b0-istio-podinfo" pod "6d6fd4b2-ad0a-4ce3-b5b3-966de34bd8b0" (UID: "6d6fd4b2-ad0a-4ce3-b5b3-966de34bd8b0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:25:59.153004 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:25:59.152965 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d6fd4b2-ad0a-4ce3-b5b3-966de34bd8b0-istio-envoy" pod "6d6fd4b2-ad0a-4ce3-b5b3-966de34bd8b0" (UID: "6d6fd4b2-ad0a-4ce3-b5b3-966de34bd8b0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:25:59.153587 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:25:59.153561 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d6fd4b2-ad0a-4ce3-b5b3-966de34bd8b0-kube-api-access-qdj6x" pod "6d6fd4b2-ad0a-4ce3-b5b3-966de34bd8b0" (UID: "6d6fd4b2-ad0a-4ce3-b5b3-966de34bd8b0"). InnerVolumeSpecName "kube-api-access-qdj6x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:25:59.153931 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:25:59.153896 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d6fd4b2-ad0a-4ce3-b5b3-966de34bd8b0-istio-token" pod "6d6fd4b2-ad0a-4ce3-b5b3-966de34bd8b0" (UID: "6d6fd4b2-ad0a-4ce3-b5b3-966de34bd8b0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:25:59.160559 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:25:59.160535 4611 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8tvvl" Oct 30 21:25:59.186460 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:25:59.186434 4611 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-cwrqt" Oct 30 21:25:59.197957 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:25:59.197935 4611 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-tg9jv" Oct 30 21:25:59.242002 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:25:59.241975 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46ae03da-7a80-4f78-8abd-391ebc697f2f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46ae03da-7a80-4f78-8abd-391ebc697f2f-istio-data\") pod \"46ae03da-7a80-4f78-8abd-391ebc697f2f\" (UID: \"46ae03da-7a80-4f78-8abd-391ebc697f2f\") " Oct 30 21:25:59.242136 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:25:59.242011 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5da54d6d-30b1-4c0c-9824-e06f7a4ae0ee-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5da54d6d-30b1-4c0c-9824-e06f7a4ae0ee-istio-data\") pod \"5da54d6d-30b1-4c0c-9824-e06f7a4ae0ee\" (UID: \"5da54d6d-30b1-4c0c-9824-e06f7a4ae0ee\") " Oct 30 21:25:59.242136 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:25:59.242039 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee56d291-6c05-4674-895a-88f339e18e44-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee56d291-6c05-4674-895a-88f339e18e44-credential-socket\") pod \"ee56d291-6c05-4674-895a-88f339e18e44\" (UID: \"ee56d291-6c05-4674-895a-88f339e18e44\") " Oct 30 21:25:59.242136 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:25:59.242066 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5da54d6d-30b1-4c0c-9824-e06f7a4ae0ee-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5da54d6d-30b1-4c0c-9824-e06f7a4ae0ee-istiod-ca-cert\") pod \"5da54d6d-30b1-4c0c-9824-e06f7a4ae0ee\" (UID: \"5da54d6d-30b1-4c0c-9824-e06f7a4ae0ee\") " Oct 30 21:25:59.242136 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:25:59.242098 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46ae03da-7a80-4f78-8abd-391ebc697f2f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46ae03da-7a80-4f78-8abd-391ebc697f2f-credential-socket\") pod \"46ae03da-7a80-4f78-8abd-391ebc697f2f\" (UID: \"46ae03da-7a80-4f78-8abd-391ebc697f2f\") " Oct 30 21:25:59.242366 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:25:59.242137 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee56d291-6c05-4674-895a-88f339e18e44-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee56d291-6c05-4674-895a-88f339e18e44-workload-certs\") pod \"ee56d291-6c05-4674-895a-88f339e18e44\" (UID: \"ee56d291-6c05-4674-895a-88f339e18e44\") " Oct 30 21:25:59.242366 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:25:59.242169 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/46ae03da-7a80-4f78-8abd-391ebc697f2f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46ae03da-7a80-4f78-8abd-391ebc697f2f-istiod-ca-cert\") pod \"46ae03da-7a80-4f78-8abd-391ebc697f2f\" (UID: \"46ae03da-7a80-4f78-8abd-391ebc697f2f\") " Oct 30 21:25:59.242366 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:25:59.242194 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee56d291-6c05-4674-895a-88f339e18e44-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee56d291-6c05-4674-895a-88f339e18e44-istio-data\") pod \"ee56d291-6c05-4674-895a-88f339e18e44\" (UID: \"ee56d291-6c05-4674-895a-88f339e18e44\") " Oct 30 21:25:59.242366 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:25:59.242232 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee56d291-6c05-4674-895a-88f339e18e44-kube-api-access-6xxsv\" (UniqueName: \"kubernetes.io/projected/ee56d291-6c05-4674-895a-88f339e18e44-kube-api-access-6xxsv\") pod \"ee56d291-6c05-4674-895a-88f339e18e44\" (UID: \"ee56d291-6c05-4674-895a-88f339e18e44\") " Oct 30 21:25:59.242366 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:25:59.242268 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46ae03da-7a80-4f78-8abd-391ebc697f2f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46ae03da-7a80-4f78-8abd-391ebc697f2f-workload-certs\") pod \"46ae03da-7a80-4f78-8abd-391ebc697f2f\" (UID: \"46ae03da-7a80-4f78-8abd-391ebc697f2f\") " Oct 30 21:25:59.242366 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:25:59.242291 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46ae03da-7a80-4f78-8abd-391ebc697f2f-istio-data" pod "46ae03da-7a80-4f78-8abd-391ebc697f2f" (UID: "46ae03da-7a80-4f78-8abd-391ebc697f2f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:25:59.242366 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:25:59.242299 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5da54d6d-30b1-4c0c-9824-e06f7a4ae0ee-kube-api-access-rxn7c\" (UniqueName: \"kubernetes.io/projected/5da54d6d-30b1-4c0c-9824-e06f7a4ae0ee-kube-api-access-rxn7c\") pod \"5da54d6d-30b1-4c0c-9824-e06f7a4ae0ee\" (UID: \"5da54d6d-30b1-4c0c-9824-e06f7a4ae0ee\") " Oct 30 21:25:59.242366 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:25:59.242320 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5da54d6d-30b1-4c0c-9824-e06f7a4ae0ee-istio-data" pod "5da54d6d-30b1-4c0c-9824-e06f7a4ae0ee" (UID: "5da54d6d-30b1-4c0c-9824-e06f7a4ae0ee"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:25:59.242366 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:25:59.242328 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5da54d6d-30b1-4c0c-9824-e06f7a4ae0ee-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5da54d6d-30b1-4c0c-9824-e06f7a4ae0ee-credential-socket\") pod \"5da54d6d-30b1-4c0c-9824-e06f7a4ae0ee\" (UID: \"5da54d6d-30b1-4c0c-9824-e06f7a4ae0ee\") " Oct 30 21:25:59.242870 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:25:59.242375 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee56d291-6c05-4674-895a-88f339e18e44-credential-socket" pod "ee56d291-6c05-4674-895a-88f339e18e44" (UID: "ee56d291-6c05-4674-895a-88f339e18e44"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:25:59.242870 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:25:59.242404 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee56d291-6c05-4674-895a-88f339e18e44-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee56d291-6c05-4674-895a-88f339e18e44-istio-envoy\") pod \"ee56d291-6c05-4674-895a-88f339e18e44\" (UID: \"ee56d291-6c05-4674-895a-88f339e18e44\") " Oct 30 21:25:59.242870 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:25:59.242453 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5da54d6d-30b1-4c0c-9824-e06f7a4ae0ee-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5da54d6d-30b1-4c0c-9824-e06f7a4ae0ee-istio-envoy\") pod \"5da54d6d-30b1-4c0c-9824-e06f7a4ae0ee\" (UID: \"5da54d6d-30b1-4c0c-9824-e06f7a4ae0ee\") " Oct 30 21:25:59.242870 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:25:59.242546 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee56d291-6c05-4674-895a-88f339e18e44-istio-data" pod "ee56d291-6c05-4674-895a-88f339e18e44" (UID: "ee56d291-6c05-4674-895a-88f339e18e44"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:25:59.242870 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:25:59.242552 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5da54d6d-30b1-4c0c-9824-e06f7a4ae0ee-credential-socket" pod "5da54d6d-30b1-4c0c-9824-e06f7a4ae0ee" (UID: "5da54d6d-30b1-4c0c-9824-e06f7a4ae0ee"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:25:59.242870 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:25:59.242571 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46ae03da-7a80-4f78-8abd-391ebc697f2f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46ae03da-7a80-4f78-8abd-391ebc697f2f-workload-socket\") pod \"46ae03da-7a80-4f78-8abd-391ebc697f2f\" (UID: \"46ae03da-7a80-4f78-8abd-391ebc697f2f\") " Oct 30 21:25:59.242870 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:25:59.242607 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46ae03da-7a80-4f78-8abd-391ebc697f2f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46ae03da-7a80-4f78-8abd-391ebc697f2f-istio-envoy\") pod \"46ae03da-7a80-4f78-8abd-391ebc697f2f\" (UID: \"46ae03da-7a80-4f78-8abd-391ebc697f2f\") " Oct 30 21:25:59.242870 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:25:59.242635 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46ae03da-7a80-4f78-8abd-391ebc697f2f-istio-token\" (UniqueName: \"kubernetes.io/projected/46ae03da-7a80-4f78-8abd-391ebc697f2f-istio-token\") pod \"46ae03da-7a80-4f78-8abd-391ebc697f2f\" (UID: \"46ae03da-7a80-4f78-8abd-391ebc697f2f\") " Oct 30 21:25:59.242870 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:25:59.242583 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5da54d6d-30b1-4c0c-9824-e06f7a4ae0ee-istiod-ca-cert" pod "5da54d6d-30b1-4c0c-9824-e06f7a4ae0ee" (UID: "5da54d6d-30b1-4c0c-9824-e06f7a4ae0ee"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:25:59.242870 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:25:59.242664 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ee56d291-6c05-4674-895a-88f339e18e44-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee56d291-6c05-4674-895a-88f339e18e44-istio-podinfo\") pod \"ee56d291-6c05-4674-895a-88f339e18e44\" (UID: \"ee56d291-6c05-4674-895a-88f339e18e44\") " Oct 30 21:25:59.242870 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:25:59.242696 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee56d291-6c05-4674-895a-88f339e18e44-istio-token\" (UniqueName: \"kubernetes.io/projected/ee56d291-6c05-4674-895a-88f339e18e44-istio-token\") pod \"ee56d291-6c05-4674-895a-88f339e18e44\" (UID: \"ee56d291-6c05-4674-895a-88f339e18e44\") " Oct 30 21:25:59.242870 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:25:59.242733 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46ae03da-7a80-4f78-8abd-391ebc697f2f-kube-api-access-lvrsc\" (UniqueName: \"kubernetes.io/projected/46ae03da-7a80-4f78-8abd-391ebc697f2f-kube-api-access-lvrsc\") pod \"46ae03da-7a80-4f78-8abd-391ebc697f2f\" (UID: \"46ae03da-7a80-4f78-8abd-391ebc697f2f\") " Oct 30 21:25:59.242870 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:25:59.242768 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5da54d6d-30b1-4c0c-9824-e06f7a4ae0ee-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5da54d6d-30b1-4c0c-9824-e06f7a4ae0ee-workload-socket\") pod \"5da54d6d-30b1-4c0c-9824-e06f7a4ae0ee\" (UID: \"5da54d6d-30b1-4c0c-9824-e06f7a4ae0ee\") " Oct 30 21:25:59.242870 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:25:59.242803 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/46ae03da-7a80-4f78-8abd-391ebc697f2f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46ae03da-7a80-4f78-8abd-391ebc697f2f-istio-podinfo\") pod \"46ae03da-7a80-4f78-8abd-391ebc697f2f\" (UID: \"46ae03da-7a80-4f78-8abd-391ebc697f2f\") " Oct 30 21:25:59.242870 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:25:59.242832 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5da54d6d-30b1-4c0c-9824-e06f7a4ae0ee-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5da54d6d-30b1-4c0c-9824-e06f7a4ae0ee-istio-podinfo\") pod \"5da54d6d-30b1-4c0c-9824-e06f7a4ae0ee\" (UID: \"5da54d6d-30b1-4c0c-9824-e06f7a4ae0ee\") " Oct 30 21:25:59.242870 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:25:59.242867 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee56d291-6c05-4674-895a-88f339e18e44-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee56d291-6c05-4674-895a-88f339e18e44-workload-socket\") pod \"ee56d291-6c05-4674-895a-88f339e18e44\" (UID: \"ee56d291-6c05-4674-895a-88f339e18e44\") " Oct 30 21:25:59.243816 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:25:59.242929 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5da54d6d-30b1-4c0c-9824-e06f7a4ae0ee-istio-token\" (UniqueName: \"kubernetes.io/projected/5da54d6d-30b1-4c0c-9824-e06f7a4ae0ee-istio-token\") pod \"5da54d6d-30b1-4c0c-9824-e06f7a4ae0ee\" (UID: \"5da54d6d-30b1-4c0c-9824-e06f7a4ae0ee\") " Oct 30 21:25:59.243816 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:25:59.242977 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5da54d6d-30b1-4c0c-9824-e06f7a4ae0ee-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5da54d6d-30b1-4c0c-9824-e06f7a4ae0ee-workload-certs\") pod \"5da54d6d-30b1-4c0c-9824-e06f7a4ae0ee\" (UID: \"5da54d6d-30b1-4c0c-9824-e06f7a4ae0ee\") " Oct 30 21:25:59.243816 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:25:59.243015 4611 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ee56d291-6c05-4674-895a-88f339e18e44-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee56d291-6c05-4674-895a-88f339e18e44-istiod-ca-cert\") pod \"ee56d291-6c05-4674-895a-88f339e18e44\" (UID: \"ee56d291-6c05-4674-895a-88f339e18e44\") " Oct 30 21:25:59.243816 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:25:59.242664 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee56d291-6c05-4674-895a-88f339e18e44-workload-certs" pod "ee56d291-6c05-4674-895a-88f339e18e44" (UID: "ee56d291-6c05-4674-895a-88f339e18e44"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:25:59.243816 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:25:59.243437 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee56d291-6c05-4674-895a-88f339e18e44-istiod-ca-cert" pod "ee56d291-6c05-4674-895a-88f339e18e44" (UID: "ee56d291-6c05-4674-895a-88f339e18e44"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:25:59.243816 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:25:59.243441 4611 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d6fd4b2-ad0a-4ce3-b5b3-966de34bd8b0-workload-socket\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:25:59.243816 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:25:59.243479 4611 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d6fd4b2-ad0a-4ce3-b5b3-966de34bd8b0-istiod-ca-cert\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:25:59.243816 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:25:59.243494 4611 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d6fd4b2-ad0a-4ce3-b5b3-966de34bd8b0-istio-data\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:25:59.243816 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:25:59.243527 4611 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d6fd4b2-ad0a-4ce3-b5b3-966de34bd8b0-istio-token\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:25:59.243816 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:25:59.243543 4611 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d6fd4b2-ad0a-4ce3-b5b3-966de34bd8b0-istio-podinfo\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:25:59.243816 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:25:59.243558 4611 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d6fd4b2-ad0a-4ce3-b5b3-966de34bd8b0-credential-socket\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:25:59.243816 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:25:59.243572 4611 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46ae03da-7a80-4f78-8abd-391ebc697f2f-istio-data\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:25:59.243816 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:25:59.243586 4611 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5da54d6d-30b1-4c0c-9824-e06f7a4ae0ee-istio-data\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:25:59.243816 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:25:59.243599 4611 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee56d291-6c05-4674-895a-88f339e18e44-credential-socket\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:25:59.243816 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:25:59.243612 4611 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5da54d6d-30b1-4c0c-9824-e06f7a4ae0ee-istiod-ca-cert\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:25:59.243816 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:25:59.243613 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5da54d6d-30b1-4c0c-9824-e06f7a4ae0ee-workload-socket" pod "5da54d6d-30b1-4c0c-9824-e06f7a4ae0ee" (UID: "5da54d6d-30b1-4c0c-9824-e06f7a4ae0ee"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:25:59.243816 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:25:59.243626 4611 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d6fd4b2-ad0a-4ce3-b5b3-966de34bd8b0-istio-envoy\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:25:59.243816 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:25:59.243640 4611 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee56d291-6c05-4674-895a-88f339e18e44-workload-certs\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:25:59.243816 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:25:59.243655 4611 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee56d291-6c05-4674-895a-88f339e18e44-istio-data\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:25:59.243816 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:25:59.243678 4611 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qdj6x\" (UniqueName: \"kubernetes.io/projected/6d6fd4b2-ad0a-4ce3-b5b3-966de34bd8b0-kube-api-access-qdj6x\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:25:59.243816 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:25:59.243696 4611 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5da54d6d-30b1-4c0c-9824-e06f7a4ae0ee-credential-socket\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:25:59.243816 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:25:59.243710 4611 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d6fd4b2-ad0a-4ce3-b5b3-966de34bd8b0-workload-certs\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:25:59.244983 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:25:59.243828 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee56d291-6c05-4674-895a-88f339e18e44-workload-socket" pod "ee56d291-6c05-4674-895a-88f339e18e44" (UID: "ee56d291-6c05-4674-895a-88f339e18e44"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:25:59.244983 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:25:59.243973 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5da54d6d-30b1-4c0c-9824-e06f7a4ae0ee-workload-certs" pod "5da54d6d-30b1-4c0c-9824-e06f7a4ae0ee" (UID: "5da54d6d-30b1-4c0c-9824-e06f7a4ae0ee"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:25:59.244983 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:25:59.242816 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46ae03da-7a80-4f78-8abd-391ebc697f2f-workload-certs" pod "46ae03da-7a80-4f78-8abd-391ebc697f2f" (UID: "46ae03da-7a80-4f78-8abd-391ebc697f2f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:25:59.244983 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:25:59.242883 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46ae03da-7a80-4f78-8abd-391ebc697f2f-istiod-ca-cert" pod "46ae03da-7a80-4f78-8abd-391ebc697f2f" (UID: "46ae03da-7a80-4f78-8abd-391ebc697f2f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:25:59.244983 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:25:59.243091 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46ae03da-7a80-4f78-8abd-391ebc697f2f-credential-socket" pod "46ae03da-7a80-4f78-8abd-391ebc697f2f" (UID: "46ae03da-7a80-4f78-8abd-391ebc697f2f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:25:59.244983 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:25:59.243113 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46ae03da-7a80-4f78-8abd-391ebc697f2f-workload-socket" pod "46ae03da-7a80-4f78-8abd-391ebc697f2f" (UID: "46ae03da-7a80-4f78-8abd-391ebc697f2f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:25:59.245496 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:25:59.245464 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46ae03da-7a80-4f78-8abd-391ebc697f2f-istio-envoy" pod "46ae03da-7a80-4f78-8abd-391ebc697f2f" (UID: "46ae03da-7a80-4f78-8abd-391ebc697f2f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:25:59.245925 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:25:59.245899 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee56d291-6c05-4674-895a-88f339e18e44-istio-envoy" pod "ee56d291-6c05-4674-895a-88f339e18e44" (UID: "ee56d291-6c05-4674-895a-88f339e18e44"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:25:59.246030 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:25:59.245973 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee56d291-6c05-4674-895a-88f339e18e44-kube-api-access-6xxsv" pod "ee56d291-6c05-4674-895a-88f339e18e44" (UID: "ee56d291-6c05-4674-895a-88f339e18e44"). InnerVolumeSpecName "kube-api-access-6xxsv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:25:59.246631 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:25:59.246602 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5da54d6d-30b1-4c0c-9824-e06f7a4ae0ee-istio-podinfo" pod "5da54d6d-30b1-4c0c-9824-e06f7a4ae0ee" (UID: "5da54d6d-30b1-4c0c-9824-e06f7a4ae0ee"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:25:59.247868 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:25:59.247833 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5da54d6d-30b1-4c0c-9824-e06f7a4ae0ee-istio-token" pod "5da54d6d-30b1-4c0c-9824-e06f7a4ae0ee" (UID: "5da54d6d-30b1-4c0c-9824-e06f7a4ae0ee"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:25:59.248205 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:25:59.247868 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46ae03da-7a80-4f78-8abd-391ebc697f2f-istio-token" pod "46ae03da-7a80-4f78-8abd-391ebc697f2f" (UID: "46ae03da-7a80-4f78-8abd-391ebc697f2f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:25:59.248205 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:25:59.247927 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46ae03da-7a80-4f78-8abd-391ebc697f2f-kube-api-access-lvrsc" pod "46ae03da-7a80-4f78-8abd-391ebc697f2f" (UID: "46ae03da-7a80-4f78-8abd-391ebc697f2f"). InnerVolumeSpecName "kube-api-access-lvrsc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:25:59.248205 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:25:59.247996 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee56d291-6c05-4674-895a-88f339e18e44-istio-token" pod "ee56d291-6c05-4674-895a-88f339e18e44" (UID: "ee56d291-6c05-4674-895a-88f339e18e44"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:25:59.248205 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:25:59.248164 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5da54d6d-30b1-4c0c-9824-e06f7a4ae0ee-kube-api-access-rxn7c" pod "5da54d6d-30b1-4c0c-9824-e06f7a4ae0ee" (UID: "5da54d6d-30b1-4c0c-9824-e06f7a4ae0ee"). InnerVolumeSpecName "kube-api-access-rxn7c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:25:59.248205 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:25:59.248169 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/46ae03da-7a80-4f78-8abd-391ebc697f2f-istio-podinfo" pod "46ae03da-7a80-4f78-8abd-391ebc697f2f" (UID: "46ae03da-7a80-4f78-8abd-391ebc697f2f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:25:59.248205 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:25:59.248192 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5da54d6d-30b1-4c0c-9824-e06f7a4ae0ee-istio-envoy" pod "5da54d6d-30b1-4c0c-9824-e06f7a4ae0ee" (UID: "5da54d6d-30b1-4c0c-9824-e06f7a4ae0ee"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:25:59.249672 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:25:59.249656 4611 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ee56d291-6c05-4674-895a-88f339e18e44-istio-podinfo" pod "ee56d291-6c05-4674-895a-88f339e18e44" (UID: "ee56d291-6c05-4674-895a-88f339e18e44"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:25:59.344835 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:25:59.344803 4611 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46ae03da-7a80-4f78-8abd-391ebc697f2f-istio-envoy\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:25:59.344835 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:25:59.344828 4611 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46ae03da-7a80-4f78-8abd-391ebc697f2f-istio-token\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:25:59.344835 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:25:59.344838 4611 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee56d291-6c05-4674-895a-88f339e18e44-istio-podinfo\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:25:59.345036 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:25:59.344846 4611 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee56d291-6c05-4674-895a-88f339e18e44-istio-token\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:25:59.345036 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:25:59.344856 4611 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lvrsc\" (UniqueName: \"kubernetes.io/projected/46ae03da-7a80-4f78-8abd-391ebc697f2f-kube-api-access-lvrsc\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:25:59.345036 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:25:59.344865 4611 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5da54d6d-30b1-4c0c-9824-e06f7a4ae0ee-workload-socket\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:25:59.345036 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:25:59.344874 4611 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46ae03da-7a80-4f78-8abd-391ebc697f2f-istio-podinfo\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:25:59.345036 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:25:59.344884 4611 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5da54d6d-30b1-4c0c-9824-e06f7a4ae0ee-istio-podinfo\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:25:59.345036 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:25:59.344892 4611 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee56d291-6c05-4674-895a-88f339e18e44-workload-socket\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:25:59.345036 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:25:59.344901 4611 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5da54d6d-30b1-4c0c-9824-e06f7a4ae0ee-istio-token\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:25:59.345036 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:25:59.344909 4611 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5da54d6d-30b1-4c0c-9824-e06f7a4ae0ee-workload-certs\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:25:59.345036 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:25:59.344918 4611 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee56d291-6c05-4674-895a-88f339e18e44-istiod-ca-cert\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:25:59.345036 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:25:59.344926 4611 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46ae03da-7a80-4f78-8abd-391ebc697f2f-credential-socket\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:25:59.345036 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:25:59.344934 4611 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46ae03da-7a80-4f78-8abd-391ebc697f2f-istiod-ca-cert\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:25:59.345036 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:25:59.344942 4611 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6xxsv\" (UniqueName: \"kubernetes.io/projected/ee56d291-6c05-4674-895a-88f339e18e44-kube-api-access-6xxsv\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:25:59.345036 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:25:59.344950 4611 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46ae03da-7a80-4f78-8abd-391ebc697f2f-workload-certs\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:25:59.345036 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:25:59.344957 4611 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rxn7c\" (UniqueName: \"kubernetes.io/projected/5da54d6d-30b1-4c0c-9824-e06f7a4ae0ee-kube-api-access-rxn7c\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:25:59.345036 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:25:59.344965 4611 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee56d291-6c05-4674-895a-88f339e18e44-istio-envoy\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:25:59.345036 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:25:59.344973 4611 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5da54d6d-30b1-4c0c-9824-e06f7a4ae0ee-istio-envoy\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:25:59.345036 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:25:59.344982 4611 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46ae03da-7a80-4f78-8abd-391ebc697f2f-workload-socket\") on node \"ip-10-0-133-192.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:25:59.641948 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:25:59.641918 4611 generic.go:356] "Generic (PLEG): container finished" podID="6d6fd4b2-ad0a-4ce3-b5b3-966de34bd8b0" containerID="294c6631ef8999b7e72bd3c483e1a97ecc657e94a3d9ccc2c0234f5fbc212c1e" exitCode=0 Oct 30 21:25:59.642061 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:25:59.641972 4611 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-w44qs" Oct 30 21:25:59.642061 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:25:59.642002 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-w44qs" event={"ID":"6d6fd4b2-ad0a-4ce3-b5b3-966de34bd8b0","Type":"ContainerDied","Data":"294c6631ef8999b7e72bd3c483e1a97ecc657e94a3d9ccc2c0234f5fbc212c1e"} Oct 30 21:25:59.642152 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:25:59.642078 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-w44qs" event={"ID":"6d6fd4b2-ad0a-4ce3-b5b3-966de34bd8b0","Type":"ContainerDied","Data":"3c4088239c57821501aa98ef69e2f7811ae80690b15b5844c65bdbd8df07c35a"} Oct 30 21:25:59.642152 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:25:59.642106 4611 scope.go:122] "RemoveContainer" containerID="294c6631ef8999b7e72bd3c483e1a97ecc657e94a3d9ccc2c0234f5fbc212c1e" Oct 30 21:25:59.643465 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:25:59.643441 4611 generic.go:356] "Generic (PLEG): container finished" podID="5da54d6d-30b1-4c0c-9824-e06f7a4ae0ee" containerID="9c714c781d32fade35666567542cb168264cfee21b36c83a7f19bf26970f441b" exitCode=0 Oct 30 21:25:59.643601 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:25:59.643519 4611 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-cwrqt" Oct 30 21:25:59.643601 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:25:59.643533 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-cwrqt" event={"ID":"5da54d6d-30b1-4c0c-9824-e06f7a4ae0ee","Type":"ContainerDied","Data":"9c714c781d32fade35666567542cb168264cfee21b36c83a7f19bf26970f441b"} Oct 30 21:25:59.643601 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:25:59.643585 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-cwrqt" event={"ID":"5da54d6d-30b1-4c0c-9824-e06f7a4ae0ee","Type":"ContainerDied","Data":"fdbf7de60cf0ba663da5d4339c441692ff5dba593481f701014d1efd8f9776ca"} Oct 30 21:25:59.645139 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:25:59.645114 4611 generic.go:356] "Generic (PLEG): container finished" podID="46ae03da-7a80-4f78-8abd-391ebc697f2f" containerID="052ae2e95eed9bc6ddfcf31320811ce5a003836a7d00de45f65a7973d480d9bc" exitCode=0 Oct 30 21:25:59.645259 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:25:59.645158 4611 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8tvvl" Oct 30 21:25:59.645259 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:25:59.645218 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8tvvl" event={"ID":"46ae03da-7a80-4f78-8abd-391ebc697f2f","Type":"ContainerDied","Data":"052ae2e95eed9bc6ddfcf31320811ce5a003836a7d00de45f65a7973d480d9bc"} Oct 30 21:25:59.645259 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:25:59.645253 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8tvvl" event={"ID":"46ae03da-7a80-4f78-8abd-391ebc697f2f","Type":"ContainerDied","Data":"61c7a3fe4fc7b7eb860da5962197197912379df9ccb1dc5e4034f16d6995cf25"} Oct 30 21:25:59.646904 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:25:59.646884 4611 generic.go:356] "Generic (PLEG): container finished" podID="ee56d291-6c05-4674-895a-88f339e18e44" containerID="289f6b0c2f22dcd6e6d93f4870c65c6516738680641d94232273824ef00a4158" exitCode=0 Oct 30 21:25:59.646996 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:25:59.646911 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-tg9jv" event={"ID":"ee56d291-6c05-4674-895a-88f339e18e44","Type":"ContainerDied","Data":"289f6b0c2f22dcd6e6d93f4870c65c6516738680641d94232273824ef00a4158"} Oct 30 21:25:59.646996 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:25:59.646940 4611 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-tg9jv" Oct 30 21:25:59.647108 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:25:59.646945 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-tg9jv" event={"ID":"ee56d291-6c05-4674-895a-88f339e18e44","Type":"ContainerDied","Data":"3119a90975846ee6602ff878b35a462eedb94cb1507167a3f57921f1741c9b77"} Oct 30 21:25:59.652482 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:25:59.652466 4611 scope.go:122] "RemoveContainer" containerID="294c6631ef8999b7e72bd3c483e1a97ecc657e94a3d9ccc2c0234f5fbc212c1e" Oct 30 21:25:59.652758 ip-10-0-133-192 kubenswrapper[4611]: E1030 21:25:59.652739 4611 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"294c6631ef8999b7e72bd3c483e1a97ecc657e94a3d9ccc2c0234f5fbc212c1e\": container with ID starting with 294c6631ef8999b7e72bd3c483e1a97ecc657e94a3d9ccc2c0234f5fbc212c1e not found: ID does not exist" containerID="294c6631ef8999b7e72bd3c483e1a97ecc657e94a3d9ccc2c0234f5fbc212c1e" Oct 30 21:25:59.652823 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:25:59.652766 4611 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"294c6631ef8999b7e72bd3c483e1a97ecc657e94a3d9ccc2c0234f5fbc212c1e"} err="failed to get container status \"294c6631ef8999b7e72bd3c483e1a97ecc657e94a3d9ccc2c0234f5fbc212c1e\": rpc error: code = NotFound desc = could not find container \"294c6631ef8999b7e72bd3c483e1a97ecc657e94a3d9ccc2c0234f5fbc212c1e\": container with ID starting with 294c6631ef8999b7e72bd3c483e1a97ecc657e94a3d9ccc2c0234f5fbc212c1e not found: ID does not exist" Oct 30 21:25:59.652823 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:25:59.652781 4611 scope.go:122] "RemoveContainer" containerID="9c714c781d32fade35666567542cb168264cfee21b36c83a7f19bf26970f441b" Oct 30 21:25:59.660430 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:25:59.660407 4611 scope.go:122] "RemoveContainer" containerID="9c714c781d32fade35666567542cb168264cfee21b36c83a7f19bf26970f441b" Oct 30 21:25:59.660701 ip-10-0-133-192 kubenswrapper[4611]: E1030 21:25:59.660683 4611 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9c714c781d32fade35666567542cb168264cfee21b36c83a7f19bf26970f441b\": container with ID starting with 9c714c781d32fade35666567542cb168264cfee21b36c83a7f19bf26970f441b not found: ID does not exist" containerID="9c714c781d32fade35666567542cb168264cfee21b36c83a7f19bf26970f441b" Oct 30 21:25:59.660778 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:25:59.660707 4611 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c714c781d32fade35666567542cb168264cfee21b36c83a7f19bf26970f441b"} err="failed to get container status \"9c714c781d32fade35666567542cb168264cfee21b36c83a7f19bf26970f441b\": rpc error: code = NotFound desc = could not find container \"9c714c781d32fade35666567542cb168264cfee21b36c83a7f19bf26970f441b\": container with ID starting with 9c714c781d32fade35666567542cb168264cfee21b36c83a7f19bf26970f441b not found: ID does not exist" Oct 30 21:25:59.660778 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:25:59.660721 4611 scope.go:122] "RemoveContainer" containerID="052ae2e95eed9bc6ddfcf31320811ce5a003836a7d00de45f65a7973d480d9bc" Oct 30 21:25:59.664399 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:25:59.664379 4611 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8tvvl"] Oct 30 21:25:59.668131 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:25:59.668107 4611 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8tvvl"] Oct 30 21:25:59.668721 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:25:59.668704 4611 scope.go:122] "RemoveContainer" containerID="052ae2e95eed9bc6ddfcf31320811ce5a003836a7d00de45f65a7973d480d9bc" Oct 30 21:25:59.668996 ip-10-0-133-192 kubenswrapper[4611]: E1030 21:25:59.668975 4611 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"052ae2e95eed9bc6ddfcf31320811ce5a003836a7d00de45f65a7973d480d9bc\": container with ID starting with 052ae2e95eed9bc6ddfcf31320811ce5a003836a7d00de45f65a7973d480d9bc not found: ID does not exist" containerID="052ae2e95eed9bc6ddfcf31320811ce5a003836a7d00de45f65a7973d480d9bc" Oct 30 21:25:59.669081 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:25:59.669000 4611 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"052ae2e95eed9bc6ddfcf31320811ce5a003836a7d00de45f65a7973d480d9bc"} err="failed to get container status \"052ae2e95eed9bc6ddfcf31320811ce5a003836a7d00de45f65a7973d480d9bc\": rpc error: code = NotFound desc = could not find container \"052ae2e95eed9bc6ddfcf31320811ce5a003836a7d00de45f65a7973d480d9bc\": container with ID starting with 052ae2e95eed9bc6ddfcf31320811ce5a003836a7d00de45f65a7973d480d9bc not found: ID does not exist" Oct 30 21:25:59.669081 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:25:59.669015 4611 scope.go:122] "RemoveContainer" containerID="289f6b0c2f22dcd6e6d93f4870c65c6516738680641d94232273824ef00a4158" Oct 30 21:25:59.676888 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:25:59.676870 4611 scope.go:122] "RemoveContainer" containerID="289f6b0c2f22dcd6e6d93f4870c65c6516738680641d94232273824ef00a4158" Oct 30 21:25:59.677160 ip-10-0-133-192 kubenswrapper[4611]: E1030 21:25:59.677143 4611 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"289f6b0c2f22dcd6e6d93f4870c65c6516738680641d94232273824ef00a4158\": container with ID starting with 289f6b0c2f22dcd6e6d93f4870c65c6516738680641d94232273824ef00a4158 not found: ID does not exist" containerID="289f6b0c2f22dcd6e6d93f4870c65c6516738680641d94232273824ef00a4158" Oct 30 21:25:59.677215 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:25:59.677167 4611 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"289f6b0c2f22dcd6e6d93f4870c65c6516738680641d94232273824ef00a4158"} err="failed to get container status \"289f6b0c2f22dcd6e6d93f4870c65c6516738680641d94232273824ef00a4158\": rpc error: code = NotFound desc = could not find container \"289f6b0c2f22dcd6e6d93f4870c65c6516738680641d94232273824ef00a4158\": container with ID starting with 289f6b0c2f22dcd6e6d93f4870c65c6516738680641d94232273824ef00a4158 not found: ID does not exist" Oct 30 21:25:59.678255 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:25:59.678237 4611 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-w44qs"] Oct 30 21:25:59.684099 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:25:59.684078 4611 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-w44qs"] Oct 30 21:25:59.694627 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:25:59.694607 4611 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-cwrqt"] Oct 30 21:25:59.698639 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:25:59.698619 4611 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-cwrqt"] Oct 30 21:25:59.709995 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:25:59.709971 4611 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-tg9jv"] Oct 30 21:25:59.711542 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:25:59.711495 4611 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-tg9jv"] Oct 30 21:26:01.604004 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:26:01.603972 4611 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="46ae03da-7a80-4f78-8abd-391ebc697f2f" path="/var/lib/kubelet/pods/46ae03da-7a80-4f78-8abd-391ebc697f2f/volumes" Oct 30 21:26:01.604394 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:26:01.604381 4611 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5da54d6d-30b1-4c0c-9824-e06f7a4ae0ee" path="/var/lib/kubelet/pods/5da54d6d-30b1-4c0c-9824-e06f7a4ae0ee/volumes" Oct 30 21:26:01.604791 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:26:01.604777 4611 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6d6fd4b2-ad0a-4ce3-b5b3-966de34bd8b0" path="/var/lib/kubelet/pods/6d6fd4b2-ad0a-4ce3-b5b3-966de34bd8b0/volumes" Oct 30 21:26:01.605475 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:26:01.605461 4611 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ee56d291-6c05-4674-895a-88f339e18e44" path="/var/lib/kubelet/pods/ee56d291-6c05-4674-895a-88f339e18e44/volumes" Oct 30 21:26:03.916478 ip-10-0-133-192 kubenswrapper[4611]: E1030 21:26:03.916445 4611 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hwzc6" containerName="extractor" Oct 30 21:26:11.083935 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:26:11.083902 4611 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-2hgxj/perf-node-gather-daemonset-859k5"] Oct 30 21:26:11.084359 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:26:11.084239 4611 cpu_manager.go:414] "containerMap: removing container" podUID="35be0c3e-0dce-47c2-8bb8-d0d82d75ddb3" containerName="istio-proxy" Oct 30 21:26:11.084359 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:26:11.084250 4611 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="35be0c3e-0dce-47c2-8bb8-d0d82d75ddb3" containerName="istio-proxy" Oct 30 21:26:11.084359 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:26:11.084263 4611 cpu_manager.go:414] "containerMap: removing container" podUID="46ae03da-7a80-4f78-8abd-391ebc697f2f" containerName="istio-proxy" Oct 30 21:26:11.084359 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:26:11.084269 4611 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46ae03da-7a80-4f78-8abd-391ebc697f2f" containerName="istio-proxy" Oct 30 21:26:11.084359 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:26:11.084279 4611 cpu_manager.go:414] "containerMap: removing container" podUID="f4155e16-c4c3-48c9-b05f-85fe3a094805" containerName="payload-pre-processing" Oct 30 21:26:11.084359 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:26:11.084284 4611 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f4155e16-c4c3-48c9-b05f-85fe3a094805" containerName="payload-pre-processing" Oct 30 21:26:11.084359 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:26:11.084291 4611 cpu_manager.go:414] "containerMap: removing container" podUID="ee56d291-6c05-4674-895a-88f339e18e44" containerName="istio-proxy" Oct 30 21:26:11.084359 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:26:11.084296 4611 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee56d291-6c05-4674-895a-88f339e18e44" containerName="istio-proxy" Oct 30 21:26:11.084359 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:26:11.084310 4611 cpu_manager.go:414] "containerMap: removing container" podUID="5da54d6d-30b1-4c0c-9824-e06f7a4ae0ee" containerName="istio-proxy" Oct 30 21:26:11.084359 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:26:11.084315 4611 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5da54d6d-30b1-4c0c-9824-e06f7a4ae0ee" containerName="istio-proxy" Oct 30 21:26:11.084359 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:26:11.084339 4611 cpu_manager.go:414] "containerMap: removing container" podUID="25df30b0-86a7-4259-b29f-fc9c9d01c268" containerName="payload-processing" Oct 30 21:26:11.084359 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:26:11.084344 4611 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="25df30b0-86a7-4259-b29f-fc9c9d01c268" containerName="payload-processing" Oct 30 21:26:11.084359 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:26:11.084350 4611 cpu_manager.go:414] "containerMap: removing container" podUID="6d6fd4b2-ad0a-4ce3-b5b3-966de34bd8b0" containerName="istio-proxy" Oct 30 21:26:11.084359 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:26:11.084354 4611 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d6fd4b2-ad0a-4ce3-b5b3-966de34bd8b0" containerName="istio-proxy" Oct 30 21:26:11.084847 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:26:11.084414 4611 memory_manager.go:365] "RemoveStaleState removing state" podUID="25df30b0-86a7-4259-b29f-fc9c9d01c268" containerName="payload-processing" Oct 30 21:26:11.084847 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:26:11.084422 4611 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d6fd4b2-ad0a-4ce3-b5b3-966de34bd8b0" containerName="istio-proxy" Oct 30 21:26:11.084847 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:26:11.084428 4611 memory_manager.go:365] "RemoveStaleState removing state" podUID="35be0c3e-0dce-47c2-8bb8-d0d82d75ddb3" containerName="istio-proxy" Oct 30 21:26:11.084847 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:26:11.084434 4611 memory_manager.go:365] "RemoveStaleState removing state" podUID="46ae03da-7a80-4f78-8abd-391ebc697f2f" containerName="istio-proxy" Oct 30 21:26:11.084847 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:26:11.084442 4611 memory_manager.go:365] "RemoveStaleState removing state" podUID="f4155e16-c4c3-48c9-b05f-85fe3a094805" containerName="payload-pre-processing" Oct 30 21:26:11.084847 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:26:11.084447 4611 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee56d291-6c05-4674-895a-88f339e18e44" containerName="istio-proxy" Oct 30 21:26:11.084847 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:26:11.084454 4611 memory_manager.go:365] "RemoveStaleState removing state" podUID="5da54d6d-30b1-4c0c-9824-e06f7a4ae0ee" containerName="istio-proxy" Oct 30 21:26:11.088943 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:26:11.088923 4611 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-2hgxj/perf-node-gather-daemonset-859k5" Oct 30 21:26:11.099133 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:26:11.099096 4611 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-2hgxj/perf-node-gather-daemonset-859k5"] Oct 30 21:26:11.144289 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:26:11.144264 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/a354ba71-3aea-4150-9ded-dcf285deb1ab-proc\") pod \"perf-node-gather-daemonset-859k5\" (UID: \"a354ba71-3aea-4150-9ded-dcf285deb1ab\") " pod="openshift-must-gather-2hgxj/perf-node-gather-daemonset-859k5" Oct 30 21:26:11.144423 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:26:11.144301 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a354ba71-3aea-4150-9ded-dcf285deb1ab-sys\") pod \"perf-node-gather-daemonset-859k5\" (UID: \"a354ba71-3aea-4150-9ded-dcf285deb1ab\") " pod="openshift-must-gather-2hgxj/perf-node-gather-daemonset-859k5" Oct 30 21:26:11.144423 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:26:11.144374 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a354ba71-3aea-4150-9ded-dcf285deb1ab-lib-modules\") pod \"perf-node-gather-daemonset-859k5\" (UID: \"a354ba71-3aea-4150-9ded-dcf285deb1ab\") " pod="openshift-must-gather-2hgxj/perf-node-gather-daemonset-859k5" Oct 30 21:26:11.144423 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:26:11.144410 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/a354ba71-3aea-4150-9ded-dcf285deb1ab-podres\") pod \"perf-node-gather-daemonset-859k5\" (UID: \"a354ba71-3aea-4150-9ded-dcf285deb1ab\") " pod="openshift-must-gather-2hgxj/perf-node-gather-daemonset-859k5" Oct 30 21:26:11.144631 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:26:11.144436 4611 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hx7dn\" (UniqueName: \"kubernetes.io/projected/a354ba71-3aea-4150-9ded-dcf285deb1ab-kube-api-access-hx7dn\") pod \"perf-node-gather-daemonset-859k5\" (UID: \"a354ba71-3aea-4150-9ded-dcf285deb1ab\") " pod="openshift-must-gather-2hgxj/perf-node-gather-daemonset-859k5" Oct 30 21:26:11.185482 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:26:11.185456 4611 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-2hgxj\"/\"kube-root-ca.crt\"" Oct 30 21:26:11.185482 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:26:11.185468 4611 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-2hgxj\"/\"openshift-service-ca.crt\"" Oct 30 21:26:11.186569 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:26:11.186554 4611 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-2hgxj\"/\"default-dockercfg-z7szz\"" Oct 30 21:26:11.245756 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:26:11.245736 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a354ba71-3aea-4150-9ded-dcf285deb1ab-lib-modules\") pod \"perf-node-gather-daemonset-859k5\" (UID: \"a354ba71-3aea-4150-9ded-dcf285deb1ab\") " pod="openshift-must-gather-2hgxj/perf-node-gather-daemonset-859k5" Oct 30 21:26:11.245876 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:26:11.245771 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/a354ba71-3aea-4150-9ded-dcf285deb1ab-podres\") pod \"perf-node-gather-daemonset-859k5\" (UID: \"a354ba71-3aea-4150-9ded-dcf285deb1ab\") " pod="openshift-must-gather-2hgxj/perf-node-gather-daemonset-859k5" Oct 30 21:26:11.245876 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:26:11.245798 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hx7dn\" (UniqueName: \"kubernetes.io/projected/a354ba71-3aea-4150-9ded-dcf285deb1ab-kube-api-access-hx7dn\") pod \"perf-node-gather-daemonset-859k5\" (UID: \"a354ba71-3aea-4150-9ded-dcf285deb1ab\") " pod="openshift-must-gather-2hgxj/perf-node-gather-daemonset-859k5" Oct 30 21:26:11.245876 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:26:11.245828 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/a354ba71-3aea-4150-9ded-dcf285deb1ab-proc\") pod \"perf-node-gather-daemonset-859k5\" (UID: \"a354ba71-3aea-4150-9ded-dcf285deb1ab\") " pod="openshift-must-gather-2hgxj/perf-node-gather-daemonset-859k5" Oct 30 21:26:11.245876 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:26:11.245849 4611 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a354ba71-3aea-4150-9ded-dcf285deb1ab-sys\") pod \"perf-node-gather-daemonset-859k5\" (UID: \"a354ba71-3aea-4150-9ded-dcf285deb1ab\") " pod="openshift-must-gather-2hgxj/perf-node-gather-daemonset-859k5" Oct 30 21:26:11.245876 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:26:11.245868 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/a354ba71-3aea-4150-9ded-dcf285deb1ab-podres\") pod \"perf-node-gather-daemonset-859k5\" (UID: \"a354ba71-3aea-4150-9ded-dcf285deb1ab\") " pod="openshift-must-gather-2hgxj/perf-node-gather-daemonset-859k5" Oct 30 21:26:11.246064 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:26:11.245880 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a354ba71-3aea-4150-9ded-dcf285deb1ab-lib-modules\") pod \"perf-node-gather-daemonset-859k5\" (UID: \"a354ba71-3aea-4150-9ded-dcf285deb1ab\") " pod="openshift-must-gather-2hgxj/perf-node-gather-daemonset-859k5" Oct 30 21:26:11.246064 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:26:11.245916 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a354ba71-3aea-4150-9ded-dcf285deb1ab-sys\") pod \"perf-node-gather-daemonset-859k5\" (UID: \"a354ba71-3aea-4150-9ded-dcf285deb1ab\") " pod="openshift-must-gather-2hgxj/perf-node-gather-daemonset-859k5" Oct 30 21:26:11.246064 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:26:11.245921 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/a354ba71-3aea-4150-9ded-dcf285deb1ab-proc\") pod \"perf-node-gather-daemonset-859k5\" (UID: \"a354ba71-3aea-4150-9ded-dcf285deb1ab\") " pod="openshift-must-gather-2hgxj/perf-node-gather-daemonset-859k5" Oct 30 21:26:11.257572 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:26:11.257550 4611 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hx7dn\" (UniqueName: \"kubernetes.io/projected/a354ba71-3aea-4150-9ded-dcf285deb1ab-kube-api-access-hx7dn\") pod \"perf-node-gather-daemonset-859k5\" (UID: \"a354ba71-3aea-4150-9ded-dcf285deb1ab\") " pod="openshift-must-gather-2hgxj/perf-node-gather-daemonset-859k5" Oct 30 21:26:11.404810 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:26:11.404784 4611 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-2hgxj/perf-node-gather-daemonset-859k5" Oct 30 21:26:11.677030 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:26:11.677006 4611 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-2hgxj/perf-node-gather-daemonset-859k5"] Oct 30 21:26:11.679783 ip-10-0-133-192 kubenswrapper[4611]: W1030 21:26:11.679758 4611 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-poda354ba71_3aea_4150_9ded_dcf285deb1ab.slice/crio-93843712b8f7fbf784e2f7fa614c5b4ef614283312ec4750473ec3f9579f5bcd WatchSource:0}: Error finding container 93843712b8f7fbf784e2f7fa614c5b4ef614283312ec4750473ec3f9579f5bcd: Status 404 returned error can't find the container with id 93843712b8f7fbf784e2f7fa614c5b4ef614283312ec4750473ec3f9579f5bcd Oct 30 21:26:11.706088 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:26:11.706064 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-2hgxj/perf-node-gather-daemonset-859k5" event={"ID":"a354ba71-3aea-4150-9ded-dcf285deb1ab","Type":"ContainerStarted","Data":"93843712b8f7fbf784e2f7fa614c5b4ef614283312ec4750473ec3f9579f5bcd"} Oct 30 21:26:12.712170 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:26:12.712132 4611 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-2hgxj/perf-node-gather-daemonset-859k5" event={"ID":"a354ba71-3aea-4150-9ded-dcf285deb1ab","Type":"ContainerStarted","Data":"ee74006bed0e25e61ce0a33d6b229beef0add7d98d076a245c7cffdd5ff307e2"} Oct 30 21:26:12.712659 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:26:12.712270 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-2hgxj/perf-node-gather-daemonset-859k5" Oct 30 21:26:12.730449 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:26:12.730403 4611 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-2hgxj/perf-node-gather-daemonset-859k5" podStartSLOduration=1.730390636 podStartE2EDuration="1.730390636s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:26:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:26:12.728923039 +0000 UTC m=+2827.262949655" watchObservedRunningTime="2026-10-30 21:26:12.730390636 +0000 UTC m=+2827.264417252" Oct 30 21:26:18.726768 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:26:18.726703 4611 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-2hgxj/perf-node-gather-daemonset-859k5" Oct 30 21:26:33.916016 ip-10-0-133-192 kubenswrapper[4611]: E1030 21:26:33.915978 4611 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hwzc6" containerName="extractor" Oct 30 21:26:50.233222 ip-10-0-133-192 kubenswrapper[4611]: I1030 21:26:50.233177 4611 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jfxnj_da50258d-eef3-4976-8e22-be0e936b9993/konnectivity-agent/0.log" limit=1024 Oct 30 21:27:03.916179 ip-10-0-133-192 kubenswrapper[4611]: E1030 21:27:03.916141 4611 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hwzc6" containerName="extractor"