Jul 22 02:37:13.572840 ip-10-0-133-31 systemd[1]: Starting Kubernetes Kubelet... Jul 22 02:37:13.937197 ip-10-0-133-31 kubenswrapper[2578]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 22 02:37:13.937197 ip-10-0-133-31 kubenswrapper[2578]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 22 02:37:13.937197 ip-10-0-133-31 kubenswrapper[2578]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 22 02:37:13.937197 ip-10-0-133-31 kubenswrapper[2578]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 22 02:37:13.937197 ip-10-0-133-31 kubenswrapper[2578]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 22 02:37:13.938131 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:13.938058 2578 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 22 02:37:13.944042 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.944020 2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 22 02:37:13.944042 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.944036 2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 22 02:37:13.944042 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.944040 2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 22 02:37:13.944042 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.944043 2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 22 02:37:13.944042 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.944046 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 22 02:37:13.944042 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.944049 2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 22 02:37:13.944042 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.944052 2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 22 02:37:13.944271 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.944055 2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 22 02:37:13.944271 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.944057 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 22 02:37:13.944271 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.944060 2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 22 02:37:13.944271 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.944062 2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 22 02:37:13.944271 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.944064 2578 feature_gate.go:328] unrecognized feature gate: Example Jul 22 02:37:13.944271 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.944067 2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 22 02:37:13.944271 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.944070 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 22 02:37:13.944271 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.944072 2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 22 02:37:13.944271 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.944074 2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 22 02:37:13.944271 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.944077 2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 22 02:37:13.944271 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.944079 2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 22 02:37:13.944271 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.944082 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 22 02:37:13.944271 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.944084 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 22 02:37:13.944271 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.944086 2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 22 02:37:13.944271 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.944089 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 22 02:37:13.944271 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.944091 2578 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 22 02:37:13.944271 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.944094 2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 22 02:37:13.944271 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.944096 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 22 02:37:13.944271 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.944099 2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 22 02:37:13.944735 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.944102 2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 22 02:37:13.944735 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.944106 2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 22 02:37:13.944735 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.944110 2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 22 02:37:13.944735 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.944116 2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 22 02:37:13.944735 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.944119 2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 22 02:37:13.944735 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.944126 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 22 02:37:13.944735 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.944128 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 22 02:37:13.944735 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.944131 2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 22 02:37:13.944735 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.944133 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 22 02:37:13.944735 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.944135 2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 22 02:37:13.944735 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.944138 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 22 02:37:13.944735 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.944140 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 22 02:37:13.944735 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.944142 2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 22 02:37:13.944735 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.944145 2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 22 02:37:13.944735 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.944148 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 22 02:37:13.944735 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.944151 2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 22 02:37:13.944735 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.944154 2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 22 02:37:13.944735 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.944156 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 22 02:37:13.944735 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.944159 2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 22 02:37:13.945181 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.944161 2578 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 22 02:37:13.945181 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.944163 2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 22 02:37:13.945181 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.944166 2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 22 02:37:13.945181 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.944168 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 22 02:37:13.945181 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.944170 2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 22 02:37:13.945181 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.944173 2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 22 02:37:13.945181 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.944175 2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 22 02:37:13.945181 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.944177 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 22 02:37:13.945181 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.944180 2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 22 02:37:13.945181 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.944182 2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 22 02:37:13.945181 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.944184 2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 22 02:37:13.945181 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.944189 2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 22 02:37:13.945181 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.944192 2578 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 22 02:37:13.945181 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.944195 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 22 02:37:13.945181 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.944198 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 22 02:37:13.945181 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.944200 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 22 02:37:13.945181 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.944203 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 22 02:37:13.945181 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.944205 2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 22 02:37:13.945181 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.944207 2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 22 02:37:13.945181 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.944210 2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 22 02:37:13.945789 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.944212 2578 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 22 02:37:13.945789 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.944214 2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 22 02:37:13.945789 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.944216 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 22 02:37:13.945789 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.944219 2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 22 02:37:13.945789 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.944221 2578 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 22 02:37:13.945789 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.944223 2578 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 22 02:37:13.945789 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.944226 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 22 02:37:13.945789 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.944228 2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 22 02:37:13.945789 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.944231 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 22 02:37:13.945789 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.944234 2578 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 22 02:37:13.945789 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.944236 2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 22 02:37:13.945789 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.944239 2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 22 02:37:13.945789 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.944241 2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 22 02:37:13.945789 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.944243 2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 22 02:37:13.945789 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.944246 2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 22 02:37:13.945789 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.944248 2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 22 02:37:13.945789 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.944250 2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 22 02:37:13.945789 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.944253 2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 22 02:37:13.945789 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.944256 2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 22 02:37:13.945789 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.944259 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 22 02:37:13.946278 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.944261 2578 feature_gate.go:328] unrecognized feature gate: Example2 Jul 22 02:37:13.946278 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.944263 2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 22 02:37:13.946278 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.944266 2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 22 02:37:13.946278 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.944268 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 22 02:37:13.946278 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.944270 2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 22 02:37:13.946278 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.944273 2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 22 02:37:13.946278 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.944275 2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 22 02:37:13.946278 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.944278 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 22 02:37:13.946278 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.944280 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 22 02:37:13.946278 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.944283 2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 22 02:37:13.946278 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.944285 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 22 02:37:13.946278 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.944288 2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 22 02:37:13.946278 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.944291 2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 22 02:37:13.946278 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.944293 2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 22 02:37:13.946278 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.944972 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 22 02:37:13.946278 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.944978 2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 22 02:37:13.946278 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.944981 2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 22 02:37:13.946278 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.944984 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 22 02:37:13.946278 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.944986 2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 22 02:37:13.946776 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.944989 2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 22 02:37:13.946776 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.944991 2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 22 02:37:13.946776 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.944994 2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 22 02:37:13.946776 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.944996 2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 22 02:37:13.946776 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.944999 2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 22 02:37:13.946776 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.945001 2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 22 02:37:13.946776 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.945003 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 22 02:37:13.946776 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.945006 2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 22 02:37:13.946776 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.945008 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 22 02:37:13.946776 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.945011 2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 22 02:37:13.946776 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.945013 2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 22 02:37:13.946776 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.945016 2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 22 02:37:13.946776 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.945019 2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 22 02:37:13.946776 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.945022 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 22 02:37:13.946776 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.945024 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 22 02:37:13.946776 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.945026 2578 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 22 02:37:13.946776 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.945029 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 22 02:37:13.946776 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.945032 2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 22 02:37:13.946776 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.945036 2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 22 02:37:13.946776 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.945040 2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 22 02:37:13.947245 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.945043 2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 22 02:37:13.947245 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.945046 2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 22 02:37:13.947245 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.945048 2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 22 02:37:13.947245 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.945051 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 22 02:37:13.947245 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.945053 2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 22 02:37:13.947245 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.945056 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 22 02:37:13.947245 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.945058 2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 22 02:37:13.947245 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.945060 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 22 02:37:13.947245 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.945063 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 22 02:37:13.947245 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.945065 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 22 02:37:13.947245 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.945068 2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 22 02:37:13.947245 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.945070 2578 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 22 02:37:13.947245 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.945082 2578 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 22 02:37:13.947245 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.945085 2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 22 02:37:13.947245 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.945088 2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 22 02:37:13.947245 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.945090 2578 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 22 02:37:13.947245 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.945092 2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 22 02:37:13.947245 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.945095 2578 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 22 02:37:13.947245 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.945097 2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 22 02:37:13.947245 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.945100 2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 22 02:37:13.947770 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.945102 2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 22 02:37:13.947770 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.945104 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 22 02:37:13.947770 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.945107 2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 22 02:37:13.947770 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.945109 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 22 02:37:13.947770 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.945112 2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 22 02:37:13.947770 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.945115 2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 22 02:37:13.947770 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.945117 2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 22 02:37:13.947770 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.945120 2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 22 02:37:13.947770 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.945123 2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 22 02:37:13.947770 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.945125 2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 22 02:37:13.947770 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.945127 2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 22 02:37:13.947770 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.945129 2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 22 02:37:13.947770 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.945132 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 22 02:37:13.947770 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.945134 2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 22 02:37:13.947770 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.945136 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 22 02:37:13.947770 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.945138 2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 22 02:37:13.947770 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.945141 2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 22 02:37:13.947770 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.945143 2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 22 02:37:13.947770 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.945146 2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 22 02:37:13.947770 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.945149 2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 22 02:37:13.948231 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.945151 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 22 02:37:13.948231 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.945154 2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 22 02:37:13.948231 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.945157 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 22 02:37:13.948231 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.945159 2578 feature_gate.go:328] unrecognized feature gate: Example Jul 22 02:37:13.948231 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.945162 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 22 02:37:13.948231 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.945165 2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 22 02:37:13.948231 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.945167 2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 22 02:37:13.948231 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.945170 2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 22 02:37:13.948231 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.945172 2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 22 02:37:13.948231 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.945176 2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 22 02:37:13.948231 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.945179 2578 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 22 02:37:13.948231 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.945183 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 22 02:37:13.948231 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.945186 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 22 02:37:13.948231 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.945188 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 22 02:37:13.948231 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.945191 2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 22 02:37:13.948231 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.945193 2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 22 02:37:13.948231 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.945196 2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 22 02:37:13.948231 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.945198 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 22 02:37:13.948231 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.945201 2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 22 02:37:13.948689 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.945204 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 22 02:37:13.948689 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.945207 2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 22 02:37:13.948689 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.945209 2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 22 02:37:13.948689 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.945212 2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 22 02:37:13.948689 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.945214 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 22 02:37:13.948689 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.945216 2578 feature_gate.go:328] unrecognized feature gate: Example2 Jul 22 02:37:13.948689 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.945219 2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 22 02:37:13.948689 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.945221 2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 22 02:37:13.948689 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.945224 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 22 02:37:13.948689 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.945226 2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 22 02:37:13.948689 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.945229 2578 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 22 02:37:13.948689 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.945231 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 22 02:37:13.948689 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.945233 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 22 02:37:13.948689 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.945235 2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 22 02:37:13.948689 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.945238 2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 22 02:37:13.948689 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.945620 2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 22 02:37:13.948689 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.945626 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 22 02:37:13.948689 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.945629 2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 22 02:37:13.948689 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.945634 2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 22 02:37:13.949135 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.945637 2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 22 02:37:13.949135 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.945640 2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 22 02:37:13.949135 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.945642 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 22 02:37:13.949135 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.945645 2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 22 02:37:13.949135 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.945648 2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 22 02:37:13.949135 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.945652 2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 22 02:37:13.949135 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.945655 2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 22 02:37:13.949135 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.945658 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 22 02:37:13.949135 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.945660 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 22 02:37:13.949135 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.945663 2578 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 22 02:37:13.949135 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.945665 2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 22 02:37:13.949135 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.945668 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 22 02:37:13.949135 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.945671 2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 22 02:37:13.949135 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.945674 2578 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 22 02:37:13.949135 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.945678 2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 22 02:37:13.949135 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.945681 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 22 02:37:13.949135 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.945683 2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 22 02:37:13.949135 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.945686 2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 22 02:37:13.949135 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.945689 2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 22 02:37:13.949599 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.945692 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 22 02:37:13.949599 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.945694 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 22 02:37:13.949599 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.945697 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 22 02:37:13.949599 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.945700 2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 22 02:37:13.949599 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.945702 2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 22 02:37:13.949599 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.945705 2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 22 02:37:13.949599 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.945708 2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 22 02:37:13.949599 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.945710 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 22 02:37:13.949599 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.945712 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 22 02:37:13.949599 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.945715 2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 22 02:37:13.949599 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.945717 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 22 02:37:13.949599 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.945720 2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 22 02:37:13.949599 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.945723 2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 22 02:37:13.949599 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.945725 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 22 02:37:13.949599 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.945728 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 22 02:37:13.949599 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.945730 2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 22 02:37:13.949599 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.945733 2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 22 02:37:13.949599 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.945735 2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 22 02:37:13.949599 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.945739 2578 feature_gate.go:328] unrecognized feature gate: Example2 Jul 22 02:37:13.949599 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.945741 2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 22 02:37:13.950065 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.945743 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 22 02:37:13.950065 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.945747 2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 22 02:37:13.950065 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.945749 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 22 02:37:13.950065 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.945751 2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 22 02:37:13.950065 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.945754 2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 22 02:37:13.950065 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.945756 2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 22 02:37:13.950065 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.945759 2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 22 02:37:13.950065 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.945761 2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 22 02:37:13.950065 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.945763 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 22 02:37:13.950065 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.945766 2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 22 02:37:13.950065 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.945768 2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 22 02:37:13.950065 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.945770 2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 22 02:37:13.950065 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.945772 2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 22 02:37:13.950065 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.945775 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 22 02:37:13.950065 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.945777 2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 22 02:37:13.950065 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.945780 2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 22 02:37:13.950065 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.945782 2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 22 02:37:13.950065 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.945785 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 22 02:37:13.950065 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.945787 2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 22 02:37:13.950065 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.945789 2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 22 02:37:13.950526 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.945792 2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 22 02:37:13.950526 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.945794 2578 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 22 02:37:13.950526 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.945797 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 22 02:37:13.950526 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.945800 2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 22 02:37:13.950526 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.945802 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 22 02:37:13.950526 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.945805 2578 feature_gate.go:328] unrecognized feature gate: Example Jul 22 02:37:13.950526 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.945807 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 22 02:37:13.950526 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.945810 2578 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 22 02:37:13.950526 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.945813 2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 22 02:37:13.950526 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.945816 2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 22 02:37:13.950526 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.945819 2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 22 02:37:13.950526 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.945821 2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 22 02:37:13.950526 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.945824 2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 22 02:37:13.950526 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.945826 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 22 02:37:13.950526 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.945828 2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 22 02:37:13.950526 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.945831 2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 22 02:37:13.950526 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.945833 2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 22 02:37:13.950526 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.945835 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 22 02:37:13.950526 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.945837 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 22 02:37:13.950526 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.945840 2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 22 02:37:13.951004 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.945844 2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 22 02:37:13.951004 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.945847 2578 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 22 02:37:13.951004 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.945849 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 22 02:37:13.951004 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.945851 2578 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 22 02:37:13.951004 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.945854 2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 22 02:37:13.951004 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.945856 2578 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 22 02:37:13.951004 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.945858 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 22 02:37:13.951004 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.945860 2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 22 02:37:13.951004 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.945863 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 22 02:37:13.951004 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.945865 2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 22 02:37:13.951004 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.945868 2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 22 02:37:13.951004 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.945870 2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 22 02:37:13.951004 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.945872 2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 22 02:37:13.951004 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.945874 2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 22 02:37:13.951004 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.945877 2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 22 02:37:13.951004 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.945880 2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 22 02:37:13.951004 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:13.945947 2578 flags.go:64] FLAG: --address="0.0.0.0" Jul 22 02:37:13.951004 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:13.945954 2578 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 22 02:37:13.951004 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:13.945959 2578 flags.go:64] FLAG: --anonymous-auth="true" Jul 22 02:37:13.951004 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:13.945963 2578 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 22 02:37:13.951493 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:13.945967 2578 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 22 02:37:13.951493 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:13.945970 2578 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 22 02:37:13.951493 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:13.945974 2578 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 22 02:37:13.951493 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:13.945979 2578 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 22 02:37:13.951493 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:13.945981 2578 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 22 02:37:13.951493 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:13.945984 2578 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 22 02:37:13.951493 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:13.945987 2578 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 22 02:37:13.951493 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:13.945990 2578 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 22 02:37:13.951493 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:13.945993 2578 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 22 02:37:13.951493 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:13.945996 2578 flags.go:64] FLAG: --cgroup-root="" Jul 22 02:37:13.951493 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:13.945998 2578 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 22 02:37:13.951493 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:13.946001 2578 flags.go:64] FLAG: --client-ca-file="" Jul 22 02:37:13.951493 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:13.946004 2578 flags.go:64] FLAG: --cloud-provider="external" Jul 22 02:37:13.951493 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:13.946007 2578 flags.go:64] FLAG: --cluster-dns="[]" Jul 22 02:37:13.951493 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:13.946011 2578 flags.go:64] FLAG: --cluster-domain="" Jul 22 02:37:13.951493 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:13.946014 2578 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 22 02:37:13.951493 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:13.946016 2578 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 22 02:37:13.951493 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:13.946019 2578 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 22 02:37:13.951493 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:13.946022 2578 flags.go:64] FLAG: --container-log-max-files="5" Jul 22 02:37:13.951493 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:13.946025 2578 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 22 02:37:13.951493 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:13.946028 2578 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 22 02:37:13.951493 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:13.946031 2578 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 22 02:37:13.951493 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:13.946034 2578 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 22 02:37:13.952043 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:13.946036 2578 flags.go:64] FLAG: --contention-profiling="false" Jul 22 02:37:13.952043 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:13.946039 2578 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 22 02:37:13.952043 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:13.946041 2578 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 22 02:37:13.952043 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:13.946044 2578 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 22 02:37:13.952043 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:13.946048 2578 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 22 02:37:13.952043 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:13.946051 2578 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 22 02:37:13.952043 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:13.946054 2578 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 22 02:37:13.952043 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:13.946057 2578 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 22 02:37:13.952043 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:13.946059 2578 flags.go:64] FLAG: --enable-load-reader="false" Jul 22 02:37:13.952043 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:13.946062 2578 flags.go:64] FLAG: --enable-server="true" Jul 22 02:37:13.952043 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:13.946064 2578 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 22 02:37:13.952043 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:13.946069 2578 flags.go:64] FLAG: --event-burst="100" Jul 22 02:37:13.952043 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:13.946072 2578 flags.go:64] FLAG: --event-qps="50" Jul 22 02:37:13.952043 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:13.946075 2578 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 22 02:37:13.952043 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:13.946077 2578 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 22 02:37:13.952043 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:13.946080 2578 flags.go:64] FLAG: --eviction-hard="" Jul 22 02:37:13.952043 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:13.946084 2578 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 22 02:37:13.952043 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:13.946086 2578 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 22 02:37:13.952043 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:13.946089 2578 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 22 02:37:13.952043 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:13.946092 2578 flags.go:64] FLAG: --eviction-soft="" Jul 22 02:37:13.952043 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:13.946094 2578 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 22 02:37:13.952043 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:13.946097 2578 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 22 02:37:13.952043 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:13.946101 2578 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 22 02:37:13.952043 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:13.946104 2578 flags.go:64] FLAG: --experimental-mounter-path="" Jul 22 02:37:13.952043 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:13.946106 2578 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 22 02:37:13.952645 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:13.946109 2578 flags.go:64] FLAG: --fail-swap-on="true" Jul 22 02:37:13.952645 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:13.946111 2578 flags.go:64] FLAG: --feature-gates="" Jul 22 02:37:13.952645 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:13.946114 2578 flags.go:64] FLAG: --file-check-frequency="20s" Jul 22 02:37:13.952645 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:13.946117 2578 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 22 02:37:13.952645 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:13.946120 2578 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 22 02:37:13.952645 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:13.946122 2578 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 22 02:37:13.952645 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:13.946125 2578 flags.go:64] FLAG: --healthz-port="10248" Jul 22 02:37:13.952645 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:13.946128 2578 flags.go:64] FLAG: --help="false" Jul 22 02:37:13.952645 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:13.946131 2578 flags.go:64] FLAG: --hostname-override="ip-10-0-133-31.ec2.internal" Jul 22 02:37:13.952645 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:13.946133 2578 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 22 02:37:13.952645 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:13.946136 2578 flags.go:64] FLAG: --http-check-frequency="20s" Jul 22 02:37:13.952645 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:13.946140 2578 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 22 02:37:13.952645 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:13.946144 2578 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 22 02:37:13.952645 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:13.946147 2578 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 22 02:37:13.952645 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:13.946149 2578 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 22 02:37:13.952645 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:13.946152 2578 flags.go:64] FLAG: --image-service-endpoint="" Jul 22 02:37:13.952645 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:13.946154 2578 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 22 02:37:13.952645 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:13.946157 2578 flags.go:64] FLAG: --kube-api-burst="100" Jul 22 02:37:13.952645 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:13.946161 2578 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 22 02:37:13.952645 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:13.946163 2578 flags.go:64] FLAG: --kube-api-qps="50" Jul 22 02:37:13.952645 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:13.946166 2578 flags.go:64] FLAG: --kube-reserved="" Jul 22 02:37:13.952645 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:13.946169 2578 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 22 02:37:13.952645 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:13.946171 2578 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 22 02:37:13.952645 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:13.946174 2578 flags.go:64] FLAG: --kubelet-cgroups="" Jul 22 02:37:13.953214 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:13.946176 2578 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 22 02:37:13.953214 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:13.946179 2578 flags.go:64] FLAG: --lock-file="" Jul 22 02:37:13.953214 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:13.946181 2578 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 22 02:37:13.953214 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:13.946184 2578 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 22 02:37:13.953214 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:13.946187 2578 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 22 02:37:13.953214 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:13.946192 2578 flags.go:64] FLAG: --log-json-split-stream="false" Jul 22 02:37:13.953214 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:13.946195 2578 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 22 02:37:13.953214 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:13.946197 2578 flags.go:64] FLAG: --log-text-split-stream="false" Jul 22 02:37:13.953214 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:13.946200 2578 flags.go:64] FLAG: --logging-format="text" Jul 22 02:37:13.953214 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:13.946202 2578 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 22 02:37:13.953214 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:13.946205 2578 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 22 02:37:13.953214 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:13.946208 2578 flags.go:64] FLAG: --manifest-url="" Jul 22 02:37:13.953214 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:13.946210 2578 flags.go:64] FLAG: --manifest-url-header="" Jul 22 02:37:13.953214 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:13.946214 2578 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 22 02:37:13.953214 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:13.946216 2578 flags.go:64] FLAG: --max-open-files="1000000" Jul 22 02:37:13.953214 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:13.946220 2578 flags.go:64] FLAG: --max-pods="110" Jul 22 02:37:13.953214 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:13.946223 2578 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 22 02:37:13.953214 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:13.946225 2578 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 22 02:37:13.953214 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:13.946228 2578 flags.go:64] FLAG: --memory-manager-policy="None" Jul 22 02:37:13.953214 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:13.946232 2578 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 22 02:37:13.953214 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:13.946234 2578 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 22 02:37:13.953214 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:13.946237 2578 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 22 02:37:13.953214 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:13.946240 2578 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 22 02:37:13.953214 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:13.946246 2578 flags.go:64] FLAG: --node-status-max-images="50" Jul 22 02:37:13.953214 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:13.946249 2578 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 22 02:37:13.953837 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:13.946252 2578 flags.go:64] FLAG: --oom-score-adj="-999" Jul 22 02:37:13.953837 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:13.946256 2578 flags.go:64] FLAG: --pod-cidr="" Jul 22 02:37:13.953837 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:13.946259 2578 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4017e99d76d7b91598d3a3430cd0eba6eda8b33653addd3838fdfe3cca69c5a0" Jul 22 02:37:13.953837 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:13.946264 2578 flags.go:64] FLAG: --pod-manifest-path="" Jul 22 02:37:13.953837 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:13.946266 2578 flags.go:64] FLAG: --pod-max-pids="-1" Jul 22 02:37:13.953837 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:13.946269 2578 flags.go:64] FLAG: --pods-per-core="0" Jul 22 02:37:13.953837 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:13.946271 2578 flags.go:64] FLAG: --port="10250" Jul 22 02:37:13.953837 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:13.946274 2578 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 22 02:37:13.953837 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:13.946277 2578 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0020d73f88884abf5" Jul 22 02:37:13.953837 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:13.946280 2578 flags.go:64] FLAG: --qos-reserved="" Jul 22 02:37:13.953837 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:13.946283 2578 flags.go:64] FLAG: --read-only-port="10255" Jul 22 02:37:13.953837 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:13.946285 2578 flags.go:64] FLAG: --register-node="true" Jul 22 02:37:13.953837 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:13.946288 2578 flags.go:64] FLAG: --register-with-taints="" Jul 22 02:37:13.953837 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:13.946291 2578 flags.go:64] FLAG: --registry-burst="10" Jul 22 02:37:13.953837 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:13.946298 2578 flags.go:64] FLAG: --registry-qps="5" Jul 22 02:37:13.953837 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:13.946300 2578 flags.go:64] FLAG: --reserved-cpus="" Jul 22 02:37:13.953837 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:13.946303 2578 flags.go:64] FLAG: --reserved-memory="" Jul 22 02:37:13.953837 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:13.946306 2578 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 22 02:37:13.953837 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:13.946309 2578 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 22 02:37:13.953837 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:13.946312 2578 flags.go:64] FLAG: --rotate-certificates="false" Jul 22 02:37:13.953837 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:13.946314 2578 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 22 02:37:13.953837 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:13.946317 2578 flags.go:64] FLAG: --runonce="false" Jul 22 02:37:13.953837 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:13.946319 2578 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 22 02:37:13.953837 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:13.946322 2578 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 22 02:37:13.953837 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:13.946325 2578 flags.go:64] FLAG: --seccomp-default="false" Jul 22 02:37:13.954396 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:13.946327 2578 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 22 02:37:13.954396 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:13.946330 2578 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 22 02:37:13.954396 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:13.946334 2578 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 22 02:37:13.954396 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:13.946336 2578 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 22 02:37:13.954396 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:13.946339 2578 flags.go:64] FLAG: --storage-driver-password="root" Jul 22 02:37:13.954396 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:13.946341 2578 flags.go:64] FLAG: --storage-driver-secure="false" Jul 22 02:37:13.954396 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:13.946344 2578 flags.go:64] FLAG: --storage-driver-table="stats" Jul 22 02:37:13.954396 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:13.946346 2578 flags.go:64] FLAG: --storage-driver-user="root" Jul 22 02:37:13.954396 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:13.946350 2578 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 22 02:37:13.954396 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:13.946352 2578 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 22 02:37:13.954396 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:13.946355 2578 flags.go:64] FLAG: --system-cgroups="" Jul 22 02:37:13.954396 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:13.946357 2578 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 22 02:37:13.954396 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:13.946363 2578 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 22 02:37:13.954396 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:13.946365 2578 flags.go:64] FLAG: --tls-cert-file="" Jul 22 02:37:13.954396 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:13.946368 2578 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 22 02:37:13.954396 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:13.946374 2578 flags.go:64] FLAG: --tls-min-version="" Jul 22 02:37:13.954396 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:13.946377 2578 flags.go:64] FLAG: --tls-private-key-file="" Jul 22 02:37:13.954396 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:13.946379 2578 flags.go:64] FLAG: --topology-manager-policy="none" Jul 22 02:37:13.954396 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:13.946382 2578 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 22 02:37:13.954396 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:13.946385 2578 flags.go:64] FLAG: --topology-manager-scope="container" Jul 22 02:37:13.954396 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:13.946387 2578 flags.go:64] FLAG: --v="2" Jul 22 02:37:13.954396 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:13.946391 2578 flags.go:64] FLAG: --version="false" Jul 22 02:37:13.954396 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:13.946395 2578 flags.go:64] FLAG: --vmodule="" Jul 22 02:37:13.954396 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:13.946398 2578 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 22 02:37:13.954396 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:13.946401 2578 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 22 02:37:13.954990 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.946515 2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 22 02:37:13.954990 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.946518 2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 22 02:37:13.954990 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.946521 2578 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 22 02:37:13.954990 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.946523 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 22 02:37:13.954990 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.946526 2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 22 02:37:13.954990 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.946528 2578 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 22 02:37:13.954990 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.946530 2578 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 22 02:37:13.954990 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.946533 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 22 02:37:13.954990 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.946536 2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 22 02:37:13.954990 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.946539 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 22 02:37:13.954990 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.946542 2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 22 02:37:13.954990 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.946544 2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 22 02:37:13.954990 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.946547 2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 22 02:37:13.954990 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.946549 2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 22 02:37:13.954990 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.946552 2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 22 02:37:13.954990 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.946554 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 22 02:37:13.954990 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.946557 2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 22 02:37:13.954990 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.946577 2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 22 02:37:13.954990 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.946580 2578 feature_gate.go:328] unrecognized feature gate: Example Jul 22 02:37:13.954990 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.946583 2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 22 02:37:13.955450 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.946585 2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 22 02:37:13.955450 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.946588 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 22 02:37:13.955450 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.946590 2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 22 02:37:13.955450 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.946592 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 22 02:37:13.955450 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.946595 2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 22 02:37:13.955450 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.946597 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 22 02:37:13.955450 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.946599 2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 22 02:37:13.955450 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.946602 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 22 02:37:13.955450 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.946605 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 22 02:37:13.955450 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.946607 2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 22 02:37:13.955450 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.946611 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 22 02:37:13.955450 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.946614 2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 22 02:37:13.955450 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.946619 2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 22 02:37:13.955450 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.946622 2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 22 02:37:13.955450 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.946625 2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 22 02:37:13.955450 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.946628 2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 22 02:37:13.955450 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.946631 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 22 02:37:13.955450 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.946634 2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 22 02:37:13.955450 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.946636 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 22 02:37:13.955907 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.946639 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 22 02:37:13.955907 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.946641 2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 22 02:37:13.955907 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.946645 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 22 02:37:13.955907 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.946647 2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 22 02:37:13.955907 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.946649 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 22 02:37:13.955907 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.946652 2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 22 02:37:13.955907 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.946654 2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 22 02:37:13.955907 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.946657 2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 22 02:37:13.955907 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.946659 2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 22 02:37:13.955907 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.946662 2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 22 02:37:13.955907 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.946664 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 22 02:37:13.955907 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.946666 2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 22 02:37:13.955907 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.946669 2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 22 02:37:13.955907 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.946671 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 22 02:37:13.955907 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.946674 2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 22 02:37:13.955907 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.946676 2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 22 02:37:13.955907 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.946679 2578 feature_gate.go:328] unrecognized feature gate: Example2 Jul 22 02:37:13.955907 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.946681 2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 22 02:37:13.955907 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.946683 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 22 02:37:13.955907 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.946686 2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 22 02:37:13.956423 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.946688 2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 22 02:37:13.956423 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.946692 2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 22 02:37:13.956423 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.946695 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 22 02:37:13.956423 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.946699 2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 22 02:37:13.956423 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.946701 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 22 02:37:13.956423 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.946704 2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 22 02:37:13.956423 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.946707 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 22 02:37:13.956423 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.946709 2578 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 22 02:37:13.956423 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.946712 2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 22 02:37:13.956423 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.946714 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 22 02:37:13.956423 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.946717 2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 22 02:37:13.956423 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.946719 2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 22 02:37:13.956423 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.946722 2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 22 02:37:13.956423 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.946724 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 22 02:37:13.956423 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.946729 2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 22 02:37:13.956423 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.946732 2578 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 22 02:37:13.956423 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.946734 2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 22 02:37:13.956423 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.946737 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 22 02:37:13.956423 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.946739 2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 22 02:37:13.956898 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.946742 2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 22 02:37:13.956898 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.946744 2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 22 02:37:13.956898 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.946746 2578 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 22 02:37:13.956898 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.946749 2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 22 02:37:13.956898 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.946751 2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 22 02:37:13.956898 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.946753 2578 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 22 02:37:13.956898 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.946756 2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 22 02:37:13.956898 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.946758 2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 22 02:37:13.956898 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.946761 2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 22 02:37:13.956898 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.946763 2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 22 02:37:13.956898 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.946765 2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 22 02:37:13.956898 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.946768 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 22 02:37:13.956898 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.946770 2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 22 02:37:13.956898 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.946772 2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 22 02:37:13.956898 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.946775 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 22 02:37:13.956898 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.946777 2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 22 02:37:13.956898 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.946780 2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 22 02:37:13.956898 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.946782 2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 22 02:37:13.956898 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.946785 2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 22 02:37:13.956898 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.946788 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 22 02:37:13.957456 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.946790 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 22 02:37:13.957456 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:13.947435 2578 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 22 02:37:13.957456 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:13.954579 2578 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 22 02:37:13.957456 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:13.954592 2578 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 22 02:37:13.957456 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:13.954606 2578 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 22 02:37:13.957456 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:13.954611 2578 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 22 02:37:13.957456 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.954662 2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 22 02:37:13.957456 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.954666 2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 22 02:37:13.957456 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.954670 2578 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 22 02:37:13.957456 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.954673 2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 22 02:37:13.957456 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.954675 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 22 02:37:13.957456 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.954678 2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 22 02:37:13.957456 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.954681 2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 22 02:37:13.957456 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.954684 2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 22 02:37:13.957456 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.954687 2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 22 02:37:13.957857 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.954689 2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 22 02:37:13.957857 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.954692 2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 22 02:37:13.957857 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.954694 2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 22 02:37:13.957857 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.954697 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 22 02:37:13.957857 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.954699 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 22 02:37:13.957857 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.954702 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 22 02:37:13.957857 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.954704 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 22 02:37:13.957857 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.954707 2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 22 02:37:13.957857 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.954710 2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 22 02:37:13.957857 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.954712 2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 22 02:37:13.957857 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.954715 2578 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 22 02:37:13.957857 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.954718 2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 22 02:37:13.957857 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.954720 2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 22 02:37:13.957857 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.954723 2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 22 02:37:13.957857 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.954725 2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 22 02:37:13.957857 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.954728 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 22 02:37:13.957857 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.954730 2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 22 02:37:13.957857 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.954732 2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 22 02:37:13.957857 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.954735 2578 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 22 02:37:13.957857 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.954737 2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 22 02:37:13.958399 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.954740 2578 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 22 02:37:13.958399 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.954743 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 22 02:37:13.958399 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.954745 2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 22 02:37:13.958399 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.954748 2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 22 02:37:13.958399 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.954750 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 22 02:37:13.958399 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.954752 2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 22 02:37:13.958399 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.954755 2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 22 02:37:13.958399 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.954757 2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 22 02:37:13.958399 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.954759 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 22 02:37:13.958399 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.954761 2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 22 02:37:13.958399 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.954764 2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 22 02:37:13.958399 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.954766 2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 22 02:37:13.958399 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.954768 2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 22 02:37:13.958399 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.954771 2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 22 02:37:13.958399 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.954773 2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 22 02:37:13.958399 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.954775 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 22 02:37:13.958399 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.954778 2578 feature_gate.go:328] unrecognized feature gate: Example Jul 22 02:37:13.958399 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.954780 2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 22 02:37:13.958399 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.954782 2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 22 02:37:13.958399 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.954785 2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 22 02:37:13.958880 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.954787 2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 22 02:37:13.958880 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.954790 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 22 02:37:13.958880 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.954792 2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 22 02:37:13.958880 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.954794 2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 22 02:37:13.958880 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.954797 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 22 02:37:13.958880 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.954799 2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 22 02:37:13.958880 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.954802 2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 22 02:37:13.958880 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.954805 2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 22 02:37:13.958880 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.954809 2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 22 02:37:13.958880 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.954814 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 22 02:37:13.958880 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.954817 2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 22 02:37:13.958880 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.954820 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 22 02:37:13.958880 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.954823 2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 22 02:37:13.958880 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.954826 2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 22 02:37:13.958880 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.954829 2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 22 02:37:13.958880 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.954832 2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 22 02:37:13.958880 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.954834 2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 22 02:37:13.958880 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.954837 2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 22 02:37:13.958880 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.954839 2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 22 02:37:13.959652 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.954841 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 22 02:37:13.959652 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.954844 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 22 02:37:13.959652 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.954846 2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 22 02:37:13.959652 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.954849 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 22 02:37:13.959652 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.954851 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 22 02:37:13.959652 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.954853 2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 22 02:37:13.959652 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.954856 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 22 02:37:13.959652 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.954858 2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 22 02:37:13.959652 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.954860 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 22 02:37:13.959652 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.954863 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 22 02:37:13.959652 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.954865 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 22 02:37:13.959652 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.954867 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 22 02:37:13.959652 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.954870 2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 22 02:37:13.959652 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.954873 2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 22 02:37:13.959652 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.954875 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 22 02:37:13.959652 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.954877 2578 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 22 02:37:13.959652 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.954880 2578 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 22 02:37:13.959652 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.954882 2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 22 02:37:13.959652 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.954885 2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 22 02:37:13.959652 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.954888 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 22 02:37:13.960393 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.954890 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 22 02:37:13.960393 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.954892 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 22 02:37:13.960393 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.954894 2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 22 02:37:13.960393 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.954897 2578 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 22 02:37:13.960393 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.954899 2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 22 02:37:13.960393 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.954902 2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 22 02:37:13.960393 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.954905 2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 22 02:37:13.960393 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.954909 2578 feature_gate.go:328] unrecognized feature gate: Example2 Jul 22 02:37:13.960393 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.954912 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 22 02:37:13.960393 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.954914 2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 22 02:37:13.960393 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.954917 2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 22 02:37:13.960393 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:13.954921 2578 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 22 02:37:13.960393 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.955024 2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 22 02:37:13.960393 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.955030 2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 22 02:37:13.960393 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.955033 2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 22 02:37:13.960946 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.955035 2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 22 02:37:13.960946 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.955038 2578 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 22 02:37:13.960946 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.955041 2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 22 02:37:13.960946 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.955043 2578 feature_gate.go:328] unrecognized feature gate: Example Jul 22 02:37:13.960946 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.955046 2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 22 02:37:13.960946 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.955048 2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 22 02:37:13.960946 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.955051 2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 22 02:37:13.960946 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.955053 2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 22 02:37:13.960946 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.955056 2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 22 02:37:13.960946 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.955059 2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 22 02:37:13.960946 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.955061 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 22 02:37:13.960946 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.955064 2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 22 02:37:13.960946 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.955066 2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 22 02:37:13.960946 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.955068 2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 22 02:37:13.960946 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.955071 2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 22 02:37:13.960946 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.955073 2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 22 02:37:13.960946 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.955075 2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 22 02:37:13.960946 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.955077 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 22 02:37:13.960946 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.955080 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 22 02:37:13.961611 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.955083 2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 22 02:37:13.961611 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.955087 2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 22 02:37:13.961611 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.955089 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 22 02:37:13.961611 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.955092 2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 22 02:37:13.961611 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.955095 2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 22 02:37:13.961611 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.955098 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 22 02:37:13.961611 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.955101 2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 22 02:37:13.961611 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.955103 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 22 02:37:13.961611 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.955106 2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 22 02:37:13.961611 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.955108 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 22 02:37:13.961611 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.955110 2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 22 02:37:13.961611 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.955113 2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 22 02:37:13.961611 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.955115 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 22 02:37:13.961611 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.955118 2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 22 02:37:13.961611 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.955120 2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 22 02:37:13.961611 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.955123 2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 22 02:37:13.961611 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.955125 2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 22 02:37:13.961611 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.955127 2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 22 02:37:13.961611 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.955129 2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 22 02:37:13.961611 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.955132 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 22 02:37:13.962071 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.955134 2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 22 02:37:13.962071 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.955136 2578 feature_gate.go:328] unrecognized feature gate: Example2 Jul 22 02:37:13.962071 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.955139 2578 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 22 02:37:13.962071 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.955141 2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 22 02:37:13.962071 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.955144 2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 22 02:37:13.962071 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.955146 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 22 02:37:13.962071 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.955149 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 22 02:37:13.962071 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.955151 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 22 02:37:13.962071 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.955153 2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 22 02:37:13.962071 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.955156 2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 22 02:37:13.962071 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.955158 2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 22 02:37:13.962071 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.955160 2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 22 02:37:13.962071 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.955164 2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 22 02:37:13.962071 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.955167 2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 22 02:37:13.962071 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.955170 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 22 02:37:13.962071 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.955172 2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 22 02:37:13.962071 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.955174 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 22 02:37:13.962071 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.955177 2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 22 02:37:13.962071 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.955179 2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 22 02:37:13.962503 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.955182 2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 22 02:37:13.962503 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.955184 2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 22 02:37:13.962503 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.955186 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 22 02:37:13.962503 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.955188 2578 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 22 02:37:13.962503 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.955191 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 22 02:37:13.962503 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.955194 2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 22 02:37:13.962503 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.955196 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 22 02:37:13.962503 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.955199 2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 22 02:37:13.962503 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.955202 2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 22 02:37:13.962503 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.955205 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 22 02:37:13.962503 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.955208 2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 22 02:37:13.962503 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.955210 2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 22 02:37:13.962503 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.955212 2578 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 22 02:37:13.962503 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.955215 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 22 02:37:13.962503 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.955217 2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 22 02:37:13.962503 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.955219 2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 22 02:37:13.962503 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.955221 2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 22 02:37:13.962503 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.955224 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 22 02:37:13.962503 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.955226 2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 22 02:37:13.962503 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.955228 2578 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 22 02:37:13.963013 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.955231 2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 22 02:37:13.963013 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.955233 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 22 02:37:13.963013 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.955236 2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 22 02:37:13.963013 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.955238 2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 22 02:37:13.963013 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.955240 2578 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 22 02:37:13.963013 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.955242 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 22 02:37:13.963013 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.955245 2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 22 02:37:13.963013 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.955247 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 22 02:37:13.963013 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.955249 2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 22 02:37:13.963013 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.955251 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 22 02:37:13.963013 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.955254 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 22 02:37:13.963013 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.955256 2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 22 02:37:13.963013 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.955258 2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 22 02:37:13.963013 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.955261 2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 22 02:37:13.963013 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.955263 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 22 02:37:13.963013 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.955265 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 22 02:37:13.963013 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.955268 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 22 02:37:13.963013 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:13.955270 2578 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 22 02:37:13.963513 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:13.955274 2578 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 22 02:37:13.963513 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:13.955357 2578 server.go:962] "Client rotation is on, will bootstrap in background" Jul 22 02:37:13.963513 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:13.956970 2578 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 22 02:37:13.963513 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:13.958094 2578 server.go:1019] "Starting client certificate rotation" Jul 22 02:37:13.963513 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:13.958213 2578 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 22 02:37:13.963513 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:13.958920 2578 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 22 02:37:13.963513 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:13.960386 2578 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 22 02:37:13.963513 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:13.963171 2578 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 22 02:37:13.970315 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:13.970299 2578 log.go:25] "Validated CRI v1 runtime API" Jul 22 02:37:13.972524 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:13.972511 2578 log.go:25] "Validated CRI v1 image API" Jul 22 02:37:13.973885 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:13.973868 2578 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 22 02:37:13.976333 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:13.976314 2578 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 aa125617-08ad-47a5-836a-a789b4f38f86:/dev/nvme0n1p4 d173ecb3-d990-4c8c-a457-d24f31a12d65:/dev/nvme0n1p3] Jul 22 02:37:13.976421 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:13.976331 2578 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 22 02:37:13.981191 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:13.981083 2578 manager.go:216] Machine: {Timestamp:2026-07-22 02:37:13.979786161 +0000 UTC m=+0.298961452 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3180456 MemoryCapacity:32812175360 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2e807ea7daca599b79bc9434ebf9cb SystemUUID:ec2e807e-a7da-ca59-9b79-bc9434ebf9cb BootID:5babf0aa-6b9b-4ac1-b01c-0132c88c3737 Filesystems:[{Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406089728 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406085632 Type:vfs Inodes:4005392 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:4c:ae:61:90:3d Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:4c:ae:61:90:3d Speed:0 Mtu:9001} {Name:ovs-system MacAddress:26:1b:2b:03:71:c2 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812175360 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 22 02:37:13.981191 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:13.981185 2578 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 22 02:37:13.981313 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:13.981251 2578 manager.go:232] Version: {KernelVersion:5.14.0-570.128.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260715-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 22 02:37:13.984387 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:13.984365 2578 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 22 02:37:13.984509 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:13.984388 2578 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-133-31.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 22 02:37:13.985340 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:13.985330 2578 topology_manager.go:138] "Creating topology manager with none policy" Jul 22 02:37:13.985377 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:13.985343 2578 container_manager_linux.go:309] "Creating device plugin manager" Jul 22 02:37:13.985377 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:13.985349 2578 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 22 02:37:13.986037 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:13.986028 2578 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 22 02:37:13.987575 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:13.987552 2578 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 22 02:37:13.990409 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:13.990396 2578 state_mem.go:36] "Initialized new in-memory state store" Jul 22 02:37:13.990550 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:13.990540 2578 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 22 02:37:13.990978 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:13.990968 2578 kubelet.go:486] "Attempting to sync node with API server" Jul 22 02:37:13.991009 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:13.990984 2578 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 22 02:37:13.991009 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:13.991001 2578 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 22 02:37:13.991056 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:13.991010 2578 kubelet.go:398] "Adding apiserver pod source" Jul 22 02:37:13.991056 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:13.991017 2578 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 22 02:37:13.991335 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:13.991323 2578 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 22 02:37:13.991394 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:13.991340 2578 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 22 02:37:13.992755 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:13.992738 2578 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 22 02:37:13.993497 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:13.993481 2578 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 22 02:37:13.995306 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:13.995294 2578 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 22 02:37:13.995353 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:13.995323 2578 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 22 02:37:13.996925 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:13.996913 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 22 02:37:13.996990 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:13.996928 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 22 02:37:13.996990 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:13.996940 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 22 02:37:13.996990 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:13.996946 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 22 02:37:13.996990 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:13.996954 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 22 02:37:13.996990 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:13.996960 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 22 02:37:13.996990 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:13.996967 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 22 02:37:13.996990 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:13.996972 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 22 02:37:13.996990 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:13.996981 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 22 02:37:13.996990 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:13.996986 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 22 02:37:13.996990 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:13.996994 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 22 02:37:13.997237 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:13.996999 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 22 02:37:13.997698 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:13.997688 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 22 02:37:13.997741 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:13.997700 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 22 02:37:14.000502 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:14.000470 2578 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-133-31.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 22 02:37:14.000648 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:14.000634 2578 server.go:1268] "Started kubelet" Jul 22 02:37:14.000785 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:14.000740 2578 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 22 02:37:14.000889 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:14.000771 2578 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 22 02:37:14.000951 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:14.000914 2578 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 22 02:37:14.001496 ip-10-0-133-31 systemd[1]: Started Kubernetes Kubelet. Jul 22 02:37:14.003578 ip-10-0-133-31 kubenswrapper[2578]: E0722 02:37:14.003348 2578 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-133-31.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 22 02:37:14.003685 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:14.003659 2578 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 22 02:37:14.003960 ip-10-0-133-31 kubenswrapper[2578]: E0722 02:37:14.003470 2578 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 22 02:37:14.005981 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:14.005961 2578 server.go:310] "Adding debug handlers to kubelet server" Jul 22 02:37:14.006844 ip-10-0-133-31 kubenswrapper[2578]: E0722 02:37:14.006824 2578 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 22 02:37:14.008524 ip-10-0-133-31 kubenswrapper[2578]: E0722 02:37:14.007423 2578 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-31.ec2.internal.18c47c211c916524 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-31.ec2.internal,UID:ip-10-0-133-31.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-133-31.ec2.internal,},FirstTimestamp:2026-07-22 02:37:14.000614692 +0000 UTC m=+0.319789983,LastTimestamp:2026-07-22 02:37:14.000614692 +0000 UTC m=+0.319789983,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-31.ec2.internal,}" Jul 22 02:37:14.011856 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:14.011845 2578 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 22 02:37:14.011914 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:14.011859 2578 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 22 02:37:14.011971 ip-10-0-133-31 kubenswrapper[2578]: E0722 02:37:14.011936 2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-31.ec2.internal\" not found" Jul 22 02:37:14.012467 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:14.012439 2578 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 22 02:37:14.012581 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:14.012467 2578 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 22 02:37:14.012581 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:14.012482 2578 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 22 02:37:14.012581 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:14.012517 2578 factory.go:55] Registering systemd factory Jul 22 02:37:14.012581 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:14.012531 2578 factory.go:223] Registration of the systemd container factory successfully Jul 22 02:37:14.012776 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:14.012589 2578 reconstruct.go:97] "Volume reconstruction finished" Jul 22 02:37:14.012776 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:14.012599 2578 reconciler.go:29] "Reconciler: start to sync state" Jul 22 02:37:14.012776 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:14.012731 2578 factory.go:153] Registering CRI-O factory Jul 22 02:37:14.012776 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:14.012743 2578 factory.go:223] Registration of the crio container factory successfully Jul 22 02:37:14.012921 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:14.012789 2578 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 22 02:37:14.012921 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:14.012811 2578 factory.go:103] Registering Raw factory Jul 22 02:37:14.012921 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:14.012823 2578 manager.go:1196] Started watching for new ooms in manager Jul 22 02:37:14.013231 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:14.013219 2578 manager.go:349] Starting recovery of all containers Jul 22 02:37:14.015826 ip-10-0-133-31 kubenswrapper[2578]: E0722 02:37:14.015772 2578 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 22 02:37:14.016007 ip-10-0-133-31 kubenswrapper[2578]: E0722 02:37:14.015872 2578 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-133-31.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 22 02:37:14.016007 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:14.015989 2578 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-gkmwf" Jul 22 02:37:14.024060 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:14.023876 2578 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-gkmwf" Jul 22 02:37:14.024149 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:14.024025 2578 manager.go:354] Recovery completed Jul 22 02:37:14.028815 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:14.028803 2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 22 02:37:14.030599 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:14.030581 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-31.ec2.internal" event="NodeHasSufficientMemory" Jul 22 02:37:14.030684 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:14.030613 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-31.ec2.internal" event="NodeHasNoDiskPressure" Jul 22 02:37:14.030684 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:14.030628 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-31.ec2.internal" event="NodeHasSufficientPID" Jul 22 02:37:14.031164 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:14.031150 2578 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 22 02:37:14.031164 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:14.031164 2578 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 22 02:37:14.031164 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:14.031177 2578 state_mem.go:36] "Initialized new in-memory state store" Jul 22 02:37:14.031333 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:14.031192 2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 22 02:37:14.031866 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:14.031852 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-31.ec2.internal" event="NodeHasSufficientMemory" Jul 22 02:37:14.031911 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:14.031880 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-31.ec2.internal" event="NodeHasNoDiskPressure" Jul 22 02:37:14.031911 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:14.031890 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-31.ec2.internal" event="NodeHasSufficientPID" Jul 22 02:37:14.032088 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:14.032077 2578 policy_none.go:49] "None policy: Start" Jul 22 02:37:14.032117 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:14.032093 2578 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 22 02:37:14.032117 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:14.032103 2578 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 22 02:37:14.032976 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:14.032964 2578 policy_none.go:47] "Start" Jul 22 02:37:14.068066 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:14.068051 2578 manager.go:337] "Starting Device Plugin manager" Jul 22 02:37:14.068128 ip-10-0-133-31 kubenswrapper[2578]: E0722 02:37:14.068088 2578 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 22 02:37:14.068128 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:14.068100 2578 server.go:87] "Starting device plugin registration server" Jul 22 02:37:14.068321 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:14.068307 2578 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 22 02:37:14.068375 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:14.068326 2578 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 22 02:37:14.068472 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:14.068451 2578 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 22 02:37:14.068620 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:14.068528 2578 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 22 02:37:14.068723 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:14.068707 2578 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 22 02:37:14.068969 ip-10-0-133-31 kubenswrapper[2578]: E0722 02:37:14.068952 2578 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 22 02:37:14.069032 ip-10-0-133-31 kubenswrapper[2578]: E0722 02:37:14.069001 2578 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-133-31.ec2.internal\" not found" Jul 22 02:37:14.134268 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:14.134247 2578 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 22 02:37:14.135456 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:14.135443 2578 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 22 02:37:14.135506 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:14.135482 2578 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 22 02:37:14.135506 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:14.135503 2578 kubelet.go:2448] "Starting kubelet main sync loop" Jul 22 02:37:14.135591 ip-10-0-133-31 kubenswrapper[2578]: E0722 02:37:14.135539 2578 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 22 02:37:14.140043 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:14.140024 2578 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 22 02:37:14.168812 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:14.168792 2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 22 02:37:14.169509 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:14.169496 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-31.ec2.internal" event="NodeHasSufficientMemory" Jul 22 02:37:14.169590 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:14.169522 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-31.ec2.internal" event="NodeHasNoDiskPressure" Jul 22 02:37:14.169590 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:14.169536 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-31.ec2.internal" event="NodeHasSufficientPID" Jul 22 02:37:14.169590 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:14.169577 2578 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-133-31.ec2.internal" Jul 22 02:37:14.177094 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:14.177081 2578 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-133-31.ec2.internal" Jul 22 02:37:14.177094 ip-10-0-133-31 kubenswrapper[2578]: E0722 02:37:14.177100 2578 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-133-31.ec2.internal\": node \"ip-10-0-133-31.ec2.internal\" not found" Jul 22 02:37:14.192919 ip-10-0-133-31 kubenswrapper[2578]: E0722 02:37:14.192870 2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-31.ec2.internal\" not found" Jul 22 02:37:14.235808 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:14.235785 2578 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-31.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-133-31.ec2.internal"] Jul 22 02:37:14.235890 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:14.235834 2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 22 02:37:14.236537 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:14.236523 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-31.ec2.internal" event="NodeHasSufficientMemory" Jul 22 02:37:14.236636 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:14.236550 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-31.ec2.internal" event="NodeHasNoDiskPressure" Jul 22 02:37:14.236636 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:14.236581 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-31.ec2.internal" event="NodeHasSufficientPID" Jul 22 02:37:14.237755 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:14.237741 2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 22 02:37:14.237875 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:14.237859 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-31.ec2.internal" Jul 22 02:37:14.237920 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:14.237887 2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 22 02:37:14.238426 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:14.238411 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-31.ec2.internal" event="NodeHasSufficientMemory" Jul 22 02:37:14.238499 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:14.238434 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-31.ec2.internal" event="NodeHasNoDiskPressure" Jul 22 02:37:14.238499 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:14.238448 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-31.ec2.internal" event="NodeHasSufficientPID" Jul 22 02:37:14.238623 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:14.238449 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-31.ec2.internal" event="NodeHasSufficientMemory" Jul 22 02:37:14.238623 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:14.238595 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-31.ec2.internal" event="NodeHasNoDiskPressure" Jul 22 02:37:14.238623 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:14.238605 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-31.ec2.internal" event="NodeHasSufficientPID" Jul 22 02:37:14.239470 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:14.239455 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-31.ec2.internal" Jul 22 02:37:14.239544 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:14.239483 2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 22 02:37:14.240165 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:14.240152 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-31.ec2.internal" event="NodeHasSufficientMemory" Jul 22 02:37:14.240253 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:14.240173 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-31.ec2.internal" event="NodeHasNoDiskPressure" Jul 22 02:37:14.240253 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:14.240185 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-31.ec2.internal" event="NodeHasSufficientPID" Jul 22 02:37:14.266153 ip-10-0-133-31 kubenswrapper[2578]: E0722 02:37:14.266132 2578 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-31.ec2.internal\" not found" node="ip-10-0-133-31.ec2.internal" Jul 22 02:37:14.270185 ip-10-0-133-31 kubenswrapper[2578]: E0722 02:37:14.270171 2578 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-31.ec2.internal\" not found" node="ip-10-0-133-31.ec2.internal" Jul 22 02:37:14.293180 ip-10-0-133-31 kubenswrapper[2578]: E0722 02:37:14.293155 2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-31.ec2.internal\" not found" Jul 22 02:37:14.314087 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:14.314065 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/86e469b6aa7347caf60b105c66d89977-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-31.ec2.internal\" (UID: \"86e469b6aa7347caf60b105c66d89977\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-31.ec2.internal" Jul 22 02:37:14.314152 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:14.314090 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/86e469b6aa7347caf60b105c66d89977-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-31.ec2.internal\" (UID: \"86e469b6aa7347caf60b105c66d89977\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-31.ec2.internal" Jul 22 02:37:14.314152 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:14.314111 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/47b7203f116608cac74abbb19cda28d8-config\") pod \"kube-apiserver-proxy-ip-10-0-133-31.ec2.internal\" (UID: \"47b7203f116608cac74abbb19cda28d8\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-31.ec2.internal" Jul 22 02:37:14.394162 ip-10-0-133-31 kubenswrapper[2578]: E0722 02:37:14.394143 2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-31.ec2.internal\" not found" Jul 22 02:37:14.414502 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:14.414477 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/86e469b6aa7347caf60b105c66d89977-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-31.ec2.internal\" (UID: \"86e469b6aa7347caf60b105c66d89977\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-31.ec2.internal" Jul 22 02:37:14.414587 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:14.414575 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/86e469b6aa7347caf60b105c66d89977-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-31.ec2.internal\" (UID: \"86e469b6aa7347caf60b105c66d89977\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-31.ec2.internal" Jul 22 02:37:14.414626 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:14.414595 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/47b7203f116608cac74abbb19cda28d8-config\") pod \"kube-apiserver-proxy-ip-10-0-133-31.ec2.internal\" (UID: \"47b7203f116608cac74abbb19cda28d8\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-31.ec2.internal" Jul 22 02:37:14.414626 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:14.414528 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/86e469b6aa7347caf60b105c66d89977-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-31.ec2.internal\" (UID: \"86e469b6aa7347caf60b105c66d89977\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-31.ec2.internal" Jul 22 02:37:14.414685 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:14.414626 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/47b7203f116608cac74abbb19cda28d8-config\") pod \"kube-apiserver-proxy-ip-10-0-133-31.ec2.internal\" (UID: \"47b7203f116608cac74abbb19cda28d8\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-31.ec2.internal" Jul 22 02:37:14.414685 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:14.414663 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/86e469b6aa7347caf60b105c66d89977-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-31.ec2.internal\" (UID: \"86e469b6aa7347caf60b105c66d89977\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-31.ec2.internal" Jul 22 02:37:14.494912 ip-10-0-133-31 kubenswrapper[2578]: E0722 02:37:14.494873 2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-31.ec2.internal\" not found" Jul 22 02:37:14.569345 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:14.569326 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-31.ec2.internal" Jul 22 02:37:14.573020 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:14.573007 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-31.ec2.internal" Jul 22 02:37:14.595097 ip-10-0-133-31 kubenswrapper[2578]: E0722 02:37:14.595080 2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-31.ec2.internal\" not found" Jul 22 02:37:14.695666 ip-10-0-133-31 kubenswrapper[2578]: E0722 02:37:14.695645 2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-31.ec2.internal\" not found" Jul 22 02:37:14.796119 ip-10-0-133-31 kubenswrapper[2578]: E0722 02:37:14.796067 2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-31.ec2.internal\" not found" Jul 22 02:37:14.896638 ip-10-0-133-31 kubenswrapper[2578]: E0722 02:37:14.896615 2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-31.ec2.internal\" not found" Jul 22 02:37:14.958065 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:14.958045 2578 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 22 02:37:14.958483 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:14.958174 2578 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 22 02:37:14.997095 ip-10-0-133-31 kubenswrapper[2578]: E0722 02:37:14.997068 2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-31.ec2.internal\" not found" Jul 22 02:37:15.028532 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:15.028504 2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-21 02:32:14 +0000 UTC" deadline="2028-01-09 04:34:03.3749924 +0000 UTC" Jul 22 02:37:15.028637 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:15.028533 2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12865h56m48.346463869s" Jul 22 02:37:15.041980 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:15.041947 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47b7203f116608cac74abbb19cda28d8.slice/crio-b02fcecc5c88e57428e8563a685a2af022e34d9701effa1e7d8bf7618a36a756 WatchSource:0}: Error finding container b02fcecc5c88e57428e8563a685a2af022e34d9701effa1e7d8bf7618a36a756: Status 404 returned error can't find the container with id b02fcecc5c88e57428e8563a685a2af022e34d9701effa1e7d8bf7618a36a756 Jul 22 02:37:15.042207 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:15.042186 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86e469b6aa7347caf60b105c66d89977.slice/crio-a9130d526780eec58f5bdc25c6c9b1746ec48109c10d4c3f4369c2137aab846b WatchSource:0}: Error finding container a9130d526780eec58f5bdc25c6c9b1746ec48109c10d4c3f4369c2137aab846b: Status 404 returned error can't find the container with id a9130d526780eec58f5bdc25c6c9b1746ec48109c10d4c3f4369c2137aab846b Jul 22 02:37:15.044261 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:15.044239 2578 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 22 02:37:15.046544 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:15.046528 2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 22 02:37:15.098400 ip-10-0-133-31 kubenswrapper[2578]: E0722 02:37:15.098377 2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-31.ec2.internal\" not found" Jul 22 02:37:15.139280 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:15.139238 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-31.ec2.internal" event={"ID":"86e469b6aa7347caf60b105c66d89977","Type":"ContainerStarted","Data":"a9130d526780eec58f5bdc25c6c9b1746ec48109c10d4c3f4369c2137aab846b"} Jul 22 02:37:15.140256 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:15.140230 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-31.ec2.internal" event={"ID":"47b7203f116608cac74abbb19cda28d8","Type":"ContainerStarted","Data":"b02fcecc5c88e57428e8563a685a2af022e34d9701effa1e7d8bf7618a36a756"} Jul 22 02:37:15.189139 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:15.189118 2578 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 22 02:37:15.199032 ip-10-0-133-31 kubenswrapper[2578]: E0722 02:37:15.199016 2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-31.ec2.internal\" not found" Jul 22 02:37:15.305678 ip-10-0-133-31 kubenswrapper[2578]: E0722 02:37:15.305633 2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-31.ec2.internal\" not found" Jul 22 02:37:15.406198 ip-10-0-133-31 kubenswrapper[2578]: E0722 02:37:15.406173 2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-31.ec2.internal\" not found" Jul 22 02:37:15.433935 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:15.433912 2578 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 22 02:37:15.512620 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:15.512594 2578 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-31.ec2.internal" Jul 22 02:37:15.526018 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:15.525991 2578 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 22 02:37:15.527325 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:15.527303 2578 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-31.ec2.internal" Jul 22 02:37:15.537328 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:15.537308 2578 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 22 02:37:15.992164 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:15.992138 2578 apiserver.go:52] "Watching apiserver" Jul 22 02:37:16.002664 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:16.002636 2578 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 22 02:37:16.003691 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:16.003655 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/node-ca-8kmfc","openshift-multus/multus-8ssxs","openshift-multus/multus-additional-cni-plugins-t9vtn","openshift-multus/network-metrics-daemon-lvbjx","openshift-network-diagnostics/network-check-target-wp7zw","kube-system/global-pull-secret-syncer-rb9sf","kube-system/kube-apiserver-proxy-ip-10-0-133-31.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wjd22","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-31.ec2.internal","openshift-network-operator/iptables-alerter-ld8dc","openshift-ovn-kubernetes/ovnkube-node-prfql","kube-system/konnectivity-agent-bxg22","openshift-cluster-node-tuning-operator/tuned-ftqsf","openshift-dns/node-resolver-9dhrs"] Jul 22 02:37:16.005336 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:16.005309 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-ld8dc" Jul 22 02:37:16.006995 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:16.006970 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-ftqsf" Jul 22 02:37:16.008351 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:16.008324 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-t9vtn" Jul 22 02:37:16.008599 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:16.008557 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-f7xr2\"" Jul 22 02:37:16.008755 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:16.008739 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 22 02:37:16.008864 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:16.008847 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 22 02:37:16.008938 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:16.008738 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 22 02:37:16.009518 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:16.009498 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 22 02:37:16.009743 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:16.009723 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lvbjx" Jul 22 02:37:16.009827 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:16.009760 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-r2t5f\"" Jul 22 02:37:16.009884 ip-10-0-133-31 kubenswrapper[2578]: E0722 02:37:16.009821 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lvbjx" podUID="d6871664-adf7-41be-b1d0-d1ec7ccae7c6" Jul 22 02:37:16.009884 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:16.009842 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 22 02:37:16.011006 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:16.010978 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wp7zw" Jul 22 02:37:16.011117 ip-10-0-133-31 kubenswrapper[2578]: E0722 02:37:16.011064 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wp7zw" podUID="6b93e551-99d3-4ee4-8560-ca6b0f26b6db" Jul 22 02:37:16.011117 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:16.011094 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 22 02:37:16.011232 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:16.011168 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 22 02:37:16.011494 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:16.011475 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 22 02:37:16.011621 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:16.011504 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 22 02:37:16.011905 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:16.011876 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 22 02:37:16.011905 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:16.011890 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-66px7\"" Jul 22 02:37:16.012214 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:16.012195 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rb9sf" Jul 22 02:37:16.012291 ip-10-0-133-31 kubenswrapper[2578]: E0722 02:37:16.012274 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rb9sf" podUID="684d87e0-1cec-4247-9446-881aff58a246" Jul 22 02:37:16.013302 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:16.013137 2578 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 22 02:37:16.013715 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:16.013465 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-9dhrs" Jul 22 02:37:16.014762 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:16.014744 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wjd22" Jul 22 02:37:16.015980 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:16.015958 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 22 02:37:16.016093 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:16.016065 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 22 02:37:16.016234 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:16.016216 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-k9wxb\"" Jul 22 02:37:16.016540 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:16.016520 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-8ssxs" Jul 22 02:37:16.017953 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:16.017789 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 22 02:37:16.017953 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:16.017819 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 22 02:37:16.017953 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:16.017792 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-8kmfc" Jul 22 02:37:16.018151 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:16.018087 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-24wk5\"" Jul 22 02:37:16.018380 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:16.018354 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 22 02:37:16.019195 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:16.019179 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-prfql" Jul 22 02:37:16.020435 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:16.020411 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 22 02:37:16.020639 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:16.020619 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-x87t8\"" Jul 22 02:37:16.020745 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:16.020677 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-bxg22" Jul 22 02:37:16.020809 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:16.020759 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 22 02:37:16.020952 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:16.020933 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-9ld7b\"" Jul 22 02:37:16.021269 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:16.021252 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 22 02:37:16.021787 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:16.021659 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 22 02:37:16.021787 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:16.021694 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 22 02:37:16.022604 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:16.022583 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 22 02:37:16.022725 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:16.022657 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 22 02:37:16.022831 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:16.022590 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 22 02:37:16.022894 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:16.022866 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/9834be29-d56c-4ce7-a542-bea10d73ccfd-iptables-alerter-script\") pod \"iptables-alerter-ld8dc\" (UID: \"9834be29-d56c-4ce7-a542-bea10d73ccfd\") " pod="openshift-network-operator/iptables-alerter-ld8dc" Jul 22 02:37:16.022944 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:16.022902 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5cxbr\" (UniqueName: \"kubernetes.io/projected/6b93e551-99d3-4ee4-8560-ca6b0f26b6db-kube-api-access-5cxbr\") pod \"network-check-target-wp7zw\" (UID: \"6b93e551-99d3-4ee4-8560-ca6b0f26b6db\") " pod="openshift-network-diagnostics/network-check-target-wp7zw" Jul 22 02:37:16.022996 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:16.022950 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/6a441682-e619-4c8d-96a2-3a391f8eac76-etc-modprobe-d\") pod \"tuned-ftqsf\" (UID: \"6a441682-e619-4c8d-96a2-3a391f8eac76\") " pod="openshift-cluster-node-tuning-operator/tuned-ftqsf" Jul 22 02:37:16.023032 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:16.022993 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/34c0277e-a51b-4c45-ae71-9f339d53d946-system-cni-dir\") pod \"multus-additional-cni-plugins-t9vtn\" (UID: \"34c0277e-a51b-4c45-ae71-9f339d53d946\") " pod="openshift-multus/multus-additional-cni-plugins-t9vtn" Jul 22 02:37:16.023032 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:16.023014 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/34c0277e-a51b-4c45-ae71-9f339d53d946-os-release\") pod \"multus-additional-cni-plugins-t9vtn\" (UID: \"34c0277e-a51b-4c45-ae71-9f339d53d946\") " pod="openshift-multus/multus-additional-cni-plugins-t9vtn" Jul 22 02:37:16.023032 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:16.023030 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/684d87e0-1cec-4247-9446-881aff58a246-kubelet-config\") pod \"global-pull-secret-syncer-rb9sf\" (UID: \"684d87e0-1cec-4247-9446-881aff58a246\") " pod="kube-system/global-pull-secret-syncer-rb9sf" Jul 22 02:37:16.023147 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:16.023045 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/6a441682-e619-4c8d-96a2-3a391f8eac76-etc-sysconfig\") pod \"tuned-ftqsf\" (UID: \"6a441682-e619-4c8d-96a2-3a391f8eac76\") " pod="openshift-cluster-node-tuning-operator/tuned-ftqsf" Jul 22 02:37:16.023147 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:16.023074 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/6a441682-e619-4c8d-96a2-3a391f8eac76-etc-systemd\") pod \"tuned-ftqsf\" (UID: \"6a441682-e619-4c8d-96a2-3a391f8eac76\") " pod="openshift-cluster-node-tuning-operator/tuned-ftqsf" Jul 22 02:37:16.023147 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:16.023094 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6a441682-e619-4c8d-96a2-3a391f8eac76-sys\") pod \"tuned-ftqsf\" (UID: \"6a441682-e619-4c8d-96a2-3a391f8eac76\") " pod="openshift-cluster-node-tuning-operator/tuned-ftqsf" Jul 22 02:37:16.023147 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:16.023117 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/34c0277e-a51b-4c45-ae71-9f339d53d946-tuning-conf-dir\") pod \"multus-additional-cni-plugins-t9vtn\" (UID: \"34c0277e-a51b-4c45-ae71-9f339d53d946\") " pod="openshift-multus/multus-additional-cni-plugins-t9vtn" Jul 22 02:37:16.023147 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:16.023131 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6a441682-e619-4c8d-96a2-3a391f8eac76-tmp\") pod \"tuned-ftqsf\" (UID: \"6a441682-e619-4c8d-96a2-3a391f8eac76\") " pod="openshift-cluster-node-tuning-operator/tuned-ftqsf" Jul 22 02:37:16.023374 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:16.023156 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jvb6v\" (UniqueName: \"kubernetes.io/projected/34c0277e-a51b-4c45-ae71-9f339d53d946-kube-api-access-jvb6v\") pod \"multus-additional-cni-plugins-t9vtn\" (UID: \"34c0277e-a51b-4c45-ae71-9f339d53d946\") " pod="openshift-multus/multus-additional-cni-plugins-t9vtn" Jul 22 02:37:16.023374 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:16.023180 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dea9d10b-9320-471c-9efd-544fabc16854-tmp-dir\") pod \"node-resolver-9dhrs\" (UID: \"dea9d10b-9320-471c-9efd-544fabc16854\") " pod="openshift-dns/node-resolver-9dhrs" Jul 22 02:37:16.023374 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:16.023210 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6a441682-e619-4c8d-96a2-3a391f8eac76-lib-modules\") pod \"tuned-ftqsf\" (UID: \"6a441682-e619-4c8d-96a2-3a391f8eac76\") " pod="openshift-cluster-node-tuning-operator/tuned-ftqsf" Jul 22 02:37:16.023374 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:16.023242 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6a441682-e619-4c8d-96a2-3a391f8eac76-host\") pod \"tuned-ftqsf\" (UID: \"6a441682-e619-4c8d-96a2-3a391f8eac76\") " pod="openshift-cluster-node-tuning-operator/tuned-ftqsf" Jul 22 02:37:16.023374 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:16.023260 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/6a441682-e619-4c8d-96a2-3a391f8eac76-etc-sysctl-conf\") pod \"tuned-ftqsf\" (UID: \"6a441682-e619-4c8d-96a2-3a391f8eac76\") " pod="openshift-cluster-node-tuning-operator/tuned-ftqsf" Jul 22 02:37:16.023374 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:16.023278 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/dea9d10b-9320-471c-9efd-544fabc16854-hosts-file\") pod \"node-resolver-9dhrs\" (UID: \"dea9d10b-9320-471c-9efd-544fabc16854\") " pod="openshift-dns/node-resolver-9dhrs" Jul 22 02:37:16.023374 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:16.023321 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9834be29-d56c-4ce7-a542-bea10d73ccfd-host-slash\") pod \"iptables-alerter-ld8dc\" (UID: \"9834be29-d56c-4ce7-a542-bea10d73ccfd\") " pod="openshift-network-operator/iptables-alerter-ld8dc" Jul 22 02:37:16.023374 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:16.023358 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d6871664-adf7-41be-b1d0-d1ec7ccae7c6-metrics-certs\") pod \"network-metrics-daemon-lvbjx\" (UID: \"d6871664-adf7-41be-b1d0-d1ec7ccae7c6\") " pod="openshift-multus/network-metrics-daemon-lvbjx" Jul 22 02:37:16.023720 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:16.023404 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/34c0277e-a51b-4c45-ae71-9f339d53d946-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-t9vtn\" (UID: \"34c0277e-a51b-4c45-ae71-9f339d53d946\") " pod="openshift-multus/multus-additional-cni-plugins-t9vtn" Jul 22 02:37:16.023720 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:16.023430 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/34c0277e-a51b-4c45-ae71-9f339d53d946-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-t9vtn\" (UID: \"34c0277e-a51b-4c45-ae71-9f339d53d946\") " pod="openshift-multus/multus-additional-cni-plugins-t9vtn" Jul 22 02:37:16.023720 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:16.023449 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6a441682-e619-4c8d-96a2-3a391f8eac76-etc-kubernetes\") pod \"tuned-ftqsf\" (UID: \"6a441682-e619-4c8d-96a2-3a391f8eac76\") " pod="openshift-cluster-node-tuning-operator/tuned-ftqsf" Jul 22 02:37:16.023720 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:16.023473 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xq6gn\" (UniqueName: \"kubernetes.io/projected/6a441682-e619-4c8d-96a2-3a391f8eac76-kube-api-access-xq6gn\") pod \"tuned-ftqsf\" (UID: \"6a441682-e619-4c8d-96a2-3a391f8eac76\") " pod="openshift-cluster-node-tuning-operator/tuned-ftqsf" Jul 22 02:37:16.023720 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:16.023497 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d97cl\" (UniqueName: \"kubernetes.io/projected/9834be29-d56c-4ce7-a542-bea10d73ccfd-kube-api-access-d97cl\") pod \"iptables-alerter-ld8dc\" (UID: \"9834be29-d56c-4ce7-a542-bea10d73ccfd\") " pod="openshift-network-operator/iptables-alerter-ld8dc" Jul 22 02:37:16.023720 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:16.023511 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/34c0277e-a51b-4c45-ae71-9f339d53d946-cnibin\") pod \"multus-additional-cni-plugins-t9vtn\" (UID: \"34c0277e-a51b-4c45-ae71-9f339d53d946\") " pod="openshift-multus/multus-additional-cni-plugins-t9vtn" Jul 22 02:37:16.023720 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:16.023551 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fmxqm\" (UniqueName: \"kubernetes.io/projected/d6871664-adf7-41be-b1d0-d1ec7ccae7c6-kube-api-access-fmxqm\") pod \"network-metrics-daemon-lvbjx\" (UID: \"d6871664-adf7-41be-b1d0-d1ec7ccae7c6\") " pod="openshift-multus/network-metrics-daemon-lvbjx" Jul 22 02:37:16.023720 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:16.023604 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/684d87e0-1cec-4247-9446-881aff58a246-dbus\") pod \"global-pull-secret-syncer-rb9sf\" (UID: \"684d87e0-1cec-4247-9446-881aff58a246\") " pod="kube-system/global-pull-secret-syncer-rb9sf" Jul 22 02:37:16.023720 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:16.023628 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6a441682-e619-4c8d-96a2-3a391f8eac76-var-lib-kubelet\") pod \"tuned-ftqsf\" (UID: \"6a441682-e619-4c8d-96a2-3a391f8eac76\") " pod="openshift-cluster-node-tuning-operator/tuned-ftqsf" Jul 22 02:37:16.023720 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:16.023652 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/34c0277e-a51b-4c45-ae71-9f339d53d946-cni-binary-copy\") pod \"multus-additional-cni-plugins-t9vtn\" (UID: \"34c0277e-a51b-4c45-ae71-9f339d53d946\") " pod="openshift-multus/multus-additional-cni-plugins-t9vtn" Jul 22 02:37:16.023720 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:16.023688 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/684d87e0-1cec-4247-9446-881aff58a246-original-pull-secret\") pod \"global-pull-secret-syncer-rb9sf\" (UID: \"684d87e0-1cec-4247-9446-881aff58a246\") " pod="kube-system/global-pull-secret-syncer-rb9sf" Jul 22 02:37:16.024196 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:16.023722 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xqsbj\" (UniqueName: \"kubernetes.io/projected/dea9d10b-9320-471c-9efd-544fabc16854-kube-api-access-xqsbj\") pod \"node-resolver-9dhrs\" (UID: \"dea9d10b-9320-471c-9efd-544fabc16854\") " pod="openshift-dns/node-resolver-9dhrs" Jul 22 02:37:16.024196 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:16.023764 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/6a441682-e619-4c8d-96a2-3a391f8eac76-etc-sysctl-d\") pod \"tuned-ftqsf\" (UID: \"6a441682-e619-4c8d-96a2-3a391f8eac76\") " pod="openshift-cluster-node-tuning-operator/tuned-ftqsf" Jul 22 02:37:16.024196 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:16.023789 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/6a441682-e619-4c8d-96a2-3a391f8eac76-run\") pod \"tuned-ftqsf\" (UID: \"6a441682-e619-4c8d-96a2-3a391f8eac76\") " pod="openshift-cluster-node-tuning-operator/tuned-ftqsf" Jul 22 02:37:16.024196 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:16.023812 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/6a441682-e619-4c8d-96a2-3a391f8eac76-etc-tuned\") pod \"tuned-ftqsf\" (UID: \"6a441682-e619-4c8d-96a2-3a391f8eac76\") " pod="openshift-cluster-node-tuning-operator/tuned-ftqsf" Jul 22 02:37:16.024196 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:16.024019 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-f2cpd\"" Jul 22 02:37:16.024196 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:16.024040 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 22 02:37:16.024196 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:16.024077 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 22 02:37:16.024528 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:16.024376 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 22 02:37:16.024779 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:16.024759 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-dgk9z\"" Jul 22 02:37:16.024849 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:16.024767 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 22 02:37:16.027662 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:16.027638 2578 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 22 02:37:16.053461 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:16.053441 2578 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-g4zq6" Jul 22 02:37:16.064836 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:16.064778 2578 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-g4zq6" Jul 22 02:37:16.113086 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:16.113061 2578 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 22 02:37:16.124901 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:16.124874 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/34c0277e-a51b-4c45-ae71-9f339d53d946-cnibin\") pod \"multus-additional-cni-plugins-t9vtn\" (UID: \"34c0277e-a51b-4c45-ae71-9f339d53d946\") " pod="openshift-multus/multus-additional-cni-plugins-t9vtn" Jul 22 02:37:16.125013 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:16.124910 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/684d87e0-1cec-4247-9446-881aff58a246-dbus\") pod \"global-pull-secret-syncer-rb9sf\" (UID: \"684d87e0-1cec-4247-9446-881aff58a246\") " pod="kube-system/global-pull-secret-syncer-rb9sf" Jul 22 02:37:16.125013 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:16.124932 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/5745b751-8d56-48ab-9e80-93190f8ef935-os-release\") pod \"multus-8ssxs\" (UID: \"5745b751-8d56-48ab-9e80-93190f8ef935\") " pod="openshift-multus/multus-8ssxs" Jul 22 02:37:16.125013 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:16.124947 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/fd7bf923-8d16-476c-a8d0-a3e6fd645061-node-log\") pod \"ovnkube-node-prfql\" (UID: \"fd7bf923-8d16-476c-a8d0-a3e6fd645061\") " pod="openshift-ovn-kubernetes/ovnkube-node-prfql" Jul 22 02:37:16.125013 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:16.124966 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/6a441682-e619-4c8d-96a2-3a391f8eac76-etc-sysctl-d\") pod \"tuned-ftqsf\" (UID: \"6a441682-e619-4c8d-96a2-3a391f8eac76\") " pod="openshift-cluster-node-tuning-operator/tuned-ftqsf" Jul 22 02:37:16.125013 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:16.124991 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/6a441682-e619-4c8d-96a2-3a391f8eac76-etc-tuned\") pod \"tuned-ftqsf\" (UID: \"6a441682-e619-4c8d-96a2-3a391f8eac76\") " pod="openshift-cluster-node-tuning-operator/tuned-ftqsf" Jul 22 02:37:16.125013 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:16.125007 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/34c0277e-a51b-4c45-ae71-9f339d53d946-cnibin\") pod \"multus-additional-cni-plugins-t9vtn\" (UID: \"34c0277e-a51b-4c45-ae71-9f339d53d946\") " pod="openshift-multus/multus-additional-cni-plugins-t9vtn" Jul 22 02:37:16.125265 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:16.125014 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/9834be29-d56c-4ce7-a542-bea10d73ccfd-iptables-alerter-script\") pod \"iptables-alerter-ld8dc\" (UID: \"9834be29-d56c-4ce7-a542-bea10d73ccfd\") " pod="openshift-network-operator/iptables-alerter-ld8dc" Jul 22 02:37:16.125265 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:16.125047 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c75eb074-c486-4152-9d81-b12360b19df0-registration-dir\") pod \"aws-ebs-csi-driver-node-wjd22\" (UID: \"c75eb074-c486-4152-9d81-b12360b19df0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wjd22" Jul 22 02:37:16.125265 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:16.125072 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-65ddv\" (UniqueName: \"kubernetes.io/projected/5745b751-8d56-48ab-9e80-93190f8ef935-kube-api-access-65ddv\") pod \"multus-8ssxs\" (UID: \"5745b751-8d56-48ab-9e80-93190f8ef935\") " pod="openshift-multus/multus-8ssxs" Jul 22 02:37:16.125265 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:16.125092 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/684d87e0-1cec-4247-9446-881aff58a246-dbus\") pod \"global-pull-secret-syncer-rb9sf\" (UID: \"684d87e0-1cec-4247-9446-881aff58a246\") " pod="kube-system/global-pull-secret-syncer-rb9sf" Jul 22 02:37:16.125265 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:16.125097 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/fd7bf923-8d16-476c-a8d0-a3e6fd645061-host-kubelet\") pod \"ovnkube-node-prfql\" (UID: \"fd7bf923-8d16-476c-a8d0-a3e6fd645061\") " pod="openshift-ovn-kubernetes/ovnkube-node-prfql" Jul 22 02:37:16.125265 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:16.125148 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fd7bf923-8d16-476c-a8d0-a3e6fd645061-run-openvswitch\") pod \"ovnkube-node-prfql\" (UID: \"fd7bf923-8d16-476c-a8d0-a3e6fd645061\") " pod="openshift-ovn-kubernetes/ovnkube-node-prfql" Jul 22 02:37:16.125265 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:16.125168 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/6a441682-e619-4c8d-96a2-3a391f8eac76-etc-sysctl-d\") pod \"tuned-ftqsf\" (UID: \"6a441682-e619-4c8d-96a2-3a391f8eac76\") " pod="openshift-cluster-node-tuning-operator/tuned-ftqsf" Jul 22 02:37:16.125265 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:16.125175 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/6a441682-e619-4c8d-96a2-3a391f8eac76-etc-modprobe-d\") pod \"tuned-ftqsf\" (UID: \"6a441682-e619-4c8d-96a2-3a391f8eac76\") " pod="openshift-cluster-node-tuning-operator/tuned-ftqsf" Jul 22 02:37:16.125265 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:16.125225 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/34c0277e-a51b-4c45-ae71-9f339d53d946-system-cni-dir\") pod \"multus-additional-cni-plugins-t9vtn\" (UID: \"34c0277e-a51b-4c45-ae71-9f339d53d946\") " pod="openshift-multus/multus-additional-cni-plugins-t9vtn" Jul 22 02:37:16.125265 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:16.125252 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/684d87e0-1cec-4247-9446-881aff58a246-kubelet-config\") pod \"global-pull-secret-syncer-rb9sf\" (UID: \"684d87e0-1cec-4247-9446-881aff58a246\") " pod="kube-system/global-pull-secret-syncer-rb9sf" Jul 22 02:37:16.125265 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:16.125263 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/6a441682-e619-4c8d-96a2-3a391f8eac76-etc-modprobe-d\") pod \"tuned-ftqsf\" (UID: \"6a441682-e619-4c8d-96a2-3a391f8eac76\") " pod="openshift-cluster-node-tuning-operator/tuned-ftqsf" Jul 22 02:37:16.125785 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:16.125279 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c75eb074-c486-4152-9d81-b12360b19df0-sys-fs\") pod \"aws-ebs-csi-driver-node-wjd22\" (UID: \"c75eb074-c486-4152-9d81-b12360b19df0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wjd22" Jul 22 02:37:16.125785 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:16.125295 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/34c0277e-a51b-4c45-ae71-9f339d53d946-system-cni-dir\") pod \"multus-additional-cni-plugins-t9vtn\" (UID: \"34c0277e-a51b-4c45-ae71-9f339d53d946\") " pod="openshift-multus/multus-additional-cni-plugins-t9vtn" Jul 22 02:37:16.125785 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:16.125327 2578 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 22 02:37:16.125785 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:16.125334 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/fd7bf923-8d16-476c-a8d0-a3e6fd645061-host-cni-netd\") pod \"ovnkube-node-prfql\" (UID: \"fd7bf923-8d16-476c-a8d0-a3e6fd645061\") " pod="openshift-ovn-kubernetes/ovnkube-node-prfql" Jul 22 02:37:16.125785 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:16.125349 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/684d87e0-1cec-4247-9446-881aff58a246-kubelet-config\") pod \"global-pull-secret-syncer-rb9sf\" (UID: \"684d87e0-1cec-4247-9446-881aff58a246\") " pod="kube-system/global-pull-secret-syncer-rb9sf" Jul 22 02:37:16.125785 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:16.125389 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/6a441682-e619-4c8d-96a2-3a391f8eac76-etc-systemd\") pod \"tuned-ftqsf\" (UID: \"6a441682-e619-4c8d-96a2-3a391f8eac76\") " pod="openshift-cluster-node-tuning-operator/tuned-ftqsf" Jul 22 02:37:16.125785 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:16.125421 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6a441682-e619-4c8d-96a2-3a391f8eac76-sys\") pod \"tuned-ftqsf\" (UID: \"6a441682-e619-4c8d-96a2-3a391f8eac76\") " pod="openshift-cluster-node-tuning-operator/tuned-ftqsf" Jul 22 02:37:16.125785 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:16.125450 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c75eb074-c486-4152-9d81-b12360b19df0-etc-selinux\") pod \"aws-ebs-csi-driver-node-wjd22\" (UID: \"c75eb074-c486-4152-9d81-b12360b19df0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wjd22" Jul 22 02:37:16.125785 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:16.125473 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6a441682-e619-4c8d-96a2-3a391f8eac76-sys\") pod \"tuned-ftqsf\" (UID: \"6a441682-e619-4c8d-96a2-3a391f8eac76\") " pod="openshift-cluster-node-tuning-operator/tuned-ftqsf" Jul 22 02:37:16.125785 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:16.125476 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5745b751-8d56-48ab-9e80-93190f8ef935-host-var-lib-cni-multus\") pod \"multus-8ssxs\" (UID: \"5745b751-8d56-48ab-9e80-93190f8ef935\") " pod="openshift-multus/multus-8ssxs" Jul 22 02:37:16.125785 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:16.125481 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/6a441682-e619-4c8d-96a2-3a391f8eac76-etc-systemd\") pod \"tuned-ftqsf\" (UID: \"6a441682-e619-4c8d-96a2-3a391f8eac76\") " pod="openshift-cluster-node-tuning-operator/tuned-ftqsf" Jul 22 02:37:16.125785 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:16.125539 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d7953894-cdfc-48a3-afe8-89503dfc3c57-host\") pod \"node-ca-8kmfc\" (UID: \"d7953894-cdfc-48a3-afe8-89503dfc3c57\") " pod="openshift-image-registry/node-ca-8kmfc" Jul 22 02:37:16.125785 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:16.125603 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fd7bf923-8d16-476c-a8d0-a3e6fd645061-host-cni-bin\") pod \"ovnkube-node-prfql\" (UID: \"fd7bf923-8d16-476c-a8d0-a3e6fd645061\") " pod="openshift-ovn-kubernetes/ovnkube-node-prfql" Jul 22 02:37:16.125785 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:16.125607 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/9834be29-d56c-4ce7-a542-bea10d73ccfd-iptables-alerter-script\") pod \"iptables-alerter-ld8dc\" (UID: \"9834be29-d56c-4ce7-a542-bea10d73ccfd\") " pod="openshift-network-operator/iptables-alerter-ld8dc" Jul 22 02:37:16.125785 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:16.125633 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6a441682-e619-4c8d-96a2-3a391f8eac76-tmp\") pod \"tuned-ftqsf\" (UID: \"6a441682-e619-4c8d-96a2-3a391f8eac76\") " pod="openshift-cluster-node-tuning-operator/tuned-ftqsf" Jul 22 02:37:16.125785 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:16.125692 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5745b751-8d56-48ab-9e80-93190f8ef935-system-cni-dir\") pod \"multus-8ssxs\" (UID: \"5745b751-8d56-48ab-9e80-93190f8ef935\") " pod="openshift-multus/multus-8ssxs" Jul 22 02:37:16.125785 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:16.125712 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5745b751-8d56-48ab-9e80-93190f8ef935-multus-cni-dir\") pod \"multus-8ssxs\" (UID: \"5745b751-8d56-48ab-9e80-93190f8ef935\") " pod="openshift-multus/multus-8ssxs" Jul 22 02:37:16.126810 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:16.125727 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5745b751-8d56-48ab-9e80-93190f8ef935-host-var-lib-cni-bin\") pod \"multus-8ssxs\" (UID: \"5745b751-8d56-48ab-9e80-93190f8ef935\") " pod="openshift-multus/multus-8ssxs" Jul 22 02:37:16.126810 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:16.125749 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5745b751-8d56-48ab-9e80-93190f8ef935-etc-kubernetes\") pod \"multus-8ssxs\" (UID: \"5745b751-8d56-48ab-9e80-93190f8ef935\") " pod="openshift-multus/multus-8ssxs" Jul 22 02:37:16.126810 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:16.125775 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/fd7bf923-8d16-476c-a8d0-a3e6fd645061-run-systemd\") pod \"ovnkube-node-prfql\" (UID: \"fd7bf923-8d16-476c-a8d0-a3e6fd645061\") " pod="openshift-ovn-kubernetes/ovnkube-node-prfql" Jul 22 02:37:16.126810 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:16.125800 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6a441682-e619-4c8d-96a2-3a391f8eac76-lib-modules\") pod \"tuned-ftqsf\" (UID: \"6a441682-e619-4c8d-96a2-3a391f8eac76\") " pod="openshift-cluster-node-tuning-operator/tuned-ftqsf" Jul 22 02:37:16.126810 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:16.125825 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6a441682-e619-4c8d-96a2-3a391f8eac76-host\") pod \"tuned-ftqsf\" (UID: \"6a441682-e619-4c8d-96a2-3a391f8eac76\") " pod="openshift-cluster-node-tuning-operator/tuned-ftqsf" Jul 22 02:37:16.126810 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:16.125850 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e56db148-7293-423e-b9fb-0ebd5db740a3-agent-certs\") pod \"konnectivity-agent-bxg22\" (UID: \"e56db148-7293-423e-b9fb-0ebd5db740a3\") " pod="kube-system/konnectivity-agent-bxg22" Jul 22 02:37:16.126810 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:16.125917 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6a441682-e619-4c8d-96a2-3a391f8eac76-host\") pod \"tuned-ftqsf\" (UID: \"6a441682-e619-4c8d-96a2-3a391f8eac76\") " pod="openshift-cluster-node-tuning-operator/tuned-ftqsf" Jul 22 02:37:16.126810 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:16.125954 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c75eb074-c486-4152-9d81-b12360b19df0-device-dir\") pod \"aws-ebs-csi-driver-node-wjd22\" (UID: \"c75eb074-c486-4152-9d81-b12360b19df0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wjd22" Jul 22 02:37:16.126810 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:16.125958 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6a441682-e619-4c8d-96a2-3a391f8eac76-lib-modules\") pod \"tuned-ftqsf\" (UID: \"6a441682-e619-4c8d-96a2-3a391f8eac76\") " pod="openshift-cluster-node-tuning-operator/tuned-ftqsf" Jul 22 02:37:16.126810 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:16.125977 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fd7bf923-8d16-476c-a8d0-a3e6fd645061-var-lib-openvswitch\") pod \"ovnkube-node-prfql\" (UID: \"fd7bf923-8d16-476c-a8d0-a3e6fd645061\") " pod="openshift-ovn-kubernetes/ovnkube-node-prfql" Jul 22 02:37:16.126810 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:16.125999 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/fd7bf923-8d16-476c-a8d0-a3e6fd645061-ovn-node-metrics-cert\") pod \"ovnkube-node-prfql\" (UID: \"fd7bf923-8d16-476c-a8d0-a3e6fd645061\") " pod="openshift-ovn-kubernetes/ovnkube-node-prfql" Jul 22 02:37:16.126810 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:16.126030 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d6871664-adf7-41be-b1d0-d1ec7ccae7c6-metrics-certs\") pod \"network-metrics-daemon-lvbjx\" (UID: \"d6871664-adf7-41be-b1d0-d1ec7ccae7c6\") " pod="openshift-multus/network-metrics-daemon-lvbjx" Jul 22 02:37:16.126810 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:16.126080 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5745b751-8d56-48ab-9e80-93190f8ef935-host-run-netns\") pod \"multus-8ssxs\" (UID: \"5745b751-8d56-48ab-9e80-93190f8ef935\") " pod="openshift-multus/multus-8ssxs" Jul 22 02:37:16.126810 ip-10-0-133-31 kubenswrapper[2578]: E0722 02:37:16.126107 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 02:37:16.126810 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:16.126116 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5745b751-8d56-48ab-9e80-93190f8ef935-hostroot\") pod \"multus-8ssxs\" (UID: \"5745b751-8d56-48ab-9e80-93190f8ef935\") " pod="openshift-multus/multus-8ssxs" Jul 22 02:37:16.126810 ip-10-0-133-31 kubenswrapper[2578]: E0722 02:37:16.126173 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d6871664-adf7-41be-b1d0-d1ec7ccae7c6-metrics-certs podName:d6871664-adf7-41be-b1d0-d1ec7ccae7c6 nodeName:}" failed. No retries permitted until 2026-07-22 02:37:16.626154696 +0000 UTC m=+2.945329977 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d6871664-adf7-41be-b1d0-d1ec7ccae7c6-metrics-certs") pod "network-metrics-daemon-lvbjx" (UID: "d6871664-adf7-41be-b1d0-d1ec7ccae7c6") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 02:37:16.126810 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:16.126240 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xqsbj\" (UniqueName: \"kubernetes.io/projected/dea9d10b-9320-471c-9efd-544fabc16854-kube-api-access-xqsbj\") pod \"node-resolver-9dhrs\" (UID: \"dea9d10b-9320-471c-9efd-544fabc16854\") " pod="openshift-dns/node-resolver-9dhrs" Jul 22 02:37:16.127664 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:16.126364 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5745b751-8d56-48ab-9e80-93190f8ef935-cni-binary-copy\") pod \"multus-8ssxs\" (UID: \"5745b751-8d56-48ab-9e80-93190f8ef935\") " pod="openshift-multus/multus-8ssxs" Jul 22 02:37:16.127664 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:16.126407 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/34c0277e-a51b-4c45-ae71-9f339d53d946-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-t9vtn\" (UID: \"34c0277e-a51b-4c45-ae71-9f339d53d946\") " pod="openshift-multus/multus-additional-cni-plugins-t9vtn" Jul 22 02:37:16.127664 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:16.126436 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/34c0277e-a51b-4c45-ae71-9f339d53d946-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-t9vtn\" (UID: \"34c0277e-a51b-4c45-ae71-9f339d53d946\") " pod="openshift-multus/multus-additional-cni-plugins-t9vtn" Jul 22 02:37:16.127664 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:16.126462 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6a441682-e619-4c8d-96a2-3a391f8eac76-etc-kubernetes\") pod \"tuned-ftqsf\" (UID: \"6a441682-e619-4c8d-96a2-3a391f8eac76\") " pod="openshift-cluster-node-tuning-operator/tuned-ftqsf" Jul 22 02:37:16.127664 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:16.126511 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xq6gn\" (UniqueName: \"kubernetes.io/projected/6a441682-e619-4c8d-96a2-3a391f8eac76-kube-api-access-xq6gn\") pod \"tuned-ftqsf\" (UID: \"6a441682-e619-4c8d-96a2-3a391f8eac76\") " pod="openshift-cluster-node-tuning-operator/tuned-ftqsf" Jul 22 02:37:16.127664 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:16.126537 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d97cl\" (UniqueName: \"kubernetes.io/projected/9834be29-d56c-4ce7-a542-bea10d73ccfd-kube-api-access-d97cl\") pod \"iptables-alerter-ld8dc\" (UID: \"9834be29-d56c-4ce7-a542-bea10d73ccfd\") " pod="openshift-network-operator/iptables-alerter-ld8dc" Jul 22 02:37:16.127664 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:16.126562 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fmxqm\" (UniqueName: \"kubernetes.io/projected/d6871664-adf7-41be-b1d0-d1ec7ccae7c6-kube-api-access-fmxqm\") pod \"network-metrics-daemon-lvbjx\" (UID: \"d6871664-adf7-41be-b1d0-d1ec7ccae7c6\") " pod="openshift-multus/network-metrics-daemon-lvbjx" Jul 22 02:37:16.127664 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:16.126603 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6a441682-e619-4c8d-96a2-3a391f8eac76-var-lib-kubelet\") pod \"tuned-ftqsf\" (UID: \"6a441682-e619-4c8d-96a2-3a391f8eac76\") " pod="openshift-cluster-node-tuning-operator/tuned-ftqsf" Jul 22 02:37:16.127664 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:16.126628 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/34c0277e-a51b-4c45-ae71-9f339d53d946-cni-binary-copy\") pod \"multus-additional-cni-plugins-t9vtn\" (UID: \"34c0277e-a51b-4c45-ae71-9f339d53d946\") " pod="openshift-multus/multus-additional-cni-plugins-t9vtn" Jul 22 02:37:16.127664 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:16.126655 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/684d87e0-1cec-4247-9446-881aff58a246-original-pull-secret\") pod \"global-pull-secret-syncer-rb9sf\" (UID: \"684d87e0-1cec-4247-9446-881aff58a246\") " pod="kube-system/global-pull-secret-syncer-rb9sf" Jul 22 02:37:16.127664 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:16.126701 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fd7bf923-8d16-476c-a8d0-a3e6fd645061-host-run-netns\") pod \"ovnkube-node-prfql\" (UID: \"fd7bf923-8d16-476c-a8d0-a3e6fd645061\") " pod="openshift-ovn-kubernetes/ovnkube-node-prfql" Jul 22 02:37:16.127664 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:16.126726 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fd7bf923-8d16-476c-a8d0-a3e6fd645061-etc-openvswitch\") pod \"ovnkube-node-prfql\" (UID: \"fd7bf923-8d16-476c-a8d0-a3e6fd645061\") " pod="openshift-ovn-kubernetes/ovnkube-node-prfql" Jul 22 02:37:16.127664 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:16.126750 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/fd7bf923-8d16-476c-a8d0-a3e6fd645061-ovnkube-script-lib\") pod \"ovnkube-node-prfql\" (UID: \"fd7bf923-8d16-476c-a8d0-a3e6fd645061\") " pod="openshift-ovn-kubernetes/ovnkube-node-prfql" Jul 22 02:37:16.127664 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:16.126772 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wgd6p\" (UniqueName: \"kubernetes.io/projected/fd7bf923-8d16-476c-a8d0-a3e6fd645061-kube-api-access-wgd6p\") pod \"ovnkube-node-prfql\" (UID: \"fd7bf923-8d16-476c-a8d0-a3e6fd645061\") " pod="openshift-ovn-kubernetes/ovnkube-node-prfql" Jul 22 02:37:16.127664 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:16.126815 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/6a441682-e619-4c8d-96a2-3a391f8eac76-run\") pod \"tuned-ftqsf\" (UID: \"6a441682-e619-4c8d-96a2-3a391f8eac76\") " pod="openshift-cluster-node-tuning-operator/tuned-ftqsf" Jul 22 02:37:16.127664 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:16.126834 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5cxbr\" (UniqueName: \"kubernetes.io/projected/6b93e551-99d3-4ee4-8560-ca6b0f26b6db-kube-api-access-5cxbr\") pod \"network-check-target-wp7zw\" (UID: \"6b93e551-99d3-4ee4-8560-ca6b0f26b6db\") " pod="openshift-network-diagnostics/network-check-target-wp7zw" Jul 22 02:37:16.128468 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:16.126854 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/5745b751-8d56-48ab-9e80-93190f8ef935-multus-daemon-config\") pod \"multus-8ssxs\" (UID: \"5745b751-8d56-48ab-9e80-93190f8ef935\") " pod="openshift-multus/multus-8ssxs" Jul 22 02:37:16.128468 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:16.126878 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/34c0277e-a51b-4c45-ae71-9f339d53d946-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-t9vtn\" (UID: \"34c0277e-a51b-4c45-ae71-9f339d53d946\") " pod="openshift-multus/multus-additional-cni-plugins-t9vtn" Jul 22 02:37:16.128468 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:16.126890 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fd7bf923-8d16-476c-a8d0-a3e6fd645061-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-prfql\" (UID: \"fd7bf923-8d16-476c-a8d0-a3e6fd645061\") " pod="openshift-ovn-kubernetes/ovnkube-node-prfql" Jul 22 02:37:16.128468 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:16.126923 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/34c0277e-a51b-4c45-ae71-9f339d53d946-os-release\") pod \"multus-additional-cni-plugins-t9vtn\" (UID: \"34c0277e-a51b-4c45-ae71-9f339d53d946\") " pod="openshift-multus/multus-additional-cni-plugins-t9vtn" Jul 22 02:37:16.128468 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:16.126945 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vtwmb\" (UniqueName: \"kubernetes.io/projected/c75eb074-c486-4152-9d81-b12360b19df0-kube-api-access-vtwmb\") pod \"aws-ebs-csi-driver-node-wjd22\" (UID: \"c75eb074-c486-4152-9d81-b12360b19df0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wjd22" Jul 22 02:37:16.128468 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:16.126961 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5745b751-8d56-48ab-9e80-93190f8ef935-host-var-lib-kubelet\") pod \"multus-8ssxs\" (UID: \"5745b751-8d56-48ab-9e80-93190f8ef935\") " pod="openshift-multus/multus-8ssxs" Jul 22 02:37:16.128468 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:16.126979 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/6a441682-e619-4c8d-96a2-3a391f8eac76-etc-sysconfig\") pod \"tuned-ftqsf\" (UID: \"6a441682-e619-4c8d-96a2-3a391f8eac76\") " pod="openshift-cluster-node-tuning-operator/tuned-ftqsf" Jul 22 02:37:16.128468 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:16.126995 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/34c0277e-a51b-4c45-ae71-9f339d53d946-tuning-conf-dir\") pod \"multus-additional-cni-plugins-t9vtn\" (UID: \"34c0277e-a51b-4c45-ae71-9f339d53d946\") " pod="openshift-multus/multus-additional-cni-plugins-t9vtn" Jul 22 02:37:16.128468 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:16.127016 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5745b751-8d56-48ab-9e80-93190f8ef935-multus-conf-dir\") pod \"multus-8ssxs\" (UID: \"5745b751-8d56-48ab-9e80-93190f8ef935\") " pod="openshift-multus/multus-8ssxs" Jul 22 02:37:16.128468 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:16.127033 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/fd7bf923-8d16-476c-a8d0-a3e6fd645061-systemd-units\") pod \"ovnkube-node-prfql\" (UID: \"fd7bf923-8d16-476c-a8d0-a3e6fd645061\") " pod="openshift-ovn-kubernetes/ovnkube-node-prfql" Jul 22 02:37:16.128468 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:16.127051 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvb6v\" (UniqueName: \"kubernetes.io/projected/34c0277e-a51b-4c45-ae71-9f339d53d946-kube-api-access-jvb6v\") pod \"multus-additional-cni-plugins-t9vtn\" (UID: \"34c0277e-a51b-4c45-ae71-9f339d53d946\") " pod="openshift-multus/multus-additional-cni-plugins-t9vtn" Jul 22 02:37:16.128468 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:16.127066 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dea9d10b-9320-471c-9efd-544fabc16854-tmp-dir\") pod \"node-resolver-9dhrs\" (UID: \"dea9d10b-9320-471c-9efd-544fabc16854\") " pod="openshift-dns/node-resolver-9dhrs" Jul 22 02:37:16.128468 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:16.127081 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e56db148-7293-423e-b9fb-0ebd5db740a3-konnectivity-ca\") pod \"konnectivity-agent-bxg22\" (UID: \"e56db148-7293-423e-b9fb-0ebd5db740a3\") " pod="kube-system/konnectivity-agent-bxg22" Jul 22 02:37:16.128468 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:16.127114 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c75eb074-c486-4152-9d81-b12360b19df0-socket-dir\") pod \"aws-ebs-csi-driver-node-wjd22\" (UID: \"c75eb074-c486-4152-9d81-b12360b19df0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wjd22" Jul 22 02:37:16.128468 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:16.127133 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d7953894-cdfc-48a3-afe8-89503dfc3c57-serviceca\") pod \"node-ca-8kmfc\" (UID: \"d7953894-cdfc-48a3-afe8-89503dfc3c57\") " pod="openshift-image-registry/node-ca-8kmfc" Jul 22 02:37:16.128468 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:16.127149 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/6a441682-e619-4c8d-96a2-3a391f8eac76-etc-sysctl-conf\") pod \"tuned-ftqsf\" (UID: \"6a441682-e619-4c8d-96a2-3a391f8eac76\") " pod="openshift-cluster-node-tuning-operator/tuned-ftqsf" Jul 22 02:37:16.129220 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:16.127165 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/dea9d10b-9320-471c-9efd-544fabc16854-hosts-file\") pod \"node-resolver-9dhrs\" (UID: \"dea9d10b-9320-471c-9efd-544fabc16854\") " pod="openshift-dns/node-resolver-9dhrs" Jul 22 02:37:16.129220 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:16.127196 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c75eb074-c486-4152-9d81-b12360b19df0-kubelet-dir\") pod \"aws-ebs-csi-driver-node-wjd22\" (UID: \"c75eb074-c486-4152-9d81-b12360b19df0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wjd22" Jul 22 02:37:16.129220 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:16.127218 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5745b751-8d56-48ab-9e80-93190f8ef935-cnibin\") pod \"multus-8ssxs\" (UID: \"5745b751-8d56-48ab-9e80-93190f8ef935\") " pod="openshift-multus/multus-8ssxs" Jul 22 02:37:16.129220 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:16.127235 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5745b751-8d56-48ab-9e80-93190f8ef935-multus-socket-dir-parent\") pod \"multus-8ssxs\" (UID: \"5745b751-8d56-48ab-9e80-93190f8ef935\") " pod="openshift-multus/multus-8ssxs" Jul 22 02:37:16.129220 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:16.127250 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5745b751-8d56-48ab-9e80-93190f8ef935-host-run-k8s-cni-cncf-io\") pod \"multus-8ssxs\" (UID: \"5745b751-8d56-48ab-9e80-93190f8ef935\") " pod="openshift-multus/multus-8ssxs" Jul 22 02:37:16.129220 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:16.127286 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dms5p\" (UniqueName: \"kubernetes.io/projected/d7953894-cdfc-48a3-afe8-89503dfc3c57-kube-api-access-dms5p\") pod \"node-ca-8kmfc\" (UID: \"d7953894-cdfc-48a3-afe8-89503dfc3c57\") " pod="openshift-image-registry/node-ca-8kmfc" Jul 22 02:37:16.129220 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:16.127303 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fd7bf923-8d16-476c-a8d0-a3e6fd645061-host-slash\") pod \"ovnkube-node-prfql\" (UID: \"fd7bf923-8d16-476c-a8d0-a3e6fd645061\") " pod="openshift-ovn-kubernetes/ovnkube-node-prfql" Jul 22 02:37:16.129220 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:16.127320 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9834be29-d56c-4ce7-a542-bea10d73ccfd-host-slash\") pod \"iptables-alerter-ld8dc\" (UID: \"9834be29-d56c-4ce7-a542-bea10d73ccfd\") " pod="openshift-network-operator/iptables-alerter-ld8dc" Jul 22 02:37:16.129220 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:16.127335 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5745b751-8d56-48ab-9e80-93190f8ef935-host-run-multus-certs\") pod \"multus-8ssxs\" (UID: \"5745b751-8d56-48ab-9e80-93190f8ef935\") " pod="openshift-multus/multus-8ssxs" Jul 22 02:37:16.129220 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:16.127344 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/34c0277e-a51b-4c45-ae71-9f339d53d946-cni-binary-copy\") pod \"multus-additional-cni-plugins-t9vtn\" (UID: \"34c0277e-a51b-4c45-ae71-9f339d53d946\") " pod="openshift-multus/multus-additional-cni-plugins-t9vtn" Jul 22 02:37:16.129220 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:16.127372 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/fd7bf923-8d16-476c-a8d0-a3e6fd645061-run-ovn\") pod \"ovnkube-node-prfql\" (UID: \"fd7bf923-8d16-476c-a8d0-a3e6fd645061\") " pod="openshift-ovn-kubernetes/ovnkube-node-prfql" Jul 22 02:37:16.129220 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:16.127390 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/fd7bf923-8d16-476c-a8d0-a3e6fd645061-log-socket\") pod \"ovnkube-node-prfql\" (UID: \"fd7bf923-8d16-476c-a8d0-a3e6fd645061\") " pod="openshift-ovn-kubernetes/ovnkube-node-prfql" Jul 22 02:37:16.129220 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:16.127405 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fd7bf923-8d16-476c-a8d0-a3e6fd645061-host-run-ovn-kubernetes\") pod \"ovnkube-node-prfql\" (UID: \"fd7bf923-8d16-476c-a8d0-a3e6fd645061\") " pod="openshift-ovn-kubernetes/ovnkube-node-prfql" Jul 22 02:37:16.129220 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:16.127433 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/fd7bf923-8d16-476c-a8d0-a3e6fd645061-ovnkube-config\") pod \"ovnkube-node-prfql\" (UID: \"fd7bf923-8d16-476c-a8d0-a3e6fd645061\") " pod="openshift-ovn-kubernetes/ovnkube-node-prfql" Jul 22 02:37:16.129220 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:16.127452 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/fd7bf923-8d16-476c-a8d0-a3e6fd645061-env-overrides\") pod \"ovnkube-node-prfql\" (UID: \"fd7bf923-8d16-476c-a8d0-a3e6fd645061\") " pod="openshift-ovn-kubernetes/ovnkube-node-prfql" Jul 22 02:37:16.129220 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:16.127509 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6a441682-e619-4c8d-96a2-3a391f8eac76-etc-kubernetes\") pod \"tuned-ftqsf\" (UID: \"6a441682-e619-4c8d-96a2-3a391f8eac76\") " pod="openshift-cluster-node-tuning-operator/tuned-ftqsf" Jul 22 02:37:16.129220 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:16.127540 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/6a441682-e619-4c8d-96a2-3a391f8eac76-run\") pod \"tuned-ftqsf\" (UID: \"6a441682-e619-4c8d-96a2-3a391f8eac76\") " pod="openshift-cluster-node-tuning-operator/tuned-ftqsf" Jul 22 02:37:16.129865 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:16.127610 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6a441682-e619-4c8d-96a2-3a391f8eac76-var-lib-kubelet\") pod \"tuned-ftqsf\" (UID: \"6a441682-e619-4c8d-96a2-3a391f8eac76\") " pod="openshift-cluster-node-tuning-operator/tuned-ftqsf" Jul 22 02:37:16.129865 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:16.127654 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9834be29-d56c-4ce7-a542-bea10d73ccfd-host-slash\") pod \"iptables-alerter-ld8dc\" (UID: \"9834be29-d56c-4ce7-a542-bea10d73ccfd\") " pod="openshift-network-operator/iptables-alerter-ld8dc" Jul 22 02:37:16.129865 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:16.127659 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/34c0277e-a51b-4c45-ae71-9f339d53d946-os-release\") pod \"multus-additional-cni-plugins-t9vtn\" (UID: \"34c0277e-a51b-4c45-ae71-9f339d53d946\") " pod="openshift-multus/multus-additional-cni-plugins-t9vtn" Jul 22 02:37:16.129865 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:16.127671 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/6a441682-e619-4c8d-96a2-3a391f8eac76-etc-sysctl-conf\") pod \"tuned-ftqsf\" (UID: \"6a441682-e619-4c8d-96a2-3a391f8eac76\") " pod="openshift-cluster-node-tuning-operator/tuned-ftqsf" Jul 22 02:37:16.129865 ip-10-0-133-31 kubenswrapper[2578]: E0722 02:37:16.126962 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 22 02:37:16.129865 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:16.127737 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/dea9d10b-9320-471c-9efd-544fabc16854-hosts-file\") pod \"node-resolver-9dhrs\" (UID: \"dea9d10b-9320-471c-9efd-544fabc16854\") " pod="openshift-dns/node-resolver-9dhrs" Jul 22 02:37:16.129865 ip-10-0-133-31 kubenswrapper[2578]: E0722 02:37:16.127785 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/684d87e0-1cec-4247-9446-881aff58a246-original-pull-secret podName:684d87e0-1cec-4247-9446-881aff58a246 nodeName:}" failed. No retries permitted until 2026-07-22 02:37:16.627766494 +0000 UTC m=+2.946941791 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/684d87e0-1cec-4247-9446-881aff58a246-original-pull-secret") pod "global-pull-secret-syncer-rb9sf" (UID: "684d87e0-1cec-4247-9446-881aff58a246") : object "kube-system"/"original-pull-secret" not registered Jul 22 02:37:16.129865 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:16.127785 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/6a441682-e619-4c8d-96a2-3a391f8eac76-etc-sysconfig\") pod \"tuned-ftqsf\" (UID: \"6a441682-e619-4c8d-96a2-3a391f8eac76\") " pod="openshift-cluster-node-tuning-operator/tuned-ftqsf" Jul 22 02:37:16.129865 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:16.127913 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/34c0277e-a51b-4c45-ae71-9f339d53d946-tuning-conf-dir\") pod \"multus-additional-cni-plugins-t9vtn\" (UID: \"34c0277e-a51b-4c45-ae71-9f339d53d946\") " pod="openshift-multus/multus-additional-cni-plugins-t9vtn" Jul 22 02:37:16.129865 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:16.127938 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dea9d10b-9320-471c-9efd-544fabc16854-tmp-dir\") pod \"node-resolver-9dhrs\" (UID: \"dea9d10b-9320-471c-9efd-544fabc16854\") " pod="openshift-dns/node-resolver-9dhrs" Jul 22 02:37:16.129865 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:16.128049 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/34c0277e-a51b-4c45-ae71-9f339d53d946-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-t9vtn\" (UID: \"34c0277e-a51b-4c45-ae71-9f339d53d946\") " pod="openshift-multus/multus-additional-cni-plugins-t9vtn" Jul 22 02:37:16.129865 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:16.129162 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6a441682-e619-4c8d-96a2-3a391f8eac76-tmp\") pod \"tuned-ftqsf\" (UID: \"6a441682-e619-4c8d-96a2-3a391f8eac76\") " pod="openshift-cluster-node-tuning-operator/tuned-ftqsf" Jul 22 02:37:16.129865 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:16.129212 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/6a441682-e619-4c8d-96a2-3a391f8eac76-etc-tuned\") pod \"tuned-ftqsf\" (UID: \"6a441682-e619-4c8d-96a2-3a391f8eac76\") " pod="openshift-cluster-node-tuning-operator/tuned-ftqsf" Jul 22 02:37:16.141726 ip-10-0-133-31 kubenswrapper[2578]: E0722 02:37:16.141390 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 22 02:37:16.141726 ip-10-0-133-31 kubenswrapper[2578]: E0722 02:37:16.141418 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 22 02:37:16.141726 ip-10-0-133-31 kubenswrapper[2578]: E0722 02:37:16.141434 2578 projected.go:196] Error preparing data for projected volume kube-api-access-5cxbr for pod openshift-network-diagnostics/network-check-target-wp7zw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 02:37:16.141726 ip-10-0-133-31 kubenswrapper[2578]: E0722 02:37:16.141505 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6b93e551-99d3-4ee4-8560-ca6b0f26b6db-kube-api-access-5cxbr podName:6b93e551-99d3-4ee4-8560-ca6b0f26b6db nodeName:}" failed. No retries permitted until 2026-07-22 02:37:16.641485573 +0000 UTC m=+2.960660870 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-5cxbr" (UniqueName: "kubernetes.io/projected/6b93e551-99d3-4ee4-8560-ca6b0f26b6db-kube-api-access-5cxbr") pod "network-check-target-wp7zw" (UID: "6b93e551-99d3-4ee4-8560-ca6b0f26b6db") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 02:37:16.141726 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:16.141601 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xqsbj\" (UniqueName: \"kubernetes.io/projected/dea9d10b-9320-471c-9efd-544fabc16854-kube-api-access-xqsbj\") pod \"node-resolver-9dhrs\" (UID: \"dea9d10b-9320-471c-9efd-544fabc16854\") " pod="openshift-dns/node-resolver-9dhrs" Jul 22 02:37:16.142172 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:16.142152 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xq6gn\" (UniqueName: \"kubernetes.io/projected/6a441682-e619-4c8d-96a2-3a391f8eac76-kube-api-access-xq6gn\") pod \"tuned-ftqsf\" (UID: \"6a441682-e619-4c8d-96a2-3a391f8eac76\") " pod="openshift-cluster-node-tuning-operator/tuned-ftqsf" Jul 22 02:37:16.143193 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:16.143169 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fmxqm\" (UniqueName: \"kubernetes.io/projected/d6871664-adf7-41be-b1d0-d1ec7ccae7c6-kube-api-access-fmxqm\") pod \"network-metrics-daemon-lvbjx\" (UID: \"d6871664-adf7-41be-b1d0-d1ec7ccae7c6\") " pod="openshift-multus/network-metrics-daemon-lvbjx" Jul 22 02:37:16.143431 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:16.143410 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvb6v\" (UniqueName: \"kubernetes.io/projected/34c0277e-a51b-4c45-ae71-9f339d53d946-kube-api-access-jvb6v\") pod \"multus-additional-cni-plugins-t9vtn\" (UID: \"34c0277e-a51b-4c45-ae71-9f339d53d946\") " pod="openshift-multus/multus-additional-cni-plugins-t9vtn" Jul 22 02:37:16.143613 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:16.143591 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d97cl\" (UniqueName: \"kubernetes.io/projected/9834be29-d56c-4ce7-a542-bea10d73ccfd-kube-api-access-d97cl\") pod \"iptables-alerter-ld8dc\" (UID: \"9834be29-d56c-4ce7-a542-bea10d73ccfd\") " pod="openshift-network-operator/iptables-alerter-ld8dc" Jul 22 02:37:16.228503 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:16.228476 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e56db148-7293-423e-b9fb-0ebd5db740a3-konnectivity-ca\") pod \"konnectivity-agent-bxg22\" (UID: \"e56db148-7293-423e-b9fb-0ebd5db740a3\") " pod="kube-system/konnectivity-agent-bxg22" Jul 22 02:37:16.228503 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:16.228507 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c75eb074-c486-4152-9d81-b12360b19df0-socket-dir\") pod \"aws-ebs-csi-driver-node-wjd22\" (UID: \"c75eb074-c486-4152-9d81-b12360b19df0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wjd22" Jul 22 02:37:16.228730 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:16.228523 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d7953894-cdfc-48a3-afe8-89503dfc3c57-serviceca\") pod \"node-ca-8kmfc\" (UID: \"d7953894-cdfc-48a3-afe8-89503dfc3c57\") " pod="openshift-image-registry/node-ca-8kmfc" Jul 22 02:37:16.228730 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:16.228539 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c75eb074-c486-4152-9d81-b12360b19df0-kubelet-dir\") pod \"aws-ebs-csi-driver-node-wjd22\" (UID: \"c75eb074-c486-4152-9d81-b12360b19df0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wjd22" Jul 22 02:37:16.228730 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:16.228558 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5745b751-8d56-48ab-9e80-93190f8ef935-cnibin\") pod \"multus-8ssxs\" (UID: \"5745b751-8d56-48ab-9e80-93190f8ef935\") " pod="openshift-multus/multus-8ssxs" Jul 22 02:37:16.228730 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:16.228638 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5745b751-8d56-48ab-9e80-93190f8ef935-cnibin\") pod \"multus-8ssxs\" (UID: \"5745b751-8d56-48ab-9e80-93190f8ef935\") " pod="openshift-multus/multus-8ssxs" Jul 22 02:37:16.228730 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:16.228660 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c75eb074-c486-4152-9d81-b12360b19df0-kubelet-dir\") pod \"aws-ebs-csi-driver-node-wjd22\" (UID: \"c75eb074-c486-4152-9d81-b12360b19df0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wjd22" Jul 22 02:37:16.228730 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:16.228663 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c75eb074-c486-4152-9d81-b12360b19df0-socket-dir\") pod \"aws-ebs-csi-driver-node-wjd22\" (UID: \"c75eb074-c486-4152-9d81-b12360b19df0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wjd22" Jul 22 02:37:16.228730 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:16.228685 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5745b751-8d56-48ab-9e80-93190f8ef935-multus-socket-dir-parent\") pod \"multus-8ssxs\" (UID: \"5745b751-8d56-48ab-9e80-93190f8ef935\") " pod="openshift-multus/multus-8ssxs" Jul 22 02:37:16.228730 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:16.228729 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5745b751-8d56-48ab-9e80-93190f8ef935-host-run-k8s-cni-cncf-io\") pod \"multus-8ssxs\" (UID: \"5745b751-8d56-48ab-9e80-93190f8ef935\") " pod="openshift-multus/multus-8ssxs" Jul 22 02:37:16.229126 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:16.228736 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5745b751-8d56-48ab-9e80-93190f8ef935-multus-socket-dir-parent\") pod \"multus-8ssxs\" (UID: \"5745b751-8d56-48ab-9e80-93190f8ef935\") " pod="openshift-multus/multus-8ssxs" Jul 22 02:37:16.229126 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:16.228754 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dms5p\" (UniqueName: \"kubernetes.io/projected/d7953894-cdfc-48a3-afe8-89503dfc3c57-kube-api-access-dms5p\") pod \"node-ca-8kmfc\" (UID: \"d7953894-cdfc-48a3-afe8-89503dfc3c57\") " pod="openshift-image-registry/node-ca-8kmfc" Jul 22 02:37:16.229126 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:16.228779 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fd7bf923-8d16-476c-a8d0-a3e6fd645061-host-slash\") pod \"ovnkube-node-prfql\" (UID: \"fd7bf923-8d16-476c-a8d0-a3e6fd645061\") " pod="openshift-ovn-kubernetes/ovnkube-node-prfql" Jul 22 02:37:16.229126 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:16.228789 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5745b751-8d56-48ab-9e80-93190f8ef935-host-run-k8s-cni-cncf-io\") pod \"multus-8ssxs\" (UID: \"5745b751-8d56-48ab-9e80-93190f8ef935\") " pod="openshift-multus/multus-8ssxs" Jul 22 02:37:16.229126 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:16.228809 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5745b751-8d56-48ab-9e80-93190f8ef935-host-run-multus-certs\") pod \"multus-8ssxs\" (UID: \"5745b751-8d56-48ab-9e80-93190f8ef935\") " pod="openshift-multus/multus-8ssxs" Jul 22 02:37:16.229126 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:16.228838 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/fd7bf923-8d16-476c-a8d0-a3e6fd645061-run-ovn\") pod \"ovnkube-node-prfql\" (UID: \"fd7bf923-8d16-476c-a8d0-a3e6fd645061\") " pod="openshift-ovn-kubernetes/ovnkube-node-prfql" Jul 22 02:37:16.229126 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:16.228842 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fd7bf923-8d16-476c-a8d0-a3e6fd645061-host-slash\") pod \"ovnkube-node-prfql\" (UID: \"fd7bf923-8d16-476c-a8d0-a3e6fd645061\") " pod="openshift-ovn-kubernetes/ovnkube-node-prfql" Jul 22 02:37:16.229126 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:16.228845 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5745b751-8d56-48ab-9e80-93190f8ef935-host-run-multus-certs\") pod \"multus-8ssxs\" (UID: \"5745b751-8d56-48ab-9e80-93190f8ef935\") " pod="openshift-multus/multus-8ssxs" Jul 22 02:37:16.229126 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:16.228862 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/fd7bf923-8d16-476c-a8d0-a3e6fd645061-log-socket\") pod \"ovnkube-node-prfql\" (UID: \"fd7bf923-8d16-476c-a8d0-a3e6fd645061\") " pod="openshift-ovn-kubernetes/ovnkube-node-prfql" Jul 22 02:37:16.229126 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:16.228890 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fd7bf923-8d16-476c-a8d0-a3e6fd645061-host-run-ovn-kubernetes\") pod \"ovnkube-node-prfql\" (UID: \"fd7bf923-8d16-476c-a8d0-a3e6fd645061\") " pod="openshift-ovn-kubernetes/ovnkube-node-prfql" Jul 22 02:37:16.229126 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:16.228898 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/fd7bf923-8d16-476c-a8d0-a3e6fd645061-log-socket\") pod \"ovnkube-node-prfql\" (UID: \"fd7bf923-8d16-476c-a8d0-a3e6fd645061\") " pod="openshift-ovn-kubernetes/ovnkube-node-prfql" Jul 22 02:37:16.229126 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:16.228914 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/fd7bf923-8d16-476c-a8d0-a3e6fd645061-ovnkube-config\") pod \"ovnkube-node-prfql\" (UID: \"fd7bf923-8d16-476c-a8d0-a3e6fd645061\") " pod="openshift-ovn-kubernetes/ovnkube-node-prfql" Jul 22 02:37:16.229126 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:16.228934 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/fd7bf923-8d16-476c-a8d0-a3e6fd645061-env-overrides\") pod \"ovnkube-node-prfql\" (UID: \"fd7bf923-8d16-476c-a8d0-a3e6fd645061\") " pod="openshift-ovn-kubernetes/ovnkube-node-prfql" Jul 22 02:37:16.229126 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:16.228941 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fd7bf923-8d16-476c-a8d0-a3e6fd645061-host-run-ovn-kubernetes\") pod \"ovnkube-node-prfql\" (UID: \"fd7bf923-8d16-476c-a8d0-a3e6fd645061\") " pod="openshift-ovn-kubernetes/ovnkube-node-prfql" Jul 22 02:37:16.229126 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:16.228953 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/5745b751-8d56-48ab-9e80-93190f8ef935-os-release\") pod \"multus-8ssxs\" (UID: \"5745b751-8d56-48ab-9e80-93190f8ef935\") " pod="openshift-multus/multus-8ssxs" Jul 22 02:37:16.229126 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:16.228970 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/fd7bf923-8d16-476c-a8d0-a3e6fd645061-node-log\") pod \"ovnkube-node-prfql\" (UID: \"fd7bf923-8d16-476c-a8d0-a3e6fd645061\") " pod="openshift-ovn-kubernetes/ovnkube-node-prfql" Jul 22 02:37:16.229126 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:16.228915 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/fd7bf923-8d16-476c-a8d0-a3e6fd645061-run-ovn\") pod \"ovnkube-node-prfql\" (UID: \"fd7bf923-8d16-476c-a8d0-a3e6fd645061\") " pod="openshift-ovn-kubernetes/ovnkube-node-prfql" Jul 22 02:37:16.229126 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:16.228997 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c75eb074-c486-4152-9d81-b12360b19df0-registration-dir\") pod \"aws-ebs-csi-driver-node-wjd22\" (UID: \"c75eb074-c486-4152-9d81-b12360b19df0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wjd22" Jul 22 02:37:16.229852 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:16.229024 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-65ddv\" (UniqueName: \"kubernetes.io/projected/5745b751-8d56-48ab-9e80-93190f8ef935-kube-api-access-65ddv\") pod \"multus-8ssxs\" (UID: \"5745b751-8d56-48ab-9e80-93190f8ef935\") " pod="openshift-multus/multus-8ssxs" Jul 22 02:37:16.229852 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:16.229037 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/5745b751-8d56-48ab-9e80-93190f8ef935-os-release\") pod \"multus-8ssxs\" (UID: \"5745b751-8d56-48ab-9e80-93190f8ef935\") " pod="openshift-multus/multus-8ssxs" Jul 22 02:37:16.229852 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:16.229051 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/fd7bf923-8d16-476c-a8d0-a3e6fd645061-host-kubelet\") pod \"ovnkube-node-prfql\" (UID: \"fd7bf923-8d16-476c-a8d0-a3e6fd645061\") " pod="openshift-ovn-kubernetes/ovnkube-node-prfql" Jul 22 02:37:16.229852 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:16.229081 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fd7bf923-8d16-476c-a8d0-a3e6fd645061-run-openvswitch\") pod \"ovnkube-node-prfql\" (UID: \"fd7bf923-8d16-476c-a8d0-a3e6fd645061\") " pod="openshift-ovn-kubernetes/ovnkube-node-prfql" Jul 22 02:37:16.229852 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:16.229061 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c75eb074-c486-4152-9d81-b12360b19df0-registration-dir\") pod \"aws-ebs-csi-driver-node-wjd22\" (UID: \"c75eb074-c486-4152-9d81-b12360b19df0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wjd22" Jul 22 02:37:16.229852 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:16.229076 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/fd7bf923-8d16-476c-a8d0-a3e6fd645061-node-log\") pod \"ovnkube-node-prfql\" (UID: \"fd7bf923-8d16-476c-a8d0-a3e6fd645061\") " pod="openshift-ovn-kubernetes/ovnkube-node-prfql" Jul 22 02:37:16.229852 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:16.229141 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fd7bf923-8d16-476c-a8d0-a3e6fd645061-run-openvswitch\") pod \"ovnkube-node-prfql\" (UID: \"fd7bf923-8d16-476c-a8d0-a3e6fd645061\") " pod="openshift-ovn-kubernetes/ovnkube-node-prfql" Jul 22 02:37:16.229852 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:16.229140 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c75eb074-c486-4152-9d81-b12360b19df0-sys-fs\") pod \"aws-ebs-csi-driver-node-wjd22\" (UID: \"c75eb074-c486-4152-9d81-b12360b19df0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wjd22" Jul 22 02:37:16.229852 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:16.229173 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e56db148-7293-423e-b9fb-0ebd5db740a3-konnectivity-ca\") pod \"konnectivity-agent-bxg22\" (UID: \"e56db148-7293-423e-b9fb-0ebd5db740a3\") " pod="kube-system/konnectivity-agent-bxg22" Jul 22 02:37:16.229852 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:16.229182 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c75eb074-c486-4152-9d81-b12360b19df0-sys-fs\") pod \"aws-ebs-csi-driver-node-wjd22\" (UID: \"c75eb074-c486-4152-9d81-b12360b19df0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wjd22" Jul 22 02:37:16.229852 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:16.229210 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/fd7bf923-8d16-476c-a8d0-a3e6fd645061-host-kubelet\") pod \"ovnkube-node-prfql\" (UID: \"fd7bf923-8d16-476c-a8d0-a3e6fd645061\") " pod="openshift-ovn-kubernetes/ovnkube-node-prfql" Jul 22 02:37:16.229852 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:16.229212 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/fd7bf923-8d16-476c-a8d0-a3e6fd645061-host-cni-netd\") pod \"ovnkube-node-prfql\" (UID: \"fd7bf923-8d16-476c-a8d0-a3e6fd645061\") " pod="openshift-ovn-kubernetes/ovnkube-node-prfql" Jul 22 02:37:16.229852 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:16.229244 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/fd7bf923-8d16-476c-a8d0-a3e6fd645061-host-cni-netd\") pod \"ovnkube-node-prfql\" (UID: \"fd7bf923-8d16-476c-a8d0-a3e6fd645061\") " pod="openshift-ovn-kubernetes/ovnkube-node-prfql" Jul 22 02:37:16.229852 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:16.229249 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c75eb074-c486-4152-9d81-b12360b19df0-etc-selinux\") pod \"aws-ebs-csi-driver-node-wjd22\" (UID: \"c75eb074-c486-4152-9d81-b12360b19df0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wjd22" Jul 22 02:37:16.229852 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:16.229279 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5745b751-8d56-48ab-9e80-93190f8ef935-host-var-lib-cni-multus\") pod \"multus-8ssxs\" (UID: \"5745b751-8d56-48ab-9e80-93190f8ef935\") " pod="openshift-multus/multus-8ssxs" Jul 22 02:37:16.229852 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:16.229304 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d7953894-cdfc-48a3-afe8-89503dfc3c57-host\") pod \"node-ca-8kmfc\" (UID: \"d7953894-cdfc-48a3-afe8-89503dfc3c57\") " pod="openshift-image-registry/node-ca-8kmfc" Jul 22 02:37:16.229852 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:16.229314 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5745b751-8d56-48ab-9e80-93190f8ef935-host-var-lib-cni-multus\") pod \"multus-8ssxs\" (UID: \"5745b751-8d56-48ab-9e80-93190f8ef935\") " pod="openshift-multus/multus-8ssxs" Jul 22 02:37:16.229852 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:16.229327 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fd7bf923-8d16-476c-a8d0-a3e6fd645061-host-cni-bin\") pod \"ovnkube-node-prfql\" (UID: \"fd7bf923-8d16-476c-a8d0-a3e6fd645061\") " pod="openshift-ovn-kubernetes/ovnkube-node-prfql" Jul 22 02:37:16.230509 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:16.229341 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c75eb074-c486-4152-9d81-b12360b19df0-etc-selinux\") pod \"aws-ebs-csi-driver-node-wjd22\" (UID: \"c75eb074-c486-4152-9d81-b12360b19df0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wjd22" Jul 22 02:37:16.230509 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:16.229353 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5745b751-8d56-48ab-9e80-93190f8ef935-system-cni-dir\") pod \"multus-8ssxs\" (UID: \"5745b751-8d56-48ab-9e80-93190f8ef935\") " pod="openshift-multus/multus-8ssxs" Jul 22 02:37:16.230509 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:16.229363 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fd7bf923-8d16-476c-a8d0-a3e6fd645061-host-cni-bin\") pod \"ovnkube-node-prfql\" (UID: \"fd7bf923-8d16-476c-a8d0-a3e6fd645061\") " pod="openshift-ovn-kubernetes/ovnkube-node-prfql" Jul 22 02:37:16.230509 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:16.229367 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d7953894-cdfc-48a3-afe8-89503dfc3c57-host\") pod \"node-ca-8kmfc\" (UID: \"d7953894-cdfc-48a3-afe8-89503dfc3c57\") " pod="openshift-image-registry/node-ca-8kmfc" Jul 22 02:37:16.230509 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:16.229377 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5745b751-8d56-48ab-9e80-93190f8ef935-multus-cni-dir\") pod \"multus-8ssxs\" (UID: \"5745b751-8d56-48ab-9e80-93190f8ef935\") " pod="openshift-multus/multus-8ssxs" Jul 22 02:37:16.230509 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:16.229414 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5745b751-8d56-48ab-9e80-93190f8ef935-host-var-lib-cni-bin\") pod \"multus-8ssxs\" (UID: \"5745b751-8d56-48ab-9e80-93190f8ef935\") " pod="openshift-multus/multus-8ssxs" Jul 22 02:37:16.230509 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:16.229416 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5745b751-8d56-48ab-9e80-93190f8ef935-system-cni-dir\") pod \"multus-8ssxs\" (UID: \"5745b751-8d56-48ab-9e80-93190f8ef935\") " pod="openshift-multus/multus-8ssxs" Jul 22 02:37:16.230509 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:16.229424 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5745b751-8d56-48ab-9e80-93190f8ef935-multus-cni-dir\") pod \"multus-8ssxs\" (UID: \"5745b751-8d56-48ab-9e80-93190f8ef935\") " pod="openshift-multus/multus-8ssxs" Jul 22 02:37:16.230509 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:16.229453 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5745b751-8d56-48ab-9e80-93190f8ef935-host-var-lib-cni-bin\") pod \"multus-8ssxs\" (UID: \"5745b751-8d56-48ab-9e80-93190f8ef935\") " pod="openshift-multus/multus-8ssxs" Jul 22 02:37:16.230509 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:16.229455 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5745b751-8d56-48ab-9e80-93190f8ef935-etc-kubernetes\") pod \"multus-8ssxs\" (UID: \"5745b751-8d56-48ab-9e80-93190f8ef935\") " pod="openshift-multus/multus-8ssxs" Jul 22 02:37:16.230509 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:16.229473 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/fd7bf923-8d16-476c-a8d0-a3e6fd645061-env-overrides\") pod \"ovnkube-node-prfql\" (UID: \"fd7bf923-8d16-476c-a8d0-a3e6fd645061\") " pod="openshift-ovn-kubernetes/ovnkube-node-prfql" Jul 22 02:37:16.230509 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:16.229483 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5745b751-8d56-48ab-9e80-93190f8ef935-etc-kubernetes\") pod \"multus-8ssxs\" (UID: \"5745b751-8d56-48ab-9e80-93190f8ef935\") " pod="openshift-multus/multus-8ssxs" Jul 22 02:37:16.230509 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:16.229479 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/fd7bf923-8d16-476c-a8d0-a3e6fd645061-run-systemd\") pod \"ovnkube-node-prfql\" (UID: \"fd7bf923-8d16-476c-a8d0-a3e6fd645061\") " pod="openshift-ovn-kubernetes/ovnkube-node-prfql" Jul 22 02:37:16.230509 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:16.229498 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d7953894-cdfc-48a3-afe8-89503dfc3c57-serviceca\") pod \"node-ca-8kmfc\" (UID: \"d7953894-cdfc-48a3-afe8-89503dfc3c57\") " pod="openshift-image-registry/node-ca-8kmfc" Jul 22 02:37:16.230509 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:16.229509 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/fd7bf923-8d16-476c-a8d0-a3e6fd645061-run-systemd\") pod \"ovnkube-node-prfql\" (UID: \"fd7bf923-8d16-476c-a8d0-a3e6fd645061\") " pod="openshift-ovn-kubernetes/ovnkube-node-prfql" Jul 22 02:37:16.230509 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:16.229528 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e56db148-7293-423e-b9fb-0ebd5db740a3-agent-certs\") pod \"konnectivity-agent-bxg22\" (UID: \"e56db148-7293-423e-b9fb-0ebd5db740a3\") " pod="kube-system/konnectivity-agent-bxg22" Jul 22 02:37:16.230509 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:16.229548 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/fd7bf923-8d16-476c-a8d0-a3e6fd645061-ovnkube-config\") pod \"ovnkube-node-prfql\" (UID: \"fd7bf923-8d16-476c-a8d0-a3e6fd645061\") " pod="openshift-ovn-kubernetes/ovnkube-node-prfql" Jul 22 02:37:16.230509 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:16.229663 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c75eb074-c486-4152-9d81-b12360b19df0-device-dir\") pod \"aws-ebs-csi-driver-node-wjd22\" (UID: \"c75eb074-c486-4152-9d81-b12360b19df0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wjd22" Jul 22 02:37:16.231613 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:16.230055 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c75eb074-c486-4152-9d81-b12360b19df0-device-dir\") pod \"aws-ebs-csi-driver-node-wjd22\" (UID: \"c75eb074-c486-4152-9d81-b12360b19df0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wjd22" Jul 22 02:37:16.231613 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:16.230090 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fd7bf923-8d16-476c-a8d0-a3e6fd645061-var-lib-openvswitch\") pod \"ovnkube-node-prfql\" (UID: \"fd7bf923-8d16-476c-a8d0-a3e6fd645061\") " pod="openshift-ovn-kubernetes/ovnkube-node-prfql" Jul 22 02:37:16.231613 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:16.230106 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/fd7bf923-8d16-476c-a8d0-a3e6fd645061-ovn-node-metrics-cert\") pod \"ovnkube-node-prfql\" (UID: \"fd7bf923-8d16-476c-a8d0-a3e6fd645061\") " pod="openshift-ovn-kubernetes/ovnkube-node-prfql" Jul 22 02:37:16.231613 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:16.230134 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5745b751-8d56-48ab-9e80-93190f8ef935-host-run-netns\") pod \"multus-8ssxs\" (UID: \"5745b751-8d56-48ab-9e80-93190f8ef935\") " pod="openshift-multus/multus-8ssxs" Jul 22 02:37:16.231613 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:16.230149 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5745b751-8d56-48ab-9e80-93190f8ef935-hostroot\") pod \"multus-8ssxs\" (UID: \"5745b751-8d56-48ab-9e80-93190f8ef935\") " pod="openshift-multus/multus-8ssxs" Jul 22 02:37:16.231613 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:16.230173 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5745b751-8d56-48ab-9e80-93190f8ef935-cni-binary-copy\") pod \"multus-8ssxs\" (UID: \"5745b751-8d56-48ab-9e80-93190f8ef935\") " pod="openshift-multus/multus-8ssxs" Jul 22 02:37:16.231613 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:16.230211 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5745b751-8d56-48ab-9e80-93190f8ef935-host-run-netns\") pod \"multus-8ssxs\" (UID: \"5745b751-8d56-48ab-9e80-93190f8ef935\") " pod="openshift-multus/multus-8ssxs" Jul 22 02:37:16.231613 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:16.230235 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5745b751-8d56-48ab-9e80-93190f8ef935-hostroot\") pod \"multus-8ssxs\" (UID: \"5745b751-8d56-48ab-9e80-93190f8ef935\") " pod="openshift-multus/multus-8ssxs" Jul 22 02:37:16.231613 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:16.230240 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fd7bf923-8d16-476c-a8d0-a3e6fd645061-var-lib-openvswitch\") pod \"ovnkube-node-prfql\" (UID: \"fd7bf923-8d16-476c-a8d0-a3e6fd645061\") " pod="openshift-ovn-kubernetes/ovnkube-node-prfql" Jul 22 02:37:16.231613 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:16.230277 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fd7bf923-8d16-476c-a8d0-a3e6fd645061-host-run-netns\") pod \"ovnkube-node-prfql\" (UID: \"fd7bf923-8d16-476c-a8d0-a3e6fd645061\") " pod="openshift-ovn-kubernetes/ovnkube-node-prfql" Jul 22 02:37:16.231613 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:16.230341 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fd7bf923-8d16-476c-a8d0-a3e6fd645061-etc-openvswitch\") pod \"ovnkube-node-prfql\" (UID: \"fd7bf923-8d16-476c-a8d0-a3e6fd645061\") " pod="openshift-ovn-kubernetes/ovnkube-node-prfql" Jul 22 02:37:16.231613 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:16.230372 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fd7bf923-8d16-476c-a8d0-a3e6fd645061-etc-openvswitch\") pod \"ovnkube-node-prfql\" (UID: \"fd7bf923-8d16-476c-a8d0-a3e6fd645061\") " pod="openshift-ovn-kubernetes/ovnkube-node-prfql" Jul 22 02:37:16.231613 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:16.230462 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fd7bf923-8d16-476c-a8d0-a3e6fd645061-host-run-netns\") pod \"ovnkube-node-prfql\" (UID: \"fd7bf923-8d16-476c-a8d0-a3e6fd645061\") " pod="openshift-ovn-kubernetes/ovnkube-node-prfql" Jul 22 02:37:16.231613 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:16.230481 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/fd7bf923-8d16-476c-a8d0-a3e6fd645061-ovnkube-script-lib\") pod \"ovnkube-node-prfql\" (UID: \"fd7bf923-8d16-476c-a8d0-a3e6fd645061\") " pod="openshift-ovn-kubernetes/ovnkube-node-prfql" Jul 22 02:37:16.231613 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:16.230506 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wgd6p\" (UniqueName: \"kubernetes.io/projected/fd7bf923-8d16-476c-a8d0-a3e6fd645061-kube-api-access-wgd6p\") pod \"ovnkube-node-prfql\" (UID: \"fd7bf923-8d16-476c-a8d0-a3e6fd645061\") " pod="openshift-ovn-kubernetes/ovnkube-node-prfql" Jul 22 02:37:16.231613 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:16.230531 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/5745b751-8d56-48ab-9e80-93190f8ef935-multus-daemon-config\") pod \"multus-8ssxs\" (UID: \"5745b751-8d56-48ab-9e80-93190f8ef935\") " pod="openshift-multus/multus-8ssxs" Jul 22 02:37:16.231613 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:16.230548 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fd7bf923-8d16-476c-a8d0-a3e6fd645061-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-prfql\" (UID: \"fd7bf923-8d16-476c-a8d0-a3e6fd645061\") " pod="openshift-ovn-kubernetes/ovnkube-node-prfql" Jul 22 02:37:16.232513 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:16.230591 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vtwmb\" (UniqueName: \"kubernetes.io/projected/c75eb074-c486-4152-9d81-b12360b19df0-kube-api-access-vtwmb\") pod \"aws-ebs-csi-driver-node-wjd22\" (UID: \"c75eb074-c486-4152-9d81-b12360b19df0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wjd22" Jul 22 02:37:16.232513 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:16.230618 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5745b751-8d56-48ab-9e80-93190f8ef935-host-var-lib-kubelet\") pod \"multus-8ssxs\" (UID: \"5745b751-8d56-48ab-9e80-93190f8ef935\") " pod="openshift-multus/multus-8ssxs" Jul 22 02:37:16.232513 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:16.230646 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5745b751-8d56-48ab-9e80-93190f8ef935-multus-conf-dir\") pod \"multus-8ssxs\" (UID: \"5745b751-8d56-48ab-9e80-93190f8ef935\") " pod="openshift-multus/multus-8ssxs" Jul 22 02:37:16.232513 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:16.230672 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/fd7bf923-8d16-476c-a8d0-a3e6fd645061-systemd-units\") pod \"ovnkube-node-prfql\" (UID: \"fd7bf923-8d16-476c-a8d0-a3e6fd645061\") " pod="openshift-ovn-kubernetes/ovnkube-node-prfql" Jul 22 02:37:16.232513 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:16.230744 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/fd7bf923-8d16-476c-a8d0-a3e6fd645061-systemd-units\") pod \"ovnkube-node-prfql\" (UID: \"fd7bf923-8d16-476c-a8d0-a3e6fd645061\") " pod="openshift-ovn-kubernetes/ovnkube-node-prfql" Jul 22 02:37:16.232513 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:16.230792 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fd7bf923-8d16-476c-a8d0-a3e6fd645061-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-prfql\" (UID: \"fd7bf923-8d16-476c-a8d0-a3e6fd645061\") " pod="openshift-ovn-kubernetes/ovnkube-node-prfql" Jul 22 02:37:16.232513 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:16.231008 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5745b751-8d56-48ab-9e80-93190f8ef935-host-var-lib-kubelet\") pod \"multus-8ssxs\" (UID: \"5745b751-8d56-48ab-9e80-93190f8ef935\") " pod="openshift-multus/multus-8ssxs" Jul 22 02:37:16.232513 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:16.231046 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/fd7bf923-8d16-476c-a8d0-a3e6fd645061-ovnkube-script-lib\") pod \"ovnkube-node-prfql\" (UID: \"fd7bf923-8d16-476c-a8d0-a3e6fd645061\") " pod="openshift-ovn-kubernetes/ovnkube-node-prfql" Jul 22 02:37:16.232513 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:16.231056 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5745b751-8d56-48ab-9e80-93190f8ef935-multus-conf-dir\") pod \"multus-8ssxs\" (UID: \"5745b751-8d56-48ab-9e80-93190f8ef935\") " pod="openshift-multus/multus-8ssxs" Jul 22 02:37:16.232513 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:16.231229 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/5745b751-8d56-48ab-9e80-93190f8ef935-multus-daemon-config\") pod \"multus-8ssxs\" (UID: \"5745b751-8d56-48ab-9e80-93190f8ef935\") " pod="openshift-multus/multus-8ssxs" Jul 22 02:37:16.232513 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:16.231380 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5745b751-8d56-48ab-9e80-93190f8ef935-cni-binary-copy\") pod \"multus-8ssxs\" (UID: \"5745b751-8d56-48ab-9e80-93190f8ef935\") " pod="openshift-multus/multus-8ssxs" Jul 22 02:37:16.232513 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:16.232197 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e56db148-7293-423e-b9fb-0ebd5db740a3-agent-certs\") pod \"konnectivity-agent-bxg22\" (UID: \"e56db148-7293-423e-b9fb-0ebd5db740a3\") " pod="kube-system/konnectivity-agent-bxg22" Jul 22 02:37:16.232985 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:16.232961 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/fd7bf923-8d16-476c-a8d0-a3e6fd645061-ovn-node-metrics-cert\") pod \"ovnkube-node-prfql\" (UID: \"fd7bf923-8d16-476c-a8d0-a3e6fd645061\") " pod="openshift-ovn-kubernetes/ovnkube-node-prfql" Jul 22 02:37:16.239904 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:16.239880 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-65ddv\" (UniqueName: \"kubernetes.io/projected/5745b751-8d56-48ab-9e80-93190f8ef935-kube-api-access-65ddv\") pod \"multus-8ssxs\" (UID: \"5745b751-8d56-48ab-9e80-93190f8ef935\") " pod="openshift-multus/multus-8ssxs" Jul 22 02:37:16.240000 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:16.239922 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wgd6p\" (UniqueName: \"kubernetes.io/projected/fd7bf923-8d16-476c-a8d0-a3e6fd645061-kube-api-access-wgd6p\") pod \"ovnkube-node-prfql\" (UID: \"fd7bf923-8d16-476c-a8d0-a3e6fd645061\") " pod="openshift-ovn-kubernetes/ovnkube-node-prfql" Jul 22 02:37:16.240392 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:16.240372 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dms5p\" (UniqueName: \"kubernetes.io/projected/d7953894-cdfc-48a3-afe8-89503dfc3c57-kube-api-access-dms5p\") pod \"node-ca-8kmfc\" (UID: \"d7953894-cdfc-48a3-afe8-89503dfc3c57\") " pod="openshift-image-registry/node-ca-8kmfc" Jul 22 02:37:16.241073 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:16.241055 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vtwmb\" (UniqueName: \"kubernetes.io/projected/c75eb074-c486-4152-9d81-b12360b19df0-kube-api-access-vtwmb\") pod \"aws-ebs-csi-driver-node-wjd22\" (UID: \"c75eb074-c486-4152-9d81-b12360b19df0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wjd22" Jul 22 02:37:16.318789 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:16.318726 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-ld8dc" Jul 22 02:37:16.325805 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:16.325776 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-ftqsf" Jul 22 02:37:16.333280 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:16.333257 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-t9vtn" Jul 22 02:37:16.338836 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:16.338818 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-9dhrs" Jul 22 02:37:16.345570 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:16.345552 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wjd22" Jul 22 02:37:16.355944 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:16.355923 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-8ssxs" Jul 22 02:37:16.361625 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:16.361600 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-8kmfc" Jul 22 02:37:16.368523 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:16.368500 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-prfql" Jul 22 02:37:16.375322 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:16.375302 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-bxg22" Jul 22 02:37:16.406011 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:16.405990 2578 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 22 02:37:16.633529 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:16.633481 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d6871664-adf7-41be-b1d0-d1ec7ccae7c6-metrics-certs\") pod \"network-metrics-daemon-lvbjx\" (UID: \"d6871664-adf7-41be-b1d0-d1ec7ccae7c6\") " pod="openshift-multus/network-metrics-daemon-lvbjx" Jul 22 02:37:16.633705 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:16.633543 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/684d87e0-1cec-4247-9446-881aff58a246-original-pull-secret\") pod \"global-pull-secret-syncer-rb9sf\" (UID: \"684d87e0-1cec-4247-9446-881aff58a246\") " pod="kube-system/global-pull-secret-syncer-rb9sf" Jul 22 02:37:16.633705 ip-10-0-133-31 kubenswrapper[2578]: E0722 02:37:16.633651 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 02:37:16.633705 ip-10-0-133-31 kubenswrapper[2578]: E0722 02:37:16.633663 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 22 02:37:16.633878 ip-10-0-133-31 kubenswrapper[2578]: E0722 02:37:16.633720 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/684d87e0-1cec-4247-9446-881aff58a246-original-pull-secret podName:684d87e0-1cec-4247-9446-881aff58a246 nodeName:}" failed. No retries permitted until 2026-07-22 02:37:17.633704666 +0000 UTC m=+3.952879946 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/684d87e0-1cec-4247-9446-881aff58a246-original-pull-secret") pod "global-pull-secret-syncer-rb9sf" (UID: "684d87e0-1cec-4247-9446-881aff58a246") : object "kube-system"/"original-pull-secret" not registered Jul 22 02:37:16.633878 ip-10-0-133-31 kubenswrapper[2578]: E0722 02:37:16.633735 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d6871664-adf7-41be-b1d0-d1ec7ccae7c6-metrics-certs podName:d6871664-adf7-41be-b1d0-d1ec7ccae7c6 nodeName:}" failed. No retries permitted until 2026-07-22 02:37:17.633728352 +0000 UTC m=+3.952903632 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d6871664-adf7-41be-b1d0-d1ec7ccae7c6-metrics-certs") pod "network-metrics-daemon-lvbjx" (UID: "d6871664-adf7-41be-b1d0-d1ec7ccae7c6") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 02:37:16.734454 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:16.734414 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5cxbr\" (UniqueName: \"kubernetes.io/projected/6b93e551-99d3-4ee4-8560-ca6b0f26b6db-kube-api-access-5cxbr\") pod \"network-check-target-wp7zw\" (UID: \"6b93e551-99d3-4ee4-8560-ca6b0f26b6db\") " pod="openshift-network-diagnostics/network-check-target-wp7zw" Jul 22 02:37:16.734614 ip-10-0-133-31 kubenswrapper[2578]: E0722 02:37:16.734542 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 22 02:37:16.734614 ip-10-0-133-31 kubenswrapper[2578]: E0722 02:37:16.734566 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 22 02:37:16.734614 ip-10-0-133-31 kubenswrapper[2578]: E0722 02:37:16.734601 2578 projected.go:196] Error preparing data for projected volume kube-api-access-5cxbr for pod openshift-network-diagnostics/network-check-target-wp7zw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 02:37:16.734757 ip-10-0-133-31 kubenswrapper[2578]: E0722 02:37:16.734663 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6b93e551-99d3-4ee4-8560-ca6b0f26b6db-kube-api-access-5cxbr podName:6b93e551-99d3-4ee4-8560-ca6b0f26b6db nodeName:}" failed. No retries permitted until 2026-07-22 02:37:17.73464242 +0000 UTC m=+4.053817700 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-5cxbr" (UniqueName: "kubernetes.io/projected/6b93e551-99d3-4ee4-8560-ca6b0f26b6db-kube-api-access-5cxbr") pod "network-check-target-wp7zw" (UID: "6b93e551-99d3-4ee4-8560-ca6b0f26b6db") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 02:37:16.929260 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:16.929131 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9834be29_d56c_4ce7_a542_bea10d73ccfd.slice/crio-a9d3d215382fe1a8765776eac4d87f2358b4b6101516fed79a5ab79570577b12 WatchSource:0}: Error finding container a9d3d215382fe1a8765776eac4d87f2358b4b6101516fed79a5ab79570577b12: Status 404 returned error can't find the container with id a9d3d215382fe1a8765776eac4d87f2358b4b6101516fed79a5ab79570577b12 Jul 22 02:37:16.937827 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:16.937075 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7953894_cdfc_48a3_afe8_89503dfc3c57.slice/crio-2c16802d060b73ebf6eb286c39254d885db4c190e5b44f9181c0375741c4b038 WatchSource:0}: Error finding container 2c16802d060b73ebf6eb286c39254d885db4c190e5b44f9181c0375741c4b038: Status 404 returned error can't find the container with id 2c16802d060b73ebf6eb286c39254d885db4c190e5b44f9181c0375741c4b038 Jul 22 02:37:16.938644 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:16.938583 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a441682_e619_4c8d_96a2_3a391f8eac76.slice/crio-fd37940f2c54d4d7a37082c8014904ebf006d004ca152730557dcc606f1f754a WatchSource:0}: Error finding container fd37940f2c54d4d7a37082c8014904ebf006d004ca152730557dcc606f1f754a: Status 404 returned error can't find the container with id fd37940f2c54d4d7a37082c8014904ebf006d004ca152730557dcc606f1f754a Jul 22 02:37:16.940647 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:16.939707 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode56db148_7293_423e_b9fb_0ebd5db740a3.slice/crio-9b0773fa1fde0142017e3e8dae3f0134c3fa31f582fc855207acb24caabcbb3c WatchSource:0}: Error finding container 9b0773fa1fde0142017e3e8dae3f0134c3fa31f582fc855207acb24caabcbb3c: Status 404 returned error can't find the container with id 9b0773fa1fde0142017e3e8dae3f0134c3fa31f582fc855207acb24caabcbb3c Jul 22 02:37:16.941958 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:16.941318 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc75eb074_c486_4152_9d81_b12360b19df0.slice/crio-59c5cd62acf5cfe676293225ee1fc38e5bd547f9eb42387f8a2daec14e3ab63a WatchSource:0}: Error finding container 59c5cd62acf5cfe676293225ee1fc38e5bd547f9eb42387f8a2daec14e3ab63a: Status 404 returned error can't find the container with id 59c5cd62acf5cfe676293225ee1fc38e5bd547f9eb42387f8a2daec14e3ab63a Jul 22 02:37:16.941958 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:16.941511 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34c0277e_a51b_4c45_ae71_9f339d53d946.slice/crio-bfa4f091eff6f463e8c95c51e3d8ae1bc8844c032f62900f8bd07e7095676f9f WatchSource:0}: Error finding container bfa4f091eff6f463e8c95c51e3d8ae1bc8844c032f62900f8bd07e7095676f9f: Status 404 returned error can't find the container with id bfa4f091eff6f463e8c95c51e3d8ae1bc8844c032f62900f8bd07e7095676f9f Jul 22 02:37:17.066784 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:17.066619 2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-21 02:32:16 +0000 UTC" deadline="2028-03-19 00:12:21.328812456 +0000 UTC" Jul 22 02:37:17.067194 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:17.066791 2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14541h35m4.262029917s" Jul 22 02:37:17.145069 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:17.145002 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-9dhrs" event={"ID":"dea9d10b-9320-471c-9efd-544fabc16854","Type":"ContainerStarted","Data":"2f6f2ae9e6fb3523d37b83addae9693d0038a9cd33a7dc00f4eaeb675414cfcf"} Jul 22 02:37:17.146727 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:17.146706 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-31.ec2.internal" event={"ID":"47b7203f116608cac74abbb19cda28d8","Type":"ContainerStarted","Data":"97d6e28c00a13bfd25f7026990647a0ef6e7f729224337d902a65b33d2c3d2cd"} Jul 22 02:37:17.147794 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:17.147773 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-bxg22" event={"ID":"e56db148-7293-423e-b9fb-0ebd5db740a3","Type":"ContainerStarted","Data":"9b0773fa1fde0142017e3e8dae3f0134c3fa31f582fc855207acb24caabcbb3c"} Jul 22 02:37:17.148828 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:17.148801 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-ftqsf" event={"ID":"6a441682-e619-4c8d-96a2-3a391f8eac76","Type":"ContainerStarted","Data":"fd37940f2c54d4d7a37082c8014904ebf006d004ca152730557dcc606f1f754a"} Jul 22 02:37:17.150024 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:17.150002 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-8ssxs" event={"ID":"5745b751-8d56-48ab-9e80-93190f8ef935","Type":"ContainerStarted","Data":"9ca38709fa7d7dc98a1b4e84d1db929ee269c7ef685f14834a7acda38fe9dd0e"} Jul 22 02:37:17.151291 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:17.151271 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-prfql" event={"ID":"fd7bf923-8d16-476c-a8d0-a3e6fd645061","Type":"ContainerStarted","Data":"81f6d96a5e8827f972babab55f4f06b7816accd2b120bc856e81f1d9f2a57c0b"} Jul 22 02:37:17.152329 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:17.152307 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-ld8dc" event={"ID":"9834be29-d56c-4ce7-a542-bea10d73ccfd","Type":"ContainerStarted","Data":"a9d3d215382fe1a8765776eac4d87f2358b4b6101516fed79a5ab79570577b12"} Jul 22 02:37:17.153805 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:17.153777 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-31.ec2.internal" event={"ID":"86e469b6aa7347caf60b105c66d89977","Type":"ContainerStarted","Data":"e398a36eff80c24aa9f3ef2c39ad566018276ed161bb1ad6a812f04895328394"} Jul 22 02:37:17.155371 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:17.155351 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-t9vtn" event={"ID":"34c0277e-a51b-4c45-ae71-9f339d53d946","Type":"ContainerStarted","Data":"bfa4f091eff6f463e8c95c51e3d8ae1bc8844c032f62900f8bd07e7095676f9f"} Jul 22 02:37:17.156358 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:17.156340 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wjd22" event={"ID":"c75eb074-c486-4152-9d81-b12360b19df0","Type":"ContainerStarted","Data":"59c5cd62acf5cfe676293225ee1fc38e5bd547f9eb42387f8a2daec14e3ab63a"} Jul 22 02:37:17.157272 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:17.157254 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-8kmfc" event={"ID":"d7953894-cdfc-48a3-afe8-89503dfc3c57","Type":"ContainerStarted","Data":"2c16802d060b73ebf6eb286c39254d885db4c190e5b44f9181c0375741c4b038"} Jul 22 02:37:17.164104 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:17.164054 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-31.ec2.internal" podStartSLOduration=2.16404253 podStartE2EDuration="2.16404253s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 02:37:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:37:17.163762561 +0000 UTC m=+3.482937866" watchObservedRunningTime="2026-07-22 02:37:17.16404253 +0000 UTC m=+3.483217904" Jul 22 02:37:17.642383 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:17.642342 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d6871664-adf7-41be-b1d0-d1ec7ccae7c6-metrics-certs\") pod \"network-metrics-daemon-lvbjx\" (UID: \"d6871664-adf7-41be-b1d0-d1ec7ccae7c6\") " pod="openshift-multus/network-metrics-daemon-lvbjx" Jul 22 02:37:17.642607 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:17.642408 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/684d87e0-1cec-4247-9446-881aff58a246-original-pull-secret\") pod \"global-pull-secret-syncer-rb9sf\" (UID: \"684d87e0-1cec-4247-9446-881aff58a246\") " pod="kube-system/global-pull-secret-syncer-rb9sf" Jul 22 02:37:17.642694 ip-10-0-133-31 kubenswrapper[2578]: E0722 02:37:17.642592 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 22 02:37:17.642694 ip-10-0-133-31 kubenswrapper[2578]: E0722 02:37:17.642675 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/684d87e0-1cec-4247-9446-881aff58a246-original-pull-secret podName:684d87e0-1cec-4247-9446-881aff58a246 nodeName:}" failed. No retries permitted until 2026-07-22 02:37:19.642655513 +0000 UTC m=+5.961830796 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/684d87e0-1cec-4247-9446-881aff58a246-original-pull-secret") pod "global-pull-secret-syncer-rb9sf" (UID: "684d87e0-1cec-4247-9446-881aff58a246") : object "kube-system"/"original-pull-secret" not registered Jul 22 02:37:17.642839 ip-10-0-133-31 kubenswrapper[2578]: E0722 02:37:17.642745 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 02:37:17.642839 ip-10-0-133-31 kubenswrapper[2578]: E0722 02:37:17.642779 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d6871664-adf7-41be-b1d0-d1ec7ccae7c6-metrics-certs podName:d6871664-adf7-41be-b1d0-d1ec7ccae7c6 nodeName:}" failed. No retries permitted until 2026-07-22 02:37:19.642767184 +0000 UTC m=+5.961942467 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d6871664-adf7-41be-b1d0-d1ec7ccae7c6-metrics-certs") pod "network-metrics-daemon-lvbjx" (UID: "d6871664-adf7-41be-b1d0-d1ec7ccae7c6") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 02:37:17.743420 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:17.743385 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5cxbr\" (UniqueName: \"kubernetes.io/projected/6b93e551-99d3-4ee4-8560-ca6b0f26b6db-kube-api-access-5cxbr\") pod \"network-check-target-wp7zw\" (UID: \"6b93e551-99d3-4ee4-8560-ca6b0f26b6db\") " pod="openshift-network-diagnostics/network-check-target-wp7zw" Jul 22 02:37:17.743595 ip-10-0-133-31 kubenswrapper[2578]: E0722 02:37:17.743567 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 22 02:37:17.743680 ip-10-0-133-31 kubenswrapper[2578]: E0722 02:37:17.743607 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 22 02:37:17.743680 ip-10-0-133-31 kubenswrapper[2578]: E0722 02:37:17.743621 2578 projected.go:196] Error preparing data for projected volume kube-api-access-5cxbr for pod openshift-network-diagnostics/network-check-target-wp7zw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 02:37:17.743680 ip-10-0-133-31 kubenswrapper[2578]: E0722 02:37:17.743680 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6b93e551-99d3-4ee4-8560-ca6b0f26b6db-kube-api-access-5cxbr podName:6b93e551-99d3-4ee4-8560-ca6b0f26b6db nodeName:}" failed. No retries permitted until 2026-07-22 02:37:19.74366271 +0000 UTC m=+6.062837992 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-5cxbr" (UniqueName: "kubernetes.io/projected/6b93e551-99d3-4ee4-8560-ca6b0f26b6db-kube-api-access-5cxbr") pod "network-check-target-wp7zw" (UID: "6b93e551-99d3-4ee4-8560-ca6b0f26b6db") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 02:37:18.067393 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:18.067276 2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-21 02:32:16 +0000 UTC" deadline="2028-04-06 12:47:16.640566234 +0000 UTC" Jul 22 02:37:18.067393 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:18.067342 2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14986h9m58.573228619s" Jul 22 02:37:18.138134 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:18.138088 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rb9sf" Jul 22 02:37:18.138315 ip-10-0-133-31 kubenswrapper[2578]: E0722 02:37:18.138189 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rb9sf" podUID="684d87e0-1cec-4247-9446-881aff58a246" Jul 22 02:37:18.138500 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:18.138482 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lvbjx" Jul 22 02:37:18.138601 ip-10-0-133-31 kubenswrapper[2578]: E0722 02:37:18.138556 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lvbjx" podUID="d6871664-adf7-41be-b1d0-d1ec7ccae7c6" Jul 22 02:37:18.138669 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:18.138600 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wp7zw" Jul 22 02:37:18.138721 ip-10-0-133-31 kubenswrapper[2578]: E0722 02:37:18.138687 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wp7zw" podUID="6b93e551-99d3-4ee4-8560-ca6b0f26b6db" Jul 22 02:37:18.174298 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:18.174260 2578 generic.go:356] "Generic (PLEG): container finished" podID="86e469b6aa7347caf60b105c66d89977" containerID="e398a36eff80c24aa9f3ef2c39ad566018276ed161bb1ad6a812f04895328394" exitCode=0 Jul 22 02:37:18.175329 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:18.175301 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-31.ec2.internal" event={"ID":"86e469b6aa7347caf60b105c66d89977","Type":"ContainerDied","Data":"e398a36eff80c24aa9f3ef2c39ad566018276ed161bb1ad6a812f04895328394"} Jul 22 02:37:19.188672 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:19.188583 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-31.ec2.internal" event={"ID":"86e469b6aa7347caf60b105c66d89977","Type":"ContainerStarted","Data":"06f5f69d43594debec9ba46baf6a53037a345c8871a13e23effc403e8ffb510c"} Jul 22 02:37:19.208342 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:19.208282 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-31.ec2.internal" podStartSLOduration=4.2082630309999995 podStartE2EDuration="4.208263031s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 02:37:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:37:19.207793234 +0000 UTC m=+5.526968539" watchObservedRunningTime="2026-07-22 02:37:19.208263031 +0000 UTC m=+5.527438336" Jul 22 02:37:19.659891 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:19.659849 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d6871664-adf7-41be-b1d0-d1ec7ccae7c6-metrics-certs\") pod \"network-metrics-daemon-lvbjx\" (UID: \"d6871664-adf7-41be-b1d0-d1ec7ccae7c6\") " pod="openshift-multus/network-metrics-daemon-lvbjx" Jul 22 02:37:19.660076 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:19.659916 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/684d87e0-1cec-4247-9446-881aff58a246-original-pull-secret\") pod \"global-pull-secret-syncer-rb9sf\" (UID: \"684d87e0-1cec-4247-9446-881aff58a246\") " pod="kube-system/global-pull-secret-syncer-rb9sf" Jul 22 02:37:19.660127 ip-10-0-133-31 kubenswrapper[2578]: E0722 02:37:19.660097 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 22 02:37:19.660201 ip-10-0-133-31 kubenswrapper[2578]: E0722 02:37:19.660167 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/684d87e0-1cec-4247-9446-881aff58a246-original-pull-secret podName:684d87e0-1cec-4247-9446-881aff58a246 nodeName:}" failed. No retries permitted until 2026-07-22 02:37:23.660145476 +0000 UTC m=+9.979320759 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/684d87e0-1cec-4247-9446-881aff58a246-original-pull-secret") pod "global-pull-secret-syncer-rb9sf" (UID: "684d87e0-1cec-4247-9446-881aff58a246") : object "kube-system"/"original-pull-secret" not registered Jul 22 02:37:19.660715 ip-10-0-133-31 kubenswrapper[2578]: E0722 02:37:19.660688 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 02:37:19.660834 ip-10-0-133-31 kubenswrapper[2578]: E0722 02:37:19.660758 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d6871664-adf7-41be-b1d0-d1ec7ccae7c6-metrics-certs podName:d6871664-adf7-41be-b1d0-d1ec7ccae7c6 nodeName:}" failed. No retries permitted until 2026-07-22 02:37:23.660737309 +0000 UTC m=+9.979912589 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d6871664-adf7-41be-b1d0-d1ec7ccae7c6-metrics-certs") pod "network-metrics-daemon-lvbjx" (UID: "d6871664-adf7-41be-b1d0-d1ec7ccae7c6") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 02:37:19.761232 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:19.761193 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5cxbr\" (UniqueName: \"kubernetes.io/projected/6b93e551-99d3-4ee4-8560-ca6b0f26b6db-kube-api-access-5cxbr\") pod \"network-check-target-wp7zw\" (UID: \"6b93e551-99d3-4ee4-8560-ca6b0f26b6db\") " pod="openshift-network-diagnostics/network-check-target-wp7zw" Jul 22 02:37:19.761407 ip-10-0-133-31 kubenswrapper[2578]: E0722 02:37:19.761381 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 22 02:37:19.761407 ip-10-0-133-31 kubenswrapper[2578]: E0722 02:37:19.761406 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 22 02:37:19.761503 ip-10-0-133-31 kubenswrapper[2578]: E0722 02:37:19.761421 2578 projected.go:196] Error preparing data for projected volume kube-api-access-5cxbr for pod openshift-network-diagnostics/network-check-target-wp7zw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 02:37:19.761503 ip-10-0-133-31 kubenswrapper[2578]: E0722 02:37:19.761482 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6b93e551-99d3-4ee4-8560-ca6b0f26b6db-kube-api-access-5cxbr podName:6b93e551-99d3-4ee4-8560-ca6b0f26b6db nodeName:}" failed. No retries permitted until 2026-07-22 02:37:23.761463387 +0000 UTC m=+10.080638669 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-5cxbr" (UniqueName: "kubernetes.io/projected/6b93e551-99d3-4ee4-8560-ca6b0f26b6db-kube-api-access-5cxbr") pod "network-check-target-wp7zw" (UID: "6b93e551-99d3-4ee4-8560-ca6b0f26b6db") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 02:37:20.136713 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:20.136676 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rb9sf" Jul 22 02:37:20.136878 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:20.136676 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wp7zw" Jul 22 02:37:20.136878 ip-10-0-133-31 kubenswrapper[2578]: E0722 02:37:20.136805 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rb9sf" podUID="684d87e0-1cec-4247-9446-881aff58a246" Jul 22 02:37:20.136878 ip-10-0-133-31 kubenswrapper[2578]: E0722 02:37:20.136868 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wp7zw" podUID="6b93e551-99d3-4ee4-8560-ca6b0f26b6db" Jul 22 02:37:20.137033 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:20.136917 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lvbjx" Jul 22 02:37:20.137128 ip-10-0-133-31 kubenswrapper[2578]: E0722 02:37:20.137090 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lvbjx" podUID="d6871664-adf7-41be-b1d0-d1ec7ccae7c6" Jul 22 02:37:22.138449 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:22.137684 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lvbjx" Jul 22 02:37:22.138449 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:22.137718 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wp7zw" Jul 22 02:37:22.138449 ip-10-0-133-31 kubenswrapper[2578]: E0722 02:37:22.137806 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lvbjx" podUID="d6871664-adf7-41be-b1d0-d1ec7ccae7c6" Jul 22 02:37:22.138449 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:22.137849 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rb9sf" Jul 22 02:37:22.138449 ip-10-0-133-31 kubenswrapper[2578]: E0722 02:37:22.137960 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rb9sf" podUID="684d87e0-1cec-4247-9446-881aff58a246" Jul 22 02:37:22.138449 ip-10-0-133-31 kubenswrapper[2578]: E0722 02:37:22.138041 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wp7zw" podUID="6b93e551-99d3-4ee4-8560-ca6b0f26b6db" Jul 22 02:37:23.694069 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:23.694035 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d6871664-adf7-41be-b1d0-d1ec7ccae7c6-metrics-certs\") pod \"network-metrics-daemon-lvbjx\" (UID: \"d6871664-adf7-41be-b1d0-d1ec7ccae7c6\") " pod="openshift-multus/network-metrics-daemon-lvbjx" Jul 22 02:37:23.694508 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:23.694084 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/684d87e0-1cec-4247-9446-881aff58a246-original-pull-secret\") pod \"global-pull-secret-syncer-rb9sf\" (UID: \"684d87e0-1cec-4247-9446-881aff58a246\") " pod="kube-system/global-pull-secret-syncer-rb9sf" Jul 22 02:37:23.694508 ip-10-0-133-31 kubenswrapper[2578]: E0722 02:37:23.694209 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 22 02:37:23.694508 ip-10-0-133-31 kubenswrapper[2578]: E0722 02:37:23.694252 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/684d87e0-1cec-4247-9446-881aff58a246-original-pull-secret podName:684d87e0-1cec-4247-9446-881aff58a246 nodeName:}" failed. No retries permitted until 2026-07-22 02:37:31.69423913 +0000 UTC m=+18.013414412 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/684d87e0-1cec-4247-9446-881aff58a246-original-pull-secret") pod "global-pull-secret-syncer-rb9sf" (UID: "684d87e0-1cec-4247-9446-881aff58a246") : object "kube-system"/"original-pull-secret" not registered Jul 22 02:37:23.694508 ip-10-0-133-31 kubenswrapper[2578]: E0722 02:37:23.694302 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 02:37:23.694508 ip-10-0-133-31 kubenswrapper[2578]: E0722 02:37:23.694332 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d6871664-adf7-41be-b1d0-d1ec7ccae7c6-metrics-certs podName:d6871664-adf7-41be-b1d0-d1ec7ccae7c6 nodeName:}" failed. No retries permitted until 2026-07-22 02:37:31.694323606 +0000 UTC m=+18.013498891 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d6871664-adf7-41be-b1d0-d1ec7ccae7c6-metrics-certs") pod "network-metrics-daemon-lvbjx" (UID: "d6871664-adf7-41be-b1d0-d1ec7ccae7c6") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 02:37:23.795296 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:23.795259 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5cxbr\" (UniqueName: \"kubernetes.io/projected/6b93e551-99d3-4ee4-8560-ca6b0f26b6db-kube-api-access-5cxbr\") pod \"network-check-target-wp7zw\" (UID: \"6b93e551-99d3-4ee4-8560-ca6b0f26b6db\") " pod="openshift-network-diagnostics/network-check-target-wp7zw" Jul 22 02:37:23.795453 ip-10-0-133-31 kubenswrapper[2578]: E0722 02:37:23.795434 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 22 02:37:23.795453 ip-10-0-133-31 kubenswrapper[2578]: E0722 02:37:23.795449 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 22 02:37:23.795590 ip-10-0-133-31 kubenswrapper[2578]: E0722 02:37:23.795458 2578 projected.go:196] Error preparing data for projected volume kube-api-access-5cxbr for pod openshift-network-diagnostics/network-check-target-wp7zw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 02:37:23.795590 ip-10-0-133-31 kubenswrapper[2578]: E0722 02:37:23.795509 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6b93e551-99d3-4ee4-8560-ca6b0f26b6db-kube-api-access-5cxbr podName:6b93e551-99d3-4ee4-8560-ca6b0f26b6db nodeName:}" failed. No retries permitted until 2026-07-22 02:37:31.795491651 +0000 UTC m=+18.114666942 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-5cxbr" (UniqueName: "kubernetes.io/projected/6b93e551-99d3-4ee4-8560-ca6b0f26b6db-kube-api-access-5cxbr") pod "network-check-target-wp7zw" (UID: "6b93e551-99d3-4ee4-8560-ca6b0f26b6db") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 02:37:24.137289 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:24.136896 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lvbjx" Jul 22 02:37:24.137289 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:24.136933 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wp7zw" Jul 22 02:37:24.137289 ip-10-0-133-31 kubenswrapper[2578]: E0722 02:37:24.136994 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lvbjx" podUID="d6871664-adf7-41be-b1d0-d1ec7ccae7c6" Jul 22 02:37:24.137289 ip-10-0-133-31 kubenswrapper[2578]: E0722 02:37:24.137061 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wp7zw" podUID="6b93e551-99d3-4ee4-8560-ca6b0f26b6db" Jul 22 02:37:24.137289 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:24.137180 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rb9sf" Jul 22 02:37:24.137289 ip-10-0-133-31 kubenswrapper[2578]: E0722 02:37:24.137260 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rb9sf" podUID="684d87e0-1cec-4247-9446-881aff58a246" Jul 22 02:37:26.135951 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:26.135863 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lvbjx" Jul 22 02:37:26.135951 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:26.135901 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wp7zw" Jul 22 02:37:26.136467 ip-10-0-133-31 kubenswrapper[2578]: E0722 02:37:26.136277 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lvbjx" podUID="d6871664-adf7-41be-b1d0-d1ec7ccae7c6" Jul 22 02:37:26.136467 ip-10-0-133-31 kubenswrapper[2578]: E0722 02:37:26.136345 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wp7zw" podUID="6b93e551-99d3-4ee4-8560-ca6b0f26b6db" Jul 22 02:37:26.136467 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:26.136425 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rb9sf" Jul 22 02:37:26.136639 ip-10-0-133-31 kubenswrapper[2578]: E0722 02:37:26.136622 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rb9sf" podUID="684d87e0-1cec-4247-9446-881aff58a246" Jul 22 02:37:28.136781 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:28.136638 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wp7zw" Jul 22 02:37:28.136781 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:28.136656 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rb9sf" Jul 22 02:37:28.136781 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:28.136638 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lvbjx" Jul 22 02:37:28.136781 ip-10-0-133-31 kubenswrapper[2578]: E0722 02:37:28.136744 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wp7zw" podUID="6b93e551-99d3-4ee4-8560-ca6b0f26b6db" Jul 22 02:37:28.137335 ip-10-0-133-31 kubenswrapper[2578]: E0722 02:37:28.136858 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lvbjx" podUID="d6871664-adf7-41be-b1d0-d1ec7ccae7c6" Jul 22 02:37:28.137335 ip-10-0-133-31 kubenswrapper[2578]: E0722 02:37:28.136930 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rb9sf" podUID="684d87e0-1cec-4247-9446-881aff58a246" Jul 22 02:37:30.136554 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:30.136519 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wp7zw" Jul 22 02:37:30.137032 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:30.136519 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lvbjx" Jul 22 02:37:30.137032 ip-10-0-133-31 kubenswrapper[2578]: E0722 02:37:30.136660 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wp7zw" podUID="6b93e551-99d3-4ee4-8560-ca6b0f26b6db" Jul 22 02:37:30.137032 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:30.136526 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rb9sf" Jul 22 02:37:30.137032 ip-10-0-133-31 kubenswrapper[2578]: E0722 02:37:30.136727 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lvbjx" podUID="d6871664-adf7-41be-b1d0-d1ec7ccae7c6" Jul 22 02:37:30.137032 ip-10-0-133-31 kubenswrapper[2578]: E0722 02:37:30.136801 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rb9sf" podUID="684d87e0-1cec-4247-9446-881aff58a246" Jul 22 02:37:31.764220 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:31.764183 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d6871664-adf7-41be-b1d0-d1ec7ccae7c6-metrics-certs\") pod \"network-metrics-daemon-lvbjx\" (UID: \"d6871664-adf7-41be-b1d0-d1ec7ccae7c6\") " pod="openshift-multus/network-metrics-daemon-lvbjx" Jul 22 02:37:31.764722 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:31.764229 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/684d87e0-1cec-4247-9446-881aff58a246-original-pull-secret\") pod \"global-pull-secret-syncer-rb9sf\" (UID: \"684d87e0-1cec-4247-9446-881aff58a246\") " pod="kube-system/global-pull-secret-syncer-rb9sf" Jul 22 02:37:31.764722 ip-10-0-133-31 kubenswrapper[2578]: E0722 02:37:31.764344 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 22 02:37:31.764722 ip-10-0-133-31 kubenswrapper[2578]: E0722 02:37:31.764346 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 02:37:31.764722 ip-10-0-133-31 kubenswrapper[2578]: E0722 02:37:31.764415 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/684d87e0-1cec-4247-9446-881aff58a246-original-pull-secret podName:684d87e0-1cec-4247-9446-881aff58a246 nodeName:}" failed. No retries permitted until 2026-07-22 02:37:47.764397111 +0000 UTC m=+34.083572394 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/684d87e0-1cec-4247-9446-881aff58a246-original-pull-secret") pod "global-pull-secret-syncer-rb9sf" (UID: "684d87e0-1cec-4247-9446-881aff58a246") : object "kube-system"/"original-pull-secret" not registered Jul 22 02:37:31.764722 ip-10-0-133-31 kubenswrapper[2578]: E0722 02:37:31.764438 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d6871664-adf7-41be-b1d0-d1ec7ccae7c6-metrics-certs podName:d6871664-adf7-41be-b1d0-d1ec7ccae7c6 nodeName:}" failed. No retries permitted until 2026-07-22 02:37:47.764426372 +0000 UTC m=+34.083601651 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d6871664-adf7-41be-b1d0-d1ec7ccae7c6-metrics-certs") pod "network-metrics-daemon-lvbjx" (UID: "d6871664-adf7-41be-b1d0-d1ec7ccae7c6") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 02:37:31.865129 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:31.865098 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5cxbr\" (UniqueName: \"kubernetes.io/projected/6b93e551-99d3-4ee4-8560-ca6b0f26b6db-kube-api-access-5cxbr\") pod \"network-check-target-wp7zw\" (UID: \"6b93e551-99d3-4ee4-8560-ca6b0f26b6db\") " pod="openshift-network-diagnostics/network-check-target-wp7zw" Jul 22 02:37:31.865267 ip-10-0-133-31 kubenswrapper[2578]: E0722 02:37:31.865235 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 22 02:37:31.865267 ip-10-0-133-31 kubenswrapper[2578]: E0722 02:37:31.865252 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 22 02:37:31.865267 ip-10-0-133-31 kubenswrapper[2578]: E0722 02:37:31.865262 2578 projected.go:196] Error preparing data for projected volume kube-api-access-5cxbr for pod openshift-network-diagnostics/network-check-target-wp7zw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 02:37:31.865388 ip-10-0-133-31 kubenswrapper[2578]: E0722 02:37:31.865303 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6b93e551-99d3-4ee4-8560-ca6b0f26b6db-kube-api-access-5cxbr podName:6b93e551-99d3-4ee4-8560-ca6b0f26b6db nodeName:}" failed. No retries permitted until 2026-07-22 02:37:47.86529193 +0000 UTC m=+34.184467221 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-5cxbr" (UniqueName: "kubernetes.io/projected/6b93e551-99d3-4ee4-8560-ca6b0f26b6db-kube-api-access-5cxbr") pod "network-check-target-wp7zw" (UID: "6b93e551-99d3-4ee4-8560-ca6b0f26b6db") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 02:37:32.136069 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:32.136034 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wp7zw" Jul 22 02:37:32.136069 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:32.136035 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lvbjx" Jul 22 02:37:32.136069 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:32.136074 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rb9sf" Jul 22 02:37:32.136321 ip-10-0-133-31 kubenswrapper[2578]: E0722 02:37:32.136154 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wp7zw" podUID="6b93e551-99d3-4ee4-8560-ca6b0f26b6db" Jul 22 02:37:32.136321 ip-10-0-133-31 kubenswrapper[2578]: E0722 02:37:32.136251 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lvbjx" podUID="d6871664-adf7-41be-b1d0-d1ec7ccae7c6" Jul 22 02:37:32.136407 ip-10-0-133-31 kubenswrapper[2578]: E0722 02:37:32.136325 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rb9sf" podUID="684d87e0-1cec-4247-9446-881aff58a246" Jul 22 02:37:34.136636 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:34.136611 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lvbjx" Jul 22 02:37:34.137091 ip-10-0-133-31 kubenswrapper[2578]: E0722 02:37:34.136704 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lvbjx" podUID="d6871664-adf7-41be-b1d0-d1ec7ccae7c6" Jul 22 02:37:34.137091 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:34.136772 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wp7zw" Jul 22 02:37:34.137091 ip-10-0-133-31 kubenswrapper[2578]: E0722 02:37:34.136848 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wp7zw" podUID="6b93e551-99d3-4ee4-8560-ca6b0f26b6db" Jul 22 02:37:34.137091 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:34.136878 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rb9sf" Jul 22 02:37:34.137091 ip-10-0-133-31 kubenswrapper[2578]: E0722 02:37:34.136996 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rb9sf" podUID="684d87e0-1cec-4247-9446-881aff58a246" Jul 22 02:37:36.136100 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:36.135948 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lvbjx" Jul 22 02:37:36.136100 ip-10-0-133-31 kubenswrapper[2578]: E0722 02:37:36.136081 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lvbjx" podUID="d6871664-adf7-41be-b1d0-d1ec7ccae7c6" Jul 22 02:37:36.136502 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:36.136418 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wp7zw" Jul 22 02:37:36.136556 ip-10-0-133-31 kubenswrapper[2578]: E0722 02:37:36.136511 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wp7zw" podUID="6b93e551-99d3-4ee4-8560-ca6b0f26b6db" Jul 22 02:37:36.136640 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:36.136590 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rb9sf" Jul 22 02:37:36.136690 ip-10-0-133-31 kubenswrapper[2578]: E0722 02:37:36.136675 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rb9sf" podUID="684d87e0-1cec-4247-9446-881aff58a246" Jul 22 02:37:36.221865 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:36.221669 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-bxg22" event={"ID":"e56db148-7293-423e-b9fb-0ebd5db740a3","Type":"ContainerStarted","Data":"bfd4531eed824314ae8e6325089fc025efd2a4cf1e04992dd7213184e28fbd43"} Jul 22 02:37:36.227195 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:36.227067 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-9dhrs" event={"ID":"dea9d10b-9320-471c-9efd-544fabc16854","Type":"ContainerStarted","Data":"554c81cebb923d4d2dd62b25b514ca24705b277d59a65e8f75e1433e029d805d"} Jul 22 02:37:36.243549 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:36.243510 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-bxg22" podStartSLOduration=11.352594787 podStartE2EDuration="22.243495568s" totalImagesPullingTime="10.890900781s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 02:37:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:37:36.242953595 +0000 UTC m=+22.562128894" watchObservedRunningTime="2026-07-22 02:37:36.243495568 +0000 UTC m=+22.562670867" Jul 22 02:37:36.928737 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:36.928536 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-bxg22" Jul 22 02:37:36.929434 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:36.929416 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-bxg22" Jul 22 02:37:36.954618 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:36.954557 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-9dhrs" podStartSLOduration=3.879335983 podStartE2EDuration="22.954543856s" totalImagesPullingTime="19.075207873s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 02:37:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:37:36.261425636 +0000 UTC m=+22.580600937" watchObservedRunningTime="2026-07-22 02:37:36.954543856 +0000 UTC m=+23.273719155" Jul 22 02:37:37.230097 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:37.230064 2578 generic.go:356] "Generic (PLEG): container finished" podID="34c0277e-a51b-4c45-ae71-9f339d53d946" containerID="48508601a174ee42b0cd57c5d6439898dc2e1b11cf9e8fa59f0b03fbb0beac17" exitCode=0 Jul 22 02:37:37.230849 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:37.230147 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-t9vtn" event={"ID":"34c0277e-a51b-4c45-ae71-9f339d53d946","Type":"ContainerDied","Data":"48508601a174ee42b0cd57c5d6439898dc2e1b11cf9e8fa59f0b03fbb0beac17"} Jul 22 02:37:37.231603 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:37.231587 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wjd22" event={"ID":"c75eb074-c486-4152-9d81-b12360b19df0","Type":"ContainerStarted","Data":"36eec5e6834cbe76379185b840c49946f6d69ec9c1164fa279178cb7bf61f50b"} Jul 22 02:37:37.232903 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:37.232885 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-8kmfc" event={"ID":"d7953894-cdfc-48a3-afe8-89503dfc3c57","Type":"ContainerStarted","Data":"f968bfd31f1ddb4ea838ece9f0d9ddeb1c0762cfba2c9ecb81946a110eea9fc1"} Jul 22 02:37:37.234141 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:37.234125 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-ftqsf" event={"ID":"6a441682-e619-4c8d-96a2-3a391f8eac76","Type":"ContainerStarted","Data":"a2d9a5df592cab061536bca041a69d1af5dc58c79793c9f5425079909e8af1ac"} Jul 22 02:37:37.235477 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:37.235460 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-8ssxs" event={"ID":"5745b751-8d56-48ab-9e80-93190f8ef935","Type":"ContainerStarted","Data":"703471b95072e472e62e9e26dde4ef9de081c3ef276fa23555ef04f103d92e41"} Jul 22 02:37:37.237784 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:37.237770 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-prfql_fd7bf923-8d16-476c-a8d0-a3e6fd645061/ovn-acl-logging/0.log" Jul 22 02:37:37.238093 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:37.238077 2578 generic.go:356] "Generic (PLEG): container finished" podID="fd7bf923-8d16-476c-a8d0-a3e6fd645061" containerID="ce9383c1bbe5678fabcac7f410178146bea07f2539eabd4774f571f777045115" exitCode=1 Jul 22 02:37:37.238169 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:37.238152 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-prfql" event={"ID":"fd7bf923-8d16-476c-a8d0-a3e6fd645061","Type":"ContainerStarted","Data":"b830f37cb1cd1ab87ae7d3585924acb4577533d4d77fb15fe921a8ee96165f00"} Jul 22 02:37:37.238244 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:37.238230 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-prfql" event={"ID":"fd7bf923-8d16-476c-a8d0-a3e6fd645061","Type":"ContainerStarted","Data":"359fb78a825e217fa36a9bdd7a20b0836b7a93fea1602c38cee5146ed97254b0"} Jul 22 02:37:37.238298 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:37.238282 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-prfql" event={"ID":"fd7bf923-8d16-476c-a8d0-a3e6fd645061","Type":"ContainerStarted","Data":"645760e232be4a71e27bb6b326a2caf0db4e64acf3ec646397c547a6a9d5e1c5"} Jul 22 02:37:37.238351 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:37.238337 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-bxg22" Jul 22 02:37:37.238351 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:37.238359 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-prfql" event={"ID":"fd7bf923-8d16-476c-a8d0-a3e6fd645061","Type":"ContainerStarted","Data":"452fe3f5f177fa3f67823f30613fdc773f8fef487c94c6cd78d30020cf4a2920"} Jul 22 02:37:37.238497 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:37.238407 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-prfql" event={"ID":"fd7bf923-8d16-476c-a8d0-a3e6fd645061","Type":"ContainerDied","Data":"ce9383c1bbe5678fabcac7f410178146bea07f2539eabd4774f571f777045115"} Jul 22 02:37:37.238497 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:37.238458 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-prfql" event={"ID":"fd7bf923-8d16-476c-a8d0-a3e6fd645061","Type":"ContainerStarted","Data":"31007ce1d6e68e688576074f223aea102ce0512347c769b4ea5c81e8b6246304"} Jul 22 02:37:37.238778 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:37.238763 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-bxg22" Jul 22 02:37:37.354640 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:37.354605 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-ftqsf" podStartSLOduration=4.284120854 podStartE2EDuration="23.354591847s" totalImagesPullingTime="19.070470993s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 02:37:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:37:37.29978357 +0000 UTC m=+23.618958870" watchObservedRunningTime="2026-07-22 02:37:37.354591847 +0000 UTC m=+23.673767144" Jul 22 02:37:37.384800 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:37.384771 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-8kmfc" podStartSLOduration=4.337843966 podStartE2EDuration="23.38475963s" totalImagesPullingTime="19.046915664s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 02:37:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:37:37.384322327 +0000 UTC m=+23.703497639" watchObservedRunningTime="2026-07-22 02:37:37.38475963 +0000 UTC m=+23.703934912" Jul 22 02:37:37.384863 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:37.384844 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-8ssxs" podStartSLOduration=4.297244822 podStartE2EDuration="23.384839277s" totalImagesPullingTime="19.087594455s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 02:37:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:37:37.356510661 +0000 UTC m=+23.675685961" watchObservedRunningTime="2026-07-22 02:37:37.384839277 +0000 UTC m=+23.704014576" Jul 22 02:37:37.446095 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:37.446076 2578 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 22 02:37:38.084914 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:38.084813 2578 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-22T02:37:37.446090422Z","UUID":"b240cfa5-b59a-4da4-b1a2-422cf10e4186","Handler":null,"Name":"","Endpoint":""} Jul 22 02:37:38.086678 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:38.086653 2578 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 22 02:37:38.086797 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:38.086702 2578 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 22 02:37:38.138387 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:38.138362 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wp7zw" Jul 22 02:37:38.138548 ip-10-0-133-31 kubenswrapper[2578]: E0722 02:37:38.138503 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wp7zw" podUID="6b93e551-99d3-4ee4-8560-ca6b0f26b6db" Jul 22 02:37:38.138817 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:38.138792 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lvbjx" Jul 22 02:37:38.138817 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:38.138805 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rb9sf" Jul 22 02:37:38.138944 ip-10-0-133-31 kubenswrapper[2578]: E0722 02:37:38.138909 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lvbjx" podUID="d6871664-adf7-41be-b1d0-d1ec7ccae7c6" Jul 22 02:37:38.139027 ip-10-0-133-31 kubenswrapper[2578]: E0722 02:37:38.138991 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rb9sf" podUID="684d87e0-1cec-4247-9446-881aff58a246" Jul 22 02:37:38.241685 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:38.241653 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-ld8dc" event={"ID":"9834be29-d56c-4ce7-a542-bea10d73ccfd","Type":"ContainerStarted","Data":"217ec40f5de9ac62f091d0e2b4ccfe5f5891efe1981f1c7b4ac270190265bb23"} Jul 22 02:37:38.243532 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:38.243506 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wjd22" event={"ID":"c75eb074-c486-4152-9d81-b12360b19df0","Type":"ContainerStarted","Data":"d27cdaed3593b6319034d11c00f20865ac74d5917b1b42d5f6ffbf78e50e820d"} Jul 22 02:37:38.260995 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:38.260954 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-ld8dc" podStartSLOduration=5.17849287 podStartE2EDuration="24.260942256s" totalImagesPullingTime="19.082449386s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 02:37:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:37:38.26017318 +0000 UTC m=+24.579348480" watchObservedRunningTime="2026-07-22 02:37:38.260942256 +0000 UTC m=+24.580117555" Jul 22 02:37:39.249180 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:39.249111 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-prfql_fd7bf923-8d16-476c-a8d0-a3e6fd645061/ovn-acl-logging/0.log" Jul 22 02:37:39.249588 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:39.249520 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-prfql" event={"ID":"fd7bf923-8d16-476c-a8d0-a3e6fd645061","Type":"ContainerStarted","Data":"5d181cfb5f05fa3bd63764aeb3589c67ff63f57e1265ec6699e0fbc7e288f7a1"} Jul 22 02:37:39.252020 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:39.251998 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wjd22" event={"ID":"c75eb074-c486-4152-9d81-b12360b19df0","Type":"ContainerStarted","Data":"168e81643e0797d5c4e66f2e869d1e76aabb0c35837ec8957a0436fea5d77e54"} Jul 22 02:37:39.278872 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:39.278828 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wjd22" podStartSLOduration=4.208742488 podStartE2EDuration="25.278812029s" totalImagesPullingTime="21.070069541s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 02:37:14 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:37:39.278187975 +0000 UTC m=+25.597363276" watchObservedRunningTime="2026-07-22 02:37:39.278812029 +0000 UTC m=+25.597987331" Jul 22 02:37:40.136414 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:40.136228 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wp7zw" Jul 22 02:37:40.136594 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:40.136228 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lvbjx" Jul 22 02:37:40.136594 ip-10-0-133-31 kubenswrapper[2578]: E0722 02:37:40.136511 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wp7zw" podUID="6b93e551-99d3-4ee4-8560-ca6b0f26b6db" Jul 22 02:37:40.136594 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:40.136254 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rb9sf" Jul 22 02:37:40.136757 ip-10-0-133-31 kubenswrapper[2578]: E0722 02:37:40.136599 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lvbjx" podUID="d6871664-adf7-41be-b1d0-d1ec7ccae7c6" Jul 22 02:37:40.136757 ip-10-0-133-31 kubenswrapper[2578]: E0722 02:37:40.136675 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rb9sf" podUID="684d87e0-1cec-4247-9446-881aff58a246" Jul 22 02:37:42.136445 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:42.136227 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lvbjx" Jul 22 02:37:42.137220 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:42.136271 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wp7zw" Jul 22 02:37:42.137220 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:42.136339 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rb9sf" Jul 22 02:37:42.137220 ip-10-0-133-31 kubenswrapper[2578]: E0722 02:37:42.136532 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lvbjx" podUID="d6871664-adf7-41be-b1d0-d1ec7ccae7c6" Jul 22 02:37:42.137220 ip-10-0-133-31 kubenswrapper[2578]: E0722 02:37:42.136636 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rb9sf" podUID="684d87e0-1cec-4247-9446-881aff58a246" Jul 22 02:37:42.137220 ip-10-0-133-31 kubenswrapper[2578]: E0722 02:37:42.136716 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wp7zw" podUID="6b93e551-99d3-4ee4-8560-ca6b0f26b6db" Jul 22 02:37:42.261102 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:42.261022 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-prfql_fd7bf923-8d16-476c-a8d0-a3e6fd645061/ovn-acl-logging/0.log" Jul 22 02:37:42.261450 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:42.261431 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-prfql" event={"ID":"fd7bf923-8d16-476c-a8d0-a3e6fd645061","Type":"ContainerStarted","Data":"4480d4eba7c777f421fe38cf7274f644214851faf4a5426ad5bd86433d74bbc6"} Jul 22 02:37:42.261703 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:42.261673 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-prfql" Jul 22 02:37:42.261906 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:42.261889 2578 scope.go:117] "RemoveContainer" containerID="ce9383c1bbe5678fabcac7f410178146bea07f2539eabd4774f571f777045115" Jul 22 02:37:42.263437 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:42.263411 2578 generic.go:356] "Generic (PLEG): container finished" podID="34c0277e-a51b-4c45-ae71-9f339d53d946" containerID="d7a29ebbb264ccd96617573a39f5a73b2d5989a66a0bb7d7a59c0675d4fa1cc8" exitCode=0 Jul 22 02:37:42.263523 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:42.263459 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-t9vtn" event={"ID":"34c0277e-a51b-4c45-ae71-9f339d53d946","Type":"ContainerDied","Data":"d7a29ebbb264ccd96617573a39f5a73b2d5989a66a0bb7d7a59c0675d4fa1cc8"} Jul 22 02:37:42.278020 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:42.277988 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-prfql" Jul 22 02:37:43.269079 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:43.269056 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-prfql_fd7bf923-8d16-476c-a8d0-a3e6fd645061/ovn-acl-logging/0.log" Jul 22 02:37:43.269483 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:43.269456 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-prfql" event={"ID":"fd7bf923-8d16-476c-a8d0-a3e6fd645061","Type":"ContainerStarted","Data":"fb48f3a7227f70a74b70e2c212c96323135f0daaa6537f2d47c4e1adf019f6d8"} Jul 22 02:37:43.269804 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:43.269780 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-prfql" Jul 22 02:37:43.269902 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:43.269856 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-prfql" Jul 22 02:37:43.284807 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:43.284789 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-prfql" Jul 22 02:37:43.317757 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:43.317707 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-prfql" podStartSLOduration=10.014720298 podStartE2EDuration="29.317689128s" totalImagesPullingTime="19.30296883s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 02:37:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:37:43.315667742 +0000 UTC m=+29.634843054" watchObservedRunningTime="2026-07-22 02:37:43.317689128 +0000 UTC m=+29.636864429" Jul 22 02:37:43.556506 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:43.556442 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-lvbjx"] Jul 22 02:37:43.556663 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:43.556539 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lvbjx" Jul 22 02:37:43.556663 ip-10-0-133-31 kubenswrapper[2578]: E0722 02:37:43.556633 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lvbjx" podUID="d6871664-adf7-41be-b1d0-d1ec7ccae7c6" Jul 22 02:37:43.559974 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:43.559948 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-wp7zw"] Jul 22 02:37:43.560104 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:43.560042 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wp7zw" Jul 22 02:37:43.560163 ip-10-0-133-31 kubenswrapper[2578]: E0722 02:37:43.560127 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wp7zw" podUID="6b93e551-99d3-4ee4-8560-ca6b0f26b6db" Jul 22 02:37:43.560578 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:43.560537 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-rb9sf"] Jul 22 02:37:43.560660 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:43.560643 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rb9sf" Jul 22 02:37:43.560752 ip-10-0-133-31 kubenswrapper[2578]: E0722 02:37:43.560733 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rb9sf" podUID="684d87e0-1cec-4247-9446-881aff58a246" Jul 22 02:37:44.273490 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:44.273463 2578 generic.go:356] "Generic (PLEG): container finished" podID="34c0277e-a51b-4c45-ae71-9f339d53d946" containerID="061cd726812631f9d1ae4fceea90d42e0233922c7377cbffd2b4a57454adcc64" exitCode=0 Jul 22 02:37:44.273989 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:44.273551 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-t9vtn" event={"ID":"34c0277e-a51b-4c45-ae71-9f339d53d946","Type":"ContainerDied","Data":"061cd726812631f9d1ae4fceea90d42e0233922c7377cbffd2b4a57454adcc64"} Jul 22 02:37:45.136489 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:45.136454 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rb9sf" Jul 22 02:37:45.136489 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:45.136471 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wp7zw" Jul 22 02:37:45.136670 ip-10-0-133-31 kubenswrapper[2578]: E0722 02:37:45.136586 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rb9sf" podUID="684d87e0-1cec-4247-9446-881aff58a246" Jul 22 02:37:45.136670 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:45.136621 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lvbjx" Jul 22 02:37:45.136768 ip-10-0-133-31 kubenswrapper[2578]: E0722 02:37:45.136717 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lvbjx" podUID="d6871664-adf7-41be-b1d0-d1ec7ccae7c6" Jul 22 02:37:45.136846 ip-10-0-133-31 kubenswrapper[2578]: E0722 02:37:45.136818 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wp7zw" podUID="6b93e551-99d3-4ee4-8560-ca6b0f26b6db" Jul 22 02:37:46.280148 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:46.279957 2578 generic.go:356] "Generic (PLEG): container finished" podID="34c0277e-a51b-4c45-ae71-9f339d53d946" containerID="e8d6eb79f92428a143a5255f6735d90961b1c9630ec63043af180bf4534be12d" exitCode=0 Jul 22 02:37:46.280148 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:46.280038 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-t9vtn" event={"ID":"34c0277e-a51b-4c45-ae71-9f339d53d946","Type":"ContainerDied","Data":"e8d6eb79f92428a143a5255f6735d90961b1c9630ec63043af180bf4534be12d"} Jul 22 02:37:47.135981 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:47.135906 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lvbjx" Jul 22 02:37:47.136133 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:47.135905 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rb9sf" Jul 22 02:37:47.136133 ip-10-0-133-31 kubenswrapper[2578]: E0722 02:37:47.136056 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lvbjx" podUID="d6871664-adf7-41be-b1d0-d1ec7ccae7c6" Jul 22 02:37:47.136133 ip-10-0-133-31 kubenswrapper[2578]: E0722 02:37:47.136114 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rb9sf" podUID="684d87e0-1cec-4247-9446-881aff58a246" Jul 22 02:37:47.136133 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:47.135905 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wp7zw" Jul 22 02:37:47.136300 ip-10-0-133-31 kubenswrapper[2578]: E0722 02:37:47.136210 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wp7zw" podUID="6b93e551-99d3-4ee4-8560-ca6b0f26b6db" Jul 22 02:37:47.788441 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:47.788399 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d6871664-adf7-41be-b1d0-d1ec7ccae7c6-metrics-certs\") pod \"network-metrics-daemon-lvbjx\" (UID: \"d6871664-adf7-41be-b1d0-d1ec7ccae7c6\") " pod="openshift-multus/network-metrics-daemon-lvbjx" Jul 22 02:37:47.788871 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:47.788465 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/684d87e0-1cec-4247-9446-881aff58a246-original-pull-secret\") pod \"global-pull-secret-syncer-rb9sf\" (UID: \"684d87e0-1cec-4247-9446-881aff58a246\") " pod="kube-system/global-pull-secret-syncer-rb9sf" Jul 22 02:37:47.788871 ip-10-0-133-31 kubenswrapper[2578]: E0722 02:37:47.788554 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 02:37:47.788871 ip-10-0-133-31 kubenswrapper[2578]: E0722 02:37:47.788620 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 22 02:37:47.788871 ip-10-0-133-31 kubenswrapper[2578]: E0722 02:37:47.788637 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d6871664-adf7-41be-b1d0-d1ec7ccae7c6-metrics-certs podName:d6871664-adf7-41be-b1d0-d1ec7ccae7c6 nodeName:}" failed. No retries permitted until 2026-07-22 02:38:19.788621434 +0000 UTC m=+66.107796714 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d6871664-adf7-41be-b1d0-d1ec7ccae7c6-metrics-certs") pod "network-metrics-daemon-lvbjx" (UID: "d6871664-adf7-41be-b1d0-d1ec7ccae7c6") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 02:37:47.788871 ip-10-0-133-31 kubenswrapper[2578]: E0722 02:37:47.788677 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/684d87e0-1cec-4247-9446-881aff58a246-original-pull-secret podName:684d87e0-1cec-4247-9446-881aff58a246 nodeName:}" failed. No retries permitted until 2026-07-22 02:38:19.788660343 +0000 UTC m=+66.107835622 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/684d87e0-1cec-4247-9446-881aff58a246-original-pull-secret") pod "global-pull-secret-syncer-rb9sf" (UID: "684d87e0-1cec-4247-9446-881aff58a246") : object "kube-system"/"original-pull-secret" not registered Jul 22 02:37:47.889083 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:47.889048 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5cxbr\" (UniqueName: \"kubernetes.io/projected/6b93e551-99d3-4ee4-8560-ca6b0f26b6db-kube-api-access-5cxbr\") pod \"network-check-target-wp7zw\" (UID: \"6b93e551-99d3-4ee4-8560-ca6b0f26b6db\") " pod="openshift-network-diagnostics/network-check-target-wp7zw" Jul 22 02:37:47.889250 ip-10-0-133-31 kubenswrapper[2578]: E0722 02:37:47.889204 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 22 02:37:47.889250 ip-10-0-133-31 kubenswrapper[2578]: E0722 02:37:47.889221 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 22 02:37:47.889250 ip-10-0-133-31 kubenswrapper[2578]: E0722 02:37:47.889231 2578 projected.go:196] Error preparing data for projected volume kube-api-access-5cxbr for pod openshift-network-diagnostics/network-check-target-wp7zw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 02:37:47.889376 ip-10-0-133-31 kubenswrapper[2578]: E0722 02:37:47.889287 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6b93e551-99d3-4ee4-8560-ca6b0f26b6db-kube-api-access-5cxbr podName:6b93e551-99d3-4ee4-8560-ca6b0f26b6db nodeName:}" failed. No retries permitted until 2026-07-22 02:38:19.889269553 +0000 UTC m=+66.208444844 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-5cxbr" (UniqueName: "kubernetes.io/projected/6b93e551-99d3-4ee4-8560-ca6b0f26b6db-kube-api-access-5cxbr") pod "network-check-target-wp7zw" (UID: "6b93e551-99d3-4ee4-8560-ca6b0f26b6db") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 02:37:49.034588 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:49.034548 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-31.ec2.internal" event="NodeReady" Jul 22 02:37:49.035105 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:49.034718 2578 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 22 02:37:49.109774 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:49.109745 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-trlg7"] Jul 22 02:37:49.112629 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:49.112071 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-trlg7" Jul 22 02:37:49.112903 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:49.112880 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-d469874bc-bnc45"] Jul 22 02:37:49.114623 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:49.114603 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-d469874bc-bnc45" Jul 22 02:37:49.115223 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:49.115198 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 22 02:37:49.115775 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:49.115756 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 22 02:37:49.115775 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:49.115767 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-7k6rz\"" Jul 22 02:37:49.118834 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:49.118814 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 22 02:37:49.119328 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:49.119125 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-swntg\"" Jul 22 02:37:49.119328 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:49.119188 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 22 02:37:49.119328 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:49.119192 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 22 02:37:49.124922 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:49.124904 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-8fzdj"] Jul 22 02:37:49.125286 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:49.125268 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 22 02:37:49.126550 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:49.126532 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-8fzdj" Jul 22 02:37:49.131748 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:49.131727 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 22 02:37:49.131915 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:49.131893 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-gq2kr\"" Jul 22 02:37:49.131987 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:49.131977 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 22 02:37:49.132166 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:49.132148 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 22 02:37:49.135166 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:49.135148 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-d469874bc-bnc45"] Jul 22 02:37:49.135757 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:49.135710 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wp7zw" Jul 22 02:37:49.135858 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:49.135817 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rb9sf" Jul 22 02:37:49.135961 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:49.135940 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lvbjx" Jul 22 02:37:49.138622 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:49.138461 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-trlg7"] Jul 22 02:37:49.139178 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:49.139161 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 22 02:37:49.140627 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:49.140606 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 22 02:37:49.140866 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:49.140848 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 22 02:37:49.140963 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:49.140896 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 22 02:37:49.141491 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:49.141474 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-d24b6\"" Jul 22 02:37:49.141491 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:49.141481 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-48l7s\"" Jul 22 02:37:49.151322 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:49.151302 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-8fzdj"] Jul 22 02:37:49.200910 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:49.200887 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/08aea90a-8858-4443-a3cc-34e6d72a705c-ca-trust-extracted\") pod \"image-registry-d469874bc-bnc45\" (UID: \"08aea90a-8858-4443-a3cc-34e6d72a705c\") " pod="openshift-image-registry/image-registry-d469874bc-bnc45" Jul 22 02:37:49.201015 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:49.200925 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/08aea90a-8858-4443-a3cc-34e6d72a705c-registry-tls\") pod \"image-registry-d469874bc-bnc45\" (UID: \"08aea90a-8858-4443-a3cc-34e6d72a705c\") " pod="openshift-image-registry/image-registry-d469874bc-bnc45" Jul 22 02:37:49.201015 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:49.200954 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/08aea90a-8858-4443-a3cc-34e6d72a705c-installation-pull-secrets\") pod \"image-registry-d469874bc-bnc45\" (UID: \"08aea90a-8858-4443-a3cc-34e6d72a705c\") " pod="openshift-image-registry/image-registry-d469874bc-bnc45" Jul 22 02:37:49.201015 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:49.200973 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/26d80b89-26cd-4736-9a65-753db280e9fd-tmp-dir\") pod \"dns-default-trlg7\" (UID: \"26d80b89-26cd-4736-9a65-753db280e9fd\") " pod="openshift-dns/dns-default-trlg7" Jul 22 02:37:49.201355 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:49.201023 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wcd29\" (UniqueName: \"kubernetes.io/projected/26d80b89-26cd-4736-9a65-753db280e9fd-kube-api-access-wcd29\") pod \"dns-default-trlg7\" (UID: \"26d80b89-26cd-4736-9a65-753db280e9fd\") " pod="openshift-dns/dns-default-trlg7" Jul 22 02:37:49.201355 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:49.201082 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/08aea90a-8858-4443-a3cc-34e6d72a705c-image-registry-private-configuration\") pod \"image-registry-d469874bc-bnc45\" (UID: \"08aea90a-8858-4443-a3cc-34e6d72a705c\") " pod="openshift-image-registry/image-registry-d469874bc-bnc45" Jul 22 02:37:49.201355 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:49.201112 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/08aea90a-8858-4443-a3cc-34e6d72a705c-registry-certificates\") pod \"image-registry-d469874bc-bnc45\" (UID: \"08aea90a-8858-4443-a3cc-34e6d72a705c\") " pod="openshift-image-registry/image-registry-d469874bc-bnc45" Jul 22 02:37:49.201355 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:49.201131 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/08aea90a-8858-4443-a3cc-34e6d72a705c-trusted-ca\") pod \"image-registry-d469874bc-bnc45\" (UID: \"08aea90a-8858-4443-a3cc-34e6d72a705c\") " pod="openshift-image-registry/image-registry-d469874bc-bnc45" Jul 22 02:37:49.201355 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:49.201178 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/26d80b89-26cd-4736-9a65-753db280e9fd-config-volume\") pod \"dns-default-trlg7\" (UID: \"26d80b89-26cd-4736-9a65-753db280e9fd\") " pod="openshift-dns/dns-default-trlg7" Jul 22 02:37:49.201355 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:49.201208 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/26d80b89-26cd-4736-9a65-753db280e9fd-metrics-tls\") pod \"dns-default-trlg7\" (UID: \"26d80b89-26cd-4736-9a65-753db280e9fd\") " pod="openshift-dns/dns-default-trlg7" Jul 22 02:37:49.201355 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:49.201234 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/08aea90a-8858-4443-a3cc-34e6d72a705c-bound-sa-token\") pod \"image-registry-d469874bc-bnc45\" (UID: \"08aea90a-8858-4443-a3cc-34e6d72a705c\") " pod="openshift-image-registry/image-registry-d469874bc-bnc45" Jul 22 02:37:49.201355 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:49.201261 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nwdff\" (UniqueName: \"kubernetes.io/projected/08aea90a-8858-4443-a3cc-34e6d72a705c-kube-api-access-nwdff\") pod \"image-registry-d469874bc-bnc45\" (UID: \"08aea90a-8858-4443-a3cc-34e6d72a705c\") " pod="openshift-image-registry/image-registry-d469874bc-bnc45" Jul 22 02:37:49.302504 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:49.302417 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/08aea90a-8858-4443-a3cc-34e6d72a705c-bound-sa-token\") pod \"image-registry-d469874bc-bnc45\" (UID: \"08aea90a-8858-4443-a3cc-34e6d72a705c\") " pod="openshift-image-registry/image-registry-d469874bc-bnc45" Jul 22 02:37:49.302504 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:49.302452 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nwdff\" (UniqueName: \"kubernetes.io/projected/08aea90a-8858-4443-a3cc-34e6d72a705c-kube-api-access-nwdff\") pod \"image-registry-d469874bc-bnc45\" (UID: \"08aea90a-8858-4443-a3cc-34e6d72a705c\") " pod="openshift-image-registry/image-registry-d469874bc-bnc45" Jul 22 02:37:49.302504 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:49.302486 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j6njd\" (UniqueName: \"kubernetes.io/projected/2bb410f4-fbcd-43d0-bfa1-e0cf2eaf4289-kube-api-access-j6njd\") pod \"ingress-canary-8fzdj\" (UID: \"2bb410f4-fbcd-43d0-bfa1-e0cf2eaf4289\") " pod="openshift-ingress-canary/ingress-canary-8fzdj" Jul 22 02:37:49.302775 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:49.302524 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/08aea90a-8858-4443-a3cc-34e6d72a705c-ca-trust-extracted\") pod \"image-registry-d469874bc-bnc45\" (UID: \"08aea90a-8858-4443-a3cc-34e6d72a705c\") " pod="openshift-image-registry/image-registry-d469874bc-bnc45" Jul 22 02:37:49.302775 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:49.302546 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/08aea90a-8858-4443-a3cc-34e6d72a705c-registry-tls\") pod \"image-registry-d469874bc-bnc45\" (UID: \"08aea90a-8858-4443-a3cc-34e6d72a705c\") " pod="openshift-image-registry/image-registry-d469874bc-bnc45" Jul 22 02:37:49.302775 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:49.302585 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/08aea90a-8858-4443-a3cc-34e6d72a705c-installation-pull-secrets\") pod \"image-registry-d469874bc-bnc45\" (UID: \"08aea90a-8858-4443-a3cc-34e6d72a705c\") " pod="openshift-image-registry/image-registry-d469874bc-bnc45" Jul 22 02:37:49.302775 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:49.302601 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/26d80b89-26cd-4736-9a65-753db280e9fd-tmp-dir\") pod \"dns-default-trlg7\" (UID: \"26d80b89-26cd-4736-9a65-753db280e9fd\") " pod="openshift-dns/dns-default-trlg7" Jul 22 02:37:49.302775 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:49.302616 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wcd29\" (UniqueName: \"kubernetes.io/projected/26d80b89-26cd-4736-9a65-753db280e9fd-kube-api-access-wcd29\") pod \"dns-default-trlg7\" (UID: \"26d80b89-26cd-4736-9a65-753db280e9fd\") " pod="openshift-dns/dns-default-trlg7" Jul 22 02:37:49.302775 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:49.302677 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/08aea90a-8858-4443-a3cc-34e6d72a705c-image-registry-private-configuration\") pod \"image-registry-d469874bc-bnc45\" (UID: \"08aea90a-8858-4443-a3cc-34e6d72a705c\") " pod="openshift-image-registry/image-registry-d469874bc-bnc45" Jul 22 02:37:49.302775 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:49.302727 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/08aea90a-8858-4443-a3cc-34e6d72a705c-registry-certificates\") pod \"image-registry-d469874bc-bnc45\" (UID: \"08aea90a-8858-4443-a3cc-34e6d72a705c\") " pod="openshift-image-registry/image-registry-d469874bc-bnc45" Jul 22 02:37:49.302775 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:49.302754 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/08aea90a-8858-4443-a3cc-34e6d72a705c-trusted-ca\") pod \"image-registry-d469874bc-bnc45\" (UID: \"08aea90a-8858-4443-a3cc-34e6d72a705c\") " pod="openshift-image-registry/image-registry-d469874bc-bnc45" Jul 22 02:37:49.303131 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:49.302792 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2bb410f4-fbcd-43d0-bfa1-e0cf2eaf4289-cert\") pod \"ingress-canary-8fzdj\" (UID: \"2bb410f4-fbcd-43d0-bfa1-e0cf2eaf4289\") " pod="openshift-ingress-canary/ingress-canary-8fzdj" Jul 22 02:37:49.303131 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:49.302855 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/26d80b89-26cd-4736-9a65-753db280e9fd-config-volume\") pod \"dns-default-trlg7\" (UID: \"26d80b89-26cd-4736-9a65-753db280e9fd\") " pod="openshift-dns/dns-default-trlg7" Jul 22 02:37:49.303131 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:49.302891 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/26d80b89-26cd-4736-9a65-753db280e9fd-metrics-tls\") pod \"dns-default-trlg7\" (UID: \"26d80b89-26cd-4736-9a65-753db280e9fd\") " pod="openshift-dns/dns-default-trlg7" Jul 22 02:37:49.303131 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:49.302935 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/26d80b89-26cd-4736-9a65-753db280e9fd-tmp-dir\") pod \"dns-default-trlg7\" (UID: \"26d80b89-26cd-4736-9a65-753db280e9fd\") " pod="openshift-dns/dns-default-trlg7" Jul 22 02:37:49.303131 ip-10-0-133-31 kubenswrapper[2578]: E0722 02:37:49.302995 2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 22 02:37:49.303131 ip-10-0-133-31 kubenswrapper[2578]: E0722 02:37:49.303005 2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 22 02:37:49.303131 ip-10-0-133-31 kubenswrapper[2578]: E0722 02:37:49.303018 2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-d469874bc-bnc45: secret "image-registry-tls" not found Jul 22 02:37:49.303131 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:49.303024 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/08aea90a-8858-4443-a3cc-34e6d72a705c-ca-trust-extracted\") pod \"image-registry-d469874bc-bnc45\" (UID: \"08aea90a-8858-4443-a3cc-34e6d72a705c\") " pod="openshift-image-registry/image-registry-d469874bc-bnc45" Jul 22 02:37:49.303131 ip-10-0-133-31 kubenswrapper[2578]: E0722 02:37:49.303049 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/26d80b89-26cd-4736-9a65-753db280e9fd-metrics-tls podName:26d80b89-26cd-4736-9a65-753db280e9fd nodeName:}" failed. No retries permitted until 2026-07-22 02:37:49.803031663 +0000 UTC m=+36.122206959 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/26d80b89-26cd-4736-9a65-753db280e9fd-metrics-tls") pod "dns-default-trlg7" (UID: "26d80b89-26cd-4736-9a65-753db280e9fd") : secret "dns-default-metrics-tls" not found Jul 22 02:37:49.303131 ip-10-0-133-31 kubenswrapper[2578]: E0722 02:37:49.303067 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/08aea90a-8858-4443-a3cc-34e6d72a705c-registry-tls podName:08aea90a-8858-4443-a3cc-34e6d72a705c nodeName:}" failed. No retries permitted until 2026-07-22 02:37:49.803057553 +0000 UTC m=+36.122232837 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/08aea90a-8858-4443-a3cc-34e6d72a705c-registry-tls") pod "image-registry-d469874bc-bnc45" (UID: "08aea90a-8858-4443-a3cc-34e6d72a705c") : secret "image-registry-tls" not found Jul 22 02:37:49.303544 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:49.303240 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/08aea90a-8858-4443-a3cc-34e6d72a705c-registry-certificates\") pod \"image-registry-d469874bc-bnc45\" (UID: \"08aea90a-8858-4443-a3cc-34e6d72a705c\") " pod="openshift-image-registry/image-registry-d469874bc-bnc45" Jul 22 02:37:49.303688 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:49.303660 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/26d80b89-26cd-4736-9a65-753db280e9fd-config-volume\") pod \"dns-default-trlg7\" (UID: \"26d80b89-26cd-4736-9a65-753db280e9fd\") " pod="openshift-dns/dns-default-trlg7" Jul 22 02:37:49.303835 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:49.303810 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/08aea90a-8858-4443-a3cc-34e6d72a705c-trusted-ca\") pod \"image-registry-d469874bc-bnc45\" (UID: \"08aea90a-8858-4443-a3cc-34e6d72a705c\") " pod="openshift-image-registry/image-registry-d469874bc-bnc45" Jul 22 02:37:49.307666 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:49.307630 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/08aea90a-8858-4443-a3cc-34e6d72a705c-image-registry-private-configuration\") pod \"image-registry-d469874bc-bnc45\" (UID: \"08aea90a-8858-4443-a3cc-34e6d72a705c\") " pod="openshift-image-registry/image-registry-d469874bc-bnc45" Jul 22 02:37:49.307767 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:49.307706 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/08aea90a-8858-4443-a3cc-34e6d72a705c-installation-pull-secrets\") pod \"image-registry-d469874bc-bnc45\" (UID: \"08aea90a-8858-4443-a3cc-34e6d72a705c\") " pod="openshift-image-registry/image-registry-d469874bc-bnc45" Jul 22 02:37:49.313489 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:49.313466 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wcd29\" (UniqueName: \"kubernetes.io/projected/26d80b89-26cd-4736-9a65-753db280e9fd-kube-api-access-wcd29\") pod \"dns-default-trlg7\" (UID: \"26d80b89-26cd-4736-9a65-753db280e9fd\") " pod="openshift-dns/dns-default-trlg7" Jul 22 02:37:49.313702 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:49.313680 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/08aea90a-8858-4443-a3cc-34e6d72a705c-bound-sa-token\") pod \"image-registry-d469874bc-bnc45\" (UID: \"08aea90a-8858-4443-a3cc-34e6d72a705c\") " pod="openshift-image-registry/image-registry-d469874bc-bnc45" Jul 22 02:37:49.315130 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:49.315110 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nwdff\" (UniqueName: \"kubernetes.io/projected/08aea90a-8858-4443-a3cc-34e6d72a705c-kube-api-access-nwdff\") pod \"image-registry-d469874bc-bnc45\" (UID: \"08aea90a-8858-4443-a3cc-34e6d72a705c\") " pod="openshift-image-registry/image-registry-d469874bc-bnc45" Jul 22 02:37:49.403771 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:49.403735 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2bb410f4-fbcd-43d0-bfa1-e0cf2eaf4289-cert\") pod \"ingress-canary-8fzdj\" (UID: \"2bb410f4-fbcd-43d0-bfa1-e0cf2eaf4289\") " pod="openshift-ingress-canary/ingress-canary-8fzdj" Jul 22 02:37:49.403931 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:49.403825 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j6njd\" (UniqueName: \"kubernetes.io/projected/2bb410f4-fbcd-43d0-bfa1-e0cf2eaf4289-kube-api-access-j6njd\") pod \"ingress-canary-8fzdj\" (UID: \"2bb410f4-fbcd-43d0-bfa1-e0cf2eaf4289\") " pod="openshift-ingress-canary/ingress-canary-8fzdj" Jul 22 02:37:49.406339 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:49.406316 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2bb410f4-fbcd-43d0-bfa1-e0cf2eaf4289-cert\") pod \"ingress-canary-8fzdj\" (UID: \"2bb410f4-fbcd-43d0-bfa1-e0cf2eaf4289\") " pod="openshift-ingress-canary/ingress-canary-8fzdj" Jul 22 02:37:49.413173 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:49.413151 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j6njd\" (UniqueName: \"kubernetes.io/projected/2bb410f4-fbcd-43d0-bfa1-e0cf2eaf4289-kube-api-access-j6njd\") pod \"ingress-canary-8fzdj\" (UID: \"2bb410f4-fbcd-43d0-bfa1-e0cf2eaf4289\") " pod="openshift-ingress-canary/ingress-canary-8fzdj" Jul 22 02:37:49.444226 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:49.444201 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-8fzdj" Jul 22 02:37:49.806625 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:49.806588 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/26d80b89-26cd-4736-9a65-753db280e9fd-metrics-tls\") pod \"dns-default-trlg7\" (UID: \"26d80b89-26cd-4736-9a65-753db280e9fd\") " pod="openshift-dns/dns-default-trlg7" Jul 22 02:37:49.806812 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:49.806641 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/08aea90a-8858-4443-a3cc-34e6d72a705c-registry-tls\") pod \"image-registry-d469874bc-bnc45\" (UID: \"08aea90a-8858-4443-a3cc-34e6d72a705c\") " pod="openshift-image-registry/image-registry-d469874bc-bnc45" Jul 22 02:37:49.806812 ip-10-0-133-31 kubenswrapper[2578]: E0722 02:37:49.806731 2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 22 02:37:49.806917 ip-10-0-133-31 kubenswrapper[2578]: E0722 02:37:49.806815 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/26d80b89-26cd-4736-9a65-753db280e9fd-metrics-tls podName:26d80b89-26cd-4736-9a65-753db280e9fd nodeName:}" failed. No retries permitted until 2026-07-22 02:37:50.806794021 +0000 UTC m=+37.125969309 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/26d80b89-26cd-4736-9a65-753db280e9fd-metrics-tls") pod "dns-default-trlg7" (UID: "26d80b89-26cd-4736-9a65-753db280e9fd") : secret "dns-default-metrics-tls" not found Jul 22 02:37:49.806917 ip-10-0-133-31 kubenswrapper[2578]: E0722 02:37:49.806738 2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 22 02:37:49.806917 ip-10-0-133-31 kubenswrapper[2578]: E0722 02:37:49.806848 2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-d469874bc-bnc45: secret "image-registry-tls" not found Jul 22 02:37:49.806917 ip-10-0-133-31 kubenswrapper[2578]: E0722 02:37:49.806893 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/08aea90a-8858-4443-a3cc-34e6d72a705c-registry-tls podName:08aea90a-8858-4443-a3cc-34e6d72a705c nodeName:}" failed. No retries permitted until 2026-07-22 02:37:50.806881517 +0000 UTC m=+37.126056795 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/08aea90a-8858-4443-a3cc-34e6d72a705c-registry-tls") pod "image-registry-d469874bc-bnc45" (UID: "08aea90a-8858-4443-a3cc-34e6d72a705c") : secret "image-registry-tls" not found Jul 22 02:37:50.249772 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:50.249456 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-8fzdj"] Jul 22 02:37:50.254408 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:37:50.254384 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2bb410f4_fbcd_43d0_bfa1_e0cf2eaf4289.slice/crio-016e46b848d16ec29eaba3c00f92baaabdfa5bc9f991de1e411fe9760a987f50 WatchSource:0}: Error finding container 016e46b848d16ec29eaba3c00f92baaabdfa5bc9f991de1e411fe9760a987f50: Status 404 returned error can't find the container with id 016e46b848d16ec29eaba3c00f92baaabdfa5bc9f991de1e411fe9760a987f50 Jul 22 02:37:50.290700 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:50.290650 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-8fzdj" event={"ID":"2bb410f4-fbcd-43d0-bfa1-e0cf2eaf4289","Type":"ContainerStarted","Data":"016e46b848d16ec29eaba3c00f92baaabdfa5bc9f991de1e411fe9760a987f50"} Jul 22 02:37:50.293329 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:50.293300 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-t9vtn" event={"ID":"34c0277e-a51b-4c45-ae71-9f339d53d946","Type":"ContainerStarted","Data":"10972f96751fd7f4c80144b18e381a64875d6c0cc38ea99a3958842577c9bd81"} Jul 22 02:37:50.812272 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:50.812233 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/08aea90a-8858-4443-a3cc-34e6d72a705c-registry-tls\") pod \"image-registry-d469874bc-bnc45\" (UID: \"08aea90a-8858-4443-a3cc-34e6d72a705c\") " pod="openshift-image-registry/image-registry-d469874bc-bnc45" Jul 22 02:37:50.812498 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:50.812344 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/26d80b89-26cd-4736-9a65-753db280e9fd-metrics-tls\") pod \"dns-default-trlg7\" (UID: \"26d80b89-26cd-4736-9a65-753db280e9fd\") " pod="openshift-dns/dns-default-trlg7" Jul 22 02:37:50.812498 ip-10-0-133-31 kubenswrapper[2578]: E0722 02:37:50.812391 2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 22 02:37:50.812498 ip-10-0-133-31 kubenswrapper[2578]: E0722 02:37:50.812411 2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-d469874bc-bnc45: secret "image-registry-tls" not found Jul 22 02:37:50.812498 ip-10-0-133-31 kubenswrapper[2578]: E0722 02:37:50.812453 2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 22 02:37:50.812498 ip-10-0-133-31 kubenswrapper[2578]: E0722 02:37:50.812477 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/08aea90a-8858-4443-a3cc-34e6d72a705c-registry-tls podName:08aea90a-8858-4443-a3cc-34e6d72a705c nodeName:}" failed. No retries permitted until 2026-07-22 02:37:52.812459729 +0000 UTC m=+39.131635031 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/08aea90a-8858-4443-a3cc-34e6d72a705c-registry-tls") pod "image-registry-d469874bc-bnc45" (UID: "08aea90a-8858-4443-a3cc-34e6d72a705c") : secret "image-registry-tls" not found Jul 22 02:37:50.812498 ip-10-0-133-31 kubenswrapper[2578]: E0722 02:37:50.812495 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/26d80b89-26cd-4736-9a65-753db280e9fd-metrics-tls podName:26d80b89-26cd-4736-9a65-753db280e9fd nodeName:}" failed. No retries permitted until 2026-07-22 02:37:52.812484239 +0000 UTC m=+39.131659518 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/26d80b89-26cd-4736-9a65-753db280e9fd-metrics-tls") pod "dns-default-trlg7" (UID: "26d80b89-26cd-4736-9a65-753db280e9fd") : secret "dns-default-metrics-tls" not found Jul 22 02:37:51.298484 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:51.298452 2578 generic.go:356] "Generic (PLEG): container finished" podID="34c0277e-a51b-4c45-ae71-9f339d53d946" containerID="10972f96751fd7f4c80144b18e381a64875d6c0cc38ea99a3958842577c9bd81" exitCode=0 Jul 22 02:37:51.298950 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:51.298519 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-t9vtn" event={"ID":"34c0277e-a51b-4c45-ae71-9f339d53d946","Type":"ContainerDied","Data":"10972f96751fd7f4c80144b18e381a64875d6c0cc38ea99a3958842577c9bd81"} Jul 22 02:37:52.304438 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:52.304398 2578 generic.go:356] "Generic (PLEG): container finished" podID="34c0277e-a51b-4c45-ae71-9f339d53d946" containerID="64afb6a236bedd27b27d2066db7300bfa371c670afb680082a317ea9287760a3" exitCode=0 Jul 22 02:37:52.304922 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:52.304473 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-t9vtn" event={"ID":"34c0277e-a51b-4c45-ae71-9f339d53d946","Type":"ContainerDied","Data":"64afb6a236bedd27b27d2066db7300bfa371c670afb680082a317ea9287760a3"} Jul 22 02:37:52.832376 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:52.832354 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/08aea90a-8858-4443-a3cc-34e6d72a705c-registry-tls\") pod \"image-registry-d469874bc-bnc45\" (UID: \"08aea90a-8858-4443-a3cc-34e6d72a705c\") " pod="openshift-image-registry/image-registry-d469874bc-bnc45" Jul 22 02:37:52.832481 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:52.832423 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/26d80b89-26cd-4736-9a65-753db280e9fd-metrics-tls\") pod \"dns-default-trlg7\" (UID: \"26d80b89-26cd-4736-9a65-753db280e9fd\") " pod="openshift-dns/dns-default-trlg7" Jul 22 02:37:52.832519 ip-10-0-133-31 kubenswrapper[2578]: E0722 02:37:52.832496 2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 22 02:37:52.832554 ip-10-0-133-31 kubenswrapper[2578]: E0722 02:37:52.832542 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/26d80b89-26cd-4736-9a65-753db280e9fd-metrics-tls podName:26d80b89-26cd-4736-9a65-753db280e9fd nodeName:}" failed. No retries permitted until 2026-07-22 02:37:56.832529364 +0000 UTC m=+43.151704642 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/26d80b89-26cd-4736-9a65-753db280e9fd-metrics-tls") pod "dns-default-trlg7" (UID: "26d80b89-26cd-4736-9a65-753db280e9fd") : secret "dns-default-metrics-tls" not found Jul 22 02:37:52.832613 ip-10-0-133-31 kubenswrapper[2578]: E0722 02:37:52.832498 2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 22 02:37:52.832613 ip-10-0-133-31 kubenswrapper[2578]: E0722 02:37:52.832591 2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-d469874bc-bnc45: secret "image-registry-tls" not found Jul 22 02:37:52.832667 ip-10-0-133-31 kubenswrapper[2578]: E0722 02:37:52.832642 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/08aea90a-8858-4443-a3cc-34e6d72a705c-registry-tls podName:08aea90a-8858-4443-a3cc-34e6d72a705c nodeName:}" failed. No retries permitted until 2026-07-22 02:37:56.832627265 +0000 UTC m=+43.151802567 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/08aea90a-8858-4443-a3cc-34e6d72a705c-registry-tls") pod "image-registry-d469874bc-bnc45" (UID: "08aea90a-8858-4443-a3cc-34e6d72a705c") : secret "image-registry-tls" not found Jul 22 02:37:53.308634 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:53.308413 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-8fzdj" event={"ID":"2bb410f4-fbcd-43d0-bfa1-e0cf2eaf4289","Type":"ContainerStarted","Data":"9333eaf1b322815cfa9037ec5a4b422e5ebbc7634fc49f598d0fc83a8cb6b81f"} Jul 22 02:37:53.311919 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:53.311888 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-t9vtn" event={"ID":"34c0277e-a51b-4c45-ae71-9f339d53d946","Type":"ContainerStarted","Data":"d7986d4d5b3a574745b672900f3dd49f5cfd069a9185ef54f738af1804a8966d"} Jul 22 02:37:53.353544 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:53.353503 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-8fzdj" podStartSLOduration=1.7586447330000001 podStartE2EDuration="4.353489461s" totalImagesPullingTime="2.594844728s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 02:37:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:37:53.353357981 +0000 UTC m=+39.672533291" watchObservedRunningTime="2026-07-22 02:37:53.353489461 +0000 UTC m=+39.672664764" Jul 22 02:37:53.413987 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:53.413944 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-t9vtn" podStartSLOduration=10.368313914 podStartE2EDuration="39.413930256s" totalImagesPullingTime="29.045616342s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 02:37:14 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:37:53.413277382 +0000 UTC m=+39.732452683" watchObservedRunningTime="2026-07-22 02:37:53.413930256 +0000 UTC m=+39.733105556" Jul 22 02:37:56.857376 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:56.857340 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/26d80b89-26cd-4736-9a65-753db280e9fd-metrics-tls\") pod \"dns-default-trlg7\" (UID: \"26d80b89-26cd-4736-9a65-753db280e9fd\") " pod="openshift-dns/dns-default-trlg7" Jul 22 02:37:56.857750 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:56.857386 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/08aea90a-8858-4443-a3cc-34e6d72a705c-registry-tls\") pod \"image-registry-d469874bc-bnc45\" (UID: \"08aea90a-8858-4443-a3cc-34e6d72a705c\") " pod="openshift-image-registry/image-registry-d469874bc-bnc45" Jul 22 02:37:56.857750 ip-10-0-133-31 kubenswrapper[2578]: E0722 02:37:56.857482 2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 22 02:37:56.857750 ip-10-0-133-31 kubenswrapper[2578]: E0722 02:37:56.857481 2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 22 02:37:56.857750 ip-10-0-133-31 kubenswrapper[2578]: E0722 02:37:56.857492 2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-d469874bc-bnc45: secret "image-registry-tls" not found Jul 22 02:37:56.857750 ip-10-0-133-31 kubenswrapper[2578]: E0722 02:37:56.857539 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/08aea90a-8858-4443-a3cc-34e6d72a705c-registry-tls podName:08aea90a-8858-4443-a3cc-34e6d72a705c nodeName:}" failed. No retries permitted until 2026-07-22 02:38:04.857527757 +0000 UTC m=+51.176703035 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/08aea90a-8858-4443-a3cc-34e6d72a705c-registry-tls") pod "image-registry-d469874bc-bnc45" (UID: "08aea90a-8858-4443-a3cc-34e6d72a705c") : secret "image-registry-tls" not found Jul 22 02:37:56.857750 ip-10-0-133-31 kubenswrapper[2578]: E0722 02:37:56.857550 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/26d80b89-26cd-4736-9a65-753db280e9fd-metrics-tls podName:26d80b89-26cd-4736-9a65-753db280e9fd nodeName:}" failed. No retries permitted until 2026-07-22 02:38:04.857545029 +0000 UTC m=+51.176720307 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/26d80b89-26cd-4736-9a65-753db280e9fd-metrics-tls") pod "dns-default-trlg7" (UID: "26d80b89-26cd-4736-9a65-753db280e9fd") : secret "dns-default-metrics-tls" not found Jul 22 02:37:58.431116 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:58.431089 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-gvwx6"] Jul 22 02:37:58.464406 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:58.464378 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-gvwx6"] Jul 22 02:37:58.464519 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:58.464492 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-gvwx6" Jul 22 02:37:58.467048 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:58.467022 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 22 02:37:58.467163 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:58.467151 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 22 02:37:58.468183 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:58.468165 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-5gmrt\"" Jul 22 02:37:58.468289 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:58.468210 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 22 02:37:58.468469 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:58.468455 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 22 02:37:58.568098 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:58.568075 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a0f7cb8b-7cc6-441e-accc-8c2061b4baab-crio-socket\") pod \"insights-runtime-extractor-gvwx6\" (UID: \"a0f7cb8b-7cc6-441e-accc-8c2061b4baab\") " pod="openshift-insights/insights-runtime-extractor-gvwx6" Jul 22 02:37:58.568193 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:58.568100 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a0f7cb8b-7cc6-441e-accc-8c2061b4baab-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gvwx6\" (UID: \"a0f7cb8b-7cc6-441e-accc-8c2061b4baab\") " pod="openshift-insights/insights-runtime-extractor-gvwx6" Jul 22 02:37:58.568193 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:58.568119 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/a0f7cb8b-7cc6-441e-accc-8c2061b4baab-data-volume\") pod \"insights-runtime-extractor-gvwx6\" (UID: \"a0f7cb8b-7cc6-441e-accc-8c2061b4baab\") " pod="openshift-insights/insights-runtime-extractor-gvwx6" Jul 22 02:37:58.568193 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:58.568187 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a0f7cb8b-7cc6-441e-accc-8c2061b4baab-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gvwx6\" (UID: \"a0f7cb8b-7cc6-441e-accc-8c2061b4baab\") " pod="openshift-insights/insights-runtime-extractor-gvwx6" Jul 22 02:37:58.568293 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:58.568203 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7ffpd\" (UniqueName: \"kubernetes.io/projected/a0f7cb8b-7cc6-441e-accc-8c2061b4baab-kube-api-access-7ffpd\") pod \"insights-runtime-extractor-gvwx6\" (UID: \"a0f7cb8b-7cc6-441e-accc-8c2061b4baab\") " pod="openshift-insights/insights-runtime-extractor-gvwx6" Jul 22 02:37:58.669412 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:58.669385 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a0f7cb8b-7cc6-441e-accc-8c2061b4baab-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gvwx6\" (UID: \"a0f7cb8b-7cc6-441e-accc-8c2061b4baab\") " pod="openshift-insights/insights-runtime-extractor-gvwx6" Jul 22 02:37:58.669495 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:58.669418 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7ffpd\" (UniqueName: \"kubernetes.io/projected/a0f7cb8b-7cc6-441e-accc-8c2061b4baab-kube-api-access-7ffpd\") pod \"insights-runtime-extractor-gvwx6\" (UID: \"a0f7cb8b-7cc6-441e-accc-8c2061b4baab\") " pod="openshift-insights/insights-runtime-extractor-gvwx6" Jul 22 02:37:58.669495 ip-10-0-133-31 kubenswrapper[2578]: E0722 02:37:58.669472 2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 22 02:37:58.669597 ip-10-0-133-31 kubenswrapper[2578]: E0722 02:37:58.669545 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a0f7cb8b-7cc6-441e-accc-8c2061b4baab-kube-rbac-proxy-cm podName:a0f7cb8b-7cc6-441e-accc-8c2061b4baab nodeName:}" failed. No retries permitted until 2026-07-22 02:37:59.169526497 +0000 UTC m=+45.488701796 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/a0f7cb8b-7cc6-441e-accc-8c2061b4baab-kube-rbac-proxy-cm") pod "insights-runtime-extractor-gvwx6" (UID: "a0f7cb8b-7cc6-441e-accc-8c2061b4baab") : configmap "kube-rbac-proxy" not found Jul 22 02:37:58.669654 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:58.669619 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a0f7cb8b-7cc6-441e-accc-8c2061b4baab-crio-socket\") pod \"insights-runtime-extractor-gvwx6\" (UID: \"a0f7cb8b-7cc6-441e-accc-8c2061b4baab\") " pod="openshift-insights/insights-runtime-extractor-gvwx6" Jul 22 02:37:58.669654 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:58.669646 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a0f7cb8b-7cc6-441e-accc-8c2061b4baab-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gvwx6\" (UID: \"a0f7cb8b-7cc6-441e-accc-8c2061b4baab\") " pod="openshift-insights/insights-runtime-extractor-gvwx6" Jul 22 02:37:58.669742 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:58.669672 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/a0f7cb8b-7cc6-441e-accc-8c2061b4baab-data-volume\") pod \"insights-runtime-extractor-gvwx6\" (UID: \"a0f7cb8b-7cc6-441e-accc-8c2061b4baab\") " pod="openshift-insights/insights-runtime-extractor-gvwx6" Jul 22 02:37:58.669781 ip-10-0-133-31 kubenswrapper[2578]: E0722 02:37:58.669773 2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 22 02:37:58.669837 ip-10-0-133-31 kubenswrapper[2578]: E0722 02:37:58.669825 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a0f7cb8b-7cc6-441e-accc-8c2061b4baab-insights-runtime-extractor-tls podName:a0f7cb8b-7cc6-441e-accc-8c2061b4baab nodeName:}" failed. No retries permitted until 2026-07-22 02:37:59.169811267 +0000 UTC m=+45.488986545 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a0f7cb8b-7cc6-441e-accc-8c2061b4baab-insights-runtime-extractor-tls") pod "insights-runtime-extractor-gvwx6" (UID: "a0f7cb8b-7cc6-441e-accc-8c2061b4baab") : secret "insights-runtime-extractor-tls" not found Jul 22 02:37:58.669894 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:58.669872 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a0f7cb8b-7cc6-441e-accc-8c2061b4baab-crio-socket\") pod \"insights-runtime-extractor-gvwx6\" (UID: \"a0f7cb8b-7cc6-441e-accc-8c2061b4baab\") " pod="openshift-insights/insights-runtime-extractor-gvwx6" Jul 22 02:37:58.669989 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:58.669976 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/a0f7cb8b-7cc6-441e-accc-8c2061b4baab-data-volume\") pod \"insights-runtime-extractor-gvwx6\" (UID: \"a0f7cb8b-7cc6-441e-accc-8c2061b4baab\") " pod="openshift-insights/insights-runtime-extractor-gvwx6" Jul 22 02:37:58.690346 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:58.690291 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7ffpd\" (UniqueName: \"kubernetes.io/projected/a0f7cb8b-7cc6-441e-accc-8c2061b4baab-kube-api-access-7ffpd\") pod \"insights-runtime-extractor-gvwx6\" (UID: \"a0f7cb8b-7cc6-441e-accc-8c2061b4baab\") " pod="openshift-insights/insights-runtime-extractor-gvwx6" Jul 22 02:37:59.172807 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:59.172779 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a0f7cb8b-7cc6-441e-accc-8c2061b4baab-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gvwx6\" (UID: \"a0f7cb8b-7cc6-441e-accc-8c2061b4baab\") " pod="openshift-insights/insights-runtime-extractor-gvwx6" Jul 22 02:37:59.172924 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:37:59.172831 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a0f7cb8b-7cc6-441e-accc-8c2061b4baab-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gvwx6\" (UID: \"a0f7cb8b-7cc6-441e-accc-8c2061b4baab\") " pod="openshift-insights/insights-runtime-extractor-gvwx6" Jul 22 02:37:59.172924 ip-10-0-133-31 kubenswrapper[2578]: E0722 02:37:59.172888 2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 22 02:37:59.172996 ip-10-0-133-31 kubenswrapper[2578]: E0722 02:37:59.172929 2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 22 02:37:59.172996 ip-10-0-133-31 kubenswrapper[2578]: E0722 02:37:59.172940 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a0f7cb8b-7cc6-441e-accc-8c2061b4baab-kube-rbac-proxy-cm podName:a0f7cb8b-7cc6-441e-accc-8c2061b4baab nodeName:}" failed. No retries permitted until 2026-07-22 02:38:00.172926262 +0000 UTC m=+46.492101540 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/a0f7cb8b-7cc6-441e-accc-8c2061b4baab-kube-rbac-proxy-cm") pod "insights-runtime-extractor-gvwx6" (UID: "a0f7cb8b-7cc6-441e-accc-8c2061b4baab") : configmap "kube-rbac-proxy" not found Jul 22 02:37:59.172996 ip-10-0-133-31 kubenswrapper[2578]: E0722 02:37:59.172979 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a0f7cb8b-7cc6-441e-accc-8c2061b4baab-insights-runtime-extractor-tls podName:a0f7cb8b-7cc6-441e-accc-8c2061b4baab nodeName:}" failed. No retries permitted until 2026-07-22 02:38:00.172968802 +0000 UTC m=+46.492144080 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a0f7cb8b-7cc6-441e-accc-8c2061b4baab-insights-runtime-extractor-tls") pod "insights-runtime-extractor-gvwx6" (UID: "a0f7cb8b-7cc6-441e-accc-8c2061b4baab") : secret "insights-runtime-extractor-tls" not found Jul 22 02:38:00.179337 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:00.179311 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a0f7cb8b-7cc6-441e-accc-8c2061b4baab-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gvwx6\" (UID: \"a0f7cb8b-7cc6-441e-accc-8c2061b4baab\") " pod="openshift-insights/insights-runtime-extractor-gvwx6" Jul 22 02:38:00.179776 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:00.179366 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a0f7cb8b-7cc6-441e-accc-8c2061b4baab-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gvwx6\" (UID: \"a0f7cb8b-7cc6-441e-accc-8c2061b4baab\") " pod="openshift-insights/insights-runtime-extractor-gvwx6" Jul 22 02:38:00.179776 ip-10-0-133-31 kubenswrapper[2578]: E0722 02:38:00.179435 2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 22 02:38:00.179776 ip-10-0-133-31 kubenswrapper[2578]: E0722 02:38:00.179451 2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 22 02:38:00.179776 ip-10-0-133-31 kubenswrapper[2578]: E0722 02:38:00.179498 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a0f7cb8b-7cc6-441e-accc-8c2061b4baab-insights-runtime-extractor-tls podName:a0f7cb8b-7cc6-441e-accc-8c2061b4baab nodeName:}" failed. No retries permitted until 2026-07-22 02:38:02.17948356 +0000 UTC m=+48.498658839 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a0f7cb8b-7cc6-441e-accc-8c2061b4baab-insights-runtime-extractor-tls") pod "insights-runtime-extractor-gvwx6" (UID: "a0f7cb8b-7cc6-441e-accc-8c2061b4baab") : secret "insights-runtime-extractor-tls" not found Jul 22 02:38:00.179776 ip-10-0-133-31 kubenswrapper[2578]: E0722 02:38:00.179512 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a0f7cb8b-7cc6-441e-accc-8c2061b4baab-kube-rbac-proxy-cm podName:a0f7cb8b-7cc6-441e-accc-8c2061b4baab nodeName:}" failed. No retries permitted until 2026-07-22 02:38:02.179505083 +0000 UTC m=+48.498680361 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/a0f7cb8b-7cc6-441e-accc-8c2061b4baab-kube-rbac-proxy-cm") pod "insights-runtime-extractor-gvwx6" (UID: "a0f7cb8b-7cc6-441e-accc-8c2061b4baab") : configmap "kube-rbac-proxy" not found Jul 22 02:38:00.205769 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:00.205754 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-9dhrs_dea9d10b-9320-471c-9efd-544fabc16854/dns-node-resolver/0.log" Jul 22 02:38:00.229589 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:00.229550 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-7bc7489cb5-gxllf"] Jul 22 02:38:00.264010 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:00.263987 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-7bc7489cb5-gxllf" Jul 22 02:38:00.264010 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:00.264006 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-7bc7489cb5-gxllf"] Jul 22 02:38:00.267144 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:00.267128 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Jul 22 02:38:00.267245 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:00.267192 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Jul 22 02:38:00.267312 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:00.267276 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-58nt6\"" Jul 22 02:38:00.337476 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:00.337456 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-58598695f7-g9dcn"] Jul 22 02:38:00.359665 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:00.359645 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-58598695f7-g9dcn"] Jul 22 02:38:00.359744 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:00.359718 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-58598695f7-g9dcn" Jul 22 02:38:00.362382 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:00.362367 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Jul 22 02:38:00.362465 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:00.362418 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-dqkpx\"" Jul 22 02:38:00.362616 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:00.362598 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Jul 22 02:38:00.362714 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:00.362671 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Jul 22 02:38:00.362926 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:00.362913 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Jul 22 02:38:00.380534 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:00.380515 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/35b750f1-f674-4130-b170-8adeb7a3fe91-nginx-conf\") pod \"networking-console-plugin-7bc7489cb5-gxllf\" (UID: \"35b750f1-f674-4130-b170-8adeb7a3fe91\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-gxllf" Jul 22 02:38:00.380639 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:00.380577 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/35b750f1-f674-4130-b170-8adeb7a3fe91-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-gxllf\" (UID: \"35b750f1-f674-4130-b170-8adeb7a3fe91\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-gxllf" Jul 22 02:38:00.480911 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:00.480866 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fndz4\" (UniqueName: \"kubernetes.io/projected/95951937-4671-40dd-a16a-aa01fb02023e-kube-api-access-fndz4\") pod \"service-ca-58598695f7-g9dcn\" (UID: \"95951937-4671-40dd-a16a-aa01fb02023e\") " pod="openshift-service-ca/service-ca-58598695f7-g9dcn" Jul 22 02:38:00.480995 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:00.480911 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/95951937-4671-40dd-a16a-aa01fb02023e-signing-key\") pod \"service-ca-58598695f7-g9dcn\" (UID: \"95951937-4671-40dd-a16a-aa01fb02023e\") " pod="openshift-service-ca/service-ca-58598695f7-g9dcn" Jul 22 02:38:00.480995 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:00.480926 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/95951937-4671-40dd-a16a-aa01fb02023e-signing-cabundle\") pod \"service-ca-58598695f7-g9dcn\" (UID: \"95951937-4671-40dd-a16a-aa01fb02023e\") " pod="openshift-service-ca/service-ca-58598695f7-g9dcn" Jul 22 02:38:00.480995 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:00.480953 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/35b750f1-f674-4130-b170-8adeb7a3fe91-nginx-conf\") pod \"networking-console-plugin-7bc7489cb5-gxllf\" (UID: \"35b750f1-f674-4130-b170-8adeb7a3fe91\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-gxllf" Jul 22 02:38:00.481107 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:00.481046 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/35b750f1-f674-4130-b170-8adeb7a3fe91-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-gxllf\" (UID: \"35b750f1-f674-4130-b170-8adeb7a3fe91\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-gxllf" Jul 22 02:38:00.481165 ip-10-0-133-31 kubenswrapper[2578]: E0722 02:38:00.481154 2578 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 22 02:38:00.481211 ip-10-0-133-31 kubenswrapper[2578]: E0722 02:38:00.481202 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/35b750f1-f674-4130-b170-8adeb7a3fe91-networking-console-plugin-cert podName:35b750f1-f674-4130-b170-8adeb7a3fe91 nodeName:}" failed. No retries permitted until 2026-07-22 02:38:00.981191737 +0000 UTC m=+47.300367028 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/35b750f1-f674-4130-b170-8adeb7a3fe91-networking-console-plugin-cert") pod "networking-console-plugin-7bc7489cb5-gxllf" (UID: "35b750f1-f674-4130-b170-8adeb7a3fe91") : secret "networking-console-plugin-cert" not found Jul 22 02:38:00.481408 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:00.481392 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/35b750f1-f674-4130-b170-8adeb7a3fe91-nginx-conf\") pod \"networking-console-plugin-7bc7489cb5-gxllf\" (UID: \"35b750f1-f674-4130-b170-8adeb7a3fe91\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-gxllf" Jul 22 02:38:00.582306 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:00.582285 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fndz4\" (UniqueName: \"kubernetes.io/projected/95951937-4671-40dd-a16a-aa01fb02023e-kube-api-access-fndz4\") pod \"service-ca-58598695f7-g9dcn\" (UID: \"95951937-4671-40dd-a16a-aa01fb02023e\") " pod="openshift-service-ca/service-ca-58598695f7-g9dcn" Jul 22 02:38:00.582368 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:00.582327 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/95951937-4671-40dd-a16a-aa01fb02023e-signing-key\") pod \"service-ca-58598695f7-g9dcn\" (UID: \"95951937-4671-40dd-a16a-aa01fb02023e\") " pod="openshift-service-ca/service-ca-58598695f7-g9dcn" Jul 22 02:38:00.582368 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:00.582343 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/95951937-4671-40dd-a16a-aa01fb02023e-signing-cabundle\") pod \"service-ca-58598695f7-g9dcn\" (UID: \"95951937-4671-40dd-a16a-aa01fb02023e\") " pod="openshift-service-ca/service-ca-58598695f7-g9dcn" Jul 22 02:38:00.583381 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:00.583353 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/95951937-4671-40dd-a16a-aa01fb02023e-signing-cabundle\") pod \"service-ca-58598695f7-g9dcn\" (UID: \"95951937-4671-40dd-a16a-aa01fb02023e\") " pod="openshift-service-ca/service-ca-58598695f7-g9dcn" Jul 22 02:38:00.586665 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:00.586645 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/95951937-4671-40dd-a16a-aa01fb02023e-signing-key\") pod \"service-ca-58598695f7-g9dcn\" (UID: \"95951937-4671-40dd-a16a-aa01fb02023e\") " pod="openshift-service-ca/service-ca-58598695f7-g9dcn" Jul 22 02:38:00.592757 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:00.592738 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fndz4\" (UniqueName: \"kubernetes.io/projected/95951937-4671-40dd-a16a-aa01fb02023e-kube-api-access-fndz4\") pod \"service-ca-58598695f7-g9dcn\" (UID: \"95951937-4671-40dd-a16a-aa01fb02023e\") " pod="openshift-service-ca/service-ca-58598695f7-g9dcn" Jul 22 02:38:00.666987 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:00.666967 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-58598695f7-g9dcn" Jul 22 02:38:00.815670 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:00.815649 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-58598695f7-g9dcn"] Jul 22 02:38:00.818038 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:38:00.818009 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95951937_4671_40dd_a16a_aa01fb02023e.slice/crio-db06b4596f69eefe47beb29863eba83b646ff4ac3db38b822abe965c65f007d3 WatchSource:0}: Error finding container db06b4596f69eefe47beb29863eba83b646ff4ac3db38b822abe965c65f007d3: Status 404 returned error can't find the container with id db06b4596f69eefe47beb29863eba83b646ff4ac3db38b822abe965c65f007d3 Jul 22 02:38:00.985685 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:00.985664 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/35b750f1-f674-4130-b170-8adeb7a3fe91-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-gxllf\" (UID: \"35b750f1-f674-4130-b170-8adeb7a3fe91\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-gxllf" Jul 22 02:38:00.985826 ip-10-0-133-31 kubenswrapper[2578]: E0722 02:38:00.985810 2578 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 22 02:38:00.985888 ip-10-0-133-31 kubenswrapper[2578]: E0722 02:38:00.985879 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/35b750f1-f674-4130-b170-8adeb7a3fe91-networking-console-plugin-cert podName:35b750f1-f674-4130-b170-8adeb7a3fe91 nodeName:}" failed. No retries permitted until 2026-07-22 02:38:01.985864536 +0000 UTC m=+48.305039814 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/35b750f1-f674-4130-b170-8adeb7a3fe91-networking-console-plugin-cert") pod "networking-console-plugin-7bc7489cb5-gxllf" (UID: "35b750f1-f674-4130-b170-8adeb7a3fe91") : secret "networking-console-plugin-cert" not found Jul 22 02:38:01.329784 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:01.329752 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-58598695f7-g9dcn" event={"ID":"95951937-4671-40dd-a16a-aa01fb02023e","Type":"ContainerStarted","Data":"db06b4596f69eefe47beb29863eba83b646ff4ac3db38b822abe965c65f007d3"} Jul 22 02:38:01.407812 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:01.407789 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-8kmfc_d7953894-cdfc-48a3-afe8-89503dfc3c57/node-ca/0.log" Jul 22 02:38:01.993690 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:01.993653 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/35b750f1-f674-4130-b170-8adeb7a3fe91-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-gxllf\" (UID: \"35b750f1-f674-4130-b170-8adeb7a3fe91\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-gxllf" Jul 22 02:38:01.993858 ip-10-0-133-31 kubenswrapper[2578]: E0722 02:38:01.993783 2578 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 22 02:38:01.993911 ip-10-0-133-31 kubenswrapper[2578]: E0722 02:38:01.993857 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/35b750f1-f674-4130-b170-8adeb7a3fe91-networking-console-plugin-cert podName:35b750f1-f674-4130-b170-8adeb7a3fe91 nodeName:}" failed. No retries permitted until 2026-07-22 02:38:03.993837457 +0000 UTC m=+50.313012798 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/35b750f1-f674-4130-b170-8adeb7a3fe91-networking-console-plugin-cert") pod "networking-console-plugin-7bc7489cb5-gxllf" (UID: "35b750f1-f674-4130-b170-8adeb7a3fe91") : secret "networking-console-plugin-cert" not found Jul 22 02:38:02.196000 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:02.195967 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a0f7cb8b-7cc6-441e-accc-8c2061b4baab-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gvwx6\" (UID: \"a0f7cb8b-7cc6-441e-accc-8c2061b4baab\") " pod="openshift-insights/insights-runtime-extractor-gvwx6" Jul 22 02:38:02.196170 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:02.196041 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a0f7cb8b-7cc6-441e-accc-8c2061b4baab-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gvwx6\" (UID: \"a0f7cb8b-7cc6-441e-accc-8c2061b4baab\") " pod="openshift-insights/insights-runtime-extractor-gvwx6" Jul 22 02:38:02.196170 ip-10-0-133-31 kubenswrapper[2578]: E0722 02:38:02.196109 2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 22 02:38:02.196170 ip-10-0-133-31 kubenswrapper[2578]: E0722 02:38:02.196130 2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 22 02:38:02.196292 ip-10-0-133-31 kubenswrapper[2578]: E0722 02:38:02.196180 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a0f7cb8b-7cc6-441e-accc-8c2061b4baab-kube-rbac-proxy-cm podName:a0f7cb8b-7cc6-441e-accc-8c2061b4baab nodeName:}" failed. No retries permitted until 2026-07-22 02:38:06.196163981 +0000 UTC m=+52.515339263 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/a0f7cb8b-7cc6-441e-accc-8c2061b4baab-kube-rbac-proxy-cm") pod "insights-runtime-extractor-gvwx6" (UID: "a0f7cb8b-7cc6-441e-accc-8c2061b4baab") : configmap "kube-rbac-proxy" not found Jul 22 02:38:02.196292 ip-10-0-133-31 kubenswrapper[2578]: E0722 02:38:02.196193 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a0f7cb8b-7cc6-441e-accc-8c2061b4baab-insights-runtime-extractor-tls podName:a0f7cb8b-7cc6-441e-accc-8c2061b4baab nodeName:}" failed. No retries permitted until 2026-07-22 02:38:06.196187391 +0000 UTC m=+52.515362670 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a0f7cb8b-7cc6-441e-accc-8c2061b4baab-insights-runtime-extractor-tls") pod "insights-runtime-extractor-gvwx6" (UID: "a0f7cb8b-7cc6-441e-accc-8c2061b4baab") : secret "insights-runtime-extractor-tls" not found Jul 22 02:38:02.413885 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:02.413852 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-8fzdj_2bb410f4-fbcd-43d0-bfa1-e0cf2eaf4289/serve-healthcheck-canary/0.log" Jul 22 02:38:03.336734 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:03.336701 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-58598695f7-g9dcn" event={"ID":"95951937-4671-40dd-a16a-aa01fb02023e","Type":"ContainerStarted","Data":"e25e92b8a31ac54f9842769ddd965797c5f2a00bfb3c7d38038acfcdf5e61179"} Jul 22 02:38:03.354300 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:03.354258 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-58598695f7-g9dcn" podStartSLOduration=1.139778174 podStartE2EDuration="3.354243917s" totalImagesPullingTime="2.214465743s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 02:38:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:38:03.35265541 +0000 UTC m=+49.671830703" watchObservedRunningTime="2026-07-22 02:38:03.354243917 +0000 UTC m=+49.673419216" Jul 22 02:38:04.014112 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:04.014073 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/35b750f1-f674-4130-b170-8adeb7a3fe91-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-gxllf\" (UID: \"35b750f1-f674-4130-b170-8adeb7a3fe91\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-gxllf" Jul 22 02:38:04.014588 ip-10-0-133-31 kubenswrapper[2578]: E0722 02:38:04.014289 2578 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 22 02:38:04.014588 ip-10-0-133-31 kubenswrapper[2578]: E0722 02:38:04.014350 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/35b750f1-f674-4130-b170-8adeb7a3fe91-networking-console-plugin-cert podName:35b750f1-f674-4130-b170-8adeb7a3fe91 nodeName:}" failed. No retries permitted until 2026-07-22 02:38:08.014332863 +0000 UTC m=+54.333508146 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/35b750f1-f674-4130-b170-8adeb7a3fe91-networking-console-plugin-cert") pod "networking-console-plugin-7bc7489cb5-gxllf" (UID: "35b750f1-f674-4130-b170-8adeb7a3fe91") : secret "networking-console-plugin-cert" not found Jul 22 02:38:04.919368 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:04.919332 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/26d80b89-26cd-4736-9a65-753db280e9fd-metrics-tls\") pod \"dns-default-trlg7\" (UID: \"26d80b89-26cd-4736-9a65-753db280e9fd\") " pod="openshift-dns/dns-default-trlg7" Jul 22 02:38:04.919546 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:04.919396 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/08aea90a-8858-4443-a3cc-34e6d72a705c-registry-tls\") pod \"image-registry-d469874bc-bnc45\" (UID: \"08aea90a-8858-4443-a3cc-34e6d72a705c\") " pod="openshift-image-registry/image-registry-d469874bc-bnc45" Jul 22 02:38:04.919546 ip-10-0-133-31 kubenswrapper[2578]: E0722 02:38:04.919514 2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 22 02:38:04.919546 ip-10-0-133-31 kubenswrapper[2578]: E0722 02:38:04.919537 2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 22 02:38:04.919710 ip-10-0-133-31 kubenswrapper[2578]: E0722 02:38:04.919551 2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-d469874bc-bnc45: secret "image-registry-tls" not found Jul 22 02:38:04.919710 ip-10-0-133-31 kubenswrapper[2578]: E0722 02:38:04.919597 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/26d80b89-26cd-4736-9a65-753db280e9fd-metrics-tls podName:26d80b89-26cd-4736-9a65-753db280e9fd nodeName:}" failed. No retries permitted until 2026-07-22 02:38:20.919577008 +0000 UTC m=+67.238752289 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/26d80b89-26cd-4736-9a65-753db280e9fd-metrics-tls") pod "dns-default-trlg7" (UID: "26d80b89-26cd-4736-9a65-753db280e9fd") : secret "dns-default-metrics-tls" not found Jul 22 02:38:04.919710 ip-10-0-133-31 kubenswrapper[2578]: E0722 02:38:04.919619 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/08aea90a-8858-4443-a3cc-34e6d72a705c-registry-tls podName:08aea90a-8858-4443-a3cc-34e6d72a705c nodeName:}" failed. No retries permitted until 2026-07-22 02:38:20.919604453 +0000 UTC m=+67.238779736 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/08aea90a-8858-4443-a3cc-34e6d72a705c-registry-tls") pod "image-registry-d469874bc-bnc45" (UID: "08aea90a-8858-4443-a3cc-34e6d72a705c") : secret "image-registry-tls" not found Jul 22 02:38:06.227239 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:06.227207 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a0f7cb8b-7cc6-441e-accc-8c2061b4baab-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gvwx6\" (UID: \"a0f7cb8b-7cc6-441e-accc-8c2061b4baab\") " pod="openshift-insights/insights-runtime-extractor-gvwx6" Jul 22 02:38:06.227647 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:06.227268 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a0f7cb8b-7cc6-441e-accc-8c2061b4baab-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gvwx6\" (UID: \"a0f7cb8b-7cc6-441e-accc-8c2061b4baab\") " pod="openshift-insights/insights-runtime-extractor-gvwx6" Jul 22 02:38:06.227647 ip-10-0-133-31 kubenswrapper[2578]: E0722 02:38:06.227388 2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 22 02:38:06.227647 ip-10-0-133-31 kubenswrapper[2578]: E0722 02:38:06.227438 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a0f7cb8b-7cc6-441e-accc-8c2061b4baab-insights-runtime-extractor-tls podName:a0f7cb8b-7cc6-441e-accc-8c2061b4baab nodeName:}" failed. No retries permitted until 2026-07-22 02:38:14.227421108 +0000 UTC m=+60.546596389 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a0f7cb8b-7cc6-441e-accc-8c2061b4baab-insights-runtime-extractor-tls") pod "insights-runtime-extractor-gvwx6" (UID: "a0f7cb8b-7cc6-441e-accc-8c2061b4baab") : secret "insights-runtime-extractor-tls" not found Jul 22 02:38:06.227757 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:06.227742 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a0f7cb8b-7cc6-441e-accc-8c2061b4baab-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gvwx6\" (UID: \"a0f7cb8b-7cc6-441e-accc-8c2061b4baab\") " pod="openshift-insights/insights-runtime-extractor-gvwx6" Jul 22 02:38:08.040714 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:08.040684 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/35b750f1-f674-4130-b170-8adeb7a3fe91-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-gxllf\" (UID: \"35b750f1-f674-4130-b170-8adeb7a3fe91\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-gxllf" Jul 22 02:38:08.041052 ip-10-0-133-31 kubenswrapper[2578]: E0722 02:38:08.040784 2578 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 22 02:38:08.041052 ip-10-0-133-31 kubenswrapper[2578]: E0722 02:38:08.040834 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/35b750f1-f674-4130-b170-8adeb7a3fe91-networking-console-plugin-cert podName:35b750f1-f674-4130-b170-8adeb7a3fe91 nodeName:}" failed. No retries permitted until 2026-07-22 02:38:16.040820901 +0000 UTC m=+62.359996179 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/35b750f1-f674-4130-b170-8adeb7a3fe91-networking-console-plugin-cert") pod "networking-console-plugin-7bc7489cb5-gxllf" (UID: "35b750f1-f674-4130-b170-8adeb7a3fe91") : secret "networking-console-plugin-cert" not found Jul 22 02:38:14.288684 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:14.288654 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a0f7cb8b-7cc6-441e-accc-8c2061b4baab-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gvwx6\" (UID: \"a0f7cb8b-7cc6-441e-accc-8c2061b4baab\") " pod="openshift-insights/insights-runtime-extractor-gvwx6" Jul 22 02:38:14.290735 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:14.290713 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a0f7cb8b-7cc6-441e-accc-8c2061b4baab-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gvwx6\" (UID: \"a0f7cb8b-7cc6-441e-accc-8c2061b4baab\") " pod="openshift-insights/insights-runtime-extractor-gvwx6" Jul 22 02:38:14.376045 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:14.376019 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-5gmrt\"" Jul 22 02:38:14.384113 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:14.384092 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-gvwx6" Jul 22 02:38:14.531209 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:14.531188 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-gvwx6"] Jul 22 02:38:14.533286 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:38:14.533261 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0f7cb8b_7cc6_441e_accc_8c2061b4baab.slice/crio-06e1715f7e7f65bc6c234f1de6582aaeb1d5158b13d6d376b6f2fd51531a09ae WatchSource:0}: Error finding container 06e1715f7e7f65bc6c234f1de6582aaeb1d5158b13d6d376b6f2fd51531a09ae: Status 404 returned error can't find the container with id 06e1715f7e7f65bc6c234f1de6582aaeb1d5158b13d6d376b6f2fd51531a09ae Jul 22 02:38:15.292405 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:15.292379 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-prfql" Jul 22 02:38:15.366005 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:15.365973 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-gvwx6" event={"ID":"a0f7cb8b-7cc6-441e-accc-8c2061b4baab","Type":"ContainerStarted","Data":"06e1715f7e7f65bc6c234f1de6582aaeb1d5158b13d6d376b6f2fd51531a09ae"} Jul 22 02:38:16.103572 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:16.103535 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/35b750f1-f674-4130-b170-8adeb7a3fe91-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-gxllf\" (UID: \"35b750f1-f674-4130-b170-8adeb7a3fe91\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-gxllf" Jul 22 02:38:16.106146 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:16.106119 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/35b750f1-f674-4130-b170-8adeb7a3fe91-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-gxllf\" (UID: \"35b750f1-f674-4130-b170-8adeb7a3fe91\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-gxllf" Jul 22 02:38:16.174809 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:16.174784 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-58nt6\"" Jul 22 02:38:16.182181 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:16.182157 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-7bc7489cb5-gxllf" Jul 22 02:38:16.316601 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:16.316554 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-7bc7489cb5-gxllf"] Jul 22 02:38:16.319761 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:38:16.319720 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35b750f1_f674_4130_b170_8adeb7a3fe91.slice/crio-5023cc3840054aa640fb06e8ffe7990c8d06a2b601cae4b8141c58766784affb WatchSource:0}: Error finding container 5023cc3840054aa640fb06e8ffe7990c8d06a2b601cae4b8141c58766784affb: Status 404 returned error can't find the container with id 5023cc3840054aa640fb06e8ffe7990c8d06a2b601cae4b8141c58766784affb Jul 22 02:38:16.369602 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:16.369521 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-7bc7489cb5-gxllf" event={"ID":"35b750f1-f674-4130-b170-8adeb7a3fe91","Type":"ContainerStarted","Data":"5023cc3840054aa640fb06e8ffe7990c8d06a2b601cae4b8141c58766784affb"} Jul 22 02:38:19.836532 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:19.836504 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d6871664-adf7-41be-b1d0-d1ec7ccae7c6-metrics-certs\") pod \"network-metrics-daemon-lvbjx\" (UID: \"d6871664-adf7-41be-b1d0-d1ec7ccae7c6\") " pod="openshift-multus/network-metrics-daemon-lvbjx" Jul 22 02:38:19.836803 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:19.836576 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/684d87e0-1cec-4247-9446-881aff58a246-original-pull-secret\") pod \"global-pull-secret-syncer-rb9sf\" (UID: \"684d87e0-1cec-4247-9446-881aff58a246\") " pod="kube-system/global-pull-secret-syncer-rb9sf" Jul 22 02:38:19.839527 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:19.839510 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 22 02:38:19.839637 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:19.839621 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 22 02:38:19.849282 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:19.849246 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d6871664-adf7-41be-b1d0-d1ec7ccae7c6-metrics-certs\") pod \"network-metrics-daemon-lvbjx\" (UID: \"d6871664-adf7-41be-b1d0-d1ec7ccae7c6\") " pod="openshift-multus/network-metrics-daemon-lvbjx" Jul 22 02:38:19.849524 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:19.849503 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/684d87e0-1cec-4247-9446-881aff58a246-original-pull-secret\") pod \"global-pull-secret-syncer-rb9sf\" (UID: \"684d87e0-1cec-4247-9446-881aff58a246\") " pod="kube-system/global-pull-secret-syncer-rb9sf" Jul 22 02:38:19.937225 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:19.937201 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5cxbr\" (UniqueName: \"kubernetes.io/projected/6b93e551-99d3-4ee4-8560-ca6b0f26b6db-kube-api-access-5cxbr\") pod \"network-check-target-wp7zw\" (UID: \"6b93e551-99d3-4ee4-8560-ca6b0f26b6db\") " pod="openshift-network-diagnostics/network-check-target-wp7zw" Jul 22 02:38:19.940284 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:19.940269 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 22 02:38:19.950934 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:19.950917 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 22 02:38:19.960045 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:19.960029 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5cxbr\" (UniqueName: \"kubernetes.io/projected/6b93e551-99d3-4ee4-8560-ca6b0f26b6db-kube-api-access-5cxbr\") pod \"network-check-target-wp7zw\" (UID: \"6b93e551-99d3-4ee4-8560-ca6b0f26b6db\") " pod="openshift-network-diagnostics/network-check-target-wp7zw" Jul 22 02:38:20.054403 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:20.054355 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-d24b6\"" Jul 22 02:38:20.057884 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:20.057873 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rb9sf" Jul 22 02:38:20.061334 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:20.061317 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wp7zw" Jul 22 02:38:20.065740 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:20.065723 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-48l7s\"" Jul 22 02:38:20.073339 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:20.073322 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lvbjx" Jul 22 02:38:20.203595 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:20.203093 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-wp7zw"] Jul 22 02:38:20.206041 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:38:20.206011 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b93e551_99d3_4ee4_8560_ca6b0f26b6db.slice/crio-25471dbf2117801936142af461fa3093707570d597bea9cc793004df8c668eab WatchSource:0}: Error finding container 25471dbf2117801936142af461fa3093707570d597bea9cc793004df8c668eab: Status 404 returned error can't find the container with id 25471dbf2117801936142af461fa3093707570d597bea9cc793004df8c668eab Jul 22 02:38:20.220394 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:20.220323 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-rb9sf"] Jul 22 02:38:20.222667 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:38:20.222643 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod684d87e0_1cec_4247_9446_881aff58a246.slice/crio-d24635fbc9db6b80fddcb0df7eb6f8f5e09159544d84eb9fbb5262c26613a46b WatchSource:0}: Error finding container d24635fbc9db6b80fddcb0df7eb6f8f5e09159544d84eb9fbb5262c26613a46b: Status 404 returned error can't find the container with id d24635fbc9db6b80fddcb0df7eb6f8f5e09159544d84eb9fbb5262c26613a46b Jul 22 02:38:20.235928 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:20.235907 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-lvbjx"] Jul 22 02:38:20.238443 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:38:20.238413 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6871664_adf7_41be_b1d0_d1ec7ccae7c6.slice/crio-f8f926a1a861cf4c45da6282f0e77964aaf8415ae3f27486fb31b6c9478885da WatchSource:0}: Error finding container f8f926a1a861cf4c45da6282f0e77964aaf8415ae3f27486fb31b6c9478885da: Status 404 returned error can't find the container with id f8f926a1a861cf4c45da6282f0e77964aaf8415ae3f27486fb31b6c9478885da Jul 22 02:38:20.383152 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:20.383076 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-wp7zw" event={"ID":"6b93e551-99d3-4ee4-8560-ca6b0f26b6db","Type":"ContainerStarted","Data":"25471dbf2117801936142af461fa3093707570d597bea9cc793004df8c668eab"} Jul 22 02:38:20.384879 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:20.384851 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-gvwx6" event={"ID":"a0f7cb8b-7cc6-441e-accc-8c2061b4baab","Type":"ContainerStarted","Data":"71f718eda4d1c8157a54ec3b678b35ffc0b7fc91fcb99cb634c59ada7a635205"} Jul 22 02:38:20.386714 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:20.386689 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-7bc7489cb5-gxllf" event={"ID":"35b750f1-f674-4130-b170-8adeb7a3fe91","Type":"ContainerStarted","Data":"7f8a08859c2feff08429aa78b2f3ad3647175a8c75ca4bdf5ddb89bc7b39d6ff"} Jul 22 02:38:20.388719 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:20.388680 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-lvbjx" event={"ID":"d6871664-adf7-41be-b1d0-d1ec7ccae7c6","Type":"ContainerStarted","Data":"f8f926a1a861cf4c45da6282f0e77964aaf8415ae3f27486fb31b6c9478885da"} Jul 22 02:38:20.389972 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:20.389923 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-rb9sf" event={"ID":"684d87e0-1cec-4247-9446-881aff58a246","Type":"ContainerStarted","Data":"d24635fbc9db6b80fddcb0df7eb6f8f5e09159544d84eb9fbb5262c26613a46b"} Jul 22 02:38:20.440338 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:20.440287 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-7bc7489cb5-gxllf" podStartSLOduration=16.947594164 podStartE2EDuration="20.440269801s" totalImagesPullingTime="3.492675637s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 02:38:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:38:20.439133284 +0000 UTC m=+66.758308588" watchObservedRunningTime="2026-07-22 02:38:20.440269801 +0000 UTC m=+66.759445102" Jul 22 02:38:20.945439 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:20.945407 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/26d80b89-26cd-4736-9a65-753db280e9fd-metrics-tls\") pod \"dns-default-trlg7\" (UID: \"26d80b89-26cd-4736-9a65-753db280e9fd\") " pod="openshift-dns/dns-default-trlg7" Jul 22 02:38:20.945880 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:20.945457 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/08aea90a-8858-4443-a3cc-34e6d72a705c-registry-tls\") pod \"image-registry-d469874bc-bnc45\" (UID: \"08aea90a-8858-4443-a3cc-34e6d72a705c\") " pod="openshift-image-registry/image-registry-d469874bc-bnc45" Jul 22 02:38:20.948467 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:20.948198 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/26d80b89-26cd-4736-9a65-753db280e9fd-metrics-tls\") pod \"dns-default-trlg7\" (UID: \"26d80b89-26cd-4736-9a65-753db280e9fd\") " pod="openshift-dns/dns-default-trlg7" Jul 22 02:38:20.949507 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:20.949152 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/08aea90a-8858-4443-a3cc-34e6d72a705c-registry-tls\") pod \"image-registry-d469874bc-bnc45\" (UID: \"08aea90a-8858-4443-a3cc-34e6d72a705c\") " pod="openshift-image-registry/image-registry-d469874bc-bnc45" Jul 22 02:38:21.228973 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:21.228896 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-7k6rz\"" Jul 22 02:38:21.236053 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:21.236028 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-trlg7" Jul 22 02:38:21.238017 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:21.237992 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-swntg\"" Jul 22 02:38:21.245619 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:21.245520 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-d469874bc-bnc45" Jul 22 02:38:21.666693 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:21.666616 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-trlg7"] Jul 22 02:38:21.672071 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:38:21.672040 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26d80b89_26cd_4736_9a65_753db280e9fd.slice/crio-72baa7f49cb82bf5c2c90c795c91b80bf3be18722866aeb1d80124216ea4eda7 WatchSource:0}: Error finding container 72baa7f49cb82bf5c2c90c795c91b80bf3be18722866aeb1d80124216ea4eda7: Status 404 returned error can't find the container with id 72baa7f49cb82bf5c2c90c795c91b80bf3be18722866aeb1d80124216ea4eda7 Jul 22 02:38:21.687295 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:21.687270 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-d469874bc-bnc45"] Jul 22 02:38:21.834598 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:38:21.834553 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08aea90a_8858_4443_a3cc_34e6d72a705c.slice/crio-3f2eaa81ccce44b34c2b1f95293fba87a4f7f7dce389efa258fa3b6e1f48a8b7 WatchSource:0}: Error finding container 3f2eaa81ccce44b34c2b1f95293fba87a4f7f7dce389efa258fa3b6e1f48a8b7: Status 404 returned error can't find the container with id 3f2eaa81ccce44b34c2b1f95293fba87a4f7f7dce389efa258fa3b6e1f48a8b7 Jul 22 02:38:22.400177 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:22.400101 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-trlg7" event={"ID":"26d80b89-26cd-4736-9a65-753db280e9fd","Type":"ContainerStarted","Data":"72baa7f49cb82bf5c2c90c795c91b80bf3be18722866aeb1d80124216ea4eda7"} Jul 22 02:38:22.401490 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:22.401462 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-d469874bc-bnc45" event={"ID":"08aea90a-8858-4443-a3cc-34e6d72a705c","Type":"ContainerStarted","Data":"3f2eaa81ccce44b34c2b1f95293fba87a4f7f7dce389efa258fa3b6e1f48a8b7"} Jul 22 02:38:23.244608 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:23.244553 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-5c5ffbc5f8-rmxz9"] Jul 22 02:38:23.336231 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:23.336207 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-79865d9d7d-lfkrv"] Jul 22 02:38:23.336396 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:23.336347 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5c5ffbc5f8-rmxz9" Jul 22 02:38:23.339971 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:23.339951 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Jul 22 02:38:23.340828 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:23.340809 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Jul 22 02:38:23.340931 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:23.340809 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\"" Jul 22 02:38:23.340931 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:23.340850 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Jul 22 02:38:23.370155 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:23.370135 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-5c5ffbc5f8-rmxz9"] Jul 22 02:38:23.370252 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:23.370173 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-79865d9d7d-lfkrv"] Jul 22 02:38:23.370252 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:23.370192 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-d469874bc-bnc45"] Jul 22 02:38:23.370350 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:23.370276 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-79865d9d7d-lfkrv" Jul 22 02:38:23.375218 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:23.375199 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\"" Jul 22 02:38:23.375454 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:23.375432 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-zswzw\"" Jul 22 02:38:23.423301 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:23.423279 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-7cf654f645-clg89"] Jul 22 02:38:23.442289 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:23.442267 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7cf654f645-clg89" Jul 22 02:38:23.446667 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:23.446647 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7cf654f645-clg89"] Jul 22 02:38:23.464663 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:23.464639 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vnnm4\" (UniqueName: \"kubernetes.io/projected/f4f791fc-687c-4f74-a126-c299a52510df-kube-api-access-vnnm4\") pod \"managed-serviceaccount-addon-agent-79865d9d7d-lfkrv\" (UID: \"f4f791fc-687c-4f74-a126-c299a52510df\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-79865d9d7d-lfkrv" Jul 22 02:38:23.464784 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:23.464697 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/c30d6d8f-3866-411a-974f-ff1748fc85b8-klusterlet-config\") pod \"klusterlet-addon-workmgr-5c5ffbc5f8-rmxz9\" (UID: \"c30d6d8f-3866-411a-974f-ff1748fc85b8\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5c5ffbc5f8-rmxz9" Jul 22 02:38:23.464784 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:23.464728 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8md89\" (UniqueName: \"kubernetes.io/projected/c30d6d8f-3866-411a-974f-ff1748fc85b8-kube-api-access-8md89\") pod \"klusterlet-addon-workmgr-5c5ffbc5f8-rmxz9\" (UID: \"c30d6d8f-3866-411a-974f-ff1748fc85b8\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5c5ffbc5f8-rmxz9" Jul 22 02:38:23.464784 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:23.464758 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f4f791fc-687c-4f74-a126-c299a52510df-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-79865d9d7d-lfkrv\" (UID: \"f4f791fc-687c-4f74-a126-c299a52510df\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-79865d9d7d-lfkrv" Jul 22 02:38:23.464902 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:23.464781 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c30d6d8f-3866-411a-974f-ff1748fc85b8-tmp\") pod \"klusterlet-addon-workmgr-5c5ffbc5f8-rmxz9\" (UID: \"c30d6d8f-3866-411a-974f-ff1748fc85b8\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5c5ffbc5f8-rmxz9" Jul 22 02:38:23.565312 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:23.565249 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9e580eb6-1fce-40b4-a59a-446daa0e0398-ca-trust-extracted\") pod \"image-registry-7cf654f645-clg89\" (UID: \"9e580eb6-1fce-40b4-a59a-446daa0e0398\") " pod="openshift-image-registry/image-registry-7cf654f645-clg89" Jul 22 02:38:23.565312 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:23.565279 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8ss8t\" (UniqueName: \"kubernetes.io/projected/9e580eb6-1fce-40b4-a59a-446daa0e0398-kube-api-access-8ss8t\") pod \"image-registry-7cf654f645-clg89\" (UID: \"9e580eb6-1fce-40b4-a59a-446daa0e0398\") " pod="openshift-image-registry/image-registry-7cf654f645-clg89" Jul 22 02:38:23.565312 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:23.565311 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/c30d6d8f-3866-411a-974f-ff1748fc85b8-klusterlet-config\") pod \"klusterlet-addon-workmgr-5c5ffbc5f8-rmxz9\" (UID: \"c30d6d8f-3866-411a-974f-ff1748fc85b8\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5c5ffbc5f8-rmxz9" Jul 22 02:38:23.565532 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:23.565338 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8md89\" (UniqueName: \"kubernetes.io/projected/c30d6d8f-3866-411a-974f-ff1748fc85b8-kube-api-access-8md89\") pod \"klusterlet-addon-workmgr-5c5ffbc5f8-rmxz9\" (UID: \"c30d6d8f-3866-411a-974f-ff1748fc85b8\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5c5ffbc5f8-rmxz9" Jul 22 02:38:23.565532 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:23.565389 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9e580eb6-1fce-40b4-a59a-446daa0e0398-bound-sa-token\") pod \"image-registry-7cf654f645-clg89\" (UID: \"9e580eb6-1fce-40b4-a59a-446daa0e0398\") " pod="openshift-image-registry/image-registry-7cf654f645-clg89" Jul 22 02:38:23.565532 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:23.565434 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9e580eb6-1fce-40b4-a59a-446daa0e0398-registry-tls\") pod \"image-registry-7cf654f645-clg89\" (UID: \"9e580eb6-1fce-40b4-a59a-446daa0e0398\") " pod="openshift-image-registry/image-registry-7cf654f645-clg89" Jul 22 02:38:23.565532 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:23.565459 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9e580eb6-1fce-40b4-a59a-446daa0e0398-installation-pull-secrets\") pod \"image-registry-7cf654f645-clg89\" (UID: \"9e580eb6-1fce-40b4-a59a-446daa0e0398\") " pod="openshift-image-registry/image-registry-7cf654f645-clg89" Jul 22 02:38:23.565532 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:23.565488 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f4f791fc-687c-4f74-a126-c299a52510df-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-79865d9d7d-lfkrv\" (UID: \"f4f791fc-687c-4f74-a126-c299a52510df\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-79865d9d7d-lfkrv" Jul 22 02:38:23.565532 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:23.565508 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c30d6d8f-3866-411a-974f-ff1748fc85b8-tmp\") pod \"klusterlet-addon-workmgr-5c5ffbc5f8-rmxz9\" (UID: \"c30d6d8f-3866-411a-974f-ff1748fc85b8\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5c5ffbc5f8-rmxz9" Jul 22 02:38:23.565768 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:23.565579 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vnnm4\" (UniqueName: \"kubernetes.io/projected/f4f791fc-687c-4f74-a126-c299a52510df-kube-api-access-vnnm4\") pod \"managed-serviceaccount-addon-agent-79865d9d7d-lfkrv\" (UID: \"f4f791fc-687c-4f74-a126-c299a52510df\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-79865d9d7d-lfkrv" Jul 22 02:38:23.565768 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:23.565602 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9e580eb6-1fce-40b4-a59a-446daa0e0398-trusted-ca\") pod \"image-registry-7cf654f645-clg89\" (UID: \"9e580eb6-1fce-40b4-a59a-446daa0e0398\") " pod="openshift-image-registry/image-registry-7cf654f645-clg89" Jul 22 02:38:23.565768 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:23.565632 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9e580eb6-1fce-40b4-a59a-446daa0e0398-image-registry-private-configuration\") pod \"image-registry-7cf654f645-clg89\" (UID: \"9e580eb6-1fce-40b4-a59a-446daa0e0398\") " pod="openshift-image-registry/image-registry-7cf654f645-clg89" Jul 22 02:38:23.565768 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:23.565653 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9e580eb6-1fce-40b4-a59a-446daa0e0398-registry-certificates\") pod \"image-registry-7cf654f645-clg89\" (UID: \"9e580eb6-1fce-40b4-a59a-446daa0e0398\") " pod="openshift-image-registry/image-registry-7cf654f645-clg89" Jul 22 02:38:23.566300 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:23.566276 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c30d6d8f-3866-411a-974f-ff1748fc85b8-tmp\") pod \"klusterlet-addon-workmgr-5c5ffbc5f8-rmxz9\" (UID: \"c30d6d8f-3866-411a-974f-ff1748fc85b8\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5c5ffbc5f8-rmxz9" Jul 22 02:38:23.568289 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:23.568256 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/c30d6d8f-3866-411a-974f-ff1748fc85b8-klusterlet-config\") pod \"klusterlet-addon-workmgr-5c5ffbc5f8-rmxz9\" (UID: \"c30d6d8f-3866-411a-974f-ff1748fc85b8\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5c5ffbc5f8-rmxz9" Jul 22 02:38:23.568403 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:23.568382 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f4f791fc-687c-4f74-a126-c299a52510df-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-79865d9d7d-lfkrv\" (UID: \"f4f791fc-687c-4f74-a126-c299a52510df\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-79865d9d7d-lfkrv" Jul 22 02:38:23.574149 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:23.574131 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8md89\" (UniqueName: \"kubernetes.io/projected/c30d6d8f-3866-411a-974f-ff1748fc85b8-kube-api-access-8md89\") pod \"klusterlet-addon-workmgr-5c5ffbc5f8-rmxz9\" (UID: \"c30d6d8f-3866-411a-974f-ff1748fc85b8\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5c5ffbc5f8-rmxz9" Jul 22 02:38:23.575298 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:23.575278 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vnnm4\" (UniqueName: \"kubernetes.io/projected/f4f791fc-687c-4f74-a126-c299a52510df-kube-api-access-vnnm4\") pod \"managed-serviceaccount-addon-agent-79865d9d7d-lfkrv\" (UID: \"f4f791fc-687c-4f74-a126-c299a52510df\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-79865d9d7d-lfkrv" Jul 22 02:38:23.645200 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:23.645182 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5c5ffbc5f8-rmxz9" Jul 22 02:38:23.666522 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:23.666497 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9e580eb6-1fce-40b4-a59a-446daa0e0398-trusted-ca\") pod \"image-registry-7cf654f645-clg89\" (UID: \"9e580eb6-1fce-40b4-a59a-446daa0e0398\") " pod="openshift-image-registry/image-registry-7cf654f645-clg89" Jul 22 02:38:23.666630 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:23.666538 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9e580eb6-1fce-40b4-a59a-446daa0e0398-image-registry-private-configuration\") pod \"image-registry-7cf654f645-clg89\" (UID: \"9e580eb6-1fce-40b4-a59a-446daa0e0398\") " pod="openshift-image-registry/image-registry-7cf654f645-clg89" Jul 22 02:38:23.666630 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:23.666583 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9e580eb6-1fce-40b4-a59a-446daa0e0398-registry-certificates\") pod \"image-registry-7cf654f645-clg89\" (UID: \"9e580eb6-1fce-40b4-a59a-446daa0e0398\") " pod="openshift-image-registry/image-registry-7cf654f645-clg89" Jul 22 02:38:23.666816 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:23.666780 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9e580eb6-1fce-40b4-a59a-446daa0e0398-ca-trust-extracted\") pod \"image-registry-7cf654f645-clg89\" (UID: \"9e580eb6-1fce-40b4-a59a-446daa0e0398\") " pod="openshift-image-registry/image-registry-7cf654f645-clg89" Jul 22 02:38:23.666921 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:23.666833 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8ss8t\" (UniqueName: \"kubernetes.io/projected/9e580eb6-1fce-40b4-a59a-446daa0e0398-kube-api-access-8ss8t\") pod \"image-registry-7cf654f645-clg89\" (UID: \"9e580eb6-1fce-40b4-a59a-446daa0e0398\") " pod="openshift-image-registry/image-registry-7cf654f645-clg89" Jul 22 02:38:23.666921 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:23.666867 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9e580eb6-1fce-40b4-a59a-446daa0e0398-bound-sa-token\") pod \"image-registry-7cf654f645-clg89\" (UID: \"9e580eb6-1fce-40b4-a59a-446daa0e0398\") " pod="openshift-image-registry/image-registry-7cf654f645-clg89" Jul 22 02:38:23.666921 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:23.666897 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9e580eb6-1fce-40b4-a59a-446daa0e0398-registry-tls\") pod \"image-registry-7cf654f645-clg89\" (UID: \"9e580eb6-1fce-40b4-a59a-446daa0e0398\") " pod="openshift-image-registry/image-registry-7cf654f645-clg89" Jul 22 02:38:23.667066 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:23.666924 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9e580eb6-1fce-40b4-a59a-446daa0e0398-installation-pull-secrets\") pod \"image-registry-7cf654f645-clg89\" (UID: \"9e580eb6-1fce-40b4-a59a-446daa0e0398\") " pod="openshift-image-registry/image-registry-7cf654f645-clg89" Jul 22 02:38:23.667237 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:23.667211 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9e580eb6-1fce-40b4-a59a-446daa0e0398-ca-trust-extracted\") pod \"image-registry-7cf654f645-clg89\" (UID: \"9e580eb6-1fce-40b4-a59a-446daa0e0398\") " pod="openshift-image-registry/image-registry-7cf654f645-clg89" Jul 22 02:38:23.667485 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:23.667455 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9e580eb6-1fce-40b4-a59a-446daa0e0398-registry-certificates\") pod \"image-registry-7cf654f645-clg89\" (UID: \"9e580eb6-1fce-40b4-a59a-446daa0e0398\") " pod="openshift-image-registry/image-registry-7cf654f645-clg89" Jul 22 02:38:23.667949 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:23.667927 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9e580eb6-1fce-40b4-a59a-446daa0e0398-trusted-ca\") pod \"image-registry-7cf654f645-clg89\" (UID: \"9e580eb6-1fce-40b4-a59a-446daa0e0398\") " pod="openshift-image-registry/image-registry-7cf654f645-clg89" Jul 22 02:38:23.669372 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:23.669354 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9e580eb6-1fce-40b4-a59a-446daa0e0398-installation-pull-secrets\") pod \"image-registry-7cf654f645-clg89\" (UID: \"9e580eb6-1fce-40b4-a59a-446daa0e0398\") " pod="openshift-image-registry/image-registry-7cf654f645-clg89" Jul 22 02:38:23.670062 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:23.670041 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9e580eb6-1fce-40b4-a59a-446daa0e0398-image-registry-private-configuration\") pod \"image-registry-7cf654f645-clg89\" (UID: \"9e580eb6-1fce-40b4-a59a-446daa0e0398\") " pod="openshift-image-registry/image-registry-7cf654f645-clg89" Jul 22 02:38:23.670135 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:23.670104 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9e580eb6-1fce-40b4-a59a-446daa0e0398-registry-tls\") pod \"image-registry-7cf654f645-clg89\" (UID: \"9e580eb6-1fce-40b4-a59a-446daa0e0398\") " pod="openshift-image-registry/image-registry-7cf654f645-clg89" Jul 22 02:38:23.678063 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:23.678040 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9e580eb6-1fce-40b4-a59a-446daa0e0398-bound-sa-token\") pod \"image-registry-7cf654f645-clg89\" (UID: \"9e580eb6-1fce-40b4-a59a-446daa0e0398\") " pod="openshift-image-registry/image-registry-7cf654f645-clg89" Jul 22 02:38:23.678579 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:23.678544 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8ss8t\" (UniqueName: \"kubernetes.io/projected/9e580eb6-1fce-40b4-a59a-446daa0e0398-kube-api-access-8ss8t\") pod \"image-registry-7cf654f645-clg89\" (UID: \"9e580eb6-1fce-40b4-a59a-446daa0e0398\") " pod="openshift-image-registry/image-registry-7cf654f645-clg89" Jul 22 02:38:23.687273 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:23.687254 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-79865d9d7d-lfkrv" Jul 22 02:38:23.755979 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:23.755959 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7cf654f645-clg89" Jul 22 02:38:27.138793 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:27.138767 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7cf654f645-clg89"] Jul 22 02:38:27.150121 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:38:27.149879 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e580eb6_1fce_40b4_a59a_446daa0e0398.slice/crio-f0a96d6e1c73902438a276de6d859b84b44a4c28f75b9fe41bdfe64745201ea2 WatchSource:0}: Error finding container f0a96d6e1c73902438a276de6d859b84b44a4c28f75b9fe41bdfe64745201ea2: Status 404 returned error can't find the container with id f0a96d6e1c73902438a276de6d859b84b44a4c28f75b9fe41bdfe64745201ea2 Jul 22 02:38:27.176069 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:27.175655 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-5c5ffbc5f8-rmxz9"] Jul 22 02:38:27.254269 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:27.254246 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-79865d9d7d-lfkrv"] Jul 22 02:38:27.258491 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:38:27.258465 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf4f791fc_687c_4f74_a126_c299a52510df.slice/crio-68712275a3ca41750bb643d7b4cc125bfffa88bc4f473d35b0efdb566d9c4fbb WatchSource:0}: Error finding container 68712275a3ca41750bb643d7b4cc125bfffa88bc4f473d35b0efdb566d9c4fbb: Status 404 returned error can't find the container with id 68712275a3ca41750bb643d7b4cc125bfffa88bc4f473d35b0efdb566d9c4fbb Jul 22 02:38:27.418265 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:27.418179 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5c5ffbc5f8-rmxz9" event={"ID":"c30d6d8f-3866-411a-974f-ff1748fc85b8","Type":"ContainerStarted","Data":"986f81cb476164ae1cb284a0796dfd033495e872e982ae50394994289d7387dc"} Jul 22 02:38:27.419836 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:27.419802 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7cf654f645-clg89" event={"ID":"9e580eb6-1fce-40b4-a59a-446daa0e0398","Type":"ContainerStarted","Data":"c60a1197f0086ffeecbbdc3aeca55cfc3531bfd69e1fe59645dd63ae00dbcde8"} Jul 22 02:38:27.419980 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:27.419851 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7cf654f645-clg89" event={"ID":"9e580eb6-1fce-40b4-a59a-446daa0e0398","Type":"ContainerStarted","Data":"f0a96d6e1c73902438a276de6d859b84b44a4c28f75b9fe41bdfe64745201ea2"} Jul 22 02:38:27.419980 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:27.419875 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-7cf654f645-clg89" Jul 22 02:38:27.422078 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:27.421643 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-lvbjx" event={"ID":"d6871664-adf7-41be-b1d0-d1ec7ccae7c6","Type":"ContainerStarted","Data":"af009babeb4641edfe25635480a4a1450b66b51246c202c41091826fcade4f91"} Jul 22 02:38:27.422078 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:27.421689 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-lvbjx" event={"ID":"d6871664-adf7-41be-b1d0-d1ec7ccae7c6","Type":"ContainerStarted","Data":"be9ceaf751ee916ea3db088e46613255382a4e1a338651aab49eb79b57003945"} Jul 22 02:38:27.423161 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:27.423133 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-rb9sf" event={"ID":"684d87e0-1cec-4247-9446-881aff58a246","Type":"ContainerStarted","Data":"59b16d950b1a169142c905c0e3cb620a3324fed8db2f29a5f979c9cd4041c9f3"} Jul 22 02:38:27.425018 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:27.424996 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-trlg7" event={"ID":"26d80b89-26cd-4736-9a65-753db280e9fd","Type":"ContainerStarted","Data":"692d77591f2f57867acc07409911276f9a50e6e271657f30e8a427dbac88830d"} Jul 22 02:38:27.425133 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:27.425039 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-trlg7" event={"ID":"26d80b89-26cd-4736-9a65-753db280e9fd","Type":"ContainerStarted","Data":"c6c0b8add1a4e89036691ad0f3a5304fc1b5c146c8dc2c52e91de8b85502807d"} Jul 22 02:38:27.425133 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:27.425122 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-trlg7" Jul 22 02:38:27.426994 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:27.426974 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-wp7zw" event={"ID":"6b93e551-99d3-4ee4-8560-ca6b0f26b6db","Type":"ContainerStarted","Data":"593e934207fee1a2105a68c9697e6a8e7d815cc7a42a2d36cb1dd96a045f55b0"} Jul 22 02:38:27.427125 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:27.427111 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-wp7zw" Jul 22 02:38:27.428504 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:27.428481 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-d469874bc-bnc45" event={"ID":"08aea90a-8858-4443-a3cc-34e6d72a705c","Type":"ContainerStarted","Data":"ead01b724705c51553476914bb3cb18ca2f482172292ec8c4dc61ea200fe507e"} Jul 22 02:38:27.428632 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:27.428590 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-d469874bc-bnc45" Jul 22 02:38:27.429882 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:27.429854 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-79865d9d7d-lfkrv" event={"ID":"f4f791fc-687c-4f74-a126-c299a52510df","Type":"ContainerStarted","Data":"68712275a3ca41750bb643d7b4cc125bfffa88bc4f473d35b0efdb566d9c4fbb"} Jul 22 02:38:27.431693 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:27.431674 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-gvwx6" event={"ID":"a0f7cb8b-7cc6-441e-accc-8c2061b4baab","Type":"ContainerStarted","Data":"e4b302ca070ae18154605113c5526fa9916c54d0f67f552ee317cbe231346554"} Jul 22 02:38:27.444798 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:27.444748 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-7cf654f645-clg89" podStartSLOduration=4.444730526 podStartE2EDuration="4.444730526s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 02:38:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:38:27.44276251 +0000 UTC m=+73.761937812" watchObservedRunningTime="2026-07-22 02:38:27.444730526 +0000 UTC m=+73.763905826" Jul 22 02:38:27.468449 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:27.468407 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-d469874bc-bnc45" podStartSLOduration=49.468391177 podStartE2EDuration="49.468391177s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 02:37:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:38:27.467365772 +0000 UTC m=+73.786541073" watchObservedRunningTime="2026-07-22 02:38:27.468391177 +0000 UTC m=+73.787566455" Jul 22 02:38:27.489182 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:27.489130 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-wp7zw" podStartSLOduration=66.825247215 podStartE2EDuration="1m13.489111817s" totalImagesPullingTime="6.663864602s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 02:37:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:38:27.488851221 +0000 UTC m=+73.808026521" watchObservedRunningTime="2026-07-22 02:38:27.489111817 +0000 UTC m=+73.808287112" Jul 22 02:38:27.509340 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:27.509292 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-trlg7" podStartSLOduration=33.402539466 podStartE2EDuration="38.509273915s" totalImagesPullingTime="5.106734449s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 02:37:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:38:27.508790382 +0000 UTC m=+73.827965682" watchObservedRunningTime="2026-07-22 02:38:27.509273915 +0000 UTC m=+73.828449217" Jul 22 02:38:27.527839 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:27.527804 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-lvbjx" podStartSLOduration=67.940645887 podStartE2EDuration="1m13.527791772s" totalImagesPullingTime="5.587145885s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 02:37:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:38:27.527549504 +0000 UTC m=+73.846724815" watchObservedRunningTime="2026-07-22 02:38:27.527791772 +0000 UTC m=+73.846967071" Jul 22 02:38:27.545197 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:27.545159 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-rb9sf" podStartSLOduration=66.843879331 podStartE2EDuration="1m13.54514705s" totalImagesPullingTime="6.701267719s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 02:37:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:38:27.544980633 +0000 UTC m=+73.864155932" watchObservedRunningTime="2026-07-22 02:38:27.54514705 +0000 UTC m=+73.864322349" Jul 22 02:38:30.839472 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:30.839439 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-858cc4d8c5-85zzs"] Jul 22 02:38:30.852778 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:30.852751 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-858cc4d8c5-85zzs"] Jul 22 02:38:30.852922 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:30.852859 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-858cc4d8c5-85zzs" Jul 22 02:38:30.855929 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:30.855904 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 22 02:38:30.857343 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:30.857199 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 22 02:38:30.857343 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:30.857292 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-449kn\"" Jul 22 02:38:30.857488 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:30.857416 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 22 02:38:30.857540 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:30.857506 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 22 02:38:30.857755 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:30.857736 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 22 02:38:30.857827 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:30.857806 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 22 02:38:30.857827 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:30.857815 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 22 02:38:30.929019 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:30.928983 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2fefb9b9-18bd-41b3-807a-21131181d0f1-console-serving-cert\") pod \"console-858cc4d8c5-85zzs\" (UID: \"2fefb9b9-18bd-41b3-807a-21131181d0f1\") " pod="openshift-console/console-858cc4d8c5-85zzs" Jul 22 02:38:30.929195 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:30.929048 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2fefb9b9-18bd-41b3-807a-21131181d0f1-oauth-serving-cert\") pod \"console-858cc4d8c5-85zzs\" (UID: \"2fefb9b9-18bd-41b3-807a-21131181d0f1\") " pod="openshift-console/console-858cc4d8c5-85zzs" Jul 22 02:38:30.929195 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:30.929091 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2fefb9b9-18bd-41b3-807a-21131181d0f1-service-ca\") pod \"console-858cc4d8c5-85zzs\" (UID: \"2fefb9b9-18bd-41b3-807a-21131181d0f1\") " pod="openshift-console/console-858cc4d8c5-85zzs" Jul 22 02:38:30.929195 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:30.929123 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xjmtt\" (UniqueName: \"kubernetes.io/projected/2fefb9b9-18bd-41b3-807a-21131181d0f1-kube-api-access-xjmtt\") pod \"console-858cc4d8c5-85zzs\" (UID: \"2fefb9b9-18bd-41b3-807a-21131181d0f1\") " pod="openshift-console/console-858cc4d8c5-85zzs" Jul 22 02:38:30.929195 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:30.929186 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2fefb9b9-18bd-41b3-807a-21131181d0f1-console-config\") pod \"console-858cc4d8c5-85zzs\" (UID: \"2fefb9b9-18bd-41b3-807a-21131181d0f1\") " pod="openshift-console/console-858cc4d8c5-85zzs" Jul 22 02:38:30.929360 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:30.929224 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2fefb9b9-18bd-41b3-807a-21131181d0f1-console-oauth-config\") pod \"console-858cc4d8c5-85zzs\" (UID: \"2fefb9b9-18bd-41b3-807a-21131181d0f1\") " pod="openshift-console/console-858cc4d8c5-85zzs" Jul 22 02:38:31.029928 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:31.029893 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2fefb9b9-18bd-41b3-807a-21131181d0f1-console-oauth-config\") pod \"console-858cc4d8c5-85zzs\" (UID: \"2fefb9b9-18bd-41b3-807a-21131181d0f1\") " pod="openshift-console/console-858cc4d8c5-85zzs" Jul 22 02:38:31.029928 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:31.029931 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2fefb9b9-18bd-41b3-807a-21131181d0f1-console-serving-cert\") pod \"console-858cc4d8c5-85zzs\" (UID: \"2fefb9b9-18bd-41b3-807a-21131181d0f1\") " pod="openshift-console/console-858cc4d8c5-85zzs" Jul 22 02:38:31.030136 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:31.029962 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2fefb9b9-18bd-41b3-807a-21131181d0f1-oauth-serving-cert\") pod \"console-858cc4d8c5-85zzs\" (UID: \"2fefb9b9-18bd-41b3-807a-21131181d0f1\") " pod="openshift-console/console-858cc4d8c5-85zzs" Jul 22 02:38:31.030136 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:31.030001 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2fefb9b9-18bd-41b3-807a-21131181d0f1-service-ca\") pod \"console-858cc4d8c5-85zzs\" (UID: \"2fefb9b9-18bd-41b3-807a-21131181d0f1\") " pod="openshift-console/console-858cc4d8c5-85zzs" Jul 22 02:38:31.030136 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:31.030034 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xjmtt\" (UniqueName: \"kubernetes.io/projected/2fefb9b9-18bd-41b3-807a-21131181d0f1-kube-api-access-xjmtt\") pod \"console-858cc4d8c5-85zzs\" (UID: \"2fefb9b9-18bd-41b3-807a-21131181d0f1\") " pod="openshift-console/console-858cc4d8c5-85zzs" Jul 22 02:38:31.030136 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:31.030073 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2fefb9b9-18bd-41b3-807a-21131181d0f1-console-config\") pod \"console-858cc4d8c5-85zzs\" (UID: \"2fefb9b9-18bd-41b3-807a-21131181d0f1\") " pod="openshift-console/console-858cc4d8c5-85zzs" Jul 22 02:38:31.030907 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:31.030879 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2fefb9b9-18bd-41b3-807a-21131181d0f1-service-ca\") pod \"console-858cc4d8c5-85zzs\" (UID: \"2fefb9b9-18bd-41b3-807a-21131181d0f1\") " pod="openshift-console/console-858cc4d8c5-85zzs" Jul 22 02:38:31.031033 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:31.030913 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2fefb9b9-18bd-41b3-807a-21131181d0f1-oauth-serving-cert\") pod \"console-858cc4d8c5-85zzs\" (UID: \"2fefb9b9-18bd-41b3-807a-21131181d0f1\") " pod="openshift-console/console-858cc4d8c5-85zzs" Jul 22 02:38:31.031033 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:31.030871 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2fefb9b9-18bd-41b3-807a-21131181d0f1-console-config\") pod \"console-858cc4d8c5-85zzs\" (UID: \"2fefb9b9-18bd-41b3-807a-21131181d0f1\") " pod="openshift-console/console-858cc4d8c5-85zzs" Jul 22 02:38:31.032822 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:31.032799 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2fefb9b9-18bd-41b3-807a-21131181d0f1-console-serving-cert\") pod \"console-858cc4d8c5-85zzs\" (UID: \"2fefb9b9-18bd-41b3-807a-21131181d0f1\") " pod="openshift-console/console-858cc4d8c5-85zzs" Jul 22 02:38:31.032921 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:31.032868 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2fefb9b9-18bd-41b3-807a-21131181d0f1-console-oauth-config\") pod \"console-858cc4d8c5-85zzs\" (UID: \"2fefb9b9-18bd-41b3-807a-21131181d0f1\") " pod="openshift-console/console-858cc4d8c5-85zzs" Jul 22 02:38:31.040829 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:31.040804 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xjmtt\" (UniqueName: \"kubernetes.io/projected/2fefb9b9-18bd-41b3-807a-21131181d0f1-kube-api-access-xjmtt\") pod \"console-858cc4d8c5-85zzs\" (UID: \"2fefb9b9-18bd-41b3-807a-21131181d0f1\") " pod="openshift-console/console-858cc4d8c5-85zzs" Jul 22 02:38:31.165962 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:31.165896 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-858cc4d8c5-85zzs" Jul 22 02:38:31.781637 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:31.781611 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-858cc4d8c5-85zzs"] Jul 22 02:38:31.784115 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:38:31.784092 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2fefb9b9_18bd_41b3_807a_21131181d0f1.slice/crio-32795d816830973e7d618201280500c8bbc269ec9a35e532ea2a3b89012c2a5f WatchSource:0}: Error finding container 32795d816830973e7d618201280500c8bbc269ec9a35e532ea2a3b89012c2a5f: Status 404 returned error can't find the container with id 32795d816830973e7d618201280500c8bbc269ec9a35e532ea2a3b89012c2a5f Jul 22 02:38:32.458062 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:32.458021 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-79865d9d7d-lfkrv" event={"ID":"f4f791fc-687c-4f74-a126-c299a52510df","Type":"ContainerStarted","Data":"24a8161168f182645bdd8f2418d7a76d2e7ffd5e7ef36d93b44b38698842aee7"} Jul 22 02:38:32.460147 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:32.460125 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-gvwx6" event={"ID":"a0f7cb8b-7cc6-441e-accc-8c2061b4baab","Type":"ContainerStarted","Data":"2bbf23036fe6ca7b29a31b4dbdc524071d3264de23dd2cc84f69d7c0468d47c9"} Jul 22 02:38:32.461509 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:32.461484 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-858cc4d8c5-85zzs" event={"ID":"2fefb9b9-18bd-41b3-807a-21131181d0f1","Type":"ContainerStarted","Data":"32795d816830973e7d618201280500c8bbc269ec9a35e532ea2a3b89012c2a5f"} Jul 22 02:38:32.476805 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:32.476766 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-79865d9d7d-lfkrv" podStartSLOduration=5.079241646 podStartE2EDuration="9.476754301s" totalImagesPullingTime="4.397512655s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 02:38:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:38:32.474990555 +0000 UTC m=+78.794165856" watchObservedRunningTime="2026-07-22 02:38:32.476754301 +0000 UTC m=+78.795929614" Jul 22 02:38:32.497855 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:32.497811 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-gvwx6" podStartSLOduration=17.662788061 podStartE2EDuration="34.497798582s" totalImagesPullingTime="16.835010521s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 02:37:58 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:38:32.496123292 +0000 UTC m=+78.815298593" watchObservedRunningTime="2026-07-22 02:38:32.497798582 +0000 UTC m=+78.816973881" Jul 22 02:38:35.472529 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:35.472500 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-858cc4d8c5-85zzs" event={"ID":"2fefb9b9-18bd-41b3-807a-21131181d0f1","Type":"ContainerStarted","Data":"d0ab20818f489af04adc61d1ad5324b8c3db03dcff2cb5e0d66534fe840f6fd1"} Jul 22 02:38:35.518109 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:35.518063 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-858cc4d8c5-85zzs" podStartSLOduration=2.666807038 podStartE2EDuration="5.51804763s" totalImagesPullingTime="2.851240592s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 02:38:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:38:35.51507298 +0000 UTC m=+81.834248279" watchObservedRunningTime="2026-07-22 02:38:35.51804763 +0000 UTC m=+81.837222930" Jul 22 02:38:37.442719 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:37.442687 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-trlg7" Jul 22 02:38:38.449251 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:38.449219 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-bq2jp"] Jul 22 02:38:38.451613 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:38.451596 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-bq2jp" Jul 22 02:38:38.454764 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:38.454741 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 22 02:38:38.455061 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:38.455043 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-9tpt9\"" Jul 22 02:38:38.455372 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:38.455354 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 22 02:38:38.455468 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:38.455436 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 22 02:38:38.456007 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:38.455989 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 22 02:38:38.456602 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:38.456584 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 22 02:38:38.456771 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:38.456758 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 22 02:38:38.484773 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:38.484740 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5c5ffbc5f8-rmxz9" event={"ID":"c30d6d8f-3866-411a-974f-ff1748fc85b8","Type":"ContainerStarted","Data":"2f4afc4c8ba0fc2377b7a1bce5a5090c798ec35f088a2af117f66d3670726d27"} Jul 22 02:38:38.485515 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:38.485498 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5c5ffbc5f8-rmxz9" Jul 22 02:38:38.487076 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:38.487049 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5c5ffbc5f8-rmxz9" Jul 22 02:38:38.507854 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:38.507813 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5c5ffbc5f8-rmxz9" podStartSLOduration=4.8492954600000004 podStartE2EDuration="15.507799653s" totalImagesPullingTime="10.658504193s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 02:38:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:38:38.506961167 +0000 UTC m=+84.826136467" watchObservedRunningTime="2026-07-22 02:38:38.507799653 +0000 UTC m=+84.826974953" Jul 22 02:38:38.590832 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:38.590805 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/04566074-17d2-481d-bf22-d5d63462ad91-node-exporter-tls\") pod \"node-exporter-bq2jp\" (UID: \"04566074-17d2-481d-bf22-d5d63462ad91\") " pod="openshift-monitoring/node-exporter-bq2jp" Jul 22 02:38:38.590953 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:38.590845 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/04566074-17d2-481d-bf22-d5d63462ad91-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-bq2jp\" (UID: \"04566074-17d2-481d-bf22-d5d63462ad91\") " pod="openshift-monitoring/node-exporter-bq2jp" Jul 22 02:38:38.590953 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:38.590873 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/04566074-17d2-481d-bf22-d5d63462ad91-root\") pod \"node-exporter-bq2jp\" (UID: \"04566074-17d2-481d-bf22-d5d63462ad91\") " pod="openshift-monitoring/node-exporter-bq2jp" Jul 22 02:38:38.590953 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:38.590931 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/04566074-17d2-481d-bf22-d5d63462ad91-metrics-client-ca\") pod \"node-exporter-bq2jp\" (UID: \"04566074-17d2-481d-bf22-d5d63462ad91\") " pod="openshift-monitoring/node-exporter-bq2jp" Jul 22 02:38:38.591103 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:38.590960 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/04566074-17d2-481d-bf22-d5d63462ad91-sys\") pod \"node-exporter-bq2jp\" (UID: \"04566074-17d2-481d-bf22-d5d63462ad91\") " pod="openshift-monitoring/node-exporter-bq2jp" Jul 22 02:38:38.591103 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:38.590990 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/04566074-17d2-481d-bf22-d5d63462ad91-node-exporter-accelerators-collector-config\") pod \"node-exporter-bq2jp\" (UID: \"04566074-17d2-481d-bf22-d5d63462ad91\") " pod="openshift-monitoring/node-exporter-bq2jp" Jul 22 02:38:38.591103 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:38.591075 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/04566074-17d2-481d-bf22-d5d63462ad91-node-exporter-wtmp\") pod \"node-exporter-bq2jp\" (UID: \"04566074-17d2-481d-bf22-d5d63462ad91\") " pod="openshift-monitoring/node-exporter-bq2jp" Jul 22 02:38:38.591242 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:38.591114 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/04566074-17d2-481d-bf22-d5d63462ad91-node-exporter-textfile\") pod \"node-exporter-bq2jp\" (UID: \"04566074-17d2-481d-bf22-d5d63462ad91\") " pod="openshift-monitoring/node-exporter-bq2jp" Jul 22 02:38:38.591242 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:38.591147 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9vpxv\" (UniqueName: \"kubernetes.io/projected/04566074-17d2-481d-bf22-d5d63462ad91-kube-api-access-9vpxv\") pod \"node-exporter-bq2jp\" (UID: \"04566074-17d2-481d-bf22-d5d63462ad91\") " pod="openshift-monitoring/node-exporter-bq2jp" Jul 22 02:38:38.692088 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:38.692067 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/04566074-17d2-481d-bf22-d5d63462ad91-metrics-client-ca\") pod \"node-exporter-bq2jp\" (UID: \"04566074-17d2-481d-bf22-d5d63462ad91\") " pod="openshift-monitoring/node-exporter-bq2jp" Jul 22 02:38:38.692203 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:38.692092 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/04566074-17d2-481d-bf22-d5d63462ad91-sys\") pod \"node-exporter-bq2jp\" (UID: \"04566074-17d2-481d-bf22-d5d63462ad91\") " pod="openshift-monitoring/node-exporter-bq2jp" Jul 22 02:38:38.692203 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:38.692112 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/04566074-17d2-481d-bf22-d5d63462ad91-node-exporter-accelerators-collector-config\") pod \"node-exporter-bq2jp\" (UID: \"04566074-17d2-481d-bf22-d5d63462ad91\") " pod="openshift-monitoring/node-exporter-bq2jp" Jul 22 02:38:38.692203 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:38.692130 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/04566074-17d2-481d-bf22-d5d63462ad91-node-exporter-wtmp\") pod \"node-exporter-bq2jp\" (UID: \"04566074-17d2-481d-bf22-d5d63462ad91\") " pod="openshift-monitoring/node-exporter-bq2jp" Jul 22 02:38:38.692203 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:38.692145 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/04566074-17d2-481d-bf22-d5d63462ad91-node-exporter-textfile\") pod \"node-exporter-bq2jp\" (UID: \"04566074-17d2-481d-bf22-d5d63462ad91\") " pod="openshift-monitoring/node-exporter-bq2jp" Jul 22 02:38:38.692203 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:38.692160 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9vpxv\" (UniqueName: \"kubernetes.io/projected/04566074-17d2-481d-bf22-d5d63462ad91-kube-api-access-9vpxv\") pod \"node-exporter-bq2jp\" (UID: \"04566074-17d2-481d-bf22-d5d63462ad91\") " pod="openshift-monitoring/node-exporter-bq2jp" Jul 22 02:38:38.692437 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:38.692192 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/04566074-17d2-481d-bf22-d5d63462ad91-sys\") pod \"node-exporter-bq2jp\" (UID: \"04566074-17d2-481d-bf22-d5d63462ad91\") " pod="openshift-monitoring/node-exporter-bq2jp" Jul 22 02:38:38.692437 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:38.692217 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/04566074-17d2-481d-bf22-d5d63462ad91-node-exporter-tls\") pod \"node-exporter-bq2jp\" (UID: \"04566074-17d2-481d-bf22-d5d63462ad91\") " pod="openshift-monitoring/node-exporter-bq2jp" Jul 22 02:38:38.692437 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:38.692244 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/04566074-17d2-481d-bf22-d5d63462ad91-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-bq2jp\" (UID: \"04566074-17d2-481d-bf22-d5d63462ad91\") " pod="openshift-monitoring/node-exporter-bq2jp" Jul 22 02:38:38.692437 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:38.692272 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/04566074-17d2-481d-bf22-d5d63462ad91-root\") pod \"node-exporter-bq2jp\" (UID: \"04566074-17d2-481d-bf22-d5d63462ad91\") " pod="openshift-monitoring/node-exporter-bq2jp" Jul 22 02:38:38.692437 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:38.692279 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/04566074-17d2-481d-bf22-d5d63462ad91-node-exporter-wtmp\") pod \"node-exporter-bq2jp\" (UID: \"04566074-17d2-481d-bf22-d5d63462ad91\") " pod="openshift-monitoring/node-exporter-bq2jp" Jul 22 02:38:38.692437 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:38.692334 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/04566074-17d2-481d-bf22-d5d63462ad91-root\") pod \"node-exporter-bq2jp\" (UID: \"04566074-17d2-481d-bf22-d5d63462ad91\") " pod="openshift-monitoring/node-exporter-bq2jp" Jul 22 02:38:38.692437 ip-10-0-133-31 kubenswrapper[2578]: E0722 02:38:38.692362 2578 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Jul 22 02:38:38.692437 ip-10-0-133-31 kubenswrapper[2578]: E0722 02:38:38.692434 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/04566074-17d2-481d-bf22-d5d63462ad91-node-exporter-tls podName:04566074-17d2-481d-bf22-d5d63462ad91 nodeName:}" failed. No retries permitted until 2026-07-22 02:38:39.192414636 +0000 UTC m=+85.511589915 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/04566074-17d2-481d-bf22-d5d63462ad91-node-exporter-tls") pod "node-exporter-bq2jp" (UID: "04566074-17d2-481d-bf22-d5d63462ad91") : secret "node-exporter-tls" not found Jul 22 02:38:38.692831 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:38.692611 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/04566074-17d2-481d-bf22-d5d63462ad91-node-exporter-textfile\") pod \"node-exporter-bq2jp\" (UID: \"04566074-17d2-481d-bf22-d5d63462ad91\") " pod="openshift-monitoring/node-exporter-bq2jp" Jul 22 02:38:38.692831 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:38.692763 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/04566074-17d2-481d-bf22-d5d63462ad91-metrics-client-ca\") pod \"node-exporter-bq2jp\" (UID: \"04566074-17d2-481d-bf22-d5d63462ad91\") " pod="openshift-monitoring/node-exporter-bq2jp" Jul 22 02:38:38.692937 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:38.692857 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/04566074-17d2-481d-bf22-d5d63462ad91-node-exporter-accelerators-collector-config\") pod \"node-exporter-bq2jp\" (UID: \"04566074-17d2-481d-bf22-d5d63462ad91\") " pod="openshift-monitoring/node-exporter-bq2jp" Jul 22 02:38:38.694675 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:38.694655 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/04566074-17d2-481d-bf22-d5d63462ad91-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-bq2jp\" (UID: \"04566074-17d2-481d-bf22-d5d63462ad91\") " pod="openshift-monitoring/node-exporter-bq2jp" Jul 22 02:38:38.701359 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:38.701302 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9vpxv\" (UniqueName: \"kubernetes.io/projected/04566074-17d2-481d-bf22-d5d63462ad91-kube-api-access-9vpxv\") pod \"node-exporter-bq2jp\" (UID: \"04566074-17d2-481d-bf22-d5d63462ad91\") " pod="openshift-monitoring/node-exporter-bq2jp" Jul 22 02:38:39.195621 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:39.195595 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/04566074-17d2-481d-bf22-d5d63462ad91-node-exporter-tls\") pod \"node-exporter-bq2jp\" (UID: \"04566074-17d2-481d-bf22-d5d63462ad91\") " pod="openshift-monitoring/node-exporter-bq2jp" Jul 22 02:38:39.197747 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:39.197724 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/04566074-17d2-481d-bf22-d5d63462ad91-node-exporter-tls\") pod \"node-exporter-bq2jp\" (UID: \"04566074-17d2-481d-bf22-d5d63462ad91\") " pod="openshift-monitoring/node-exporter-bq2jp" Jul 22 02:38:39.362665 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:39.362643 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-bq2jp" Jul 22 02:38:39.371980 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:38:39.371958 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04566074_17d2_481d_bf22_d5d63462ad91.slice/crio-d6d71bbfbc5b34f978a0661c0d3c214d7fb0c4316e44db8c96210c50ec10bd77 WatchSource:0}: Error finding container d6d71bbfbc5b34f978a0661c0d3c214d7fb0c4316e44db8c96210c50ec10bd77: Status 404 returned error can't find the container with id d6d71bbfbc5b34f978a0661c0d3c214d7fb0c4316e44db8c96210c50ec10bd77 Jul 22 02:38:39.489136 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:39.489080 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-bq2jp" event={"ID":"04566074-17d2-481d-bf22-d5d63462ad91","Type":"ContainerStarted","Data":"d6d71bbfbc5b34f978a0661c0d3c214d7fb0c4316e44db8c96210c50ec10bd77"} Jul 22 02:38:40.493832 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:40.493780 2578 generic.go:356] "Generic (PLEG): container finished" podID="04566074-17d2-481d-bf22-d5d63462ad91" containerID="ac70fb21f483ca40d468e1aefdfe329bab82ca745fd9b461a71ced5d76c16aac" exitCode=0 Jul 22 02:38:40.494113 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:40.493841 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-bq2jp" event={"ID":"04566074-17d2-481d-bf22-d5d63462ad91","Type":"ContainerDied","Data":"ac70fb21f483ca40d468e1aefdfe329bab82ca745fd9b461a71ced5d76c16aac"} Jul 22 02:38:41.167059 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:41.167031 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-858cc4d8c5-85zzs" Jul 22 02:38:41.167183 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:41.167087 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-858cc4d8c5-85zzs" Jul 22 02:38:41.171624 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:41.171603 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-858cc4d8c5-85zzs" Jul 22 02:38:41.501058 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:41.500977 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-bq2jp" event={"ID":"04566074-17d2-481d-bf22-d5d63462ad91","Type":"ContainerStarted","Data":"1bf287a1d247956646f667eda9bbf0da99a377967e1deed89c5c51343c60b061"} Jul 22 02:38:41.501419 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:41.501064 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-bq2jp" event={"ID":"04566074-17d2-481d-bf22-d5d63462ad91","Type":"ContainerStarted","Data":"7273d932784fa23fc2a6dcfec8c9f29b8410ef1352c35f3b781adb46c75a800e"} Jul 22 02:38:41.504870 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:41.504848 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-858cc4d8c5-85zzs" Jul 22 02:38:41.533213 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:41.533172 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-bq2jp" podStartSLOduration=2.608314953 podStartE2EDuration="3.533160305s" totalImagesPullingTime="855.68823ms" totalInitContainerRuntime="69.157122ms" isStatefulPod=true podCreationTimestamp="2026-07-22 02:38:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:38:41.531520052 +0000 UTC m=+87.850695374" watchObservedRunningTime="2026-07-22 02:38:41.533160305 +0000 UTC m=+87.852335604" Jul 22 02:38:42.878263 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:42.878230 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-7495b9d698-s64vm"] Jul 22 02:38:42.880062 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:42.880046 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-7495b9d698-s64vm" Jul 22 02:38:42.883466 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:42.883445 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-ek23fenh3le1e\"" Jul 22 02:38:42.883466 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:42.883457 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Jul 22 02:38:42.884204 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:42.884186 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Jul 22 02:38:42.884555 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:42.884538 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Jul 22 02:38:42.884669 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:42.884589 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-kkkdv\"" Jul 22 02:38:42.884669 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:42.884593 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Jul 22 02:38:42.894406 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:42.894387 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-7495b9d698-s64vm"] Jul 22 02:38:43.024769 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:43.024745 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/b9b81360-170a-4e01-af3e-8fd2947ec5a6-audit-log\") pod \"metrics-server-7495b9d698-s64vm\" (UID: \"b9b81360-170a-4e01-af3e-8fd2947ec5a6\") " pod="openshift-monitoring/metrics-server-7495b9d698-s64vm" Jul 22 02:38:43.024860 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:43.024781 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/b9b81360-170a-4e01-af3e-8fd2947ec5a6-client-ca-bundle\") pod \"metrics-server-7495b9d698-s64vm\" (UID: \"b9b81360-170a-4e01-af3e-8fd2947ec5a6\") " pod="openshift-monitoring/metrics-server-7495b9d698-s64vm" Jul 22 02:38:43.024860 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:43.024801 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/b9b81360-170a-4e01-af3e-8fd2947ec5a6-secret-metrics-server-tls\") pod \"metrics-server-7495b9d698-s64vm\" (UID: \"b9b81360-170a-4e01-af3e-8fd2947ec5a6\") " pod="openshift-monitoring/metrics-server-7495b9d698-s64vm" Jul 22 02:38:43.024860 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:43.024817 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mx59c\" (UniqueName: \"kubernetes.io/projected/b9b81360-170a-4e01-af3e-8fd2947ec5a6-kube-api-access-mx59c\") pod \"metrics-server-7495b9d698-s64vm\" (UID: \"b9b81360-170a-4e01-af3e-8fd2947ec5a6\") " pod="openshift-monitoring/metrics-server-7495b9d698-s64vm" Jul 22 02:38:43.024980 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:43.024888 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/b9b81360-170a-4e01-af3e-8fd2947ec5a6-secret-metrics-server-client-certs\") pod \"metrics-server-7495b9d698-s64vm\" (UID: \"b9b81360-170a-4e01-af3e-8fd2947ec5a6\") " pod="openshift-monitoring/metrics-server-7495b9d698-s64vm" Jul 22 02:38:43.024980 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:43.024916 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b9b81360-170a-4e01-af3e-8fd2947ec5a6-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7495b9d698-s64vm\" (UID: \"b9b81360-170a-4e01-af3e-8fd2947ec5a6\") " pod="openshift-monitoring/metrics-server-7495b9d698-s64vm" Jul 22 02:38:43.024980 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:43.024954 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/b9b81360-170a-4e01-af3e-8fd2947ec5a6-metrics-server-audit-profiles\") pod \"metrics-server-7495b9d698-s64vm\" (UID: \"b9b81360-170a-4e01-af3e-8fd2947ec5a6\") " pod="openshift-monitoring/metrics-server-7495b9d698-s64vm" Jul 22 02:38:43.125222 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:43.125197 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/b9b81360-170a-4e01-af3e-8fd2947ec5a6-audit-log\") pod \"metrics-server-7495b9d698-s64vm\" (UID: \"b9b81360-170a-4e01-af3e-8fd2947ec5a6\") " pod="openshift-monitoring/metrics-server-7495b9d698-s64vm" Jul 22 02:38:43.125307 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:43.125229 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/b9b81360-170a-4e01-af3e-8fd2947ec5a6-client-ca-bundle\") pod \"metrics-server-7495b9d698-s64vm\" (UID: \"b9b81360-170a-4e01-af3e-8fd2947ec5a6\") " pod="openshift-monitoring/metrics-server-7495b9d698-s64vm" Jul 22 02:38:43.125307 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:43.125249 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/b9b81360-170a-4e01-af3e-8fd2947ec5a6-secret-metrics-server-tls\") pod \"metrics-server-7495b9d698-s64vm\" (UID: \"b9b81360-170a-4e01-af3e-8fd2947ec5a6\") " pod="openshift-monitoring/metrics-server-7495b9d698-s64vm" Jul 22 02:38:43.125384 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:43.125335 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mx59c\" (UniqueName: \"kubernetes.io/projected/b9b81360-170a-4e01-af3e-8fd2947ec5a6-kube-api-access-mx59c\") pod \"metrics-server-7495b9d698-s64vm\" (UID: \"b9b81360-170a-4e01-af3e-8fd2947ec5a6\") " pod="openshift-monitoring/metrics-server-7495b9d698-s64vm" Jul 22 02:38:43.125384 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:43.125369 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/b9b81360-170a-4e01-af3e-8fd2947ec5a6-secret-metrics-server-client-certs\") pod \"metrics-server-7495b9d698-s64vm\" (UID: \"b9b81360-170a-4e01-af3e-8fd2947ec5a6\") " pod="openshift-monitoring/metrics-server-7495b9d698-s64vm" Jul 22 02:38:43.125474 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:43.125386 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b9b81360-170a-4e01-af3e-8fd2947ec5a6-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7495b9d698-s64vm\" (UID: \"b9b81360-170a-4e01-af3e-8fd2947ec5a6\") " pod="openshift-monitoring/metrics-server-7495b9d698-s64vm" Jul 22 02:38:43.125474 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:43.125405 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/b9b81360-170a-4e01-af3e-8fd2947ec5a6-metrics-server-audit-profiles\") pod \"metrics-server-7495b9d698-s64vm\" (UID: \"b9b81360-170a-4e01-af3e-8fd2947ec5a6\") " pod="openshift-monitoring/metrics-server-7495b9d698-s64vm" Jul 22 02:38:43.125613 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:43.125593 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/b9b81360-170a-4e01-af3e-8fd2947ec5a6-audit-log\") pod \"metrics-server-7495b9d698-s64vm\" (UID: \"b9b81360-170a-4e01-af3e-8fd2947ec5a6\") " pod="openshift-monitoring/metrics-server-7495b9d698-s64vm" Jul 22 02:38:43.126112 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:43.126085 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b9b81360-170a-4e01-af3e-8fd2947ec5a6-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7495b9d698-s64vm\" (UID: \"b9b81360-170a-4e01-af3e-8fd2947ec5a6\") " pod="openshift-monitoring/metrics-server-7495b9d698-s64vm" Jul 22 02:38:43.126372 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:43.126353 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/b9b81360-170a-4e01-af3e-8fd2947ec5a6-metrics-server-audit-profiles\") pod \"metrics-server-7495b9d698-s64vm\" (UID: \"b9b81360-170a-4e01-af3e-8fd2947ec5a6\") " pod="openshift-monitoring/metrics-server-7495b9d698-s64vm" Jul 22 02:38:43.127646 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:43.127623 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/b9b81360-170a-4e01-af3e-8fd2947ec5a6-secret-metrics-server-tls\") pod \"metrics-server-7495b9d698-s64vm\" (UID: \"b9b81360-170a-4e01-af3e-8fd2947ec5a6\") " pod="openshift-monitoring/metrics-server-7495b9d698-s64vm" Jul 22 02:38:43.127832 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:43.127815 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/b9b81360-170a-4e01-af3e-8fd2947ec5a6-client-ca-bundle\") pod \"metrics-server-7495b9d698-s64vm\" (UID: \"b9b81360-170a-4e01-af3e-8fd2947ec5a6\") " pod="openshift-monitoring/metrics-server-7495b9d698-s64vm" Jul 22 02:38:43.128400 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:43.128352 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/b9b81360-170a-4e01-af3e-8fd2947ec5a6-secret-metrics-server-client-certs\") pod \"metrics-server-7495b9d698-s64vm\" (UID: \"b9b81360-170a-4e01-af3e-8fd2947ec5a6\") " pod="openshift-monitoring/metrics-server-7495b9d698-s64vm" Jul 22 02:38:43.134232 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:43.134209 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mx59c\" (UniqueName: \"kubernetes.io/projected/b9b81360-170a-4e01-af3e-8fd2947ec5a6-kube-api-access-mx59c\") pod \"metrics-server-7495b9d698-s64vm\" (UID: \"b9b81360-170a-4e01-af3e-8fd2947ec5a6\") " pod="openshift-monitoring/metrics-server-7495b9d698-s64vm" Jul 22 02:38:43.188236 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:43.188211 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-7495b9d698-s64vm" Jul 22 02:38:43.326196 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:43.326173 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-7495b9d698-s64vm"] Jul 22 02:38:43.327822 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:38:43.327795 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9b81360_170a_4e01_af3e_8fd2947ec5a6.slice/crio-e1bdca61360ee595304d125a38beb8665d70d05297499be48c11f52c2dfdc4d2 WatchSource:0}: Error finding container e1bdca61360ee595304d125a38beb8665d70d05297499be48c11f52c2dfdc4d2: Status 404 returned error can't find the container with id e1bdca61360ee595304d125a38beb8665d70d05297499be48c11f52c2dfdc4d2 Jul 22 02:38:43.509783 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:43.509732 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-7495b9d698-s64vm" event={"ID":"b9b81360-170a-4e01-af3e-8fd2947ec5a6","Type":"ContainerStarted","Data":"e1bdca61360ee595304d125a38beb8665d70d05297499be48c11f52c2dfdc4d2"} Jul 22 02:38:45.518529 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:45.518492 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-7495b9d698-s64vm" event={"ID":"b9b81360-170a-4e01-af3e-8fd2947ec5a6","Type":"ContainerStarted","Data":"3dc0349d9c5d64f59bd9baa093d80f2b10a8f97f2661b08c77444130923d2b37"} Jul 22 02:38:45.538321 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:45.538279 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-7495b9d698-s64vm" podStartSLOduration=2.311732515 podStartE2EDuration="3.538265963s" totalImagesPullingTime="1.226533448s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 02:38:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:38:45.537053138 +0000 UTC m=+91.856228437" watchObservedRunningTime="2026-07-22 02:38:45.538265963 +0000 UTC m=+91.857441263" Jul 22 02:38:46.084318 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:46.084288 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-76b6d645f6-j7z5j"] Jul 22 02:38:46.086239 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:46.086225 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-76b6d645f6-j7z5j" Jul 22 02:38:46.094743 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:46.094717 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 22 02:38:46.102040 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:46.102019 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-76b6d645f6-j7z5j"] Jul 22 02:38:46.247931 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:46.247904 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b2d88106-06d5-447f-bdf7-bbd6ca28e68c-service-ca\") pod \"console-76b6d645f6-j7z5j\" (UID: \"b2d88106-06d5-447f-bdf7-bbd6ca28e68c\") " pod="openshift-console/console-76b6d645f6-j7z5j" Jul 22 02:38:46.247931 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:46.247935 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zsdd4\" (UniqueName: \"kubernetes.io/projected/b2d88106-06d5-447f-bdf7-bbd6ca28e68c-kube-api-access-zsdd4\") pod \"console-76b6d645f6-j7z5j\" (UID: \"b2d88106-06d5-447f-bdf7-bbd6ca28e68c\") " pod="openshift-console/console-76b6d645f6-j7z5j" Jul 22 02:38:46.248139 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:46.247958 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b2d88106-06d5-447f-bdf7-bbd6ca28e68c-console-config\") pod \"console-76b6d645f6-j7z5j\" (UID: \"b2d88106-06d5-447f-bdf7-bbd6ca28e68c\") " pod="openshift-console/console-76b6d645f6-j7z5j" Jul 22 02:38:46.248139 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:46.247989 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b2d88106-06d5-447f-bdf7-bbd6ca28e68c-console-oauth-config\") pod \"console-76b6d645f6-j7z5j\" (UID: \"b2d88106-06d5-447f-bdf7-bbd6ca28e68c\") " pod="openshift-console/console-76b6d645f6-j7z5j" Jul 22 02:38:46.248139 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:46.248066 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b2d88106-06d5-447f-bdf7-bbd6ca28e68c-console-serving-cert\") pod \"console-76b6d645f6-j7z5j\" (UID: \"b2d88106-06d5-447f-bdf7-bbd6ca28e68c\") " pod="openshift-console/console-76b6d645f6-j7z5j" Jul 22 02:38:46.248139 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:46.248096 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b2d88106-06d5-447f-bdf7-bbd6ca28e68c-trusted-ca-bundle\") pod \"console-76b6d645f6-j7z5j\" (UID: \"b2d88106-06d5-447f-bdf7-bbd6ca28e68c\") " pod="openshift-console/console-76b6d645f6-j7z5j" Jul 22 02:38:46.248139 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:46.248119 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b2d88106-06d5-447f-bdf7-bbd6ca28e68c-oauth-serving-cert\") pod \"console-76b6d645f6-j7z5j\" (UID: \"b2d88106-06d5-447f-bdf7-bbd6ca28e68c\") " pod="openshift-console/console-76b6d645f6-j7z5j" Jul 22 02:38:46.349253 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:46.349192 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b2d88106-06d5-447f-bdf7-bbd6ca28e68c-oauth-serving-cert\") pod \"console-76b6d645f6-j7z5j\" (UID: \"b2d88106-06d5-447f-bdf7-bbd6ca28e68c\") " pod="openshift-console/console-76b6d645f6-j7z5j" Jul 22 02:38:46.349253 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:46.349225 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b2d88106-06d5-447f-bdf7-bbd6ca28e68c-service-ca\") pod \"console-76b6d645f6-j7z5j\" (UID: \"b2d88106-06d5-447f-bdf7-bbd6ca28e68c\") " pod="openshift-console/console-76b6d645f6-j7z5j" Jul 22 02:38:46.349253 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:46.349247 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zsdd4\" (UniqueName: \"kubernetes.io/projected/b2d88106-06d5-447f-bdf7-bbd6ca28e68c-kube-api-access-zsdd4\") pod \"console-76b6d645f6-j7z5j\" (UID: \"b2d88106-06d5-447f-bdf7-bbd6ca28e68c\") " pod="openshift-console/console-76b6d645f6-j7z5j" Jul 22 02:38:46.349476 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:46.349277 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b2d88106-06d5-447f-bdf7-bbd6ca28e68c-console-config\") pod \"console-76b6d645f6-j7z5j\" (UID: \"b2d88106-06d5-447f-bdf7-bbd6ca28e68c\") " pod="openshift-console/console-76b6d645f6-j7z5j" Jul 22 02:38:46.349476 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:46.349317 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b2d88106-06d5-447f-bdf7-bbd6ca28e68c-console-oauth-config\") pod \"console-76b6d645f6-j7z5j\" (UID: \"b2d88106-06d5-447f-bdf7-bbd6ca28e68c\") " pod="openshift-console/console-76b6d645f6-j7z5j" Jul 22 02:38:46.349476 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:46.349361 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b2d88106-06d5-447f-bdf7-bbd6ca28e68c-console-serving-cert\") pod \"console-76b6d645f6-j7z5j\" (UID: \"b2d88106-06d5-447f-bdf7-bbd6ca28e68c\") " pod="openshift-console/console-76b6d645f6-j7z5j" Jul 22 02:38:46.349476 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:46.349389 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b2d88106-06d5-447f-bdf7-bbd6ca28e68c-trusted-ca-bundle\") pod \"console-76b6d645f6-j7z5j\" (UID: \"b2d88106-06d5-447f-bdf7-bbd6ca28e68c\") " pod="openshift-console/console-76b6d645f6-j7z5j" Jul 22 02:38:46.349992 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:46.349973 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b2d88106-06d5-447f-bdf7-bbd6ca28e68c-console-config\") pod \"console-76b6d645f6-j7z5j\" (UID: \"b2d88106-06d5-447f-bdf7-bbd6ca28e68c\") " pod="openshift-console/console-76b6d645f6-j7z5j" Jul 22 02:38:46.350082 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:46.350007 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b2d88106-06d5-447f-bdf7-bbd6ca28e68c-oauth-serving-cert\") pod \"console-76b6d645f6-j7z5j\" (UID: \"b2d88106-06d5-447f-bdf7-bbd6ca28e68c\") " pod="openshift-console/console-76b6d645f6-j7z5j" Jul 22 02:38:46.350134 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:46.350088 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b2d88106-06d5-447f-bdf7-bbd6ca28e68c-service-ca\") pod \"console-76b6d645f6-j7z5j\" (UID: \"b2d88106-06d5-447f-bdf7-bbd6ca28e68c\") " pod="openshift-console/console-76b6d645f6-j7z5j" Jul 22 02:38:46.350182 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:46.350166 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b2d88106-06d5-447f-bdf7-bbd6ca28e68c-trusted-ca-bundle\") pod \"console-76b6d645f6-j7z5j\" (UID: \"b2d88106-06d5-447f-bdf7-bbd6ca28e68c\") " pod="openshift-console/console-76b6d645f6-j7z5j" Jul 22 02:38:46.351934 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:46.351901 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b2d88106-06d5-447f-bdf7-bbd6ca28e68c-console-oauth-config\") pod \"console-76b6d645f6-j7z5j\" (UID: \"b2d88106-06d5-447f-bdf7-bbd6ca28e68c\") " pod="openshift-console/console-76b6d645f6-j7z5j" Jul 22 02:38:46.352081 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:46.352063 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b2d88106-06d5-447f-bdf7-bbd6ca28e68c-console-serving-cert\") pod \"console-76b6d645f6-j7z5j\" (UID: \"b2d88106-06d5-447f-bdf7-bbd6ca28e68c\") " pod="openshift-console/console-76b6d645f6-j7z5j" Jul 22 02:38:46.359119 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:46.359094 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zsdd4\" (UniqueName: \"kubernetes.io/projected/b2d88106-06d5-447f-bdf7-bbd6ca28e68c-kube-api-access-zsdd4\") pod \"console-76b6d645f6-j7z5j\" (UID: \"b2d88106-06d5-447f-bdf7-bbd6ca28e68c\") " pod="openshift-console/console-76b6d645f6-j7z5j" Jul 22 02:38:46.394800 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:46.394779 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-76b6d645f6-j7z5j" Jul 22 02:38:46.524131 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:46.524103 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-76b6d645f6-j7z5j"] Jul 22 02:38:46.527740 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:38:46.527713 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2d88106_06d5_447f_bdf7_bbd6ca28e68c.slice/crio-370a2e5a476808ccb2cdc38fd0afa68fd5ec4e3b560eb45f4a1384d6ccc0efe8 WatchSource:0}: Error finding container 370a2e5a476808ccb2cdc38fd0afa68fd5ec4e3b560eb45f4a1384d6ccc0efe8: Status 404 returned error can't find the container with id 370a2e5a476808ccb2cdc38fd0afa68fd5ec4e3b560eb45f4a1384d6ccc0efe8 Jul 22 02:38:47.433162 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:47.433137 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-d469874bc-bnc45" Jul 22 02:38:47.528715 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:47.528684 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-76b6d645f6-j7z5j" event={"ID":"b2d88106-06d5-447f-bdf7-bbd6ca28e68c","Type":"ContainerStarted","Data":"e96bd8aba85425f657334af1731ca2b40e84ef11049824d06f1378abfc18855d"} Jul 22 02:38:47.529032 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:47.528733 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-76b6d645f6-j7z5j" event={"ID":"b2d88106-06d5-447f-bdf7-bbd6ca28e68c","Type":"ContainerStarted","Data":"370a2e5a476808ccb2cdc38fd0afa68fd5ec4e3b560eb45f4a1384d6ccc0efe8"} Jul 22 02:38:47.548410 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:47.548373 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-76b6d645f6-j7z5j" podStartSLOduration=1.548359472 podStartE2EDuration="1.548359472s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 02:38:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:38:47.547240772 +0000 UTC m=+93.866416070" watchObservedRunningTime="2026-07-22 02:38:47.548359472 +0000 UTC m=+93.867534770" Jul 22 02:38:48.445274 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:48.445250 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-7cf654f645-clg89" Jul 22 02:38:52.448878 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:52.448838 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-d469874bc-bnc45" podUID="08aea90a-8858-4443-a3cc-34e6d72a705c" containerName="registry" containerID="cri-o://ead01b724705c51553476914bb3cb18ca2f482172292ec8c4dc61ea200fe507e" gracePeriod=30 Jul 22 02:38:53.236969 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:53.236915 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-d469874bc-bnc45" Jul 22 02:38:53.402195 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:53.402168 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/08aea90a-8858-4443-a3cc-34e6d72a705c-registry-certificates\") pod \"08aea90a-8858-4443-a3cc-34e6d72a705c\" (UID: \"08aea90a-8858-4443-a3cc-34e6d72a705c\") " Jul 22 02:38:53.402364 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:53.402203 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/08aea90a-8858-4443-a3cc-34e6d72a705c-ca-trust-extracted\") pod \"08aea90a-8858-4443-a3cc-34e6d72a705c\" (UID: \"08aea90a-8858-4443-a3cc-34e6d72a705c\") " Jul 22 02:38:53.402364 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:53.402221 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/08aea90a-8858-4443-a3cc-34e6d72a705c-trusted-ca\") pod \"08aea90a-8858-4443-a3cc-34e6d72a705c\" (UID: \"08aea90a-8858-4443-a3cc-34e6d72a705c\") " Jul 22 02:38:53.402364 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:53.402242 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/08aea90a-8858-4443-a3cc-34e6d72a705c-registry-tls\") pod \"08aea90a-8858-4443-a3cc-34e6d72a705c\" (UID: \"08aea90a-8858-4443-a3cc-34e6d72a705c\") " Jul 22 02:38:53.402364 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:53.402263 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nwdff\" (UniqueName: \"kubernetes.io/projected/08aea90a-8858-4443-a3cc-34e6d72a705c-kube-api-access-nwdff\") pod \"08aea90a-8858-4443-a3cc-34e6d72a705c\" (UID: \"08aea90a-8858-4443-a3cc-34e6d72a705c\") " Jul 22 02:38:53.402600 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:53.402407 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/08aea90a-8858-4443-a3cc-34e6d72a705c-installation-pull-secrets\") pod \"08aea90a-8858-4443-a3cc-34e6d72a705c\" (UID: \"08aea90a-8858-4443-a3cc-34e6d72a705c\") " Jul 22 02:38:53.402600 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:53.402482 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/08aea90a-8858-4443-a3cc-34e6d72a705c-image-registry-private-configuration\") pod \"08aea90a-8858-4443-a3cc-34e6d72a705c\" (UID: \"08aea90a-8858-4443-a3cc-34e6d72a705c\") " Jul 22 02:38:53.402600 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:53.402543 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/08aea90a-8858-4443-a3cc-34e6d72a705c-bound-sa-token\") pod \"08aea90a-8858-4443-a3cc-34e6d72a705c\" (UID: \"08aea90a-8858-4443-a3cc-34e6d72a705c\") " Jul 22 02:38:53.402600 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:53.402580 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08aea90a-8858-4443-a3cc-34e6d72a705c-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "08aea90a-8858-4443-a3cc-34e6d72a705c" (UID: "08aea90a-8858-4443-a3cc-34e6d72a705c"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 02:38:53.402802 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:53.402682 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08aea90a-8858-4443-a3cc-34e6d72a705c-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "08aea90a-8858-4443-a3cc-34e6d72a705c" (UID: "08aea90a-8858-4443-a3cc-34e6d72a705c"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 02:38:53.402858 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:53.402814 2578 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/08aea90a-8858-4443-a3cc-34e6d72a705c-registry-certificates\") on node \"ip-10-0-133-31.ec2.internal\" DevicePath \"\"" Jul 22 02:38:53.402858 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:53.402832 2578 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/08aea90a-8858-4443-a3cc-34e6d72a705c-trusted-ca\") on node \"ip-10-0-133-31.ec2.internal\" DevicePath \"\"" Jul 22 02:38:53.405195 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:53.405167 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08aea90a-8858-4443-a3cc-34e6d72a705c-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "08aea90a-8858-4443-a3cc-34e6d72a705c" (UID: "08aea90a-8858-4443-a3cc-34e6d72a705c"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 02:38:53.405195 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:53.405173 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/08aea90a-8858-4443-a3cc-34e6d72a705c-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "08aea90a-8858-4443-a3cc-34e6d72a705c" (UID: "08aea90a-8858-4443-a3cc-34e6d72a705c"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 02:38:53.405368 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:53.405243 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08aea90a-8858-4443-a3cc-34e6d72a705c-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "08aea90a-8858-4443-a3cc-34e6d72a705c" (UID: "08aea90a-8858-4443-a3cc-34e6d72a705c"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 02:38:53.405368 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:53.405292 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08aea90a-8858-4443-a3cc-34e6d72a705c-kube-api-access-nwdff" (OuterVolumeSpecName: "kube-api-access-nwdff") pod "08aea90a-8858-4443-a3cc-34e6d72a705c" (UID: "08aea90a-8858-4443-a3cc-34e6d72a705c"). InnerVolumeSpecName "kube-api-access-nwdff". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 02:38:53.405471 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:53.405391 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/08aea90a-8858-4443-a3cc-34e6d72a705c-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "08aea90a-8858-4443-a3cc-34e6d72a705c" (UID: "08aea90a-8858-4443-a3cc-34e6d72a705c"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 02:38:53.410055 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:53.410033 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08aea90a-8858-4443-a3cc-34e6d72a705c-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "08aea90a-8858-4443-a3cc-34e6d72a705c" (UID: "08aea90a-8858-4443-a3cc-34e6d72a705c"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 02:38:53.504078 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:53.504029 2578 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/08aea90a-8858-4443-a3cc-34e6d72a705c-bound-sa-token\") on node \"ip-10-0-133-31.ec2.internal\" DevicePath \"\"" Jul 22 02:38:53.504078 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:53.504048 2578 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/08aea90a-8858-4443-a3cc-34e6d72a705c-ca-trust-extracted\") on node \"ip-10-0-133-31.ec2.internal\" DevicePath \"\"" Jul 22 02:38:53.504078 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:53.504058 2578 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/08aea90a-8858-4443-a3cc-34e6d72a705c-registry-tls\") on node \"ip-10-0-133-31.ec2.internal\" DevicePath \"\"" Jul 22 02:38:53.504078 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:53.504066 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nwdff\" (UniqueName: \"kubernetes.io/projected/08aea90a-8858-4443-a3cc-34e6d72a705c-kube-api-access-nwdff\") on node \"ip-10-0-133-31.ec2.internal\" DevicePath \"\"" Jul 22 02:38:53.504078 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:53.504075 2578 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/08aea90a-8858-4443-a3cc-34e6d72a705c-installation-pull-secrets\") on node \"ip-10-0-133-31.ec2.internal\" DevicePath \"\"" Jul 22 02:38:53.504456 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:53.504085 2578 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/08aea90a-8858-4443-a3cc-34e6d72a705c-image-registry-private-configuration\") on node \"ip-10-0-133-31.ec2.internal\" DevicePath \"\"" Jul 22 02:38:53.550119 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:53.550097 2578 generic.go:356] "Generic (PLEG): container finished" podID="08aea90a-8858-4443-a3cc-34e6d72a705c" containerID="ead01b724705c51553476914bb3cb18ca2f482172292ec8c4dc61ea200fe507e" exitCode=0 Jul 22 02:38:53.550255 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:53.550132 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-d469874bc-bnc45" event={"ID":"08aea90a-8858-4443-a3cc-34e6d72a705c","Type":"ContainerDied","Data":"ead01b724705c51553476914bb3cb18ca2f482172292ec8c4dc61ea200fe507e"} Jul 22 02:38:53.550255 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:53.550141 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-d469874bc-bnc45" Jul 22 02:38:53.550255 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:53.550169 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-d469874bc-bnc45" event={"ID":"08aea90a-8858-4443-a3cc-34e6d72a705c","Type":"ContainerDied","Data":"3f2eaa81ccce44b34c2b1f95293fba87a4f7f7dce389efa258fa3b6e1f48a8b7"} Jul 22 02:38:53.551762 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:53.551744 2578 scope.go:117] "RemoveContainer" containerID="ead01b724705c51553476914bb3cb18ca2f482172292ec8c4dc61ea200fe507e" Jul 22 02:38:53.560281 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:53.560262 2578 scope.go:117] "RemoveContainer" containerID="ead01b724705c51553476914bb3cb18ca2f482172292ec8c4dc61ea200fe507e" Jul 22 02:38:53.560543 ip-10-0-133-31 kubenswrapper[2578]: E0722 02:38:53.560524 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ead01b724705c51553476914bb3cb18ca2f482172292ec8c4dc61ea200fe507e\": container with ID starting with ead01b724705c51553476914bb3cb18ca2f482172292ec8c4dc61ea200fe507e not found: ID does not exist" containerID="ead01b724705c51553476914bb3cb18ca2f482172292ec8c4dc61ea200fe507e" Jul 22 02:38:53.560621 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:53.560550 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ead01b724705c51553476914bb3cb18ca2f482172292ec8c4dc61ea200fe507e"} err="failed to get container status \"ead01b724705c51553476914bb3cb18ca2f482172292ec8c4dc61ea200fe507e\": rpc error: code = NotFound desc = could not find container \"ead01b724705c51553476914bb3cb18ca2f482172292ec8c4dc61ea200fe507e\": container with ID starting with ead01b724705c51553476914bb3cb18ca2f482172292ec8c4dc61ea200fe507e not found: ID does not exist" Jul 22 02:38:53.572838 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:53.572814 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-d469874bc-bnc45"] Jul 22 02:38:53.579830 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:53.579810 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-d469874bc-bnc45"] Jul 22 02:38:54.139848 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:54.139821 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="08aea90a-8858-4443-a3cc-34e6d72a705c" path="/var/lib/kubelet/pods/08aea90a-8858-4443-a3cc-34e6d72a705c/volumes" Jul 22 02:38:56.395351 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:56.395315 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-76b6d645f6-j7z5j" Jul 22 02:38:56.395735 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:56.395414 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-76b6d645f6-j7z5j" Jul 22 02:38:56.400113 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:56.400090 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-76b6d645f6-j7z5j" Jul 22 02:38:56.563961 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:56.563938 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-76b6d645f6-j7z5j" Jul 22 02:38:56.617191 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:56.617165 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-858cc4d8c5-85zzs"] Jul 22 02:38:58.442202 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:38:58.442179 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-wp7zw" Jul 22 02:39:03.189314 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:39:03.189274 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-7495b9d698-s64vm" Jul 22 02:39:03.189811 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:39:03.189350 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-7495b9d698-s64vm" Jul 22 02:39:21.637583 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:39:21.637505 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-858cc4d8c5-85zzs" podUID="2fefb9b9-18bd-41b3-807a-21131181d0f1" containerName="console" containerID="cri-o://d0ab20818f489af04adc61d1ad5324b8c3db03dcff2cb5e0d66534fe840f6fd1" gracePeriod=15 Jul 22 02:39:21.868853 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:39:21.868832 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-858cc4d8c5-85zzs" Jul 22 02:39:21.894168 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:39:21.893939 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2fefb9b9-18bd-41b3-807a-21131181d0f1-console-serving-cert\") pod \"2fefb9b9-18bd-41b3-807a-21131181d0f1\" (UID: \"2fefb9b9-18bd-41b3-807a-21131181d0f1\") " Jul 22 02:39:21.894168 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:39:21.893987 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xjmtt\" (UniqueName: \"kubernetes.io/projected/2fefb9b9-18bd-41b3-807a-21131181d0f1-kube-api-access-xjmtt\") pod \"2fefb9b9-18bd-41b3-807a-21131181d0f1\" (UID: \"2fefb9b9-18bd-41b3-807a-21131181d0f1\") " Jul 22 02:39:21.894168 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:39:21.894009 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2fefb9b9-18bd-41b3-807a-21131181d0f1-console-oauth-config\") pod \"2fefb9b9-18bd-41b3-807a-21131181d0f1\" (UID: \"2fefb9b9-18bd-41b3-807a-21131181d0f1\") " Jul 22 02:39:21.894168 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:39:21.894057 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2fefb9b9-18bd-41b3-807a-21131181d0f1-oauth-serving-cert\") pod \"2fefb9b9-18bd-41b3-807a-21131181d0f1\" (UID: \"2fefb9b9-18bd-41b3-807a-21131181d0f1\") " Jul 22 02:39:21.894168 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:39:21.894100 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2fefb9b9-18bd-41b3-807a-21131181d0f1-service-ca\") pod \"2fefb9b9-18bd-41b3-807a-21131181d0f1\" (UID: \"2fefb9b9-18bd-41b3-807a-21131181d0f1\") " Jul 22 02:39:21.894168 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:39:21.894145 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2fefb9b9-18bd-41b3-807a-21131181d0f1-console-config\") pod \"2fefb9b9-18bd-41b3-807a-21131181d0f1\" (UID: \"2fefb9b9-18bd-41b3-807a-21131181d0f1\") " Jul 22 02:39:21.894781 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:39:21.894676 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2fefb9b9-18bd-41b3-807a-21131181d0f1-console-config" (OuterVolumeSpecName: "console-config") pod "2fefb9b9-18bd-41b3-807a-21131181d0f1" (UID: "2fefb9b9-18bd-41b3-807a-21131181d0f1"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 02:39:21.894929 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:39:21.894899 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2fefb9b9-18bd-41b3-807a-21131181d0f1-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "2fefb9b9-18bd-41b3-807a-21131181d0f1" (UID: "2fefb9b9-18bd-41b3-807a-21131181d0f1"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 02:39:21.895087 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:39:21.894997 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2fefb9b9-18bd-41b3-807a-21131181d0f1-service-ca" (OuterVolumeSpecName: "service-ca") pod "2fefb9b9-18bd-41b3-807a-21131181d0f1" (UID: "2fefb9b9-18bd-41b3-807a-21131181d0f1"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 02:39:21.896375 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:39:21.896341 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2fefb9b9-18bd-41b3-807a-21131181d0f1-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "2fefb9b9-18bd-41b3-807a-21131181d0f1" (UID: "2fefb9b9-18bd-41b3-807a-21131181d0f1"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 02:39:21.896537 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:39:21.896515 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2fefb9b9-18bd-41b3-807a-21131181d0f1-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "2fefb9b9-18bd-41b3-807a-21131181d0f1" (UID: "2fefb9b9-18bd-41b3-807a-21131181d0f1"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 02:39:21.896600 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:39:21.896556 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2fefb9b9-18bd-41b3-807a-21131181d0f1-kube-api-access-xjmtt" (OuterVolumeSpecName: "kube-api-access-xjmtt") pod "2fefb9b9-18bd-41b3-807a-21131181d0f1" (UID: "2fefb9b9-18bd-41b3-807a-21131181d0f1"). InnerVolumeSpecName "kube-api-access-xjmtt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 02:39:21.995412 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:39:21.995389 2578 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2fefb9b9-18bd-41b3-807a-21131181d0f1-console-config\") on node \"ip-10-0-133-31.ec2.internal\" DevicePath \"\"" Jul 22 02:39:21.995412 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:39:21.995409 2578 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2fefb9b9-18bd-41b3-807a-21131181d0f1-console-serving-cert\") on node \"ip-10-0-133-31.ec2.internal\" DevicePath \"\"" Jul 22 02:39:21.995526 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:39:21.995419 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xjmtt\" (UniqueName: \"kubernetes.io/projected/2fefb9b9-18bd-41b3-807a-21131181d0f1-kube-api-access-xjmtt\") on node \"ip-10-0-133-31.ec2.internal\" DevicePath \"\"" Jul 22 02:39:21.995526 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:39:21.995428 2578 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2fefb9b9-18bd-41b3-807a-21131181d0f1-console-oauth-config\") on node \"ip-10-0-133-31.ec2.internal\" DevicePath \"\"" Jul 22 02:39:21.995526 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:39:21.995436 2578 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2fefb9b9-18bd-41b3-807a-21131181d0f1-oauth-serving-cert\") on node \"ip-10-0-133-31.ec2.internal\" DevicePath \"\"" Jul 22 02:39:21.995526 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:39:21.995444 2578 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2fefb9b9-18bd-41b3-807a-21131181d0f1-service-ca\") on node \"ip-10-0-133-31.ec2.internal\" DevicePath \"\"" Jul 22 02:39:22.646391 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:39:22.646354 2578 generic.go:356] "Generic (PLEG): container finished" podID="2fefb9b9-18bd-41b3-807a-21131181d0f1" containerID="d0ab20818f489af04adc61d1ad5324b8c3db03dcff2cb5e0d66534fe840f6fd1" exitCode=0 Jul 22 02:39:22.646772 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:39:22.646402 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-858cc4d8c5-85zzs" Jul 22 02:39:22.646772 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:39:22.646419 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-858cc4d8c5-85zzs" event={"ID":"2fefb9b9-18bd-41b3-807a-21131181d0f1","Type":"ContainerDied","Data":"d0ab20818f489af04adc61d1ad5324b8c3db03dcff2cb5e0d66534fe840f6fd1"} Jul 22 02:39:22.646772 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:39:22.646479 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-858cc4d8c5-85zzs" event={"ID":"2fefb9b9-18bd-41b3-807a-21131181d0f1","Type":"ContainerDied","Data":"32795d816830973e7d618201280500c8bbc269ec9a35e532ea2a3b89012c2a5f"} Jul 22 02:39:22.646772 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:39:22.646498 2578 scope.go:117] "RemoveContainer" containerID="d0ab20818f489af04adc61d1ad5324b8c3db03dcff2cb5e0d66534fe840f6fd1" Jul 22 02:39:22.654118 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:39:22.654102 2578 scope.go:117] "RemoveContainer" containerID="d0ab20818f489af04adc61d1ad5324b8c3db03dcff2cb5e0d66534fe840f6fd1" Jul 22 02:39:22.654337 ip-10-0-133-31 kubenswrapper[2578]: E0722 02:39:22.654320 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d0ab20818f489af04adc61d1ad5324b8c3db03dcff2cb5e0d66534fe840f6fd1\": container with ID starting with d0ab20818f489af04adc61d1ad5324b8c3db03dcff2cb5e0d66534fe840f6fd1 not found: ID does not exist" containerID="d0ab20818f489af04adc61d1ad5324b8c3db03dcff2cb5e0d66534fe840f6fd1" Jul 22 02:39:22.654378 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:39:22.654345 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d0ab20818f489af04adc61d1ad5324b8c3db03dcff2cb5e0d66534fe840f6fd1"} err="failed to get container status \"d0ab20818f489af04adc61d1ad5324b8c3db03dcff2cb5e0d66534fe840f6fd1\": rpc error: code = NotFound desc = could not find container \"d0ab20818f489af04adc61d1ad5324b8c3db03dcff2cb5e0d66534fe840f6fd1\": container with ID starting with d0ab20818f489af04adc61d1ad5324b8c3db03dcff2cb5e0d66534fe840f6fd1 not found: ID does not exist" Jul 22 02:39:22.664345 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:39:22.664321 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-858cc4d8c5-85zzs"] Jul 22 02:39:22.666470 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:39:22.666454 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-858cc4d8c5-85zzs"] Jul 22 02:39:23.194206 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:39:23.194181 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-7495b9d698-s64vm" Jul 22 02:39:23.197802 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:39:23.197783 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-7495b9d698-s64vm" Jul 22 02:39:24.139734 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:39:24.139708 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2fefb9b9-18bd-41b3-807a-21131181d0f1" path="/var/lib/kubelet/pods/2fefb9b9-18bd-41b3-807a-21131181d0f1/volumes" Jul 22 02:40:08.065852 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:40:08.065751 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-69997f8c4b-9phsb"] Jul 22 02:40:08.066269 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:40:08.065981 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="08aea90a-8858-4443-a3cc-34e6d72a705c" containerName="registry" Jul 22 02:40:08.066269 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:40:08.065991 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="08aea90a-8858-4443-a3cc-34e6d72a705c" containerName="registry" Jul 22 02:40:08.066269 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:40:08.066002 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2fefb9b9-18bd-41b3-807a-21131181d0f1" containerName="console" Jul 22 02:40:08.066269 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:40:08.066008 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="2fefb9b9-18bd-41b3-807a-21131181d0f1" containerName="console" Jul 22 02:40:08.066269 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:40:08.066057 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="2fefb9b9-18bd-41b3-807a-21131181d0f1" containerName="console" Jul 22 02:40:08.066269 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:40:08.066064 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="08aea90a-8858-4443-a3cc-34e6d72a705c" containerName="registry" Jul 22 02:40:08.068848 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:40:08.068832 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-69997f8c4b-9phsb" Jul 22 02:40:08.084743 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:40:08.084721 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-69997f8c4b-9phsb"] Jul 22 02:40:08.208090 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:40:08.208052 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5463bf38-361c-4ca0-9fe2-078e86072035-oauth-serving-cert\") pod \"console-69997f8c4b-9phsb\" (UID: \"5463bf38-361c-4ca0-9fe2-078e86072035\") " pod="openshift-console/console-69997f8c4b-9phsb" Jul 22 02:40:08.208090 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:40:08.208096 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2bll8\" (UniqueName: \"kubernetes.io/projected/5463bf38-361c-4ca0-9fe2-078e86072035-kube-api-access-2bll8\") pod \"console-69997f8c4b-9phsb\" (UID: \"5463bf38-361c-4ca0-9fe2-078e86072035\") " pod="openshift-console/console-69997f8c4b-9phsb" Jul 22 02:40:08.208286 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:40:08.208114 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5463bf38-361c-4ca0-9fe2-078e86072035-console-serving-cert\") pod \"console-69997f8c4b-9phsb\" (UID: \"5463bf38-361c-4ca0-9fe2-078e86072035\") " pod="openshift-console/console-69997f8c4b-9phsb" Jul 22 02:40:08.208286 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:40:08.208137 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5463bf38-361c-4ca0-9fe2-078e86072035-console-config\") pod \"console-69997f8c4b-9phsb\" (UID: \"5463bf38-361c-4ca0-9fe2-078e86072035\") " pod="openshift-console/console-69997f8c4b-9phsb" Jul 22 02:40:08.208286 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:40:08.208163 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5463bf38-361c-4ca0-9fe2-078e86072035-console-oauth-config\") pod \"console-69997f8c4b-9phsb\" (UID: \"5463bf38-361c-4ca0-9fe2-078e86072035\") " pod="openshift-console/console-69997f8c4b-9phsb" Jul 22 02:40:08.208286 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:40:08.208183 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5463bf38-361c-4ca0-9fe2-078e86072035-trusted-ca-bundle\") pod \"console-69997f8c4b-9phsb\" (UID: \"5463bf38-361c-4ca0-9fe2-078e86072035\") " pod="openshift-console/console-69997f8c4b-9phsb" Jul 22 02:40:08.208286 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:40:08.208216 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5463bf38-361c-4ca0-9fe2-078e86072035-service-ca\") pod \"console-69997f8c4b-9phsb\" (UID: \"5463bf38-361c-4ca0-9fe2-078e86072035\") " pod="openshift-console/console-69997f8c4b-9phsb" Jul 22 02:40:08.308904 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:40:08.308870 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5463bf38-361c-4ca0-9fe2-078e86072035-oauth-serving-cert\") pod \"console-69997f8c4b-9phsb\" (UID: \"5463bf38-361c-4ca0-9fe2-078e86072035\") " pod="openshift-console/console-69997f8c4b-9phsb" Jul 22 02:40:08.309047 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:40:08.308917 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2bll8\" (UniqueName: \"kubernetes.io/projected/5463bf38-361c-4ca0-9fe2-078e86072035-kube-api-access-2bll8\") pod \"console-69997f8c4b-9phsb\" (UID: \"5463bf38-361c-4ca0-9fe2-078e86072035\") " pod="openshift-console/console-69997f8c4b-9phsb" Jul 22 02:40:08.309047 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:40:08.308942 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5463bf38-361c-4ca0-9fe2-078e86072035-console-serving-cert\") pod \"console-69997f8c4b-9phsb\" (UID: \"5463bf38-361c-4ca0-9fe2-078e86072035\") " pod="openshift-console/console-69997f8c4b-9phsb" Jul 22 02:40:08.309047 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:40:08.308964 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5463bf38-361c-4ca0-9fe2-078e86072035-console-config\") pod \"console-69997f8c4b-9phsb\" (UID: \"5463bf38-361c-4ca0-9fe2-078e86072035\") " pod="openshift-console/console-69997f8c4b-9phsb" Jul 22 02:40:08.309047 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:40:08.308998 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5463bf38-361c-4ca0-9fe2-078e86072035-console-oauth-config\") pod \"console-69997f8c4b-9phsb\" (UID: \"5463bf38-361c-4ca0-9fe2-078e86072035\") " pod="openshift-console/console-69997f8c4b-9phsb" Jul 22 02:40:08.309047 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:40:08.309029 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5463bf38-361c-4ca0-9fe2-078e86072035-trusted-ca-bundle\") pod \"console-69997f8c4b-9phsb\" (UID: \"5463bf38-361c-4ca0-9fe2-078e86072035\") " pod="openshift-console/console-69997f8c4b-9phsb" Jul 22 02:40:08.309265 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:40:08.309195 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5463bf38-361c-4ca0-9fe2-078e86072035-service-ca\") pod \"console-69997f8c4b-9phsb\" (UID: \"5463bf38-361c-4ca0-9fe2-078e86072035\") " pod="openshift-console/console-69997f8c4b-9phsb" Jul 22 02:40:08.309771 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:40:08.309737 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5463bf38-361c-4ca0-9fe2-078e86072035-oauth-serving-cert\") pod \"console-69997f8c4b-9phsb\" (UID: \"5463bf38-361c-4ca0-9fe2-078e86072035\") " pod="openshift-console/console-69997f8c4b-9phsb" Jul 22 02:40:08.309895 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:40:08.309833 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5463bf38-361c-4ca0-9fe2-078e86072035-service-ca\") pod \"console-69997f8c4b-9phsb\" (UID: \"5463bf38-361c-4ca0-9fe2-078e86072035\") " pod="openshift-console/console-69997f8c4b-9phsb" Jul 22 02:40:08.309895 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:40:08.309869 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5463bf38-361c-4ca0-9fe2-078e86072035-trusted-ca-bundle\") pod \"console-69997f8c4b-9phsb\" (UID: \"5463bf38-361c-4ca0-9fe2-078e86072035\") " pod="openshift-console/console-69997f8c4b-9phsb" Jul 22 02:40:08.309983 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:40:08.309889 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5463bf38-361c-4ca0-9fe2-078e86072035-console-config\") pod \"console-69997f8c4b-9phsb\" (UID: \"5463bf38-361c-4ca0-9fe2-078e86072035\") " pod="openshift-console/console-69997f8c4b-9phsb" Jul 22 02:40:08.311517 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:40:08.311487 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5463bf38-361c-4ca0-9fe2-078e86072035-console-oauth-config\") pod \"console-69997f8c4b-9phsb\" (UID: \"5463bf38-361c-4ca0-9fe2-078e86072035\") " pod="openshift-console/console-69997f8c4b-9phsb" Jul 22 02:40:08.311626 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:40:08.311537 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5463bf38-361c-4ca0-9fe2-078e86072035-console-serving-cert\") pod \"console-69997f8c4b-9phsb\" (UID: \"5463bf38-361c-4ca0-9fe2-078e86072035\") " pod="openshift-console/console-69997f8c4b-9phsb" Jul 22 02:40:08.318196 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:40:08.318145 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2bll8\" (UniqueName: \"kubernetes.io/projected/5463bf38-361c-4ca0-9fe2-078e86072035-kube-api-access-2bll8\") pod \"console-69997f8c4b-9phsb\" (UID: \"5463bf38-361c-4ca0-9fe2-078e86072035\") " pod="openshift-console/console-69997f8c4b-9phsb" Jul 22 02:40:08.378087 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:40:08.378054 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-69997f8c4b-9phsb" Jul 22 02:40:08.504890 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:40:08.504869 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-69997f8c4b-9phsb"] Jul 22 02:40:08.506988 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:40:08.506960 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5463bf38_361c_4ca0_9fe2_078e86072035.slice/crio-dc28baa303078177573556f7069bb77443a1c074a04f45919f1ae367e71db7db WatchSource:0}: Error finding container dc28baa303078177573556f7069bb77443a1c074a04f45919f1ae367e71db7db: Status 404 returned error can't find the container with id dc28baa303078177573556f7069bb77443a1c074a04f45919f1ae367e71db7db Jul 22 02:40:08.803454 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:40:08.803416 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-69997f8c4b-9phsb" event={"ID":"5463bf38-361c-4ca0-9fe2-078e86072035","Type":"ContainerStarted","Data":"af93d6520e92439523c928665264f78a086c361d5cde8ae017eb1a06ba732f48"} Jul 22 02:40:08.803454 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:40:08.803459 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-69997f8c4b-9phsb" event={"ID":"5463bf38-361c-4ca0-9fe2-078e86072035","Type":"ContainerStarted","Data":"dc28baa303078177573556f7069bb77443a1c074a04f45919f1ae367e71db7db"} Jul 22 02:40:08.835708 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:40:08.835666 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-69997f8c4b-9phsb" podStartSLOduration=0.835650042 podStartE2EDuration="835.650042ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 02:40:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:40:08.834493508 +0000 UTC m=+175.153668820" watchObservedRunningTime="2026-07-22 02:40:08.835650042 +0000 UTC m=+175.154825341" Jul 22 02:40:18.378495 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:40:18.378462 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-69997f8c4b-9phsb" Jul 22 02:40:18.378930 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:40:18.378518 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-69997f8c4b-9phsb" Jul 22 02:40:18.382828 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:40:18.382812 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-69997f8c4b-9phsb" Jul 22 02:40:18.839553 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:40:18.839526 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-69997f8c4b-9phsb" Jul 22 02:40:18.909164 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:40:18.909141 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-76b6d645f6-j7z5j"] Jul 22 02:40:43.933643 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:40:43.933606 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-76b6d645f6-j7z5j" podUID="b2d88106-06d5-447f-bdf7-bbd6ca28e68c" containerName="console" containerID="cri-o://e96bd8aba85425f657334af1731ca2b40e84ef11049824d06f1378abfc18855d" gracePeriod=15 Jul 22 02:40:44.168704 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:40:44.168677 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-76b6d645f6-j7z5j" Jul 22 02:40:44.359337 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:40:44.359307 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b2d88106-06d5-447f-bdf7-bbd6ca28e68c-oauth-serving-cert\") pod \"b2d88106-06d5-447f-bdf7-bbd6ca28e68c\" (UID: \"b2d88106-06d5-447f-bdf7-bbd6ca28e68c\") " Jul 22 02:40:44.359467 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:40:44.359341 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b2d88106-06d5-447f-bdf7-bbd6ca28e68c-trusted-ca-bundle\") pod \"b2d88106-06d5-447f-bdf7-bbd6ca28e68c\" (UID: \"b2d88106-06d5-447f-bdf7-bbd6ca28e68c\") " Jul 22 02:40:44.359467 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:40:44.359371 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zsdd4\" (UniqueName: \"kubernetes.io/projected/b2d88106-06d5-447f-bdf7-bbd6ca28e68c-kube-api-access-zsdd4\") pod \"b2d88106-06d5-447f-bdf7-bbd6ca28e68c\" (UID: \"b2d88106-06d5-447f-bdf7-bbd6ca28e68c\") " Jul 22 02:40:44.359467 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:40:44.359387 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b2d88106-06d5-447f-bdf7-bbd6ca28e68c-console-serving-cert\") pod \"b2d88106-06d5-447f-bdf7-bbd6ca28e68c\" (UID: \"b2d88106-06d5-447f-bdf7-bbd6ca28e68c\") " Jul 22 02:40:44.359654 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:40:44.359513 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b2d88106-06d5-447f-bdf7-bbd6ca28e68c-console-oauth-config\") pod \"b2d88106-06d5-447f-bdf7-bbd6ca28e68c\" (UID: \"b2d88106-06d5-447f-bdf7-bbd6ca28e68c\") " Jul 22 02:40:44.359654 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:40:44.359555 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b2d88106-06d5-447f-bdf7-bbd6ca28e68c-console-config\") pod \"b2d88106-06d5-447f-bdf7-bbd6ca28e68c\" (UID: \"b2d88106-06d5-447f-bdf7-bbd6ca28e68c\") " Jul 22 02:40:44.359654 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:40:44.359597 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b2d88106-06d5-447f-bdf7-bbd6ca28e68c-service-ca\") pod \"b2d88106-06d5-447f-bdf7-bbd6ca28e68c\" (UID: \"b2d88106-06d5-447f-bdf7-bbd6ca28e68c\") " Jul 22 02:40:44.359863 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:40:44.359777 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2d88106-06d5-447f-bdf7-bbd6ca28e68c-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "b2d88106-06d5-447f-bdf7-bbd6ca28e68c" (UID: "b2d88106-06d5-447f-bdf7-bbd6ca28e68c"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 02:40:44.359928 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:40:44.359866 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2d88106-06d5-447f-bdf7-bbd6ca28e68c-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "b2d88106-06d5-447f-bdf7-bbd6ca28e68c" (UID: "b2d88106-06d5-447f-bdf7-bbd6ca28e68c"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 02:40:44.359993 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:40:44.359967 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2d88106-06d5-447f-bdf7-bbd6ca28e68c-console-config" (OuterVolumeSpecName: "console-config") pod "b2d88106-06d5-447f-bdf7-bbd6ca28e68c" (UID: "b2d88106-06d5-447f-bdf7-bbd6ca28e68c"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 02:40:44.360070 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:40:44.360049 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2d88106-06d5-447f-bdf7-bbd6ca28e68c-service-ca" (OuterVolumeSpecName: "service-ca") pod "b2d88106-06d5-447f-bdf7-bbd6ca28e68c" (UID: "b2d88106-06d5-447f-bdf7-bbd6ca28e68c"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 02:40:44.361676 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:40:44.361654 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b2d88106-06d5-447f-bdf7-bbd6ca28e68c-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "b2d88106-06d5-447f-bdf7-bbd6ca28e68c" (UID: "b2d88106-06d5-447f-bdf7-bbd6ca28e68c"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 02:40:44.361760 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:40:44.361676 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2d88106-06d5-447f-bdf7-bbd6ca28e68c-kube-api-access-zsdd4" (OuterVolumeSpecName: "kube-api-access-zsdd4") pod "b2d88106-06d5-447f-bdf7-bbd6ca28e68c" (UID: "b2d88106-06d5-447f-bdf7-bbd6ca28e68c"). InnerVolumeSpecName "kube-api-access-zsdd4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 02:40:44.361760 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:40:44.361708 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b2d88106-06d5-447f-bdf7-bbd6ca28e68c-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "b2d88106-06d5-447f-bdf7-bbd6ca28e68c" (UID: "b2d88106-06d5-447f-bdf7-bbd6ca28e68c"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 02:40:44.460347 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:40:44.460315 2578 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b2d88106-06d5-447f-bdf7-bbd6ca28e68c-console-config\") on node \"ip-10-0-133-31.ec2.internal\" DevicePath \"\"" Jul 22 02:40:44.460347 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:40:44.460340 2578 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b2d88106-06d5-447f-bdf7-bbd6ca28e68c-service-ca\") on node \"ip-10-0-133-31.ec2.internal\" DevicePath \"\"" Jul 22 02:40:44.460347 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:40:44.460351 2578 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b2d88106-06d5-447f-bdf7-bbd6ca28e68c-oauth-serving-cert\") on node \"ip-10-0-133-31.ec2.internal\" DevicePath \"\"" Jul 22 02:40:44.460540 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:40:44.460364 2578 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b2d88106-06d5-447f-bdf7-bbd6ca28e68c-trusted-ca-bundle\") on node \"ip-10-0-133-31.ec2.internal\" DevicePath \"\"" Jul 22 02:40:44.460540 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:40:44.460375 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zsdd4\" (UniqueName: \"kubernetes.io/projected/b2d88106-06d5-447f-bdf7-bbd6ca28e68c-kube-api-access-zsdd4\") on node \"ip-10-0-133-31.ec2.internal\" DevicePath \"\"" Jul 22 02:40:44.460540 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:40:44.460387 2578 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b2d88106-06d5-447f-bdf7-bbd6ca28e68c-console-serving-cert\") on node \"ip-10-0-133-31.ec2.internal\" DevicePath \"\"" Jul 22 02:40:44.460540 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:40:44.460399 2578 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b2d88106-06d5-447f-bdf7-bbd6ca28e68c-console-oauth-config\") on node \"ip-10-0-133-31.ec2.internal\" DevicePath \"\"" Jul 22 02:40:44.918896 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:40:44.918862 2578 generic.go:356] "Generic (PLEG): container finished" podID="b2d88106-06d5-447f-bdf7-bbd6ca28e68c" containerID="e96bd8aba85425f657334af1731ca2b40e84ef11049824d06f1378abfc18855d" exitCode=0 Jul 22 02:40:44.918998 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:40:44.918927 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-76b6d645f6-j7z5j" Jul 22 02:40:44.918998 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:40:44.918945 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-76b6d645f6-j7z5j" event={"ID":"b2d88106-06d5-447f-bdf7-bbd6ca28e68c","Type":"ContainerDied","Data":"e96bd8aba85425f657334af1731ca2b40e84ef11049824d06f1378abfc18855d"} Jul 22 02:40:44.918998 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:40:44.918993 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-76b6d645f6-j7z5j" event={"ID":"b2d88106-06d5-447f-bdf7-bbd6ca28e68c","Type":"ContainerDied","Data":"370a2e5a476808ccb2cdc38fd0afa68fd5ec4e3b560eb45f4a1384d6ccc0efe8"} Jul 22 02:40:44.919110 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:40:44.919018 2578 scope.go:117] "RemoveContainer" containerID="e96bd8aba85425f657334af1731ca2b40e84ef11049824d06f1378abfc18855d" Jul 22 02:40:44.930660 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:40:44.930644 2578 scope.go:117] "RemoveContainer" containerID="e96bd8aba85425f657334af1731ca2b40e84ef11049824d06f1378abfc18855d" Jul 22 02:40:44.930898 ip-10-0-133-31 kubenswrapper[2578]: E0722 02:40:44.930880 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e96bd8aba85425f657334af1731ca2b40e84ef11049824d06f1378abfc18855d\": container with ID starting with e96bd8aba85425f657334af1731ca2b40e84ef11049824d06f1378abfc18855d not found: ID does not exist" containerID="e96bd8aba85425f657334af1731ca2b40e84ef11049824d06f1378abfc18855d" Jul 22 02:40:44.930950 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:40:44.930907 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e96bd8aba85425f657334af1731ca2b40e84ef11049824d06f1378abfc18855d"} err="failed to get container status \"e96bd8aba85425f657334af1731ca2b40e84ef11049824d06f1378abfc18855d\": rpc error: code = NotFound desc = could not find container \"e96bd8aba85425f657334af1731ca2b40e84ef11049824d06f1378abfc18855d\": container with ID starting with e96bd8aba85425f657334af1731ca2b40e84ef11049824d06f1378abfc18855d not found: ID does not exist" Jul 22 02:40:44.945018 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:40:44.944994 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-76b6d645f6-j7z5j"] Jul 22 02:40:44.949425 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:40:44.949402 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-76b6d645f6-j7z5j"] Jul 22 02:40:46.139743 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:40:46.139715 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b2d88106-06d5-447f-bdf7-bbd6ca28e68c" path="/var/lib/kubelet/pods/b2d88106-06d5-447f-bdf7-bbd6ca28e68c/volumes" Jul 22 02:41:17.401454 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:41:17.401424 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-567848f679-z2t2x"] Jul 22 02:41:17.401934 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:41:17.401742 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b2d88106-06d5-447f-bdf7-bbd6ca28e68c" containerName="console" Jul 22 02:41:17.401934 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:41:17.401757 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="b2d88106-06d5-447f-bdf7-bbd6ca28e68c" containerName="console" Jul 22 02:41:17.401934 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:41:17.401796 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="b2d88106-06d5-447f-bdf7-bbd6ca28e68c" containerName="console" Jul 22 02:41:17.405642 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:41:17.405625 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-567848f679-z2t2x" Jul 22 02:41:17.430245 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:41:17.430223 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-567848f679-z2t2x"] Jul 22 02:41:17.465683 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:41:17.465656 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b5c79625-adb9-4d46-b9d9-7278cd5e310d-console-oauth-config\") pod \"console-567848f679-z2t2x\" (UID: \"b5c79625-adb9-4d46-b9d9-7278cd5e310d\") " pod="openshift-console/console-567848f679-z2t2x" Jul 22 02:41:17.465683 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:41:17.465684 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b5c79625-adb9-4d46-b9d9-7278cd5e310d-trusted-ca-bundle\") pod \"console-567848f679-z2t2x\" (UID: \"b5c79625-adb9-4d46-b9d9-7278cd5e310d\") " pod="openshift-console/console-567848f679-z2t2x" Jul 22 02:41:17.465802 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:41:17.465705 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xj2h9\" (UniqueName: \"kubernetes.io/projected/b5c79625-adb9-4d46-b9d9-7278cd5e310d-kube-api-access-xj2h9\") pod \"console-567848f679-z2t2x\" (UID: \"b5c79625-adb9-4d46-b9d9-7278cd5e310d\") " pod="openshift-console/console-567848f679-z2t2x" Jul 22 02:41:17.465802 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:41:17.465735 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b5c79625-adb9-4d46-b9d9-7278cd5e310d-console-config\") pod \"console-567848f679-z2t2x\" (UID: \"b5c79625-adb9-4d46-b9d9-7278cd5e310d\") " pod="openshift-console/console-567848f679-z2t2x" Jul 22 02:41:17.465802 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:41:17.465766 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b5c79625-adb9-4d46-b9d9-7278cd5e310d-console-serving-cert\") pod \"console-567848f679-z2t2x\" (UID: \"b5c79625-adb9-4d46-b9d9-7278cd5e310d\") " pod="openshift-console/console-567848f679-z2t2x" Jul 22 02:41:17.465802 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:41:17.465781 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b5c79625-adb9-4d46-b9d9-7278cd5e310d-service-ca\") pod \"console-567848f679-z2t2x\" (UID: \"b5c79625-adb9-4d46-b9d9-7278cd5e310d\") " pod="openshift-console/console-567848f679-z2t2x" Jul 22 02:41:17.465918 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:41:17.465799 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b5c79625-adb9-4d46-b9d9-7278cd5e310d-oauth-serving-cert\") pod \"console-567848f679-z2t2x\" (UID: \"b5c79625-adb9-4d46-b9d9-7278cd5e310d\") " pod="openshift-console/console-567848f679-z2t2x" Jul 22 02:41:17.566149 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:41:17.566124 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b5c79625-adb9-4d46-b9d9-7278cd5e310d-console-config\") pod \"console-567848f679-z2t2x\" (UID: \"b5c79625-adb9-4d46-b9d9-7278cd5e310d\") " pod="openshift-console/console-567848f679-z2t2x" Jul 22 02:41:17.566236 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:41:17.566156 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b5c79625-adb9-4d46-b9d9-7278cd5e310d-console-serving-cert\") pod \"console-567848f679-z2t2x\" (UID: \"b5c79625-adb9-4d46-b9d9-7278cd5e310d\") " pod="openshift-console/console-567848f679-z2t2x" Jul 22 02:41:17.566236 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:41:17.566172 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b5c79625-adb9-4d46-b9d9-7278cd5e310d-service-ca\") pod \"console-567848f679-z2t2x\" (UID: \"b5c79625-adb9-4d46-b9d9-7278cd5e310d\") " pod="openshift-console/console-567848f679-z2t2x" Jul 22 02:41:17.566236 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:41:17.566186 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b5c79625-adb9-4d46-b9d9-7278cd5e310d-oauth-serving-cert\") pod \"console-567848f679-z2t2x\" (UID: \"b5c79625-adb9-4d46-b9d9-7278cd5e310d\") " pod="openshift-console/console-567848f679-z2t2x" Jul 22 02:41:17.566236 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:41:17.566218 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b5c79625-adb9-4d46-b9d9-7278cd5e310d-console-oauth-config\") pod \"console-567848f679-z2t2x\" (UID: \"b5c79625-adb9-4d46-b9d9-7278cd5e310d\") " pod="openshift-console/console-567848f679-z2t2x" Jul 22 02:41:17.566236 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:41:17.566232 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b5c79625-adb9-4d46-b9d9-7278cd5e310d-trusted-ca-bundle\") pod \"console-567848f679-z2t2x\" (UID: \"b5c79625-adb9-4d46-b9d9-7278cd5e310d\") " pod="openshift-console/console-567848f679-z2t2x" Jul 22 02:41:17.566440 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:41:17.566357 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xj2h9\" (UniqueName: \"kubernetes.io/projected/b5c79625-adb9-4d46-b9d9-7278cd5e310d-kube-api-access-xj2h9\") pod \"console-567848f679-z2t2x\" (UID: \"b5c79625-adb9-4d46-b9d9-7278cd5e310d\") " pod="openshift-console/console-567848f679-z2t2x" Jul 22 02:41:17.566889 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:41:17.566868 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b5c79625-adb9-4d46-b9d9-7278cd5e310d-service-ca\") pod \"console-567848f679-z2t2x\" (UID: \"b5c79625-adb9-4d46-b9d9-7278cd5e310d\") " pod="openshift-console/console-567848f679-z2t2x" Jul 22 02:41:17.567023 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:41:17.566996 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b5c79625-adb9-4d46-b9d9-7278cd5e310d-trusted-ca-bundle\") pod \"console-567848f679-z2t2x\" (UID: \"b5c79625-adb9-4d46-b9d9-7278cd5e310d\") " pod="openshift-console/console-567848f679-z2t2x" Jul 22 02:41:17.567023 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:41:17.567012 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b5c79625-adb9-4d46-b9d9-7278cd5e310d-oauth-serving-cert\") pod \"console-567848f679-z2t2x\" (UID: \"b5c79625-adb9-4d46-b9d9-7278cd5e310d\") " pod="openshift-console/console-567848f679-z2t2x" Jul 22 02:41:17.567093 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:41:17.566996 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b5c79625-adb9-4d46-b9d9-7278cd5e310d-console-config\") pod \"console-567848f679-z2t2x\" (UID: \"b5c79625-adb9-4d46-b9d9-7278cd5e310d\") " pod="openshift-console/console-567848f679-z2t2x" Jul 22 02:41:17.569023 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:41:17.568999 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b5c79625-adb9-4d46-b9d9-7278cd5e310d-console-serving-cert\") pod \"console-567848f679-z2t2x\" (UID: \"b5c79625-adb9-4d46-b9d9-7278cd5e310d\") " pod="openshift-console/console-567848f679-z2t2x" Jul 22 02:41:17.569100 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:41:17.569020 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b5c79625-adb9-4d46-b9d9-7278cd5e310d-console-oauth-config\") pod \"console-567848f679-z2t2x\" (UID: \"b5c79625-adb9-4d46-b9d9-7278cd5e310d\") " pod="openshift-console/console-567848f679-z2t2x" Jul 22 02:41:17.579854 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:41:17.579836 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xj2h9\" (UniqueName: \"kubernetes.io/projected/b5c79625-adb9-4d46-b9d9-7278cd5e310d-kube-api-access-xj2h9\") pod \"console-567848f679-z2t2x\" (UID: \"b5c79625-adb9-4d46-b9d9-7278cd5e310d\") " pod="openshift-console/console-567848f679-z2t2x" Jul 22 02:41:17.713976 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:41:17.713920 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-567848f679-z2t2x" Jul 22 02:41:17.837281 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:41:17.837260 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-567848f679-z2t2x"] Jul 22 02:41:17.839626 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:41:17.839600 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5c79625_adb9_4d46_b9d9_7278cd5e310d.slice/crio-a14127f9c7452c0c92c1ed5177785034e98421898e992eabf7a6ef015db69d8d WatchSource:0}: Error finding container a14127f9c7452c0c92c1ed5177785034e98421898e992eabf7a6ef015db69d8d: Status 404 returned error can't find the container with id a14127f9c7452c0c92c1ed5177785034e98421898e992eabf7a6ef015db69d8d Jul 22 02:41:18.021951 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:41:18.021877 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-567848f679-z2t2x" event={"ID":"b5c79625-adb9-4d46-b9d9-7278cd5e310d","Type":"ContainerStarted","Data":"f32889816e2dcc2f02f6deac9eed480ca1c5d0b0940a8a56207b7dc9ad967952"} Jul 22 02:41:18.021951 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:41:18.021929 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-567848f679-z2t2x" event={"ID":"b5c79625-adb9-4d46-b9d9-7278cd5e310d","Type":"ContainerStarted","Data":"a14127f9c7452c0c92c1ed5177785034e98421898e992eabf7a6ef015db69d8d"} Jul 22 02:41:18.049819 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:41:18.049773 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-567848f679-z2t2x" podStartSLOduration=1.049754159 podStartE2EDuration="1.049754159s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 02:41:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:41:18.048462058 +0000 UTC m=+244.367637358" watchObservedRunningTime="2026-07-22 02:41:18.049754159 +0000 UTC m=+244.368929457" Jul 22 02:41:27.714652 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:41:27.714616 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-567848f679-z2t2x" Jul 22 02:41:27.714981 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:41:27.714677 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-567848f679-z2t2x" Jul 22 02:41:27.718843 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:41:27.718819 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-567848f679-z2t2x" Jul 22 02:41:28.056175 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:41:28.056115 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-567848f679-z2t2x" Jul 22 02:41:28.118729 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:41:28.118705 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-69997f8c4b-9phsb"] Jul 22 02:41:53.138403 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:41:53.138365 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-69997f8c4b-9phsb" podUID="5463bf38-361c-4ca0-9fe2-078e86072035" containerName="console" containerID="cri-o://af93d6520e92439523c928665264f78a086c361d5cde8ae017eb1a06ba732f48" gracePeriod=15 Jul 22 02:41:53.380178 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:41:53.380158 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-69997f8c4b-9phsb" Jul 22 02:41:53.493138 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:41:53.493114 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5463bf38-361c-4ca0-9fe2-078e86072035-oauth-serving-cert\") pod \"5463bf38-361c-4ca0-9fe2-078e86072035\" (UID: \"5463bf38-361c-4ca0-9fe2-078e86072035\") " Jul 22 02:41:53.493271 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:41:53.493155 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5463bf38-361c-4ca0-9fe2-078e86072035-console-serving-cert\") pod \"5463bf38-361c-4ca0-9fe2-078e86072035\" (UID: \"5463bf38-361c-4ca0-9fe2-078e86072035\") " Jul 22 02:41:53.493271 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:41:53.493177 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5463bf38-361c-4ca0-9fe2-078e86072035-console-oauth-config\") pod \"5463bf38-361c-4ca0-9fe2-078e86072035\" (UID: \"5463bf38-361c-4ca0-9fe2-078e86072035\") " Jul 22 02:41:53.493337 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:41:53.493305 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2bll8\" (UniqueName: \"kubernetes.io/projected/5463bf38-361c-4ca0-9fe2-078e86072035-kube-api-access-2bll8\") pod \"5463bf38-361c-4ca0-9fe2-078e86072035\" (UID: \"5463bf38-361c-4ca0-9fe2-078e86072035\") " Jul 22 02:41:53.493381 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:41:53.493340 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5463bf38-361c-4ca0-9fe2-078e86072035-trusted-ca-bundle\") pod \"5463bf38-361c-4ca0-9fe2-078e86072035\" (UID: \"5463bf38-361c-4ca0-9fe2-078e86072035\") " Jul 22 02:41:53.493432 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:41:53.493377 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5463bf38-361c-4ca0-9fe2-078e86072035-console-config\") pod \"5463bf38-361c-4ca0-9fe2-078e86072035\" (UID: \"5463bf38-361c-4ca0-9fe2-078e86072035\") " Jul 22 02:41:53.493432 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:41:53.493409 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5463bf38-361c-4ca0-9fe2-078e86072035-service-ca\") pod \"5463bf38-361c-4ca0-9fe2-078e86072035\" (UID: \"5463bf38-361c-4ca0-9fe2-078e86072035\") " Jul 22 02:41:53.493536 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:41:53.493493 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5463bf38-361c-4ca0-9fe2-078e86072035-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "5463bf38-361c-4ca0-9fe2-078e86072035" (UID: "5463bf38-361c-4ca0-9fe2-078e86072035"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 02:41:53.493926 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:41:53.493887 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5463bf38-361c-4ca0-9fe2-078e86072035-console-config" (OuterVolumeSpecName: "console-config") pod "5463bf38-361c-4ca0-9fe2-078e86072035" (UID: "5463bf38-361c-4ca0-9fe2-078e86072035"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 02:41:53.493926 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:41:53.493885 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5463bf38-361c-4ca0-9fe2-078e86072035-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "5463bf38-361c-4ca0-9fe2-078e86072035" (UID: "5463bf38-361c-4ca0-9fe2-078e86072035"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 02:41:53.493926 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:41:53.493850 2578 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5463bf38-361c-4ca0-9fe2-078e86072035-oauth-serving-cert\") on node \"ip-10-0-133-31.ec2.internal\" DevicePath \"\"" Jul 22 02:41:53.494174 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:41:53.494150 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5463bf38-361c-4ca0-9fe2-078e86072035-service-ca" (OuterVolumeSpecName: "service-ca") pod "5463bf38-361c-4ca0-9fe2-078e86072035" (UID: "5463bf38-361c-4ca0-9fe2-078e86072035"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 02:41:53.495333 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:41:53.495309 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5463bf38-361c-4ca0-9fe2-078e86072035-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "5463bf38-361c-4ca0-9fe2-078e86072035" (UID: "5463bf38-361c-4ca0-9fe2-078e86072035"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 02:41:53.495495 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:41:53.495476 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5463bf38-361c-4ca0-9fe2-078e86072035-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "5463bf38-361c-4ca0-9fe2-078e86072035" (UID: "5463bf38-361c-4ca0-9fe2-078e86072035"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 02:41:53.495572 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:41:53.495538 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5463bf38-361c-4ca0-9fe2-078e86072035-kube-api-access-2bll8" (OuterVolumeSpecName: "kube-api-access-2bll8") pod "5463bf38-361c-4ca0-9fe2-078e86072035" (UID: "5463bf38-361c-4ca0-9fe2-078e86072035"). InnerVolumeSpecName "kube-api-access-2bll8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 02:41:53.595001 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:41:53.594981 2578 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5463bf38-361c-4ca0-9fe2-078e86072035-service-ca\") on node \"ip-10-0-133-31.ec2.internal\" DevicePath \"\"" Jul 22 02:41:53.595082 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:41:53.595001 2578 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5463bf38-361c-4ca0-9fe2-078e86072035-console-serving-cert\") on node \"ip-10-0-133-31.ec2.internal\" DevicePath \"\"" Jul 22 02:41:53.595082 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:41:53.595011 2578 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5463bf38-361c-4ca0-9fe2-078e86072035-console-oauth-config\") on node \"ip-10-0-133-31.ec2.internal\" DevicePath \"\"" Jul 22 02:41:53.595082 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:41:53.595019 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2bll8\" (UniqueName: \"kubernetes.io/projected/5463bf38-361c-4ca0-9fe2-078e86072035-kube-api-access-2bll8\") on node \"ip-10-0-133-31.ec2.internal\" DevicePath \"\"" Jul 22 02:41:53.595082 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:41:53.595027 2578 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5463bf38-361c-4ca0-9fe2-078e86072035-trusted-ca-bundle\") on node \"ip-10-0-133-31.ec2.internal\" DevicePath \"\"" Jul 22 02:41:53.595082 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:41:53.595035 2578 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5463bf38-361c-4ca0-9fe2-078e86072035-console-config\") on node \"ip-10-0-133-31.ec2.internal\" DevicePath \"\"" Jul 22 02:41:54.138530 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:41:54.138503 2578 generic.go:356] "Generic (PLEG): container finished" podID="5463bf38-361c-4ca0-9fe2-078e86072035" containerID="af93d6520e92439523c928665264f78a086c361d5cde8ae017eb1a06ba732f48" exitCode=0 Jul 22 02:41:54.138920 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:41:54.138629 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-69997f8c4b-9phsb" Jul 22 02:41:54.140447 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:41:54.140422 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-69997f8c4b-9phsb" event={"ID":"5463bf38-361c-4ca0-9fe2-078e86072035","Type":"ContainerDied","Data":"af93d6520e92439523c928665264f78a086c361d5cde8ae017eb1a06ba732f48"} Jul 22 02:41:54.140518 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:41:54.140469 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-69997f8c4b-9phsb" event={"ID":"5463bf38-361c-4ca0-9fe2-078e86072035","Type":"ContainerDied","Data":"dc28baa303078177573556f7069bb77443a1c074a04f45919f1ae367e71db7db"} Jul 22 02:41:54.140518 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:41:54.140496 2578 scope.go:117] "RemoveContainer" containerID="af93d6520e92439523c928665264f78a086c361d5cde8ae017eb1a06ba732f48" Jul 22 02:41:54.148478 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:41:54.148464 2578 scope.go:117] "RemoveContainer" containerID="af93d6520e92439523c928665264f78a086c361d5cde8ae017eb1a06ba732f48" Jul 22 02:41:54.148725 ip-10-0-133-31 kubenswrapper[2578]: E0722 02:41:54.148706 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"af93d6520e92439523c928665264f78a086c361d5cde8ae017eb1a06ba732f48\": container with ID starting with af93d6520e92439523c928665264f78a086c361d5cde8ae017eb1a06ba732f48 not found: ID does not exist" containerID="af93d6520e92439523c928665264f78a086c361d5cde8ae017eb1a06ba732f48" Jul 22 02:41:54.148798 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:41:54.148737 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af93d6520e92439523c928665264f78a086c361d5cde8ae017eb1a06ba732f48"} err="failed to get container status \"af93d6520e92439523c928665264f78a086c361d5cde8ae017eb1a06ba732f48\": rpc error: code = NotFound desc = could not find container \"af93d6520e92439523c928665264f78a086c361d5cde8ae017eb1a06ba732f48\": container with ID starting with af93d6520e92439523c928665264f78a086c361d5cde8ae017eb1a06ba732f48 not found: ID does not exist" Jul 22 02:41:54.167373 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:41:54.167351 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-69997f8c4b-9phsb"] Jul 22 02:41:54.170015 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:41:54.169998 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-69997f8c4b-9phsb"] Jul 22 02:41:56.140227 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:41:56.140198 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5463bf38-361c-4ca0-9fe2-078e86072035" path="/var/lib/kubelet/pods/5463bf38-361c-4ca0-9fe2-078e86072035/volumes" Jul 22 02:42:14.023181 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:42:14.023154 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-prfql_fd7bf923-8d16-476c-a8d0-a3e6fd645061/ovn-acl-logging/0.log" Jul 22 02:42:14.025922 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:42:14.025901 2578 kubelet.go:1648] "Image garbage collection succeeded" Jul 22 02:42:14.028821 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:42:14.028799 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-prfql_fd7bf923-8d16-476c-a8d0-a3e6fd645061/ovn-acl-logging/0.log" Jul 22 02:42:21.256513 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:42:21.256483 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jx5n9"] Jul 22 02:42:21.258821 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:42:21.256731 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5463bf38-361c-4ca0-9fe2-078e86072035" containerName="console" Jul 22 02:42:21.258821 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:42:21.256743 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="5463bf38-361c-4ca0-9fe2-078e86072035" containerName="console" Jul 22 02:42:21.258821 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:42:21.256792 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="5463bf38-361c-4ca0-9fe2-078e86072035" containerName="console" Jul 22 02:42:21.259686 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:42:21.259672 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jx5n9" Jul 22 02:42:21.263484 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:42:21.263463 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 22 02:42:21.264615 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:42:21.264595 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-jz75q\"" Jul 22 02:42:21.264615 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:42:21.264609 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 22 02:42:21.274948 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:42:21.274928 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jx5n9"] Jul 22 02:42:21.362180 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:42:21.362161 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0dd0bf57-5f7c-495f-9d7c-bd532c296d92-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jx5n9\" (UID: \"0dd0bf57-5f7c-495f-9d7c-bd532c296d92\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jx5n9" Jul 22 02:42:21.362284 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:42:21.362197 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0dd0bf57-5f7c-495f-9d7c-bd532c296d92-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jx5n9\" (UID: \"0dd0bf57-5f7c-495f-9d7c-bd532c296d92\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jx5n9" Jul 22 02:42:21.362284 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:42:21.362224 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8gzbh\" (UniqueName: \"kubernetes.io/projected/0dd0bf57-5f7c-495f-9d7c-bd532c296d92-kube-api-access-8gzbh\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jx5n9\" (UID: \"0dd0bf57-5f7c-495f-9d7c-bd532c296d92\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jx5n9" Jul 22 02:42:21.462741 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:42:21.462719 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8gzbh\" (UniqueName: \"kubernetes.io/projected/0dd0bf57-5f7c-495f-9d7c-bd532c296d92-kube-api-access-8gzbh\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jx5n9\" (UID: \"0dd0bf57-5f7c-495f-9d7c-bd532c296d92\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jx5n9" Jul 22 02:42:21.462826 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:42:21.462760 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0dd0bf57-5f7c-495f-9d7c-bd532c296d92-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jx5n9\" (UID: \"0dd0bf57-5f7c-495f-9d7c-bd532c296d92\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jx5n9" Jul 22 02:42:21.462826 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:42:21.462785 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0dd0bf57-5f7c-495f-9d7c-bd532c296d92-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jx5n9\" (UID: \"0dd0bf57-5f7c-495f-9d7c-bd532c296d92\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jx5n9" Jul 22 02:42:21.463056 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:42:21.463043 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0dd0bf57-5f7c-495f-9d7c-bd532c296d92-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jx5n9\" (UID: \"0dd0bf57-5f7c-495f-9d7c-bd532c296d92\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jx5n9" Jul 22 02:42:21.463093 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:42:21.463077 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0dd0bf57-5f7c-495f-9d7c-bd532c296d92-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jx5n9\" (UID: \"0dd0bf57-5f7c-495f-9d7c-bd532c296d92\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jx5n9" Jul 22 02:42:21.474774 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:42:21.474749 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8gzbh\" (UniqueName: \"kubernetes.io/projected/0dd0bf57-5f7c-495f-9d7c-bd532c296d92-kube-api-access-8gzbh\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jx5n9\" (UID: \"0dd0bf57-5f7c-495f-9d7c-bd532c296d92\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jx5n9" Jul 22 02:42:21.567984 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:42:21.567938 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jx5n9" Jul 22 02:42:21.706085 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:42:21.706057 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jx5n9"] Jul 22 02:42:21.708881 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:42:21.708851 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0dd0bf57_5f7c_495f_9d7c_bd532c296d92.slice/crio-c3690746b293db5cc80231603044b5f5cf4d6aa13b0ffa2da51219668f598e34 WatchSource:0}: Error finding container c3690746b293db5cc80231603044b5f5cf4d6aa13b0ffa2da51219668f598e34: Status 404 returned error can't find the container with id c3690746b293db5cc80231603044b5f5cf4d6aa13b0ffa2da51219668f598e34 Jul 22 02:42:21.710659 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:42:21.710644 2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 22 02:42:22.218781 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:42:22.218755 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jx5n9" event={"ID":"0dd0bf57-5f7c-495f-9d7c-bd532c296d92","Type":"ContainerStarted","Data":"c3690746b293db5cc80231603044b5f5cf4d6aa13b0ffa2da51219668f598e34"} Jul 22 02:42:29.245249 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:42:29.245216 2578 generic.go:356] "Generic (PLEG): container finished" podID="0dd0bf57-5f7c-495f-9d7c-bd532c296d92" containerID="e4bc1e28e7d4182b078041f3f020b59a75204cd328eb7a90428b420e3afa8aeb" exitCode=0 Jul 22 02:42:29.245545 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:42:29.245282 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jx5n9" event={"ID":"0dd0bf57-5f7c-495f-9d7c-bd532c296d92","Type":"ContainerDied","Data":"e4bc1e28e7d4182b078041f3f020b59a75204cd328eb7a90428b420e3afa8aeb"} Jul 22 02:42:32.256729 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:42:32.256700 2578 generic.go:356] "Generic (PLEG): container finished" podID="0dd0bf57-5f7c-495f-9d7c-bd532c296d92" containerID="bd79dcdc2af939f249b2ffba3d8a4de2b772e9f0720b60efdce0fd1da05db711" exitCode=0 Jul 22 02:42:32.257017 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:42:32.256753 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jx5n9" event={"ID":"0dd0bf57-5f7c-495f-9d7c-bd532c296d92","Type":"ContainerDied","Data":"bd79dcdc2af939f249b2ffba3d8a4de2b772e9f0720b60efdce0fd1da05db711"} Jul 22 02:42:39.286792 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:42:39.286730 2578 generic.go:356] "Generic (PLEG): container finished" podID="0dd0bf57-5f7c-495f-9d7c-bd532c296d92" containerID="4d8798660e0c2cd36b304f6c321d8366d0953f952b0a7f2b5cb273591131ecf1" exitCode=0 Jul 22 02:42:39.287094 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:42:39.286810 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jx5n9" event={"ID":"0dd0bf57-5f7c-495f-9d7c-bd532c296d92","Type":"ContainerDied","Data":"4d8798660e0c2cd36b304f6c321d8366d0953f952b0a7f2b5cb273591131ecf1"} Jul 22 02:42:40.436404 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:42:40.436374 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jx5n9" Jul 22 02:42:40.608855 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:42:40.608832 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0dd0bf57-5f7c-495f-9d7c-bd532c296d92-util\") pod \"0dd0bf57-5f7c-495f-9d7c-bd532c296d92\" (UID: \"0dd0bf57-5f7c-495f-9d7c-bd532c296d92\") " Jul 22 02:42:40.608953 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:42:40.608876 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0dd0bf57-5f7c-495f-9d7c-bd532c296d92-bundle\") pod \"0dd0bf57-5f7c-495f-9d7c-bd532c296d92\" (UID: \"0dd0bf57-5f7c-495f-9d7c-bd532c296d92\") " Jul 22 02:42:40.608953 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:42:40.608915 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8gzbh\" (UniqueName: \"kubernetes.io/projected/0dd0bf57-5f7c-495f-9d7c-bd532c296d92-kube-api-access-8gzbh\") pod \"0dd0bf57-5f7c-495f-9d7c-bd532c296d92\" (UID: \"0dd0bf57-5f7c-495f-9d7c-bd532c296d92\") " Jul 22 02:42:40.609469 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:42:40.609446 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0dd0bf57-5f7c-495f-9d7c-bd532c296d92-bundle" (OuterVolumeSpecName: "bundle") pod "0dd0bf57-5f7c-495f-9d7c-bd532c296d92" (UID: "0dd0bf57-5f7c-495f-9d7c-bd532c296d92"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 02:42:40.611030 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:42:40.611006 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0dd0bf57-5f7c-495f-9d7c-bd532c296d92-kube-api-access-8gzbh" (OuterVolumeSpecName: "kube-api-access-8gzbh") pod "0dd0bf57-5f7c-495f-9d7c-bd532c296d92" (UID: "0dd0bf57-5f7c-495f-9d7c-bd532c296d92"). InnerVolumeSpecName "kube-api-access-8gzbh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 02:42:40.613043 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:42:40.613023 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0dd0bf57-5f7c-495f-9d7c-bd532c296d92-util" (OuterVolumeSpecName: "util") pod "0dd0bf57-5f7c-495f-9d7c-bd532c296d92" (UID: "0dd0bf57-5f7c-495f-9d7c-bd532c296d92"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 02:42:40.710282 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:42:40.710260 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8gzbh\" (UniqueName: \"kubernetes.io/projected/0dd0bf57-5f7c-495f-9d7c-bd532c296d92-kube-api-access-8gzbh\") on node \"ip-10-0-133-31.ec2.internal\" DevicePath \"\"" Jul 22 02:42:40.710282 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:42:40.710280 2578 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0dd0bf57-5f7c-495f-9d7c-bd532c296d92-util\") on node \"ip-10-0-133-31.ec2.internal\" DevicePath \"\"" Jul 22 02:42:40.710386 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:42:40.710288 2578 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0dd0bf57-5f7c-495f-9d7c-bd532c296d92-bundle\") on node \"ip-10-0-133-31.ec2.internal\" DevicePath \"\"" Jul 22 02:42:41.294634 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:42:41.294617 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jx5n9" Jul 22 02:42:41.294736 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:42:41.294613 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jx5n9" event={"ID":"0dd0bf57-5f7c-495f-9d7c-bd532c296d92","Type":"ContainerDied","Data":"c3690746b293db5cc80231603044b5f5cf4d6aa13b0ffa2da51219668f598e34"} Jul 22 02:42:41.294736 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:42:41.294732 2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c3690746b293db5cc80231603044b5f5cf4d6aa13b0ffa2da51219668f598e34" Jul 22 02:42:44.957531 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:42:44.957499 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-rlsr4"] Jul 22 02:42:44.957948 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:42:44.957718 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0dd0bf57-5f7c-495f-9d7c-bd532c296d92" containerName="util" Jul 22 02:42:44.957948 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:42:44.957728 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="0dd0bf57-5f7c-495f-9d7c-bd532c296d92" containerName="util" Jul 22 02:42:44.957948 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:42:44.957742 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0dd0bf57-5f7c-495f-9d7c-bd532c296d92" containerName="extract" Jul 22 02:42:44.957948 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:42:44.957747 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="0dd0bf57-5f7c-495f-9d7c-bd532c296d92" containerName="extract" Jul 22 02:42:44.957948 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:42:44.957754 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0dd0bf57-5f7c-495f-9d7c-bd532c296d92" containerName="pull" Jul 22 02:42:44.957948 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:42:44.957760 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="0dd0bf57-5f7c-495f-9d7c-bd532c296d92" containerName="pull" Jul 22 02:42:44.957948 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:42:44.957795 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="0dd0bf57-5f7c-495f-9d7c-bd532c296d92" containerName="extract" Jul 22 02:42:44.960800 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:42:44.960783 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-rlsr4" Jul 22 02:42:44.965916 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:42:44.965898 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Jul 22 02:42:44.965916 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:42:44.965905 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Jul 22 02:42:44.966083 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:42:44.966003 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-vsppk\"" Jul 22 02:42:44.984542 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:42:44.984521 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-rlsr4"] Jul 22 02:42:45.137940 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:42:45.137914 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8zg6s\" (UniqueName: \"kubernetes.io/projected/2351539b-cc42-4181-a09d-a418184fb120-kube-api-access-8zg6s\") pod \"cert-manager-operator-controller-manager-55c444bf86-rlsr4\" (UID: \"2351539b-cc42-4181-a09d-a418184fb120\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-rlsr4" Jul 22 02:42:45.138043 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:42:45.137958 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2351539b-cc42-4181-a09d-a418184fb120-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-rlsr4\" (UID: \"2351539b-cc42-4181-a09d-a418184fb120\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-rlsr4" Jul 22 02:42:45.238198 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:42:45.238153 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2351539b-cc42-4181-a09d-a418184fb120-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-rlsr4\" (UID: \"2351539b-cc42-4181-a09d-a418184fb120\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-rlsr4" Jul 22 02:42:45.238270 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:42:45.238197 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8zg6s\" (UniqueName: \"kubernetes.io/projected/2351539b-cc42-4181-a09d-a418184fb120-kube-api-access-8zg6s\") pod \"cert-manager-operator-controller-manager-55c444bf86-rlsr4\" (UID: \"2351539b-cc42-4181-a09d-a418184fb120\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-rlsr4" Jul 22 02:42:45.238471 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:42:45.238455 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2351539b-cc42-4181-a09d-a418184fb120-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-rlsr4\" (UID: \"2351539b-cc42-4181-a09d-a418184fb120\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-rlsr4" Jul 22 02:42:45.259528 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:42:45.259505 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8zg6s\" (UniqueName: \"kubernetes.io/projected/2351539b-cc42-4181-a09d-a418184fb120-kube-api-access-8zg6s\") pod \"cert-manager-operator-controller-manager-55c444bf86-rlsr4\" (UID: \"2351539b-cc42-4181-a09d-a418184fb120\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-rlsr4" Jul 22 02:42:45.269294 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:42:45.269274 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-rlsr4" Jul 22 02:42:45.419167 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:42:45.419136 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-rlsr4"] Jul 22 02:42:45.423409 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:42:45.423384 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2351539b_cc42_4181_a09d_a418184fb120.slice/crio-da365d7b88a7d12f5b4768e3736884bcdbdcbd0db0f39671225b79d7767d7dce WatchSource:0}: Error finding container da365d7b88a7d12f5b4768e3736884bcdbdcbd0db0f39671225b79d7767d7dce: Status 404 returned error can't find the container with id da365d7b88a7d12f5b4768e3736884bcdbdcbd0db0f39671225b79d7767d7dce Jul 22 02:42:46.314249 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:42:46.314208 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-rlsr4" event={"ID":"2351539b-cc42-4181-a09d-a418184fb120","Type":"ContainerStarted","Data":"da365d7b88a7d12f5b4768e3736884bcdbdcbd0db0f39671225b79d7767d7dce"} Jul 22 02:42:48.324405 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:42:48.324364 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-rlsr4" event={"ID":"2351539b-cc42-4181-a09d-a418184fb120","Type":"ContainerStarted","Data":"0a6d3e5ef7dd33247c40465b6991c157acbf06237e3fc3b11c5cec8e2ccba5f6"} Jul 22 02:42:48.351318 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:42:48.351276 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-rlsr4" podStartSLOduration=2.074416279 podStartE2EDuration="4.351264704s" totalImagesPullingTime="2.276848425s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 02:42:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:42:48.349809603 +0000 UTC m=+334.668984904" watchObservedRunningTime="2026-07-22 02:42:48.351264704 +0000 UTC m=+334.670440006" Jul 22 02:42:52.095079 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:42:52.095046 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-xglxn"] Jul 22 02:42:52.096972 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:42:52.096954 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-xglxn" Jul 22 02:42:52.103898 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:42:52.103877 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-ds6cd\"" Jul 22 02:42:52.103995 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:42:52.103917 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Jul 22 02:42:52.103995 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:42:52.103934 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Jul 22 02:42:52.111443 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:42:52.111421 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-xglxn"] Jul 22 02:42:52.283051 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:42:52.283028 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5df3ebd9-75ab-49af-9caf-873bb1e69208-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-xglxn\" (UID: \"5df3ebd9-75ab-49af-9caf-873bb1e69208\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-xglxn" Jul 22 02:42:52.283163 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:42:52.283071 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r766d\" (UniqueName: \"kubernetes.io/projected/5df3ebd9-75ab-49af-9caf-873bb1e69208-kube-api-access-r766d\") pod \"cert-manager-cainjector-559cdd8c87-xglxn\" (UID: \"5df3ebd9-75ab-49af-9caf-873bb1e69208\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-xglxn" Jul 22 02:42:52.383947 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:42:52.383898 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r766d\" (UniqueName: \"kubernetes.io/projected/5df3ebd9-75ab-49af-9caf-873bb1e69208-kube-api-access-r766d\") pod \"cert-manager-cainjector-559cdd8c87-xglxn\" (UID: \"5df3ebd9-75ab-49af-9caf-873bb1e69208\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-xglxn" Jul 22 02:42:52.384030 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:42:52.383963 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5df3ebd9-75ab-49af-9caf-873bb1e69208-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-xglxn\" (UID: \"5df3ebd9-75ab-49af-9caf-873bb1e69208\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-xglxn" Jul 22 02:42:52.395269 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:42:52.395246 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5df3ebd9-75ab-49af-9caf-873bb1e69208-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-xglxn\" (UID: \"5df3ebd9-75ab-49af-9caf-873bb1e69208\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-xglxn" Jul 22 02:42:52.399610 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:42:52.399593 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r766d\" (UniqueName: \"kubernetes.io/projected/5df3ebd9-75ab-49af-9caf-873bb1e69208-kube-api-access-r766d\") pod \"cert-manager-cainjector-559cdd8c87-xglxn\" (UID: \"5df3ebd9-75ab-49af-9caf-873bb1e69208\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-xglxn" Jul 22 02:42:52.405393 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:42:52.405373 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-xglxn" Jul 22 02:42:52.536913 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:42:52.536889 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-xglxn"] Jul 22 02:42:52.538250 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:42:52.538227 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5df3ebd9_75ab_49af_9caf_873bb1e69208.slice/crio-ba82b92424e34bce8f20af4ad6999c0f452e9e339ae707378031cb6aea025cfe WatchSource:0}: Error finding container ba82b92424e34bce8f20af4ad6999c0f452e9e339ae707378031cb6aea025cfe: Status 404 returned error can't find the container with id ba82b92424e34bce8f20af4ad6999c0f452e9e339ae707378031cb6aea025cfe Jul 22 02:42:53.346411 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:42:53.346379 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-xglxn" event={"ID":"5df3ebd9-75ab-49af-9caf-873bb1e69208","Type":"ContainerStarted","Data":"ba82b92424e34bce8f20af4ad6999c0f452e9e339ae707378031cb6aea025cfe"} Jul 22 02:42:57.362695 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:42:57.362662 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-xglxn" event={"ID":"5df3ebd9-75ab-49af-9caf-873bb1e69208","Type":"ContainerStarted","Data":"5e763ddd68b10e4386fc0b4fd071a3ccab397470b7a37f24c63ec67510ae01aa"} Jul 22 02:42:57.385214 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:42:57.385168 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-xglxn" podStartSLOduration=0.636347642 podStartE2EDuration="5.385153327s" totalImagesPullingTime="4.748805685s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 02:42:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:42:57.382939551 +0000 UTC m=+343.702114852" watchObservedRunningTime="2026-07-22 02:42:57.385153327 +0000 UTC m=+343.704328626" Jul 22 02:42:57.751036 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:42:57.751010 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-676qx"] Jul 22 02:42:57.752780 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:42:57.752766 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-676qx" Jul 22 02:42:57.755746 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:42:57.755725 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-wl8b8\"" Jul 22 02:42:57.765020 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:42:57.765000 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-676qx"] Jul 22 02:42:57.922094 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:42:57.922069 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hgmvf\" (UniqueName: \"kubernetes.io/projected/e1fa84fd-d2b5-4452-ba6b-1161ad43de9e-kube-api-access-hgmvf\") pod \"cert-manager-webhook-6486ffd997-676qx\" (UID: \"e1fa84fd-d2b5-4452-ba6b-1161ad43de9e\") " pod="cert-manager/cert-manager-webhook-6486ffd997-676qx" Jul 22 02:42:57.922204 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:42:57.922102 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e1fa84fd-d2b5-4452-ba6b-1161ad43de9e-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-676qx\" (UID: \"e1fa84fd-d2b5-4452-ba6b-1161ad43de9e\") " pod="cert-manager/cert-manager-webhook-6486ffd997-676qx" Jul 22 02:42:58.022466 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:42:58.022414 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hgmvf\" (UniqueName: \"kubernetes.io/projected/e1fa84fd-d2b5-4452-ba6b-1161ad43de9e-kube-api-access-hgmvf\") pod \"cert-manager-webhook-6486ffd997-676qx\" (UID: \"e1fa84fd-d2b5-4452-ba6b-1161ad43de9e\") " pod="cert-manager/cert-manager-webhook-6486ffd997-676qx" Jul 22 02:42:58.022466 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:42:58.022441 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e1fa84fd-d2b5-4452-ba6b-1161ad43de9e-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-676qx\" (UID: \"e1fa84fd-d2b5-4452-ba6b-1161ad43de9e\") " pod="cert-manager/cert-manager-webhook-6486ffd997-676qx" Jul 22 02:42:58.035125 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:42:58.035100 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e1fa84fd-d2b5-4452-ba6b-1161ad43de9e-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-676qx\" (UID: \"e1fa84fd-d2b5-4452-ba6b-1161ad43de9e\") " pod="cert-manager/cert-manager-webhook-6486ffd997-676qx" Jul 22 02:42:58.036388 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:42:58.036368 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hgmvf\" (UniqueName: \"kubernetes.io/projected/e1fa84fd-d2b5-4452-ba6b-1161ad43de9e-kube-api-access-hgmvf\") pod \"cert-manager-webhook-6486ffd997-676qx\" (UID: \"e1fa84fd-d2b5-4452-ba6b-1161ad43de9e\") " pod="cert-manager/cert-manager-webhook-6486ffd997-676qx" Jul 22 02:42:58.060998 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:42:58.060972 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-676qx" Jul 22 02:42:58.189763 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:42:58.189738 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-676qx"] Jul 22 02:42:58.192575 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:42:58.192538 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode1fa84fd_d2b5_4452_ba6b_1161ad43de9e.slice/crio-9cb48fc916ac46c14af154ca684e055cd9613d02103845e4aef1f127fbeebe50 WatchSource:0}: Error finding container 9cb48fc916ac46c14af154ca684e055cd9613d02103845e4aef1f127fbeebe50: Status 404 returned error can't find the container with id 9cb48fc916ac46c14af154ca684e055cd9613d02103845e4aef1f127fbeebe50 Jul 22 02:42:58.367437 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:42:58.367408 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-676qx" event={"ID":"e1fa84fd-d2b5-4452-ba6b-1161ad43de9e","Type":"ContainerStarted","Data":"53dfd3d2fd185297d63c1c8f56f13764f1de72c7011868640d8d82ac3b2bf0ce"} Jul 22 02:42:58.367765 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:42:58.367448 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-676qx" event={"ID":"e1fa84fd-d2b5-4452-ba6b-1161ad43de9e","Type":"ContainerStarted","Data":"9cb48fc916ac46c14af154ca684e055cd9613d02103845e4aef1f127fbeebe50"} Jul 22 02:42:58.405632 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:42:58.405582 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-676qx" podStartSLOduration=1.405554664 podStartE2EDuration="1.405554664s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 02:42:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:42:58.403806734 +0000 UTC m=+344.722982033" watchObservedRunningTime="2026-07-22 02:42:58.405554664 +0000 UTC m=+344.724729964" Jul 22 02:42:59.370612 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:42:59.370586 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-676qx" Jul 22 02:43:05.376338 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:43:05.376269 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-676qx" Jul 22 02:43:22.587626 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:43:22.587590 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afz8p6s"] Jul 22 02:43:22.591678 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:43:22.591663 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afz8p6s" Jul 22 02:43:22.594625 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:43:22.594608 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 22 02:43:22.594725 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:43:22.594682 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 22 02:43:22.595754 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:43:22.595738 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-jz75q\"" Jul 22 02:43:22.601142 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:43:22.601122 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afz8p6s"] Jul 22 02:43:22.678304 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:43:22.678277 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hpv7l\" (UniqueName: \"kubernetes.io/projected/cfa0e2b4-462a-46c8-bf96-6453a811fafc-kube-api-access-hpv7l\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afz8p6s\" (UID: \"cfa0e2b4-462a-46c8-bf96-6453a811fafc\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afz8p6s" Jul 22 02:43:22.678304 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:43:22.678306 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cfa0e2b4-462a-46c8-bf96-6453a811fafc-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afz8p6s\" (UID: \"cfa0e2b4-462a-46c8-bf96-6453a811fafc\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afz8p6s" Jul 22 02:43:22.678448 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:43:22.678336 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cfa0e2b4-462a-46c8-bf96-6453a811fafc-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afz8p6s\" (UID: \"cfa0e2b4-462a-46c8-bf96-6453a811fafc\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afz8p6s" Jul 22 02:43:22.779617 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:43:22.779581 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hpv7l\" (UniqueName: \"kubernetes.io/projected/cfa0e2b4-462a-46c8-bf96-6453a811fafc-kube-api-access-hpv7l\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afz8p6s\" (UID: \"cfa0e2b4-462a-46c8-bf96-6453a811fafc\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afz8p6s" Jul 22 02:43:22.779719 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:43:22.779633 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cfa0e2b4-462a-46c8-bf96-6453a811fafc-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afz8p6s\" (UID: \"cfa0e2b4-462a-46c8-bf96-6453a811fafc\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afz8p6s" Jul 22 02:43:22.779719 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:43:22.779681 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cfa0e2b4-462a-46c8-bf96-6453a811fafc-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afz8p6s\" (UID: \"cfa0e2b4-462a-46c8-bf96-6453a811fafc\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afz8p6s" Jul 22 02:43:22.779956 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:43:22.779941 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cfa0e2b4-462a-46c8-bf96-6453a811fafc-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afz8p6s\" (UID: \"cfa0e2b4-462a-46c8-bf96-6453a811fafc\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afz8p6s" Jul 22 02:43:22.779990 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:43:22.779980 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cfa0e2b4-462a-46c8-bf96-6453a811fafc-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afz8p6s\" (UID: \"cfa0e2b4-462a-46c8-bf96-6453a811fafc\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afz8p6s" Jul 22 02:43:22.789252 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:43:22.789224 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hpv7l\" (UniqueName: \"kubernetes.io/projected/cfa0e2b4-462a-46c8-bf96-6453a811fafc-kube-api-access-hpv7l\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afz8p6s\" (UID: \"cfa0e2b4-462a-46c8-bf96-6453a811fafc\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afz8p6s" Jul 22 02:43:22.899984 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:43:22.899939 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afz8p6s" Jul 22 02:43:23.030113 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:43:23.030092 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afz8p6s"] Jul 22 02:43:23.032140 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:43:23.032114 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcfa0e2b4_462a_46c8_bf96_6453a811fafc.slice/crio-b5e6cead687332031441e5083dfed715da69169b1b0bf6d49755d7c1e05775d3 WatchSource:0}: Error finding container b5e6cead687332031441e5083dfed715da69169b1b0bf6d49755d7c1e05775d3: Status 404 returned error can't find the container with id b5e6cead687332031441e5083dfed715da69169b1b0bf6d49755d7c1e05775d3 Jul 22 02:43:23.455931 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:43:23.455900 2578 generic.go:356] "Generic (PLEG): container finished" podID="cfa0e2b4-462a-46c8-bf96-6453a811fafc" containerID="d040c7c2b502e66eb8ed482e7a5a11f1b42f1c37efcc76f35e53c34a45599820" exitCode=0 Jul 22 02:43:23.456101 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:43:23.455977 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afz8p6s" event={"ID":"cfa0e2b4-462a-46c8-bf96-6453a811fafc","Type":"ContainerDied","Data":"d040c7c2b502e66eb8ed482e7a5a11f1b42f1c37efcc76f35e53c34a45599820"} Jul 22 02:43:23.456101 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:43:23.456023 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afz8p6s" event={"ID":"cfa0e2b4-462a-46c8-bf96-6453a811fafc","Type":"ContainerStarted","Data":"b5e6cead687332031441e5083dfed715da69169b1b0bf6d49755d7c1e05775d3"} Jul 22 02:43:24.461296 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:43:24.461233 2578 generic.go:356] "Generic (PLEG): container finished" podID="cfa0e2b4-462a-46c8-bf96-6453a811fafc" containerID="5bd52e02afe7a649bac0784a9197e43da2abd7cb32e86f3c4f2c0af4a95b0add" exitCode=0 Jul 22 02:43:24.461619 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:43:24.461317 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afz8p6s" event={"ID":"cfa0e2b4-462a-46c8-bf96-6453a811fafc","Type":"ContainerDied","Data":"5bd52e02afe7a649bac0784a9197e43da2abd7cb32e86f3c4f2c0af4a95b0add"} Jul 22 02:43:25.466852 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:43:25.466825 2578 generic.go:356] "Generic (PLEG): container finished" podID="cfa0e2b4-462a-46c8-bf96-6453a811fafc" containerID="9860447e66f50fed0a7cc7439c6ce9c8572f7f5d7b88947183c0c4a008a55f47" exitCode=0 Jul 22 02:43:25.467204 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:43:25.466863 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afz8p6s" event={"ID":"cfa0e2b4-462a-46c8-bf96-6453a811fafc","Type":"ContainerDied","Data":"9860447e66f50fed0a7cc7439c6ce9c8572f7f5d7b88947183c0c4a008a55f47"} Jul 22 02:43:26.591862 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:43:26.591842 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afz8p6s" Jul 22 02:43:26.604627 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:43:26.604606 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hpv7l\" (UniqueName: \"kubernetes.io/projected/cfa0e2b4-462a-46c8-bf96-6453a811fafc-kube-api-access-hpv7l\") pod \"cfa0e2b4-462a-46c8-bf96-6453a811fafc\" (UID: \"cfa0e2b4-462a-46c8-bf96-6453a811fafc\") " Jul 22 02:43:26.604709 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:43:26.604648 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cfa0e2b4-462a-46c8-bf96-6453a811fafc-util\") pod \"cfa0e2b4-462a-46c8-bf96-6453a811fafc\" (UID: \"cfa0e2b4-462a-46c8-bf96-6453a811fafc\") " Jul 22 02:43:26.604709 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:43:26.604675 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cfa0e2b4-462a-46c8-bf96-6453a811fafc-bundle\") pod \"cfa0e2b4-462a-46c8-bf96-6453a811fafc\" (UID: \"cfa0e2b4-462a-46c8-bf96-6453a811fafc\") " Jul 22 02:43:26.605309 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:43:26.605285 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfa0e2b4-462a-46c8-bf96-6453a811fafc-bundle" (OuterVolumeSpecName: "bundle") pod "cfa0e2b4-462a-46c8-bf96-6453a811fafc" (UID: "cfa0e2b4-462a-46c8-bf96-6453a811fafc"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 02:43:26.606731 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:43:26.606708 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cfa0e2b4-462a-46c8-bf96-6453a811fafc-kube-api-access-hpv7l" (OuterVolumeSpecName: "kube-api-access-hpv7l") pod "cfa0e2b4-462a-46c8-bf96-6453a811fafc" (UID: "cfa0e2b4-462a-46c8-bf96-6453a811fafc"). InnerVolumeSpecName "kube-api-access-hpv7l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 02:43:26.609739 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:43:26.609711 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfa0e2b4-462a-46c8-bf96-6453a811fafc-util" (OuterVolumeSpecName: "util") pod "cfa0e2b4-462a-46c8-bf96-6453a811fafc" (UID: "cfa0e2b4-462a-46c8-bf96-6453a811fafc"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 02:43:26.705521 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:43:26.705501 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hpv7l\" (UniqueName: \"kubernetes.io/projected/cfa0e2b4-462a-46c8-bf96-6453a811fafc-kube-api-access-hpv7l\") on node \"ip-10-0-133-31.ec2.internal\" DevicePath \"\"" Jul 22 02:43:26.705521 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:43:26.705519 2578 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cfa0e2b4-462a-46c8-bf96-6453a811fafc-util\") on node \"ip-10-0-133-31.ec2.internal\" DevicePath \"\"" Jul 22 02:43:26.705669 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:43:26.705529 2578 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cfa0e2b4-462a-46c8-bf96-6453a811fafc-bundle\") on node \"ip-10-0-133-31.ec2.internal\" DevicePath \"\"" Jul 22 02:43:27.475376 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:43:27.475352 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afz8p6s" event={"ID":"cfa0e2b4-462a-46c8-bf96-6453a811fafc","Type":"ContainerDied","Data":"b5e6cead687332031441e5083dfed715da69169b1b0bf6d49755d7c1e05775d3"} Jul 22 02:43:27.475461 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:43:27.475382 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afz8p6s" Jul 22 02:43:27.475461 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:43:27.475391 2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b5e6cead687332031441e5083dfed715da69169b1b0bf6d49755d7c1e05775d3" Jul 22 02:43:35.757898 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:43:35.757869 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5b96c6dbc5-t7fj2"] Jul 22 02:43:35.758242 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:43:35.758097 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cfa0e2b4-462a-46c8-bf96-6453a811fafc" containerName="util" Jul 22 02:43:35.758242 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:43:35.758107 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="cfa0e2b4-462a-46c8-bf96-6453a811fafc" containerName="util" Jul 22 02:43:35.758242 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:43:35.758114 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cfa0e2b4-462a-46c8-bf96-6453a811fafc" containerName="pull" Jul 22 02:43:35.758242 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:43:35.758119 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="cfa0e2b4-462a-46c8-bf96-6453a811fafc" containerName="pull" Jul 22 02:43:35.758242 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:43:35.758125 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cfa0e2b4-462a-46c8-bf96-6453a811fafc" containerName="extract" Jul 22 02:43:35.758242 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:43:35.758130 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="cfa0e2b4-462a-46c8-bf96-6453a811fafc" containerName="extract" Jul 22 02:43:35.758242 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:43:35.758177 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="cfa0e2b4-462a-46c8-bf96-6453a811fafc" containerName="extract" Jul 22 02:43:35.760907 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:43:35.760892 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-5b96c6dbc5-t7fj2" Jul 22 02:43:35.764722 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:43:35.764695 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 22 02:43:35.764955 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:43:35.764938 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-p8xl5\"" Jul 22 02:43:35.765033 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:43:35.764941 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 22 02:43:35.765033 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:43:35.764964 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 22 02:43:35.765033 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:43:35.765005 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 22 02:43:35.782627 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:43:35.782606 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5b96c6dbc5-t7fj2"] Jul 22 02:43:35.865320 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:43:35.865297 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f60d3c35-191d-441a-8caa-95913b917e57-tmp\") pod \"opendatahub-operator-controller-manager-5b96c6dbc5-t7fj2\" (UID: \"f60d3c35-191d-441a-8caa-95913b917e57\") " pod="opendatahub/opendatahub-operator-controller-manager-5b96c6dbc5-t7fj2" Jul 22 02:43:35.865421 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:43:35.865327 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/f60d3c35-191d-441a-8caa-95913b917e57-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5b96c6dbc5-t7fj2\" (UID: \"f60d3c35-191d-441a-8caa-95913b917e57\") " pod="opendatahub/opendatahub-operator-controller-manager-5b96c6dbc5-t7fj2" Jul 22 02:43:35.865421 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:43:35.865349 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/f60d3c35-191d-441a-8caa-95913b917e57-manifests\") pod \"opendatahub-operator-controller-manager-5b96c6dbc5-t7fj2\" (UID: \"f60d3c35-191d-441a-8caa-95913b917e57\") " pod="opendatahub/opendatahub-operator-controller-manager-5b96c6dbc5-t7fj2" Jul 22 02:43:35.865421 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:43:35.865365 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7kr5g\" (UniqueName: \"kubernetes.io/projected/f60d3c35-191d-441a-8caa-95913b917e57-kube-api-access-7kr5g\") pod \"opendatahub-operator-controller-manager-5b96c6dbc5-t7fj2\" (UID: \"f60d3c35-191d-441a-8caa-95913b917e57\") " pod="opendatahub/opendatahub-operator-controller-manager-5b96c6dbc5-t7fj2" Jul 22 02:43:35.865536 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:43:35.865432 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f60d3c35-191d-441a-8caa-95913b917e57-webhook-cert\") pod \"opendatahub-operator-controller-manager-5b96c6dbc5-t7fj2\" (UID: \"f60d3c35-191d-441a-8caa-95913b917e57\") " pod="opendatahub/opendatahub-operator-controller-manager-5b96c6dbc5-t7fj2" Jul 22 02:43:35.966647 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:43:35.966626 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/f60d3c35-191d-441a-8caa-95913b917e57-manifests\") pod \"opendatahub-operator-controller-manager-5b96c6dbc5-t7fj2\" (UID: \"f60d3c35-191d-441a-8caa-95913b917e57\") " pod="opendatahub/opendatahub-operator-controller-manager-5b96c6dbc5-t7fj2" Jul 22 02:43:35.966746 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:43:35.966652 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7kr5g\" (UniqueName: \"kubernetes.io/projected/f60d3c35-191d-441a-8caa-95913b917e57-kube-api-access-7kr5g\") pod \"opendatahub-operator-controller-manager-5b96c6dbc5-t7fj2\" (UID: \"f60d3c35-191d-441a-8caa-95913b917e57\") " pod="opendatahub/opendatahub-operator-controller-manager-5b96c6dbc5-t7fj2" Jul 22 02:43:35.966746 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:43:35.966674 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f60d3c35-191d-441a-8caa-95913b917e57-webhook-cert\") pod \"opendatahub-operator-controller-manager-5b96c6dbc5-t7fj2\" (UID: \"f60d3c35-191d-441a-8caa-95913b917e57\") " pod="opendatahub/opendatahub-operator-controller-manager-5b96c6dbc5-t7fj2" Jul 22 02:43:35.966746 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:43:35.966715 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f60d3c35-191d-441a-8caa-95913b917e57-tmp\") pod \"opendatahub-operator-controller-manager-5b96c6dbc5-t7fj2\" (UID: \"f60d3c35-191d-441a-8caa-95913b917e57\") " pod="opendatahub/opendatahub-operator-controller-manager-5b96c6dbc5-t7fj2" Jul 22 02:43:35.966746 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:43:35.966739 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/f60d3c35-191d-441a-8caa-95913b917e57-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5b96c6dbc5-t7fj2\" (UID: \"f60d3c35-191d-441a-8caa-95913b917e57\") " pod="opendatahub/opendatahub-operator-controller-manager-5b96c6dbc5-t7fj2" Jul 22 02:43:35.966992 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:43:35.966935 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/f60d3c35-191d-441a-8caa-95913b917e57-manifests\") pod \"opendatahub-operator-controller-manager-5b96c6dbc5-t7fj2\" (UID: \"f60d3c35-191d-441a-8caa-95913b917e57\") " pod="opendatahub/opendatahub-operator-controller-manager-5b96c6dbc5-t7fj2" Jul 22 02:43:35.968914 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:43:35.968890 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f60d3c35-191d-441a-8caa-95913b917e57-tmp\") pod \"opendatahub-operator-controller-manager-5b96c6dbc5-t7fj2\" (UID: \"f60d3c35-191d-441a-8caa-95913b917e57\") " pod="opendatahub/opendatahub-operator-controller-manager-5b96c6dbc5-t7fj2" Jul 22 02:43:35.969151 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:43:35.969133 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/f60d3c35-191d-441a-8caa-95913b917e57-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5b96c6dbc5-t7fj2\" (UID: \"f60d3c35-191d-441a-8caa-95913b917e57\") " pod="opendatahub/opendatahub-operator-controller-manager-5b96c6dbc5-t7fj2" Jul 22 02:43:35.969211 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:43:35.969171 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f60d3c35-191d-441a-8caa-95913b917e57-webhook-cert\") pod \"opendatahub-operator-controller-manager-5b96c6dbc5-t7fj2\" (UID: \"f60d3c35-191d-441a-8caa-95913b917e57\") " pod="opendatahub/opendatahub-operator-controller-manager-5b96c6dbc5-t7fj2" Jul 22 02:43:35.979043 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:43:35.979019 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7kr5g\" (UniqueName: \"kubernetes.io/projected/f60d3c35-191d-441a-8caa-95913b917e57-kube-api-access-7kr5g\") pod \"opendatahub-operator-controller-manager-5b96c6dbc5-t7fj2\" (UID: \"f60d3c35-191d-441a-8caa-95913b917e57\") " pod="opendatahub/opendatahub-operator-controller-manager-5b96c6dbc5-t7fj2" Jul 22 02:43:36.070293 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:43:36.070241 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-5b96c6dbc5-t7fj2" Jul 22 02:43:36.211605 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:43:36.211549 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5b96c6dbc5-t7fj2"] Jul 22 02:43:36.213862 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:43:36.213834 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf60d3c35_191d_441a_8caa_95913b917e57.slice/crio-d336437f091828da52eda9871fe2d5fce1bf9e13515b48b84f0ecc99149c97e6 WatchSource:0}: Error finding container d336437f091828da52eda9871fe2d5fce1bf9e13515b48b84f0ecc99149c97e6: Status 404 returned error can't find the container with id d336437f091828da52eda9871fe2d5fce1bf9e13515b48b84f0ecc99149c97e6 Jul 22 02:43:36.513462 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:43:36.513434 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5b96c6dbc5-t7fj2" event={"ID":"f60d3c35-191d-441a-8caa-95913b917e57","Type":"ContainerStarted","Data":"d336437f091828da52eda9871fe2d5fce1bf9e13515b48b84f0ecc99149c97e6"} Jul 22 02:43:39.527431 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:43:39.527363 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5b96c6dbc5-t7fj2" event={"ID":"f60d3c35-191d-441a-8caa-95913b917e57","Type":"ContainerStarted","Data":"ef9ab4db5d7fa7b9dcd61d0f3787d64b08268cee97434b94872adf54b3cb2e0b"} Jul 22 02:43:40.533053 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:43:40.532993 2578 generic.go:356] "Generic (PLEG): container finished" podID="f60d3c35-191d-441a-8caa-95913b917e57" containerID="ef9ab4db5d7fa7b9dcd61d0f3787d64b08268cee97434b94872adf54b3cb2e0b" exitCode=0 Jul 22 02:43:40.533348 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:43:40.533072 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5b96c6dbc5-t7fj2" event={"ID":"f60d3c35-191d-441a-8caa-95913b917e57","Type":"ContainerDied","Data":"ef9ab4db5d7fa7b9dcd61d0f3787d64b08268cee97434b94872adf54b3cb2e0b"} Jul 22 02:43:41.538972 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:43:41.538941 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5b96c6dbc5-t7fj2" event={"ID":"f60d3c35-191d-441a-8caa-95913b917e57","Type":"ContainerStarted","Data":"7714fc5593c3cde50957376b8c6f5d8588d7915f8701881bbe3dd0b88787d36d"} Jul 22 02:43:41.539392 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:43:41.539080 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-5b96c6dbc5-t7fj2" Jul 22 02:43:52.545009 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:43:52.544979 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-5b96c6dbc5-t7fj2" Jul 22 02:43:52.570397 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:43:52.570353 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-5b96c6dbc5-t7fj2" podStartSLOduration=13.276858956 podStartE2EDuration="17.570339181s" totalImagesPullingTime="3.297907166s" totalInitContainerRuntime="995.573059ms" isStatefulPod=false podCreationTimestamp="2026-07-22 02:43:35 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:43:41.613981872 +0000 UTC m=+387.933157171" watchObservedRunningTime="2026-07-22 02:43:52.570339181 +0000 UTC m=+398.889514484" Jul 22 02:43:55.503739 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:43:55.503708 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-7885ddc578-vv7tg"] Jul 22 02:43:55.511770 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:43:55.511751 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7885ddc578-vv7tg" Jul 22 02:43:55.514923 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:43:55.514904 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 22 02:43:55.516304 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:43:55.516288 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 22 02:43:55.516457 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:43:55.516441 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 22 02:43:55.516533 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:43:55.516441 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-g982k\"" Jul 22 02:43:55.516626 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:43:55.516538 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 22 02:43:55.524755 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:43:55.524733 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7885ddc578-vv7tg"] Jul 22 02:43:55.593056 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:43:55.593033 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ef04119c-df6f-4ad7-8619-fb9bc10be49b-tls-certs\") pod \"kube-auth-proxy-7885ddc578-vv7tg\" (UID: \"ef04119c-df6f-4ad7-8619-fb9bc10be49b\") " pod="openshift-ingress/kube-auth-proxy-7885ddc578-vv7tg" Jul 22 02:43:55.593146 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:43:55.593076 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ef04119c-df6f-4ad7-8619-fb9bc10be49b-tmp\") pod \"kube-auth-proxy-7885ddc578-vv7tg\" (UID: \"ef04119c-df6f-4ad7-8619-fb9bc10be49b\") " pod="openshift-ingress/kube-auth-proxy-7885ddc578-vv7tg" Jul 22 02:43:55.593146 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:43:55.593100 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-chz5q\" (UniqueName: \"kubernetes.io/projected/ef04119c-df6f-4ad7-8619-fb9bc10be49b-kube-api-access-chz5q\") pod \"kube-auth-proxy-7885ddc578-vv7tg\" (UID: \"ef04119c-df6f-4ad7-8619-fb9bc10be49b\") " pod="openshift-ingress/kube-auth-proxy-7885ddc578-vv7tg" Jul 22 02:43:55.694350 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:43:55.694328 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-chz5q\" (UniqueName: \"kubernetes.io/projected/ef04119c-df6f-4ad7-8619-fb9bc10be49b-kube-api-access-chz5q\") pod \"kube-auth-proxy-7885ddc578-vv7tg\" (UID: \"ef04119c-df6f-4ad7-8619-fb9bc10be49b\") " pod="openshift-ingress/kube-auth-proxy-7885ddc578-vv7tg" Jul 22 02:43:55.694436 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:43:55.694369 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ef04119c-df6f-4ad7-8619-fb9bc10be49b-tls-certs\") pod \"kube-auth-proxy-7885ddc578-vv7tg\" (UID: \"ef04119c-df6f-4ad7-8619-fb9bc10be49b\") " pod="openshift-ingress/kube-auth-proxy-7885ddc578-vv7tg" Jul 22 02:43:55.694436 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:43:55.694409 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ef04119c-df6f-4ad7-8619-fb9bc10be49b-tmp\") pod \"kube-auth-proxy-7885ddc578-vv7tg\" (UID: \"ef04119c-df6f-4ad7-8619-fb9bc10be49b\") " pod="openshift-ingress/kube-auth-proxy-7885ddc578-vv7tg" Jul 22 02:43:55.696760 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:43:55.696740 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ef04119c-df6f-4ad7-8619-fb9bc10be49b-tmp\") pod \"kube-auth-proxy-7885ddc578-vv7tg\" (UID: \"ef04119c-df6f-4ad7-8619-fb9bc10be49b\") " pod="openshift-ingress/kube-auth-proxy-7885ddc578-vv7tg" Jul 22 02:43:55.696878 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:43:55.696860 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ef04119c-df6f-4ad7-8619-fb9bc10be49b-tls-certs\") pod \"kube-auth-proxy-7885ddc578-vv7tg\" (UID: \"ef04119c-df6f-4ad7-8619-fb9bc10be49b\") " pod="openshift-ingress/kube-auth-proxy-7885ddc578-vv7tg" Jul 22 02:43:55.716749 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:43:55.716717 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-chz5q\" (UniqueName: \"kubernetes.io/projected/ef04119c-df6f-4ad7-8619-fb9bc10be49b-kube-api-access-chz5q\") pod \"kube-auth-proxy-7885ddc578-vv7tg\" (UID: \"ef04119c-df6f-4ad7-8619-fb9bc10be49b\") " pod="openshift-ingress/kube-auth-proxy-7885ddc578-vv7tg" Jul 22 02:43:55.821053 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:43:55.821000 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7885ddc578-vv7tg" Jul 22 02:43:55.970872 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:43:55.970839 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7885ddc578-vv7tg"] Jul 22 02:43:55.973018 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:43:55.972994 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef04119c_df6f_4ad7_8619_fb9bc10be49b.slice/crio-3e5e9307949fa67f8369380d4c658b498481476fa74f937105d3a471979efdf0 WatchSource:0}: Error finding container 3e5e9307949fa67f8369380d4c658b498481476fa74f937105d3a471979efdf0: Status 404 returned error can't find the container with id 3e5e9307949fa67f8369380d4c658b498481476fa74f937105d3a471979efdf0 Jul 22 02:43:56.596736 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:43:56.596703 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7885ddc578-vv7tg" event={"ID":"ef04119c-df6f-4ad7-8619-fb9bc10be49b","Type":"ContainerStarted","Data":"3e5e9307949fa67f8369380d4c658b498481476fa74f937105d3a471979efdf0"} Jul 22 02:43:56.924294 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:43:56.924219 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-4fz8t"] Jul 22 02:43:56.927671 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:43:56.927647 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-4fz8t" Jul 22 02:43:56.930373 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:43:56.930313 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Jul 22 02:43:56.947204 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:43:56.947179 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-4fz8t"] Jul 22 02:43:57.007017 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:43:57.006985 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/27c81423-5417-4cb0-87f8-2edaa3b345d5-tls-certs\") pod \"model-serving-api-54d89bd5fc-4fz8t\" (UID: \"27c81423-5417-4cb0-87f8-2edaa3b345d5\") " pod="opendatahub/model-serving-api-54d89bd5fc-4fz8t" Jul 22 02:43:57.007152 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:43:57.007040 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9dvpp\" (UniqueName: \"kubernetes.io/projected/27c81423-5417-4cb0-87f8-2edaa3b345d5-kube-api-access-9dvpp\") pod \"model-serving-api-54d89bd5fc-4fz8t\" (UID: \"27c81423-5417-4cb0-87f8-2edaa3b345d5\") " pod="opendatahub/model-serving-api-54d89bd5fc-4fz8t" Jul 22 02:43:57.108268 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:43:57.108242 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/27c81423-5417-4cb0-87f8-2edaa3b345d5-tls-certs\") pod \"model-serving-api-54d89bd5fc-4fz8t\" (UID: \"27c81423-5417-4cb0-87f8-2edaa3b345d5\") " pod="opendatahub/model-serving-api-54d89bd5fc-4fz8t" Jul 22 02:43:57.108400 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:43:57.108280 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9dvpp\" (UniqueName: \"kubernetes.io/projected/27c81423-5417-4cb0-87f8-2edaa3b345d5-kube-api-access-9dvpp\") pod \"model-serving-api-54d89bd5fc-4fz8t\" (UID: \"27c81423-5417-4cb0-87f8-2edaa3b345d5\") " pod="opendatahub/model-serving-api-54d89bd5fc-4fz8t" Jul 22 02:43:57.110863 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:43:57.110844 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/27c81423-5417-4cb0-87f8-2edaa3b345d5-tls-certs\") pod \"model-serving-api-54d89bd5fc-4fz8t\" (UID: \"27c81423-5417-4cb0-87f8-2edaa3b345d5\") " pod="opendatahub/model-serving-api-54d89bd5fc-4fz8t" Jul 22 02:43:57.121092 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:43:57.121074 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9dvpp\" (UniqueName: \"kubernetes.io/projected/27c81423-5417-4cb0-87f8-2edaa3b345d5-kube-api-access-9dvpp\") pod \"model-serving-api-54d89bd5fc-4fz8t\" (UID: \"27c81423-5417-4cb0-87f8-2edaa3b345d5\") " pod="opendatahub/model-serving-api-54d89bd5fc-4fz8t" Jul 22 02:43:57.240351 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:43:57.240295 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-4fz8t" Jul 22 02:43:57.373248 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:43:57.373218 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-4fz8t"] Jul 22 02:43:57.375791 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:43:57.375767 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27c81423_5417_4cb0_87f8_2edaa3b345d5.slice/crio-b0e16d37d09c3983a61eb453840470dfe4807320e9c06e0d7b1742ef88c85319 WatchSource:0}: Error finding container b0e16d37d09c3983a61eb453840470dfe4807320e9c06e0d7b1742ef88c85319: Status 404 returned error can't find the container with id b0e16d37d09c3983a61eb453840470dfe4807320e9c06e0d7b1742ef88c85319 Jul 22 02:43:57.475406 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:43:57.475377 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-mhjlf"] Jul 22 02:43:57.479916 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:43:57.479894 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-mhjlf" Jul 22 02:43:57.483063 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:43:57.483039 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Jul 22 02:43:57.483230 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:43:57.483072 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Jul 22 02:43:57.483230 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:43:57.483095 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-rgj7v\"" Jul 22 02:43:57.483230 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:43:57.483040 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Jul 22 02:43:57.483633 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:43:57.483610 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 22 02:43:57.506370 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:43:57.506315 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-mhjlf"] Jul 22 02:43:57.511438 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:43:57.511417 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/3d2d9ffc-ecbe-44ca-b117-4d39b1377785-cacerts\") pod \"istiod-openshift-gateway-7574459656-mhjlf\" (UID: \"3d2d9ffc-ecbe-44ca-b117-4d39b1377785\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-mhjlf" Jul 22 02:43:57.511541 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:43:57.511456 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d2d9ffc-ecbe-44ca-b117-4d39b1377785-istio-token\") pod \"istiod-openshift-gateway-7574459656-mhjlf\" (UID: \"3d2d9ffc-ecbe-44ca-b117-4d39b1377785\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-mhjlf" Jul 22 02:43:57.511541 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:43:57.511492 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/3d2d9ffc-ecbe-44ca-b117-4d39b1377785-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-mhjlf\" (UID: \"3d2d9ffc-ecbe-44ca-b117-4d39b1377785\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-mhjlf" Jul 22 02:43:57.511652 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:43:57.511572 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3d2d9ffc-ecbe-44ca-b117-4d39b1377785-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-mhjlf\" (UID: \"3d2d9ffc-ecbe-44ca-b117-4d39b1377785\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-mhjlf" Jul 22 02:43:57.511652 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:43:57.511607 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6q56d\" (UniqueName: \"kubernetes.io/projected/3d2d9ffc-ecbe-44ca-b117-4d39b1377785-kube-api-access-6q56d\") pod \"istiod-openshift-gateway-7574459656-mhjlf\" (UID: \"3d2d9ffc-ecbe-44ca-b117-4d39b1377785\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-mhjlf" Jul 22 02:43:57.511652 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:43:57.511647 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d2d9ffc-ecbe-44ca-b117-4d39b1377785-local-certs\") pod \"istiod-openshift-gateway-7574459656-mhjlf\" (UID: \"3d2d9ffc-ecbe-44ca-b117-4d39b1377785\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-mhjlf" Jul 22 02:43:57.511768 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:43:57.511738 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/3d2d9ffc-ecbe-44ca-b117-4d39b1377785-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-mhjlf\" (UID: \"3d2d9ffc-ecbe-44ca-b117-4d39b1377785\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-mhjlf" Jul 22 02:43:57.602842 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:43:57.602813 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-4fz8t" event={"ID":"27c81423-5417-4cb0-87f8-2edaa3b345d5","Type":"ContainerStarted","Data":"b0e16d37d09c3983a61eb453840470dfe4807320e9c06e0d7b1742ef88c85319"} Jul 22 02:43:57.612147 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:43:57.612116 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d2d9ffc-ecbe-44ca-b117-4d39b1377785-local-certs\") pod \"istiod-openshift-gateway-7574459656-mhjlf\" (UID: \"3d2d9ffc-ecbe-44ca-b117-4d39b1377785\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-mhjlf" Jul 22 02:43:57.612269 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:43:57.612170 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/3d2d9ffc-ecbe-44ca-b117-4d39b1377785-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-mhjlf\" (UID: \"3d2d9ffc-ecbe-44ca-b117-4d39b1377785\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-mhjlf" Jul 22 02:43:57.612269 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:43:57.612210 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/3d2d9ffc-ecbe-44ca-b117-4d39b1377785-cacerts\") pod \"istiod-openshift-gateway-7574459656-mhjlf\" (UID: \"3d2d9ffc-ecbe-44ca-b117-4d39b1377785\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-mhjlf" Jul 22 02:43:57.612269 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:43:57.612235 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d2d9ffc-ecbe-44ca-b117-4d39b1377785-istio-token\") pod \"istiod-openshift-gateway-7574459656-mhjlf\" (UID: \"3d2d9ffc-ecbe-44ca-b117-4d39b1377785\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-mhjlf" Jul 22 02:43:57.612418 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:43:57.612273 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/3d2d9ffc-ecbe-44ca-b117-4d39b1377785-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-mhjlf\" (UID: \"3d2d9ffc-ecbe-44ca-b117-4d39b1377785\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-mhjlf" Jul 22 02:43:57.612418 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:43:57.612308 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3d2d9ffc-ecbe-44ca-b117-4d39b1377785-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-mhjlf\" (UID: \"3d2d9ffc-ecbe-44ca-b117-4d39b1377785\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-mhjlf" Jul 22 02:43:57.612524 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:43:57.612437 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6q56d\" (UniqueName: \"kubernetes.io/projected/3d2d9ffc-ecbe-44ca-b117-4d39b1377785-kube-api-access-6q56d\") pod \"istiod-openshift-gateway-7574459656-mhjlf\" (UID: \"3d2d9ffc-ecbe-44ca-b117-4d39b1377785\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-mhjlf" Jul 22 02:43:57.613000 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:43:57.612959 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/3d2d9ffc-ecbe-44ca-b117-4d39b1377785-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-mhjlf\" (UID: \"3d2d9ffc-ecbe-44ca-b117-4d39b1377785\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-mhjlf" Jul 22 02:43:57.615088 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:43:57.615062 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/3d2d9ffc-ecbe-44ca-b117-4d39b1377785-cacerts\") pod \"istiod-openshift-gateway-7574459656-mhjlf\" (UID: \"3d2d9ffc-ecbe-44ca-b117-4d39b1377785\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-mhjlf" Jul 22 02:43:57.615642 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:43:57.615618 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d2d9ffc-ecbe-44ca-b117-4d39b1377785-local-certs\") pod \"istiod-openshift-gateway-7574459656-mhjlf\" (UID: \"3d2d9ffc-ecbe-44ca-b117-4d39b1377785\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-mhjlf" Jul 22 02:43:57.615727 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:43:57.615665 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3d2d9ffc-ecbe-44ca-b117-4d39b1377785-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-mhjlf\" (UID: \"3d2d9ffc-ecbe-44ca-b117-4d39b1377785\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-mhjlf" Jul 22 02:43:57.615777 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:43:57.615754 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/3d2d9ffc-ecbe-44ca-b117-4d39b1377785-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-mhjlf\" (UID: \"3d2d9ffc-ecbe-44ca-b117-4d39b1377785\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-mhjlf" Jul 22 02:43:57.640208 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:43:57.640185 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6q56d\" (UniqueName: \"kubernetes.io/projected/3d2d9ffc-ecbe-44ca-b117-4d39b1377785-kube-api-access-6q56d\") pod \"istiod-openshift-gateway-7574459656-mhjlf\" (UID: \"3d2d9ffc-ecbe-44ca-b117-4d39b1377785\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-mhjlf" Jul 22 02:43:57.642025 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:43:57.642008 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d2d9ffc-ecbe-44ca-b117-4d39b1377785-istio-token\") pod \"istiod-openshift-gateway-7574459656-mhjlf\" (UID: \"3d2d9ffc-ecbe-44ca-b117-4d39b1377785\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-mhjlf" Jul 22 02:43:57.791719 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:43:57.791659 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-mhjlf" Jul 22 02:43:58.592770 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:43:58.592743 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-mhjlf"] Jul 22 02:43:58.595925 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:43:58.595898 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d2d9ffc_ecbe_44ca_b117_4d39b1377785.slice/crio-4c53753caaf573406581e67291218023a6d5215fd09f3498a419aadb14f16927 WatchSource:0}: Error finding container 4c53753caaf573406581e67291218023a6d5215fd09f3498a419aadb14f16927: Status 404 returned error can't find the container with id 4c53753caaf573406581e67291218023a6d5215fd09f3498a419aadb14f16927 Jul 22 02:43:58.609413 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:43:58.609310 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7885ddc578-vv7tg" event={"ID":"ef04119c-df6f-4ad7-8619-fb9bc10be49b","Type":"ContainerStarted","Data":"b5ebab0ca21b64b4c1568c65d90867e55f4c44b7d9f411cc9d2d3afcab058837"} Jul 22 02:43:58.610905 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:43:58.610880 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-mhjlf" event={"ID":"3d2d9ffc-ecbe-44ca-b117-4d39b1377785","Type":"ContainerStarted","Data":"4c53753caaf573406581e67291218023a6d5215fd09f3498a419aadb14f16927"} Jul 22 02:43:58.635145 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:43:58.635099 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-7885ddc578-vv7tg" podStartSLOduration=1.100937682 podStartE2EDuration="3.635085216s" totalImagesPullingTime="2.534147534s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 02:43:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:43:58.633647576 +0000 UTC m=+404.952822876" watchObservedRunningTime="2026-07-22 02:43:58.635085216 +0000 UTC m=+404.954260515" Jul 22 02:43:59.621156 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:43:59.621121 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-4fz8t" event={"ID":"27c81423-5417-4cb0-87f8-2edaa3b345d5","Type":"ContainerStarted","Data":"2dee893f1d46e3e13fc65c283358078a89549ebfb15e5bc9ddf7db6b895c26bf"} Jul 22 02:43:59.650519 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:43:59.650475 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-4fz8t" podStartSLOduration=2.021765277 podStartE2EDuration="3.65046191s" totalImagesPullingTime="1.628696633s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 02:43:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:43:59.649158758 +0000 UTC m=+405.968334057" watchObservedRunningTime="2026-07-22 02:43:59.65046191 +0000 UTC m=+405.969637242" Jul 22 02:44:00.625436 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:44:00.625412 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-4fz8t" Jul 22 02:44:08.523450 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:44:08.523422 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-g5rrk"] Jul 22 02:44:08.526529 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:44:08.526512 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-g5rrk" Jul 22 02:44:08.533275 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:44:08.533187 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-6zzbd\"" Jul 22 02:44:08.533425 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:44:08.533391 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Jul 22 02:44:08.563412 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:44:08.563384 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-g5rrk"] Jul 22 02:44:08.576735 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:44:08.576700 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-f6p2m"] Jul 22 02:44:08.579847 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:44:08.579824 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-f6p2m" Jul 22 02:44:08.583039 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:44:08.583023 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Jul 22 02:44:08.584018 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:44:08.584001 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-nvw65\"" Jul 22 02:44:08.592484 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:44:08.592466 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e61437db-765c-4abf-bf6b-2df9bf59faf4-cert\") pod \"kserve-controller-manager-d6f4b65b8-g5rrk\" (UID: \"e61437db-765c-4abf-bf6b-2df9bf59faf4\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-g5rrk" Jul 22 02:44:08.592587 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:44:08.592513 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xbntf\" (UniqueName: \"kubernetes.io/projected/e61437db-765c-4abf-bf6b-2df9bf59faf4-kube-api-access-xbntf\") pod \"kserve-controller-manager-d6f4b65b8-g5rrk\" (UID: \"e61437db-765c-4abf-bf6b-2df9bf59faf4\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-g5rrk" Jul 22 02:44:08.597350 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:44:08.597333 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-f6p2m"] Jul 22 02:44:08.692946 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:44:08.692919 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e61437db-765c-4abf-bf6b-2df9bf59faf4-cert\") pod \"kserve-controller-manager-d6f4b65b8-g5rrk\" (UID: \"e61437db-765c-4abf-bf6b-2df9bf59faf4\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-g5rrk" Jul 22 02:44:08.693044 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:44:08.692949 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bab76e8d-039e-45e2-8122-e402782e7cd1-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-f6p2m\" (UID: \"bab76e8d-039e-45e2-8122-e402782e7cd1\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-f6p2m" Jul 22 02:44:08.693044 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:44:08.692976 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gt8dh\" (UniqueName: \"kubernetes.io/projected/bab76e8d-039e-45e2-8122-e402782e7cd1-kube-api-access-gt8dh\") pod \"llmisvc-controller-manager-777c6bd7d5-f6p2m\" (UID: \"bab76e8d-039e-45e2-8122-e402782e7cd1\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-f6p2m" Jul 22 02:44:08.693044 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:44:08.693002 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xbntf\" (UniqueName: \"kubernetes.io/projected/e61437db-765c-4abf-bf6b-2df9bf59faf4-kube-api-access-xbntf\") pod \"kserve-controller-manager-d6f4b65b8-g5rrk\" (UID: \"e61437db-765c-4abf-bf6b-2df9bf59faf4\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-g5rrk" Jul 22 02:44:08.695390 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:44:08.695368 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e61437db-765c-4abf-bf6b-2df9bf59faf4-cert\") pod \"kserve-controller-manager-d6f4b65b8-g5rrk\" (UID: \"e61437db-765c-4abf-bf6b-2df9bf59faf4\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-g5rrk" Jul 22 02:44:08.702609 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:44:08.702588 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xbntf\" (UniqueName: \"kubernetes.io/projected/e61437db-765c-4abf-bf6b-2df9bf59faf4-kube-api-access-xbntf\") pod \"kserve-controller-manager-d6f4b65b8-g5rrk\" (UID: \"e61437db-765c-4abf-bf6b-2df9bf59faf4\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-g5rrk" Jul 22 02:44:08.793550 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:44:08.793495 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bab76e8d-039e-45e2-8122-e402782e7cd1-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-f6p2m\" (UID: \"bab76e8d-039e-45e2-8122-e402782e7cd1\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-f6p2m" Jul 22 02:44:08.793550 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:44:08.793532 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gt8dh\" (UniqueName: \"kubernetes.io/projected/bab76e8d-039e-45e2-8122-e402782e7cd1-kube-api-access-gt8dh\") pod \"llmisvc-controller-manager-777c6bd7d5-f6p2m\" (UID: \"bab76e8d-039e-45e2-8122-e402782e7cd1\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-f6p2m" Jul 22 02:44:08.795570 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:44:08.795540 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bab76e8d-039e-45e2-8122-e402782e7cd1-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-f6p2m\" (UID: \"bab76e8d-039e-45e2-8122-e402782e7cd1\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-f6p2m" Jul 22 02:44:08.811727 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:44:08.811702 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gt8dh\" (UniqueName: \"kubernetes.io/projected/bab76e8d-039e-45e2-8122-e402782e7cd1-kube-api-access-gt8dh\") pod \"llmisvc-controller-manager-777c6bd7d5-f6p2m\" (UID: \"bab76e8d-039e-45e2-8122-e402782e7cd1\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-f6p2m" Jul 22 02:44:08.838305 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:44:08.838282 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-g5rrk" Jul 22 02:44:08.890906 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:44:08.890875 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-f6p2m" Jul 22 02:44:09.010648 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:44:09.010620 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-g5rrk"] Jul 22 02:44:09.018095 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:44:09.018071 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode61437db_765c_4abf_bf6b_2df9bf59faf4.slice/crio-fb71911df53c210e0deec9de6b0898df4904d686a3ea5d3000e81cfb8dcf8c6c WatchSource:0}: Error finding container fb71911df53c210e0deec9de6b0898df4904d686a3ea5d3000e81cfb8dcf8c6c: Status 404 returned error can't find the container with id fb71911df53c210e0deec9de6b0898df4904d686a3ea5d3000e81cfb8dcf8c6c Jul 22 02:44:09.040327 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:44:09.040308 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-f6p2m"] Jul 22 02:44:09.041692 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:44:09.041669 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podbab76e8d_039e_45e2_8122_e402782e7cd1.slice/crio-d19e04fcff859805dbb24b0f3280df9e2baa92c1ee22cc53846a5cbe6b1bf4bc WatchSource:0}: Error finding container d19e04fcff859805dbb24b0f3280df9e2baa92c1ee22cc53846a5cbe6b1bf4bc: Status 404 returned error can't find the container with id d19e04fcff859805dbb24b0f3280df9e2baa92c1ee22cc53846a5cbe6b1bf4bc Jul 22 02:44:09.663228 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:44:09.663194 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-f6p2m" event={"ID":"bab76e8d-039e-45e2-8122-e402782e7cd1","Type":"ContainerStarted","Data":"d19e04fcff859805dbb24b0f3280df9e2baa92c1ee22cc53846a5cbe6b1bf4bc"} Jul 22 02:44:09.664694 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:44:09.664667 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-g5rrk" event={"ID":"e61437db-765c-4abf-bf6b-2df9bf59faf4","Type":"ContainerStarted","Data":"fb71911df53c210e0deec9de6b0898df4904d686a3ea5d3000e81cfb8dcf8c6c"} Jul 22 02:44:11.633980 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:44:11.633954 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-4fz8t" Jul 22 02:44:11.677717 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:44:11.677636 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-g5rrk" event={"ID":"e61437db-765c-4abf-bf6b-2df9bf59faf4","Type":"ContainerStarted","Data":"872120e699da6cdf0a40737c372ea55210e138fcf5a2b614fa13fda6385104f8"} Jul 22 02:44:11.677852 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:44:11.677734 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-g5rrk" Jul 22 02:44:12.693184 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:44:12.693137 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-f6p2m" event={"ID":"bab76e8d-039e-45e2-8122-e402782e7cd1","Type":"ContainerStarted","Data":"95925be795f84a2cefc84c3955e88ffc2772557a2c12f70f88d5bd38499b97f0"} Jul 22 02:44:12.773997 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:44:12.773947 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-g5rrk" podStartSLOduration=2.402402279 podStartE2EDuration="4.773932337s" totalImagesPullingTime="2.371530058s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 02:44:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:44:11.77409337 +0000 UTC m=+418.093268669" watchObservedRunningTime="2026-07-22 02:44:12.773932337 +0000 UTC m=+419.093107631" Jul 22 02:44:13.697111 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:44:13.697083 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-f6p2m" Jul 22 02:44:14.880927 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:44:14.880880 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 22 02:44:14.881194 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:44:14.880952 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 22 02:44:15.708632 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:44:15.708599 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-mhjlf" event={"ID":"3d2d9ffc-ecbe-44ca-b117-4d39b1377785","Type":"ContainerStarted","Data":"27a4ea941d12c5e465848a09711fb7892490e50b87a26554508f4e8a1ebf8989"} Jul 22 02:44:15.708783 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:44:15.708702 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7574459656-mhjlf" Jul 22 02:44:15.710019 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:44:15.709995 2578 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7574459656-mhjlf container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 22 02:44:15.710166 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:44:15.710042 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7574459656-mhjlf" podUID="3d2d9ffc-ecbe-44ca-b117-4d39b1377785" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 22 02:44:15.712485 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:44:15.712420 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-f6p2m" podStartSLOduration=4.765475998 podStartE2EDuration="7.712406652s" totalImagesPullingTime="2.946930654s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 02:44:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:44:12.773030113 +0000 UTC m=+419.092205413" watchObservedRunningTime="2026-07-22 02:44:15.712406652 +0000 UTC m=+422.031581951" Jul 22 02:44:15.712884 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:44:15.712866 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf46vpxp"] Jul 22 02:44:15.716016 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:44:15.716000 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf46vpxp" Jul 22 02:44:15.718770 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:44:15.718753 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-smnmv\"" Jul 22 02:44:15.736293 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:44:15.736275 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf46vpxp"] Jul 22 02:44:15.740189 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:44:15.740147 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7574459656-mhjlf" podStartSLOduration=2.457669047 podStartE2EDuration="18.740130502s" totalImagesPullingTime="16.282461455s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 02:43:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:44:15.739617201 +0000 UTC m=+422.058792500" watchObservedRunningTime="2026-07-22 02:44:15.740130502 +0000 UTC m=+422.059305803" Jul 22 02:44:15.853338 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:44:15.853314 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07b23859-7ee5-4299-965e-b1592f48e054-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf46vpxp\" (UID: \"07b23859-7ee5-4299-965e-b1592f48e054\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf46vpxp" Jul 22 02:44:15.853462 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:44:15.853343 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07b23859-7ee5-4299-965e-b1592f48e054-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf46vpxp\" (UID: \"07b23859-7ee5-4299-965e-b1592f48e054\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf46vpxp" Jul 22 02:44:15.853462 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:44:15.853359 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wmqvp\" (UniqueName: \"kubernetes.io/projected/07b23859-7ee5-4299-965e-b1592f48e054-kube-api-access-wmqvp\") pod \"data-science-gateway-data-science-gateway-class-6576859cf46vpxp\" (UID: \"07b23859-7ee5-4299-965e-b1592f48e054\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf46vpxp" Jul 22 02:44:15.853462 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:44:15.853376 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07b23859-7ee5-4299-965e-b1592f48e054-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf46vpxp\" (UID: \"07b23859-7ee5-4299-965e-b1592f48e054\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf46vpxp" Jul 22 02:44:15.853462 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:44:15.853425 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07b23859-7ee5-4299-965e-b1592f48e054-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf46vpxp\" (UID: \"07b23859-7ee5-4299-965e-b1592f48e054\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf46vpxp" Jul 22 02:44:15.853693 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:44:15.853468 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07b23859-7ee5-4299-965e-b1592f48e054-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf46vpxp\" (UID: \"07b23859-7ee5-4299-965e-b1592f48e054\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf46vpxp" Jul 22 02:44:15.853693 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:44:15.853488 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07b23859-7ee5-4299-965e-b1592f48e054-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf46vpxp\" (UID: \"07b23859-7ee5-4299-965e-b1592f48e054\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf46vpxp" Jul 22 02:44:15.853693 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:44:15.853602 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07b23859-7ee5-4299-965e-b1592f48e054-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf46vpxp\" (UID: \"07b23859-7ee5-4299-965e-b1592f48e054\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf46vpxp" Jul 22 02:44:15.853693 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:44:15.853653 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07b23859-7ee5-4299-965e-b1592f48e054-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf46vpxp\" (UID: \"07b23859-7ee5-4299-965e-b1592f48e054\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf46vpxp" Jul 22 02:44:15.954065 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:44:15.954036 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07b23859-7ee5-4299-965e-b1592f48e054-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf46vpxp\" (UID: \"07b23859-7ee5-4299-965e-b1592f48e054\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf46vpxp" Jul 22 02:44:15.954065 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:44:15.954066 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07b23859-7ee5-4299-965e-b1592f48e054-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf46vpxp\" (UID: \"07b23859-7ee5-4299-965e-b1592f48e054\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf46vpxp" Jul 22 02:44:15.954472 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:44:15.954084 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wmqvp\" (UniqueName: \"kubernetes.io/projected/07b23859-7ee5-4299-965e-b1592f48e054-kube-api-access-wmqvp\") pod \"data-science-gateway-data-science-gateway-class-6576859cf46vpxp\" (UID: \"07b23859-7ee5-4299-965e-b1592f48e054\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf46vpxp" Jul 22 02:44:15.954472 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:44:15.954230 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07b23859-7ee5-4299-965e-b1592f48e054-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf46vpxp\" (UID: \"07b23859-7ee5-4299-965e-b1592f48e054\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf46vpxp" Jul 22 02:44:15.954472 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:44:15.954261 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07b23859-7ee5-4299-965e-b1592f48e054-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf46vpxp\" (UID: \"07b23859-7ee5-4299-965e-b1592f48e054\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf46vpxp" Jul 22 02:44:15.954472 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:44:15.954301 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07b23859-7ee5-4299-965e-b1592f48e054-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf46vpxp\" (UID: \"07b23859-7ee5-4299-965e-b1592f48e054\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf46vpxp" Jul 22 02:44:15.954472 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:44:15.954321 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07b23859-7ee5-4299-965e-b1592f48e054-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf46vpxp\" (UID: \"07b23859-7ee5-4299-965e-b1592f48e054\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf46vpxp" Jul 22 02:44:15.954472 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:44:15.954430 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07b23859-7ee5-4299-965e-b1592f48e054-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf46vpxp\" (UID: \"07b23859-7ee5-4299-965e-b1592f48e054\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf46vpxp" Jul 22 02:44:15.954778 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:44:15.954489 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07b23859-7ee5-4299-965e-b1592f48e054-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf46vpxp\" (UID: \"07b23859-7ee5-4299-965e-b1592f48e054\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf46vpxp" Jul 22 02:44:15.954778 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:44:15.954514 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07b23859-7ee5-4299-965e-b1592f48e054-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf46vpxp\" (UID: \"07b23859-7ee5-4299-965e-b1592f48e054\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf46vpxp" Jul 22 02:44:15.954778 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:44:15.954643 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07b23859-7ee5-4299-965e-b1592f48e054-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf46vpxp\" (UID: \"07b23859-7ee5-4299-965e-b1592f48e054\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf46vpxp" Jul 22 02:44:15.954778 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:44:15.954736 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07b23859-7ee5-4299-965e-b1592f48e054-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf46vpxp\" (UID: \"07b23859-7ee5-4299-965e-b1592f48e054\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf46vpxp" Jul 22 02:44:15.954778 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:44:15.954769 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07b23859-7ee5-4299-965e-b1592f48e054-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf46vpxp\" (UID: \"07b23859-7ee5-4299-965e-b1592f48e054\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf46vpxp" Jul 22 02:44:15.955345 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:44:15.955323 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07b23859-7ee5-4299-965e-b1592f48e054-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf46vpxp\" (UID: \"07b23859-7ee5-4299-965e-b1592f48e054\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf46vpxp" Jul 22 02:44:15.956573 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:44:15.956534 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07b23859-7ee5-4299-965e-b1592f48e054-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf46vpxp\" (UID: \"07b23859-7ee5-4299-965e-b1592f48e054\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf46vpxp" Jul 22 02:44:15.956781 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:44:15.956766 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07b23859-7ee5-4299-965e-b1592f48e054-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf46vpxp\" (UID: \"07b23859-7ee5-4299-965e-b1592f48e054\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf46vpxp" Jul 22 02:44:15.969319 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:44:15.969263 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wmqvp\" (UniqueName: \"kubernetes.io/projected/07b23859-7ee5-4299-965e-b1592f48e054-kube-api-access-wmqvp\") pod \"data-science-gateway-data-science-gateway-class-6576859cf46vpxp\" (UID: \"07b23859-7ee5-4299-965e-b1592f48e054\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf46vpxp" Jul 22 02:44:15.971611 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:44:15.971595 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07b23859-7ee5-4299-965e-b1592f48e054-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf46vpxp\" (UID: \"07b23859-7ee5-4299-965e-b1592f48e054\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf46vpxp" Jul 22 02:44:16.025754 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:44:16.025733 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf46vpxp" Jul 22 02:44:16.200698 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:44:16.200675 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf46vpxp"] Jul 22 02:44:16.201592 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:44:16.201544 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07b23859_7ee5_4299_965e_b1592f48e054.slice/crio-100e007ad2e990226a69729d9c43427d40d20e9024ee83d5e0f658e053b0bcea WatchSource:0}: Error finding container 100e007ad2e990226a69729d9c43427d40d20e9024ee83d5e0f658e053b0bcea: Status 404 returned error can't find the container with id 100e007ad2e990226a69729d9c43427d40d20e9024ee83d5e0f658e053b0bcea Jul 22 02:44:16.655870 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:44:16.655836 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-mhjlf"] Jul 22 02:44:16.714127 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:44:16.714093 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf46vpxp" event={"ID":"07b23859-7ee5-4299-965e-b1592f48e054","Type":"ContainerStarted","Data":"100e007ad2e990226a69729d9c43427d40d20e9024ee83d5e0f658e053b0bcea"} Jul 22 02:44:16.714760 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:44:16.714740 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7574459656-mhjlf" Jul 22 02:44:17.720296 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:44:17.720257 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7574459656-mhjlf" podUID="3d2d9ffc-ecbe-44ca-b117-4d39b1377785" containerName="discovery" containerID="cri-o://27a4ea941d12c5e465848a09711fb7892490e50b87a26554508f4e8a1ebf8989" gracePeriod=30 Jul 22 02:44:17.981004 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:44:17.980948 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-mhjlf" Jul 22 02:44:18.071078 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:44:18.071049 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d2d9ffc-ecbe-44ca-b117-4d39b1377785-istio-token\") pod \"3d2d9ffc-ecbe-44ca-b117-4d39b1377785\" (UID: \"3d2d9ffc-ecbe-44ca-b117-4d39b1377785\") " Jul 22 02:44:18.071195 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:44:18.071098 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6q56d\" (UniqueName: \"kubernetes.io/projected/3d2d9ffc-ecbe-44ca-b117-4d39b1377785-kube-api-access-6q56d\") pod \"3d2d9ffc-ecbe-44ca-b117-4d39b1377785\" (UID: \"3d2d9ffc-ecbe-44ca-b117-4d39b1377785\") " Jul 22 02:44:18.071195 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:44:18.071153 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/3d2d9ffc-ecbe-44ca-b117-4d39b1377785-cacerts\") pod \"3d2d9ffc-ecbe-44ca-b117-4d39b1377785\" (UID: \"3d2d9ffc-ecbe-44ca-b117-4d39b1377785\") " Jul 22 02:44:18.071195 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:44:18.071189 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/3d2d9ffc-ecbe-44ca-b117-4d39b1377785-istio-csr-dns-cert\") pod \"3d2d9ffc-ecbe-44ca-b117-4d39b1377785\" (UID: \"3d2d9ffc-ecbe-44ca-b117-4d39b1377785\") " Jul 22 02:44:18.071299 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:44:18.071222 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d2d9ffc-ecbe-44ca-b117-4d39b1377785-local-certs\") pod \"3d2d9ffc-ecbe-44ca-b117-4d39b1377785\" (UID: \"3d2d9ffc-ecbe-44ca-b117-4d39b1377785\") " Jul 22 02:44:18.071299 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:44:18.071246 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3d2d9ffc-ecbe-44ca-b117-4d39b1377785-istio-kubeconfig\") pod \"3d2d9ffc-ecbe-44ca-b117-4d39b1377785\" (UID: \"3d2d9ffc-ecbe-44ca-b117-4d39b1377785\") " Jul 22 02:44:18.071299 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:44:18.071286 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/3d2d9ffc-ecbe-44ca-b117-4d39b1377785-istio-csr-ca-configmap\") pod \"3d2d9ffc-ecbe-44ca-b117-4d39b1377785\" (UID: \"3d2d9ffc-ecbe-44ca-b117-4d39b1377785\") " Jul 22 02:44:18.072075 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:44:18.071855 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d2d9ffc-ecbe-44ca-b117-4d39b1377785-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "3d2d9ffc-ecbe-44ca-b117-4d39b1377785" (UID: "3d2d9ffc-ecbe-44ca-b117-4d39b1377785"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 02:44:18.073619 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:44:18.073591 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d2d9ffc-ecbe-44ca-b117-4d39b1377785-kube-api-access-6q56d" (OuterVolumeSpecName: "kube-api-access-6q56d") pod "3d2d9ffc-ecbe-44ca-b117-4d39b1377785" (UID: "3d2d9ffc-ecbe-44ca-b117-4d39b1377785"). InnerVolumeSpecName "kube-api-access-6q56d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 02:44:18.073619 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:44:18.073604 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d2d9ffc-ecbe-44ca-b117-4d39b1377785-istio-token" (OuterVolumeSpecName: "istio-token") pod "3d2d9ffc-ecbe-44ca-b117-4d39b1377785" (UID: "3d2d9ffc-ecbe-44ca-b117-4d39b1377785"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 02:44:18.073778 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:44:18.073762 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3d2d9ffc-ecbe-44ca-b117-4d39b1377785-cacerts" (OuterVolumeSpecName: "cacerts") pod "3d2d9ffc-ecbe-44ca-b117-4d39b1377785" (UID: "3d2d9ffc-ecbe-44ca-b117-4d39b1377785"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 02:44:18.073835 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:44:18.073786 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3d2d9ffc-ecbe-44ca-b117-4d39b1377785-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "3d2d9ffc-ecbe-44ca-b117-4d39b1377785" (UID: "3d2d9ffc-ecbe-44ca-b117-4d39b1377785"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 02:44:18.073965 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:44:18.073944 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d2d9ffc-ecbe-44ca-b117-4d39b1377785-local-certs" (OuterVolumeSpecName: "local-certs") pod "3d2d9ffc-ecbe-44ca-b117-4d39b1377785" (UID: "3d2d9ffc-ecbe-44ca-b117-4d39b1377785"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 02:44:18.074065 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:44:18.074049 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3d2d9ffc-ecbe-44ca-b117-4d39b1377785-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "3d2d9ffc-ecbe-44ca-b117-4d39b1377785" (UID: "3d2d9ffc-ecbe-44ca-b117-4d39b1377785"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 02:44:18.172258 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:44:18.172238 2578 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d2d9ffc-ecbe-44ca-b117-4d39b1377785-istio-token\") on node \"ip-10-0-133-31.ec2.internal\" DevicePath \"\"" Jul 22 02:44:18.172353 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:44:18.172274 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6q56d\" (UniqueName: \"kubernetes.io/projected/3d2d9ffc-ecbe-44ca-b117-4d39b1377785-kube-api-access-6q56d\") on node \"ip-10-0-133-31.ec2.internal\" DevicePath \"\"" Jul 22 02:44:18.172353 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:44:18.172284 2578 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/3d2d9ffc-ecbe-44ca-b117-4d39b1377785-cacerts\") on node \"ip-10-0-133-31.ec2.internal\" DevicePath \"\"" Jul 22 02:44:18.172353 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:44:18.172293 2578 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/3d2d9ffc-ecbe-44ca-b117-4d39b1377785-istio-csr-dns-cert\") on node \"ip-10-0-133-31.ec2.internal\" DevicePath \"\"" Jul 22 02:44:18.172353 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:44:18.172301 2578 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d2d9ffc-ecbe-44ca-b117-4d39b1377785-local-certs\") on node \"ip-10-0-133-31.ec2.internal\" DevicePath \"\"" Jul 22 02:44:18.172353 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:44:18.172308 2578 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3d2d9ffc-ecbe-44ca-b117-4d39b1377785-istio-kubeconfig\") on node \"ip-10-0-133-31.ec2.internal\" DevicePath \"\"" Jul 22 02:44:18.172353 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:44:18.172315 2578 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/3d2d9ffc-ecbe-44ca-b117-4d39b1377785-istio-csr-ca-configmap\") on node \"ip-10-0-133-31.ec2.internal\" DevicePath \"\"" Jul 22 02:44:18.725128 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:44:18.725097 2578 generic.go:356] "Generic (PLEG): container finished" podID="3d2d9ffc-ecbe-44ca-b117-4d39b1377785" containerID="27a4ea941d12c5e465848a09711fb7892490e50b87a26554508f4e8a1ebf8989" exitCode=0 Jul 22 02:44:18.725459 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:44:18.725144 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-mhjlf" Jul 22 02:44:18.725459 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:44:18.725189 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-mhjlf" event={"ID":"3d2d9ffc-ecbe-44ca-b117-4d39b1377785","Type":"ContainerDied","Data":"27a4ea941d12c5e465848a09711fb7892490e50b87a26554508f4e8a1ebf8989"} Jul 22 02:44:18.725459 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:44:18.725259 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-mhjlf" event={"ID":"3d2d9ffc-ecbe-44ca-b117-4d39b1377785","Type":"ContainerDied","Data":"4c53753caaf573406581e67291218023a6d5215fd09f3498a419aadb14f16927"} Jul 22 02:44:18.725459 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:44:18.725278 2578 scope.go:117] "RemoveContainer" containerID="27a4ea941d12c5e465848a09711fb7892490e50b87a26554508f4e8a1ebf8989" Jul 22 02:44:18.734937 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:44:18.734877 2578 scope.go:117] "RemoveContainer" containerID="27a4ea941d12c5e465848a09711fb7892490e50b87a26554508f4e8a1ebf8989" Jul 22 02:44:18.735347 ip-10-0-133-31 kubenswrapper[2578]: E0722 02:44:18.735328 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"27a4ea941d12c5e465848a09711fb7892490e50b87a26554508f4e8a1ebf8989\": container with ID starting with 27a4ea941d12c5e465848a09711fb7892490e50b87a26554508f4e8a1ebf8989 not found: ID does not exist" containerID="27a4ea941d12c5e465848a09711fb7892490e50b87a26554508f4e8a1ebf8989" Jul 22 02:44:18.735453 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:44:18.735354 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"27a4ea941d12c5e465848a09711fb7892490e50b87a26554508f4e8a1ebf8989"} err="failed to get container status \"27a4ea941d12c5e465848a09711fb7892490e50b87a26554508f4e8a1ebf8989\": rpc error: code = NotFound desc = could not find container \"27a4ea941d12c5e465848a09711fb7892490e50b87a26554508f4e8a1ebf8989\": container with ID starting with 27a4ea941d12c5e465848a09711fb7892490e50b87a26554508f4e8a1ebf8989 not found: ID does not exist" Jul 22 02:44:18.749147 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:44:18.749126 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-mhjlf"] Jul 22 02:44:18.751337 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:44:18.751312 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-mhjlf"] Jul 22 02:44:19.961073 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:44:19.961042 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 22 02:44:19.961349 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:44:19.961106 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 22 02:44:19.961349 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:44:19.961135 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 22 02:44:20.142055 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:44:20.142015 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3d2d9ffc-ecbe-44ca-b117-4d39b1377785" path="/var/lib/kubelet/pods/3d2d9ffc-ecbe-44ca-b117-4d39b1377785/volumes" Jul 22 02:44:20.737176 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:44:20.737148 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf46vpxp" event={"ID":"07b23859-7ee5-4299-965e-b1592f48e054","Type":"ContainerStarted","Data":"9c8e07787b835623e9e548f0afdee80bef454237673a6f4db77b74b546339fb1"} Jul 22 02:44:20.769184 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:44:20.769147 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf46vpxp" podStartSLOduration=2.012091893 podStartE2EDuration="5.769133767s" totalImagesPullingTime="3.757041874s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 02:44:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:44:20.768050224 +0000 UTC m=+427.087225524" watchObservedRunningTime="2026-07-22 02:44:20.769133767 +0000 UTC m=+427.088309066" Jul 22 02:44:21.026223 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:44:21.026166 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf46vpxp" Jul 22 02:44:21.030176 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:44:21.030154 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf46vpxp" Jul 22 02:44:21.741237 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:44:21.741205 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf46vpxp" Jul 22 02:44:21.741885 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:44:21.741871 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf46vpxp" Jul 22 02:44:42.697851 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:44:42.697780 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-g5rrk" Jul 22 02:44:44.705172 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:44:44.705145 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-f6p2m" Jul 22 02:44:56.371936 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:44:56.371907 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-wfxc7"] Jul 22 02:44:56.372309 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:44:56.372167 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3d2d9ffc-ecbe-44ca-b117-4d39b1377785" containerName="discovery" Jul 22 02:44:56.372309 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:44:56.372176 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="3d2d9ffc-ecbe-44ca-b117-4d39b1377785" containerName="discovery" Jul 22 02:44:56.372309 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:44:56.372232 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="3d2d9ffc-ecbe-44ca-b117-4d39b1377785" containerName="discovery" Jul 22 02:44:56.381920 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:44:56.381897 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-wfxc7" Jul 22 02:44:56.386231 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:44:56.386196 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-catalog-dockercfg-skrsq\"" Jul 22 02:44:56.387273 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:44:56.387252 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 22 02:44:56.387377 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:44:56.387280 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 22 02:44:56.391691 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:44:56.391669 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-wfxc7"] Jul 22 02:44:56.546883 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:44:56.546858 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2qt5t\" (UniqueName: \"kubernetes.io/projected/81b36d89-3471-4dea-9a7d-0e2e8bdab2bc-kube-api-access-2qt5t\") pod \"kuadrant-operator-catalog-wfxc7\" (UID: \"81b36d89-3471-4dea-9a7d-0e2e8bdab2bc\") " pod="kuadrant-system/kuadrant-operator-catalog-wfxc7" Jul 22 02:44:56.648047 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:44:56.647995 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2qt5t\" (UniqueName: \"kubernetes.io/projected/81b36d89-3471-4dea-9a7d-0e2e8bdab2bc-kube-api-access-2qt5t\") pod \"kuadrant-operator-catalog-wfxc7\" (UID: \"81b36d89-3471-4dea-9a7d-0e2e8bdab2bc\") " pod="kuadrant-system/kuadrant-operator-catalog-wfxc7" Jul 22 02:44:56.658973 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:44:56.658956 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2qt5t\" (UniqueName: \"kubernetes.io/projected/81b36d89-3471-4dea-9a7d-0e2e8bdab2bc-kube-api-access-2qt5t\") pod \"kuadrant-operator-catalog-wfxc7\" (UID: \"81b36d89-3471-4dea-9a7d-0e2e8bdab2bc\") " pod="kuadrant-system/kuadrant-operator-catalog-wfxc7" Jul 22 02:44:56.692896 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:44:56.692876 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-wfxc7" Jul 22 02:44:56.836980 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:44:56.836956 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-wfxc7"] Jul 22 02:44:56.839075 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:44:56.839051 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81b36d89_3471_4dea_9a7d_0e2e8bdab2bc.slice/crio-5184a2309b6b3bf45c619cbbbacdc713e2b42770389a096f2a9854fe9a1b0279 WatchSource:0}: Error finding container 5184a2309b6b3bf45c619cbbbacdc713e2b42770389a096f2a9854fe9a1b0279: Status 404 returned error can't find the container with id 5184a2309b6b3bf45c619cbbbacdc713e2b42770389a096f2a9854fe9a1b0279 Jul 22 02:44:56.884525 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:44:56.884494 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-wfxc7" event={"ID":"81b36d89-3471-4dea-9a7d-0e2e8bdab2bc","Type":"ContainerStarted","Data":"5184a2309b6b3bf45c619cbbbacdc713e2b42770389a096f2a9854fe9a1b0279"} Jul 22 02:44:58.895914 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:44:58.895844 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-wfxc7" event={"ID":"81b36d89-3471-4dea-9a7d-0e2e8bdab2bc","Type":"ContainerStarted","Data":"b875159e970c6f3b26ebfc2cfc89b684da869801e3a08803e95516c041a0adfe"} Jul 22 02:44:58.916588 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:44:58.916530 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-wfxc7" podStartSLOduration=1.150752116 podStartE2EDuration="2.916518602s" totalImagesPullingTime="1.765766486s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 02:44:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:44:58.914759048 +0000 UTC m=+465.233934359" watchObservedRunningTime="2026-07-22 02:44:58.916518602 +0000 UTC m=+465.235693901" Jul 22 02:45:06.693625 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:06.693556 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kuadrant-system/kuadrant-operator-catalog-wfxc7" Jul 22 02:45:06.694077 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:06.693656 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-catalog-wfxc7" Jul 22 02:45:06.715400 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:06.715370 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kuadrant-system/kuadrant-operator-catalog-wfxc7" Jul 22 02:45:06.956833 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:06.956774 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-catalog-wfxc7" Jul 22 02:45:10.231615 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:10.231587 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735h76m"] Jul 22 02:45:10.234812 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:10.234796 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735h76m" Jul 22 02:45:10.237913 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:10.237894 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-crzfq\"" Jul 22 02:45:10.243665 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:10.243646 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/122e8d30-1f3c-4d5f-a322-54896dff729a-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735h76m\" (UID: \"122e8d30-1f3c-4d5f-a322-54896dff729a\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735h76m" Jul 22 02:45:10.243749 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:10.243705 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rdnvx\" (UniqueName: \"kubernetes.io/projected/122e8d30-1f3c-4d5f-a322-54896dff729a-kube-api-access-rdnvx\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735h76m\" (UID: \"122e8d30-1f3c-4d5f-a322-54896dff729a\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735h76m" Jul 22 02:45:10.243833 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:10.243814 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/122e8d30-1f3c-4d5f-a322-54896dff729a-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735h76m\" (UID: \"122e8d30-1f3c-4d5f-a322-54896dff729a\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735h76m" Jul 22 02:45:10.248246 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:10.248224 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735h76m"] Jul 22 02:45:10.344039 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:10.344019 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/122e8d30-1f3c-4d5f-a322-54896dff729a-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735h76m\" (UID: \"122e8d30-1f3c-4d5f-a322-54896dff729a\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735h76m" Jul 22 02:45:10.344120 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:10.344056 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/122e8d30-1f3c-4d5f-a322-54896dff729a-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735h76m\" (UID: \"122e8d30-1f3c-4d5f-a322-54896dff729a\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735h76m" Jul 22 02:45:10.344120 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:10.344097 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rdnvx\" (UniqueName: \"kubernetes.io/projected/122e8d30-1f3c-4d5f-a322-54896dff729a-kube-api-access-rdnvx\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735h76m\" (UID: \"122e8d30-1f3c-4d5f-a322-54896dff729a\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735h76m" Jul 22 02:45:10.344350 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:10.344332 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/122e8d30-1f3c-4d5f-a322-54896dff729a-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735h76m\" (UID: \"122e8d30-1f3c-4d5f-a322-54896dff729a\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735h76m" Jul 22 02:45:10.344430 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:10.344375 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/122e8d30-1f3c-4d5f-a322-54896dff729a-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735h76m\" (UID: \"122e8d30-1f3c-4d5f-a322-54896dff729a\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735h76m" Jul 22 02:45:10.353818 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:10.353793 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rdnvx\" (UniqueName: \"kubernetes.io/projected/122e8d30-1f3c-4d5f-a322-54896dff729a-kube-api-access-rdnvx\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735h76m\" (UID: \"122e8d30-1f3c-4d5f-a322-54896dff729a\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735h76m" Jul 22 02:45:10.543609 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:10.543550 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735h76m" Jul 22 02:45:10.672080 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:10.672057 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735h76m"] Jul 22 02:45:10.674007 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:45:10.673984 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod122e8d30_1f3c_4d5f_a322_54896dff729a.slice/crio-a3c9428dc425f2db4d599569ca68e13bd1ce4d11dc6e992f0493f8d610cbe10f WatchSource:0}: Error finding container a3c9428dc425f2db4d599569ca68e13bd1ce4d11dc6e992f0493f8d610cbe10f: Status 404 returned error can't find the container with id a3c9428dc425f2db4d599569ca68e13bd1ce4d11dc6e992f0493f8d610cbe10f Jul 22 02:45:10.803645 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:10.803592 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0gkpq6"] Jul 22 02:45:10.806825 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:10.806811 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0gkpq6" Jul 22 02:45:10.816487 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:10.816464 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0gkpq6"] Jul 22 02:45:10.848195 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:10.848171 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/74b87c7d-f11f-4cf4-88f4-9f6bf42b6d59-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0gkpq6\" (UID: \"74b87c7d-f11f-4cf4-88f4-9f6bf42b6d59\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0gkpq6" Jul 22 02:45:10.848294 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:10.848209 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/74b87c7d-f11f-4cf4-88f4-9f6bf42b6d59-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0gkpq6\" (UID: \"74b87c7d-f11f-4cf4-88f4-9f6bf42b6d59\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0gkpq6" Jul 22 02:45:10.848294 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:10.848246 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cqncx\" (UniqueName: \"kubernetes.io/projected/74b87c7d-f11f-4cf4-88f4-9f6bf42b6d59-kube-api-access-cqncx\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0gkpq6\" (UID: \"74b87c7d-f11f-4cf4-88f4-9f6bf42b6d59\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0gkpq6" Jul 22 02:45:10.948550 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:10.948529 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/74b87c7d-f11f-4cf4-88f4-9f6bf42b6d59-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0gkpq6\" (UID: \"74b87c7d-f11f-4cf4-88f4-9f6bf42b6d59\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0gkpq6" Jul 22 02:45:10.948680 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:10.948623 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cqncx\" (UniqueName: \"kubernetes.io/projected/74b87c7d-f11f-4cf4-88f4-9f6bf42b6d59-kube-api-access-cqncx\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0gkpq6\" (UID: \"74b87c7d-f11f-4cf4-88f4-9f6bf42b6d59\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0gkpq6" Jul 22 02:45:10.948743 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:10.948681 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/74b87c7d-f11f-4cf4-88f4-9f6bf42b6d59-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0gkpq6\" (UID: \"74b87c7d-f11f-4cf4-88f4-9f6bf42b6d59\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0gkpq6" Jul 22 02:45:10.948913 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:10.948896 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/74b87c7d-f11f-4cf4-88f4-9f6bf42b6d59-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0gkpq6\" (UID: \"74b87c7d-f11f-4cf4-88f4-9f6bf42b6d59\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0gkpq6" Jul 22 02:45:10.948980 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:10.948943 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/74b87c7d-f11f-4cf4-88f4-9f6bf42b6d59-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0gkpq6\" (UID: \"74b87c7d-f11f-4cf4-88f4-9f6bf42b6d59\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0gkpq6" Jul 22 02:45:10.953260 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:10.953238 2578 generic.go:356] "Generic (PLEG): container finished" podID="122e8d30-1f3c-4d5f-a322-54896dff729a" containerID="92c9996d35a206d8136948ae4bd1dcb63dca4308990f0a6cfe4676266682e889" exitCode=0 Jul 22 02:45:10.953354 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:10.953317 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735h76m" event={"ID":"122e8d30-1f3c-4d5f-a322-54896dff729a","Type":"ContainerDied","Data":"92c9996d35a206d8136948ae4bd1dcb63dca4308990f0a6cfe4676266682e889"} Jul 22 02:45:10.953416 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:10.953364 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735h76m" event={"ID":"122e8d30-1f3c-4d5f-a322-54896dff729a","Type":"ContainerStarted","Data":"a3c9428dc425f2db4d599569ca68e13bd1ce4d11dc6e992f0493f8d610cbe10f"} Jul 22 02:45:10.957867 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:10.957851 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cqncx\" (UniqueName: \"kubernetes.io/projected/74b87c7d-f11f-4cf4-88f4-9f6bf42b6d59-kube-api-access-cqncx\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0gkpq6\" (UID: \"74b87c7d-f11f-4cf4-88f4-9f6bf42b6d59\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0gkpq6" Jul 22 02:45:11.125536 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:11.125517 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0gkpq6" Jul 22 02:45:11.258201 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:11.258181 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0gkpq6"] Jul 22 02:45:11.260077 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:45:11.260054 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74b87c7d_f11f_4cf4_88f4_9f6bf42b6d59.slice/crio-0a167c0de1f8795ac9666923ec0cdd358a28566de492f19e01ecdffa1d427fdd WatchSource:0}: Error finding container 0a167c0de1f8795ac9666923ec0cdd358a28566de492f19e01ecdffa1d427fdd: Status 404 returned error can't find the container with id 0a167c0de1f8795ac9666923ec0cdd358a28566de492f19e01ecdffa1d427fdd Jul 22 02:45:11.417811 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:11.417760 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7598m48r"] Jul 22 02:45:11.421065 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:11.421048 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7598m48r" Jul 22 02:45:11.435661 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:11.435639 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7598m48r"] Jul 22 02:45:11.452382 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:11.452361 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/969f440a-4dbb-4982-8817-e27fde2b3f19-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7598m48r\" (UID: \"969f440a-4dbb-4982-8817-e27fde2b3f19\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7598m48r" Jul 22 02:45:11.452481 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:11.452423 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lkp95\" (UniqueName: \"kubernetes.io/projected/969f440a-4dbb-4982-8817-e27fde2b3f19-kube-api-access-lkp95\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7598m48r\" (UID: \"969f440a-4dbb-4982-8817-e27fde2b3f19\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7598m48r" Jul 22 02:45:11.452481 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:11.452469 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/969f440a-4dbb-4982-8817-e27fde2b3f19-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7598m48r\" (UID: \"969f440a-4dbb-4982-8817-e27fde2b3f19\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7598m48r" Jul 22 02:45:11.553416 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:11.553392 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lkp95\" (UniqueName: \"kubernetes.io/projected/969f440a-4dbb-4982-8817-e27fde2b3f19-kube-api-access-lkp95\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7598m48r\" (UID: \"969f440a-4dbb-4982-8817-e27fde2b3f19\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7598m48r" Jul 22 02:45:11.553534 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:11.553423 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/969f440a-4dbb-4982-8817-e27fde2b3f19-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7598m48r\" (UID: \"969f440a-4dbb-4982-8817-e27fde2b3f19\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7598m48r" Jul 22 02:45:11.553534 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:11.553492 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/969f440a-4dbb-4982-8817-e27fde2b3f19-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7598m48r\" (UID: \"969f440a-4dbb-4982-8817-e27fde2b3f19\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7598m48r" Jul 22 02:45:11.553797 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:11.553782 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/969f440a-4dbb-4982-8817-e27fde2b3f19-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7598m48r\" (UID: \"969f440a-4dbb-4982-8817-e27fde2b3f19\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7598m48r" Jul 22 02:45:11.553832 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:11.553807 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/969f440a-4dbb-4982-8817-e27fde2b3f19-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7598m48r\" (UID: \"969f440a-4dbb-4982-8817-e27fde2b3f19\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7598m48r" Jul 22 02:45:11.565864 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:11.565836 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lkp95\" (UniqueName: \"kubernetes.io/projected/969f440a-4dbb-4982-8817-e27fde2b3f19-kube-api-access-lkp95\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7598m48r\" (UID: \"969f440a-4dbb-4982-8817-e27fde2b3f19\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7598m48r" Jul 22 02:45:11.728928 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:11.728908 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7598m48r" Jul 22 02:45:11.858418 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:11.858391 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7598m48r"] Jul 22 02:45:11.862107 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:45:11.862078 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod969f440a_4dbb_4982_8817_e27fde2b3f19.slice/crio-1c6cc8bc999769b3d86b42b8dd6e8ce8de056b37d2fdd2e261419be9e0e8ac15 WatchSource:0}: Error finding container 1c6cc8bc999769b3d86b42b8dd6e8ce8de056b37d2fdd2e261419be9e0e8ac15: Status 404 returned error can't find the container with id 1c6cc8bc999769b3d86b42b8dd6e8ce8de056b37d2fdd2e261419be9e0e8ac15 Jul 22 02:45:11.959593 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:11.959544 2578 generic.go:356] "Generic (PLEG): container finished" podID="122e8d30-1f3c-4d5f-a322-54896dff729a" containerID="4ff231199150efa9eeef56c52a72a2c6e6ba471ef5ea982c1627c3063b28e98d" exitCode=0 Jul 22 02:45:11.959695 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:11.959595 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735h76m" event={"ID":"122e8d30-1f3c-4d5f-a322-54896dff729a","Type":"ContainerDied","Data":"4ff231199150efa9eeef56c52a72a2c6e6ba471ef5ea982c1627c3063b28e98d"} Jul 22 02:45:11.961097 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:11.961078 2578 generic.go:356] "Generic (PLEG): container finished" podID="969f440a-4dbb-4982-8817-e27fde2b3f19" containerID="3a5f2730bde35aac9ee4ee5c5fbc633908bbfb75a923e8cdd52ad874c0d1e93b" exitCode=0 Jul 22 02:45:11.961173 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:11.961116 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7598m48r" event={"ID":"969f440a-4dbb-4982-8817-e27fde2b3f19","Type":"ContainerDied","Data":"3a5f2730bde35aac9ee4ee5c5fbc633908bbfb75a923e8cdd52ad874c0d1e93b"} Jul 22 02:45:11.961173 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:11.961167 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7598m48r" event={"ID":"969f440a-4dbb-4982-8817-e27fde2b3f19","Type":"ContainerStarted","Data":"1c6cc8bc999769b3d86b42b8dd6e8ce8de056b37d2fdd2e261419be9e0e8ac15"} Jul 22 02:45:11.962697 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:11.962674 2578 generic.go:356] "Generic (PLEG): container finished" podID="74b87c7d-f11f-4cf4-88f4-9f6bf42b6d59" containerID="269d92bbdc3b43aa572825aee383fb8c8bafa273db9ec3bd1d6a8b6244e8415a" exitCode=0 Jul 22 02:45:11.962762 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:11.962708 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0gkpq6" event={"ID":"74b87c7d-f11f-4cf4-88f4-9f6bf42b6d59","Type":"ContainerDied","Data":"269d92bbdc3b43aa572825aee383fb8c8bafa273db9ec3bd1d6a8b6244e8415a"} Jul 22 02:45:11.962762 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:11.962739 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0gkpq6" event={"ID":"74b87c7d-f11f-4cf4-88f4-9f6bf42b6d59","Type":"ContainerStarted","Data":"0a167c0de1f8795ac9666923ec0cdd358a28566de492f19e01ecdffa1d427fdd"} Jul 22 02:45:12.024950 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:12.024928 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1n794h"] Jul 22 02:45:12.028084 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:12.028070 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1n794h" Jul 22 02:45:12.040145 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:12.040119 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1n794h"] Jul 22 02:45:12.057599 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:12.057577 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g765s\" (UniqueName: \"kubernetes.io/projected/f47eb2cd-4a19-4bec-b095-0dd6f0409ff2-kube-api-access-g765s\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1n794h\" (UID: \"f47eb2cd-4a19-4bec-b095-0dd6f0409ff2\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1n794h" Jul 22 02:45:12.057727 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:12.057659 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f47eb2cd-4a19-4bec-b095-0dd6f0409ff2-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1n794h\" (UID: \"f47eb2cd-4a19-4bec-b095-0dd6f0409ff2\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1n794h" Jul 22 02:45:12.057794 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:12.057720 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f47eb2cd-4a19-4bec-b095-0dd6f0409ff2-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1n794h\" (UID: \"f47eb2cd-4a19-4bec-b095-0dd6f0409ff2\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1n794h" Jul 22 02:45:12.158554 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:12.158530 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g765s\" (UniqueName: \"kubernetes.io/projected/f47eb2cd-4a19-4bec-b095-0dd6f0409ff2-kube-api-access-g765s\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1n794h\" (UID: \"f47eb2cd-4a19-4bec-b095-0dd6f0409ff2\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1n794h" Jul 22 02:45:12.158665 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:12.158611 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f47eb2cd-4a19-4bec-b095-0dd6f0409ff2-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1n794h\" (UID: \"f47eb2cd-4a19-4bec-b095-0dd6f0409ff2\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1n794h" Jul 22 02:45:12.158665 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:12.158631 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f47eb2cd-4a19-4bec-b095-0dd6f0409ff2-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1n794h\" (UID: \"f47eb2cd-4a19-4bec-b095-0dd6f0409ff2\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1n794h" Jul 22 02:45:12.158946 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:12.158928 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f47eb2cd-4a19-4bec-b095-0dd6f0409ff2-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1n794h\" (UID: \"f47eb2cd-4a19-4bec-b095-0dd6f0409ff2\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1n794h" Jul 22 02:45:12.158982 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:12.158952 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f47eb2cd-4a19-4bec-b095-0dd6f0409ff2-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1n794h\" (UID: \"f47eb2cd-4a19-4bec-b095-0dd6f0409ff2\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1n794h" Jul 22 02:45:12.171494 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:12.171472 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g765s\" (UniqueName: \"kubernetes.io/projected/f47eb2cd-4a19-4bec-b095-0dd6f0409ff2-kube-api-access-g765s\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1n794h\" (UID: \"f47eb2cd-4a19-4bec-b095-0dd6f0409ff2\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1n794h" Jul 22 02:45:12.338622 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:12.338602 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1n794h" Jul 22 02:45:12.476650 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:12.476483 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1n794h"] Jul 22 02:45:12.491891 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:45:12.491861 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf47eb2cd_4a19_4bec_b095_0dd6f0409ff2.slice/crio-972c1924e2941d1387eb3803d0b578f2828e5b355ccd831298a4e69153fe7a8b WatchSource:0}: Error finding container 972c1924e2941d1387eb3803d0b578f2828e5b355ccd831298a4e69153fe7a8b: Status 404 returned error can't find the container with id 972c1924e2941d1387eb3803d0b578f2828e5b355ccd831298a4e69153fe7a8b Jul 22 02:45:12.969103 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:12.969067 2578 generic.go:356] "Generic (PLEG): container finished" podID="74b87c7d-f11f-4cf4-88f4-9f6bf42b6d59" containerID="470d61341504189d7d770ba216abdc72a47a3fb898e3fad4edbe2026b84e8e23" exitCode=0 Jul 22 02:45:12.969213 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:12.969131 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0gkpq6" event={"ID":"74b87c7d-f11f-4cf4-88f4-9f6bf42b6d59","Type":"ContainerDied","Data":"470d61341504189d7d770ba216abdc72a47a3fb898e3fad4edbe2026b84e8e23"} Jul 22 02:45:12.971177 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:12.971155 2578 generic.go:356] "Generic (PLEG): container finished" podID="122e8d30-1f3c-4d5f-a322-54896dff729a" containerID="a83c26a5c19459333fa1f6633e6d0042503857eb0ebbb4c1c6ac2a35a166a955" exitCode=0 Jul 22 02:45:12.971263 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:12.971206 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735h76m" event={"ID":"122e8d30-1f3c-4d5f-a322-54896dff729a","Type":"ContainerDied","Data":"a83c26a5c19459333fa1f6633e6d0042503857eb0ebbb4c1c6ac2a35a166a955"} Jul 22 02:45:12.972934 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:12.972915 2578 generic.go:356] "Generic (PLEG): container finished" podID="969f440a-4dbb-4982-8817-e27fde2b3f19" containerID="7582173acbc32c51493c2b28092bb74cb34bafc20453c5171ac076a5108d2ae3" exitCode=0 Jul 22 02:45:12.973019 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:12.972976 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7598m48r" event={"ID":"969f440a-4dbb-4982-8817-e27fde2b3f19","Type":"ContainerDied","Data":"7582173acbc32c51493c2b28092bb74cb34bafc20453c5171ac076a5108d2ae3"} Jul 22 02:45:12.974281 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:12.974263 2578 generic.go:356] "Generic (PLEG): container finished" podID="f47eb2cd-4a19-4bec-b095-0dd6f0409ff2" containerID="0055d4b6a8b173c5fc60187ee2aa3fbe7468d55dd66cd8831cdd5a47f302a9b5" exitCode=0 Jul 22 02:45:12.974351 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:12.974303 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1n794h" event={"ID":"f47eb2cd-4a19-4bec-b095-0dd6f0409ff2","Type":"ContainerDied","Data":"0055d4b6a8b173c5fc60187ee2aa3fbe7468d55dd66cd8831cdd5a47f302a9b5"} Jul 22 02:45:12.974351 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:12.974325 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1n794h" event={"ID":"f47eb2cd-4a19-4bec-b095-0dd6f0409ff2","Type":"ContainerStarted","Data":"972c1924e2941d1387eb3803d0b578f2828e5b355ccd831298a4e69153fe7a8b"} Jul 22 02:45:13.981532 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:13.981500 2578 generic.go:356] "Generic (PLEG): container finished" podID="969f440a-4dbb-4982-8817-e27fde2b3f19" containerID="0d878f3ab97da7dd334b3b969754716a69e0abb154e16a66ca34b6683957329e" exitCode=0 Jul 22 02:45:13.981850 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:13.981592 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7598m48r" event={"ID":"969f440a-4dbb-4982-8817-e27fde2b3f19","Type":"ContainerDied","Data":"0d878f3ab97da7dd334b3b969754716a69e0abb154e16a66ca34b6683957329e"} Jul 22 02:45:13.983257 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:13.983237 2578 generic.go:356] "Generic (PLEG): container finished" podID="f47eb2cd-4a19-4bec-b095-0dd6f0409ff2" containerID="5505da604f3b389496f68e3520fe0e0a21bd1270f830c23cce39f546f498f743" exitCode=0 Jul 22 02:45:13.983332 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:13.983312 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1n794h" event={"ID":"f47eb2cd-4a19-4bec-b095-0dd6f0409ff2","Type":"ContainerDied","Data":"5505da604f3b389496f68e3520fe0e0a21bd1270f830c23cce39f546f498f743"} Jul 22 02:45:13.985502 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:13.985476 2578 generic.go:356] "Generic (PLEG): container finished" podID="74b87c7d-f11f-4cf4-88f4-9f6bf42b6d59" containerID="baab47049d0e1a7c02c1166fe3d1ad567102883748254bfc0aedb719cbde5915" exitCode=0 Jul 22 02:45:13.985613 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:13.985555 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0gkpq6" event={"ID":"74b87c7d-f11f-4cf4-88f4-9f6bf42b6d59","Type":"ContainerDied","Data":"baab47049d0e1a7c02c1166fe3d1ad567102883748254bfc0aedb719cbde5915"} Jul 22 02:45:14.110390 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:14.110369 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735h76m" Jul 22 02:45:14.173078 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:14.173058 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/122e8d30-1f3c-4d5f-a322-54896dff729a-bundle\") pod \"122e8d30-1f3c-4d5f-a322-54896dff729a\" (UID: \"122e8d30-1f3c-4d5f-a322-54896dff729a\") " Jul 22 02:45:14.173217 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:14.173086 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/122e8d30-1f3c-4d5f-a322-54896dff729a-util\") pod \"122e8d30-1f3c-4d5f-a322-54896dff729a\" (UID: \"122e8d30-1f3c-4d5f-a322-54896dff729a\") " Jul 22 02:45:14.173217 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:14.173151 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rdnvx\" (UniqueName: \"kubernetes.io/projected/122e8d30-1f3c-4d5f-a322-54896dff729a-kube-api-access-rdnvx\") pod \"122e8d30-1f3c-4d5f-a322-54896dff729a\" (UID: \"122e8d30-1f3c-4d5f-a322-54896dff729a\") " Jul 22 02:45:14.173730 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:14.173701 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/122e8d30-1f3c-4d5f-a322-54896dff729a-bundle" (OuterVolumeSpecName: "bundle") pod "122e8d30-1f3c-4d5f-a322-54896dff729a" (UID: "122e8d30-1f3c-4d5f-a322-54896dff729a"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 02:45:14.175466 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:14.175445 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/122e8d30-1f3c-4d5f-a322-54896dff729a-kube-api-access-rdnvx" (OuterVolumeSpecName: "kube-api-access-rdnvx") pod "122e8d30-1f3c-4d5f-a322-54896dff729a" (UID: "122e8d30-1f3c-4d5f-a322-54896dff729a"). InnerVolumeSpecName "kube-api-access-rdnvx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 02:45:14.178084 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:14.178052 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/122e8d30-1f3c-4d5f-a322-54896dff729a-util" (OuterVolumeSpecName: "util") pod "122e8d30-1f3c-4d5f-a322-54896dff729a" (UID: "122e8d30-1f3c-4d5f-a322-54896dff729a"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 02:45:14.274064 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:14.274042 2578 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/122e8d30-1f3c-4d5f-a322-54896dff729a-bundle\") on node \"ip-10-0-133-31.ec2.internal\" DevicePath \"\"" Jul 22 02:45:14.274064 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:14.274063 2578 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/122e8d30-1f3c-4d5f-a322-54896dff729a-util\") on node \"ip-10-0-133-31.ec2.internal\" DevicePath \"\"" Jul 22 02:45:14.274174 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:14.274071 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rdnvx\" (UniqueName: \"kubernetes.io/projected/122e8d30-1f3c-4d5f-a322-54896dff729a-kube-api-access-rdnvx\") on node \"ip-10-0-133-31.ec2.internal\" DevicePath \"\"" Jul 22 02:45:14.994289 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:14.994267 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735h76m" Jul 22 02:45:14.994289 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:14.994286 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735h76m" event={"ID":"122e8d30-1f3c-4d5f-a322-54896dff729a","Type":"ContainerDied","Data":"a3c9428dc425f2db4d599569ca68e13bd1ce4d11dc6e992f0493f8d610cbe10f"} Jul 22 02:45:14.994775 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:14.994325 2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a3c9428dc425f2db4d599569ca68e13bd1ce4d11dc6e992f0493f8d610cbe10f" Jul 22 02:45:14.996362 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:14.996340 2578 generic.go:356] "Generic (PLEG): container finished" podID="f47eb2cd-4a19-4bec-b095-0dd6f0409ff2" containerID="eed5f2a138013b54dae854678a7254c94797fd9b259a9a4429dce6e2f7d39ae3" exitCode=0 Jul 22 02:45:14.996462 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:14.996427 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1n794h" event={"ID":"f47eb2cd-4a19-4bec-b095-0dd6f0409ff2","Type":"ContainerDied","Data":"eed5f2a138013b54dae854678a7254c94797fd9b259a9a4429dce6e2f7d39ae3"} Jul 22 02:45:15.150829 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:15.150808 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7598m48r" Jul 22 02:45:15.152789 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:15.152769 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0gkpq6" Jul 22 02:45:15.181416 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:15.181393 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/74b87c7d-f11f-4cf4-88f4-9f6bf42b6d59-util\") pod \"74b87c7d-f11f-4cf4-88f4-9f6bf42b6d59\" (UID: \"74b87c7d-f11f-4cf4-88f4-9f6bf42b6d59\") " Jul 22 02:45:15.181506 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:15.181432 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lkp95\" (UniqueName: \"kubernetes.io/projected/969f440a-4dbb-4982-8817-e27fde2b3f19-kube-api-access-lkp95\") pod \"969f440a-4dbb-4982-8817-e27fde2b3f19\" (UID: \"969f440a-4dbb-4982-8817-e27fde2b3f19\") " Jul 22 02:45:15.181506 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:15.181465 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/969f440a-4dbb-4982-8817-e27fde2b3f19-bundle\") pod \"969f440a-4dbb-4982-8817-e27fde2b3f19\" (UID: \"969f440a-4dbb-4982-8817-e27fde2b3f19\") " Jul 22 02:45:15.181641 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:15.181552 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cqncx\" (UniqueName: \"kubernetes.io/projected/74b87c7d-f11f-4cf4-88f4-9f6bf42b6d59-kube-api-access-cqncx\") pod \"74b87c7d-f11f-4cf4-88f4-9f6bf42b6d59\" (UID: \"74b87c7d-f11f-4cf4-88f4-9f6bf42b6d59\") " Jul 22 02:45:15.181641 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:15.181608 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/969f440a-4dbb-4982-8817-e27fde2b3f19-util\") pod \"969f440a-4dbb-4982-8817-e27fde2b3f19\" (UID: \"969f440a-4dbb-4982-8817-e27fde2b3f19\") " Jul 22 02:45:15.181641 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:15.181636 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/74b87c7d-f11f-4cf4-88f4-9f6bf42b6d59-bundle\") pod \"74b87c7d-f11f-4cf4-88f4-9f6bf42b6d59\" (UID: \"74b87c7d-f11f-4cf4-88f4-9f6bf42b6d59\") " Jul 22 02:45:15.182101 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:15.182074 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/969f440a-4dbb-4982-8817-e27fde2b3f19-bundle" (OuterVolumeSpecName: "bundle") pod "969f440a-4dbb-4982-8817-e27fde2b3f19" (UID: "969f440a-4dbb-4982-8817-e27fde2b3f19"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 02:45:15.182261 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:15.182237 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74b87c7d-f11f-4cf4-88f4-9f6bf42b6d59-bundle" (OuterVolumeSpecName: "bundle") pod "74b87c7d-f11f-4cf4-88f4-9f6bf42b6d59" (UID: "74b87c7d-f11f-4cf4-88f4-9f6bf42b6d59"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 02:45:15.184099 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:15.184074 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/969f440a-4dbb-4982-8817-e27fde2b3f19-kube-api-access-lkp95" (OuterVolumeSpecName: "kube-api-access-lkp95") pod "969f440a-4dbb-4982-8817-e27fde2b3f19" (UID: "969f440a-4dbb-4982-8817-e27fde2b3f19"). InnerVolumeSpecName "kube-api-access-lkp95". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 02:45:15.184192 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:15.184150 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74b87c7d-f11f-4cf4-88f4-9f6bf42b6d59-kube-api-access-cqncx" (OuterVolumeSpecName: "kube-api-access-cqncx") pod "74b87c7d-f11f-4cf4-88f4-9f6bf42b6d59" (UID: "74b87c7d-f11f-4cf4-88f4-9f6bf42b6d59"). InnerVolumeSpecName "kube-api-access-cqncx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 02:45:15.188203 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:15.188181 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74b87c7d-f11f-4cf4-88f4-9f6bf42b6d59-util" (OuterVolumeSpecName: "util") pod "74b87c7d-f11f-4cf4-88f4-9f6bf42b6d59" (UID: "74b87c7d-f11f-4cf4-88f4-9f6bf42b6d59"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 02:45:15.189737 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:15.189699 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/969f440a-4dbb-4982-8817-e27fde2b3f19-util" (OuterVolumeSpecName: "util") pod "969f440a-4dbb-4982-8817-e27fde2b3f19" (UID: "969f440a-4dbb-4982-8817-e27fde2b3f19"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 02:45:15.282659 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:15.282595 2578 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/74b87c7d-f11f-4cf4-88f4-9f6bf42b6d59-util\") on node \"ip-10-0-133-31.ec2.internal\" DevicePath \"\"" Jul 22 02:45:15.282659 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:15.282618 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lkp95\" (UniqueName: \"kubernetes.io/projected/969f440a-4dbb-4982-8817-e27fde2b3f19-kube-api-access-lkp95\") on node \"ip-10-0-133-31.ec2.internal\" DevicePath \"\"" Jul 22 02:45:15.282659 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:15.282628 2578 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/969f440a-4dbb-4982-8817-e27fde2b3f19-bundle\") on node \"ip-10-0-133-31.ec2.internal\" DevicePath \"\"" Jul 22 02:45:15.282659 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:15.282636 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cqncx\" (UniqueName: \"kubernetes.io/projected/74b87c7d-f11f-4cf4-88f4-9f6bf42b6d59-kube-api-access-cqncx\") on node \"ip-10-0-133-31.ec2.internal\" DevicePath \"\"" Jul 22 02:45:15.282659 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:15.282644 2578 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/969f440a-4dbb-4982-8817-e27fde2b3f19-util\") on node \"ip-10-0-133-31.ec2.internal\" DevicePath \"\"" Jul 22 02:45:15.282659 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:15.282652 2578 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/74b87c7d-f11f-4cf4-88f4-9f6bf42b6d59-bundle\") on node \"ip-10-0-133-31.ec2.internal\" DevicePath \"\"" Jul 22 02:45:16.002676 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:16.002652 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0gkpq6" Jul 22 02:45:16.002676 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:16.002673 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0gkpq6" event={"ID":"74b87c7d-f11f-4cf4-88f4-9f6bf42b6d59","Type":"ContainerDied","Data":"0a167c0de1f8795ac9666923ec0cdd358a28566de492f19e01ecdffa1d427fdd"} Jul 22 02:45:16.003088 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:16.002713 2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0a167c0de1f8795ac9666923ec0cdd358a28566de492f19e01ecdffa1d427fdd" Jul 22 02:45:16.004449 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:16.004432 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7598m48r" Jul 22 02:45:16.004577 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:16.004466 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7598m48r" event={"ID":"969f440a-4dbb-4982-8817-e27fde2b3f19","Type":"ContainerDied","Data":"1c6cc8bc999769b3d86b42b8dd6e8ce8de056b37d2fdd2e261419be9e0e8ac15"} Jul 22 02:45:16.004577 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:16.004503 2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1c6cc8bc999769b3d86b42b8dd6e8ce8de056b37d2fdd2e261419be9e0e8ac15" Jul 22 02:45:16.127644 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:16.127623 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1n794h" Jul 22 02:45:16.189335 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:16.189316 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g765s\" (UniqueName: \"kubernetes.io/projected/f47eb2cd-4a19-4bec-b095-0dd6f0409ff2-kube-api-access-g765s\") pod \"f47eb2cd-4a19-4bec-b095-0dd6f0409ff2\" (UID: \"f47eb2cd-4a19-4bec-b095-0dd6f0409ff2\") " Jul 22 02:45:16.189428 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:16.189347 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f47eb2cd-4a19-4bec-b095-0dd6f0409ff2-util\") pod \"f47eb2cd-4a19-4bec-b095-0dd6f0409ff2\" (UID: \"f47eb2cd-4a19-4bec-b095-0dd6f0409ff2\") " Jul 22 02:45:16.189428 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:16.189366 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f47eb2cd-4a19-4bec-b095-0dd6f0409ff2-bundle\") pod \"f47eb2cd-4a19-4bec-b095-0dd6f0409ff2\" (UID: \"f47eb2cd-4a19-4bec-b095-0dd6f0409ff2\") " Jul 22 02:45:16.190058 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:16.189960 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f47eb2cd-4a19-4bec-b095-0dd6f0409ff2-bundle" (OuterVolumeSpecName: "bundle") pod "f47eb2cd-4a19-4bec-b095-0dd6f0409ff2" (UID: "f47eb2cd-4a19-4bec-b095-0dd6f0409ff2"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 02:45:16.191381 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:16.191354 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f47eb2cd-4a19-4bec-b095-0dd6f0409ff2-kube-api-access-g765s" (OuterVolumeSpecName: "kube-api-access-g765s") pod "f47eb2cd-4a19-4bec-b095-0dd6f0409ff2" (UID: "f47eb2cd-4a19-4bec-b095-0dd6f0409ff2"). InnerVolumeSpecName "kube-api-access-g765s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 02:45:16.194811 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:16.194793 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f47eb2cd-4a19-4bec-b095-0dd6f0409ff2-util" (OuterVolumeSpecName: "util") pod "f47eb2cd-4a19-4bec-b095-0dd6f0409ff2" (UID: "f47eb2cd-4a19-4bec-b095-0dd6f0409ff2"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 02:45:16.289933 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:16.289884 2578 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f47eb2cd-4a19-4bec-b095-0dd6f0409ff2-bundle\") on node \"ip-10-0-133-31.ec2.internal\" DevicePath \"\"" Jul 22 02:45:16.289933 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:16.289903 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g765s\" (UniqueName: \"kubernetes.io/projected/f47eb2cd-4a19-4bec-b095-0dd6f0409ff2-kube-api-access-g765s\") on node \"ip-10-0-133-31.ec2.internal\" DevicePath \"\"" Jul 22 02:45:16.289933 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:16.289912 2578 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f47eb2cd-4a19-4bec-b095-0dd6f0409ff2-util\") on node \"ip-10-0-133-31.ec2.internal\" DevicePath \"\"" Jul 22 02:45:17.010877 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:17.010841 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1n794h" event={"ID":"f47eb2cd-4a19-4bec-b095-0dd6f0409ff2","Type":"ContainerDied","Data":"972c1924e2941d1387eb3803d0b578f2828e5b355ccd831298a4e69153fe7a8b"} Jul 22 02:45:17.010877 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:17.010870 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1n794h" Jul 22 02:45:17.011351 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:17.010887 2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="972c1924e2941d1387eb3803d0b578f2828e5b355ccd831298a4e69153fe7a8b" Jul 22 02:45:20.759481 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:20.759449 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-dzltm"] Jul 22 02:45:20.759845 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:20.759819 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="122e8d30-1f3c-4d5f-a322-54896dff729a" containerName="extract" Jul 22 02:45:20.759845 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:20.759831 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="122e8d30-1f3c-4d5f-a322-54896dff729a" containerName="extract" Jul 22 02:45:20.759845 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:20.759838 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="969f440a-4dbb-4982-8817-e27fde2b3f19" containerName="extract" Jul 22 02:45:20.759845 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:20.759843 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="969f440a-4dbb-4982-8817-e27fde2b3f19" containerName="extract" Jul 22 02:45:20.760023 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:20.759854 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="122e8d30-1f3c-4d5f-a322-54896dff729a" containerName="util" Jul 22 02:45:20.760023 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:20.759860 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="122e8d30-1f3c-4d5f-a322-54896dff729a" containerName="util" Jul 22 02:45:20.760023 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:20.759866 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="969f440a-4dbb-4982-8817-e27fde2b3f19" containerName="pull" Jul 22 02:45:20.760023 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:20.759870 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="969f440a-4dbb-4982-8817-e27fde2b3f19" containerName="pull" Jul 22 02:45:20.760023 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:20.759876 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="969f440a-4dbb-4982-8817-e27fde2b3f19" containerName="util" Jul 22 02:45:20.760023 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:20.759881 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="969f440a-4dbb-4982-8817-e27fde2b3f19" containerName="util" Jul 22 02:45:20.760023 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:20.759892 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f47eb2cd-4a19-4bec-b095-0dd6f0409ff2" containerName="util" Jul 22 02:45:20.760023 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:20.759896 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="f47eb2cd-4a19-4bec-b095-0dd6f0409ff2" containerName="util" Jul 22 02:45:20.760023 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:20.759905 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="74b87c7d-f11f-4cf4-88f4-9f6bf42b6d59" containerName="pull" Jul 22 02:45:20.760023 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:20.759910 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="74b87c7d-f11f-4cf4-88f4-9f6bf42b6d59" containerName="pull" Jul 22 02:45:20.760023 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:20.759917 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="74b87c7d-f11f-4cf4-88f4-9f6bf42b6d59" containerName="util" Jul 22 02:45:20.760023 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:20.759922 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="74b87c7d-f11f-4cf4-88f4-9f6bf42b6d59" containerName="util" Jul 22 02:45:20.760023 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:20.759927 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="122e8d30-1f3c-4d5f-a322-54896dff729a" containerName="pull" Jul 22 02:45:20.760023 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:20.759931 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="122e8d30-1f3c-4d5f-a322-54896dff729a" containerName="pull" Jul 22 02:45:20.760023 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:20.759937 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="74b87c7d-f11f-4cf4-88f4-9f6bf42b6d59" containerName="extract" Jul 22 02:45:20.760023 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:20.759941 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="74b87c7d-f11f-4cf4-88f4-9f6bf42b6d59" containerName="extract" Jul 22 02:45:20.760023 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:20.759946 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f47eb2cd-4a19-4bec-b095-0dd6f0409ff2" containerName="pull" Jul 22 02:45:20.760023 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:20.759951 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="f47eb2cd-4a19-4bec-b095-0dd6f0409ff2" containerName="pull" Jul 22 02:45:20.760023 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:20.759957 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f47eb2cd-4a19-4bec-b095-0dd6f0409ff2" containerName="extract" Jul 22 02:45:20.760023 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:20.759961 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="f47eb2cd-4a19-4bec-b095-0dd6f0409ff2" containerName="extract" Jul 22 02:45:20.760023 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:20.760006 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="122e8d30-1f3c-4d5f-a322-54896dff729a" containerName="extract" Jul 22 02:45:20.760023 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:20.760013 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="969f440a-4dbb-4982-8817-e27fde2b3f19" containerName="extract" Jul 22 02:45:20.760023 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:20.760020 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="74b87c7d-f11f-4cf4-88f4-9f6bf42b6d59" containerName="extract" Jul 22 02:45:20.760023 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:20.760028 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="f47eb2cd-4a19-4bec-b095-0dd6f0409ff2" containerName="extract" Jul 22 02:45:20.763130 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:20.763116 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-dzltm" Jul 22 02:45:20.765928 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:20.765910 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Jul 22 02:45:20.766020 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:20.766002 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-c6xkl\"" Jul 22 02:45:20.774618 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:20.774593 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-dzltm"] Jul 22 02:45:20.829872 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:20.829834 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lmktm\" (UniqueName: \"kubernetes.io/projected/08542239-2a2d-463c-8b80-c3d5b90651b4-kube-api-access-lmktm\") pod \"dns-operator-controller-manager-864dd78c4c-dzltm\" (UID: \"08542239-2a2d-463c-8b80-c3d5b90651b4\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-dzltm" Jul 22 02:45:20.930788 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:20.930755 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lmktm\" (UniqueName: \"kubernetes.io/projected/08542239-2a2d-463c-8b80-c3d5b90651b4-kube-api-access-lmktm\") pod \"dns-operator-controller-manager-864dd78c4c-dzltm\" (UID: \"08542239-2a2d-463c-8b80-c3d5b90651b4\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-dzltm" Jul 22 02:45:20.945892 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:20.945868 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lmktm\" (UniqueName: \"kubernetes.io/projected/08542239-2a2d-463c-8b80-c3d5b90651b4-kube-api-access-lmktm\") pod \"dns-operator-controller-manager-864dd78c4c-dzltm\" (UID: \"08542239-2a2d-463c-8b80-c3d5b90651b4\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-dzltm" Jul 22 02:45:21.072962 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:21.072901 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-dzltm" Jul 22 02:45:21.212602 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:21.212511 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-dzltm"] Jul 22 02:45:21.214827 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:45:21.214798 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08542239_2a2d_463c_8b80_c3d5b90651b4.slice/crio-24716a6becf5dac6da60dd7c0474d09e1c49179874296696fd341e2762849427 WatchSource:0}: Error finding container 24716a6becf5dac6da60dd7c0474d09e1c49179874296696fd341e2762849427: Status 404 returned error can't find the container with id 24716a6becf5dac6da60dd7c0474d09e1c49179874296696fd341e2762849427 Jul 22 02:45:22.036847 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:22.036812 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-dzltm" event={"ID":"08542239-2a2d-463c-8b80-c3d5b90651b4","Type":"ContainerStarted","Data":"24716a6becf5dac6da60dd7c0474d09e1c49179874296696fd341e2762849427"} Jul 22 02:45:23.065604 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:23.065573 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-9c4f7966-xxjsg"] Jul 22 02:45:23.069549 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:23.069507 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-9c4f7966-xxjsg" Jul 22 02:45:23.090781 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:23.090760 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-9c4f7966-xxjsg"] Jul 22 02:45:23.149150 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:23.149122 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/52d794b1-1afa-4ee1-868e-abf89dc7a91f-console-oauth-config\") pod \"console-9c4f7966-xxjsg\" (UID: \"52d794b1-1afa-4ee1-868e-abf89dc7a91f\") " pod="openshift-console/console-9c4f7966-xxjsg" Jul 22 02:45:23.149150 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:23.149155 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/52d794b1-1afa-4ee1-868e-abf89dc7a91f-trusted-ca-bundle\") pod \"console-9c4f7966-xxjsg\" (UID: \"52d794b1-1afa-4ee1-868e-abf89dc7a91f\") " pod="openshift-console/console-9c4f7966-xxjsg" Jul 22 02:45:23.149319 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:23.149197 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/52d794b1-1afa-4ee1-868e-abf89dc7a91f-console-serving-cert\") pod \"console-9c4f7966-xxjsg\" (UID: \"52d794b1-1afa-4ee1-868e-abf89dc7a91f\") " pod="openshift-console/console-9c4f7966-xxjsg" Jul 22 02:45:23.149319 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:23.149230 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/52d794b1-1afa-4ee1-868e-abf89dc7a91f-oauth-serving-cert\") pod \"console-9c4f7966-xxjsg\" (UID: \"52d794b1-1afa-4ee1-868e-abf89dc7a91f\") " pod="openshift-console/console-9c4f7966-xxjsg" Jul 22 02:45:23.149319 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:23.149274 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/52d794b1-1afa-4ee1-868e-abf89dc7a91f-service-ca\") pod \"console-9c4f7966-xxjsg\" (UID: \"52d794b1-1afa-4ee1-868e-abf89dc7a91f\") " pod="openshift-console/console-9c4f7966-xxjsg" Jul 22 02:45:23.149319 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:23.149300 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kvwzn\" (UniqueName: \"kubernetes.io/projected/52d794b1-1afa-4ee1-868e-abf89dc7a91f-kube-api-access-kvwzn\") pod \"console-9c4f7966-xxjsg\" (UID: \"52d794b1-1afa-4ee1-868e-abf89dc7a91f\") " pod="openshift-console/console-9c4f7966-xxjsg" Jul 22 02:45:23.149457 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:23.149342 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/52d794b1-1afa-4ee1-868e-abf89dc7a91f-console-config\") pod \"console-9c4f7966-xxjsg\" (UID: \"52d794b1-1afa-4ee1-868e-abf89dc7a91f\") " pod="openshift-console/console-9c4f7966-xxjsg" Jul 22 02:45:23.250707 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:23.250686 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/52d794b1-1afa-4ee1-868e-abf89dc7a91f-console-oauth-config\") pod \"console-9c4f7966-xxjsg\" (UID: \"52d794b1-1afa-4ee1-868e-abf89dc7a91f\") " pod="openshift-console/console-9c4f7966-xxjsg" Jul 22 02:45:23.250803 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:23.250717 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/52d794b1-1afa-4ee1-868e-abf89dc7a91f-trusted-ca-bundle\") pod \"console-9c4f7966-xxjsg\" (UID: \"52d794b1-1afa-4ee1-868e-abf89dc7a91f\") " pod="openshift-console/console-9c4f7966-xxjsg" Jul 22 02:45:23.250803 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:23.250741 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/52d794b1-1afa-4ee1-868e-abf89dc7a91f-console-serving-cert\") pod \"console-9c4f7966-xxjsg\" (UID: \"52d794b1-1afa-4ee1-868e-abf89dc7a91f\") " pod="openshift-console/console-9c4f7966-xxjsg" Jul 22 02:45:23.250803 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:23.250783 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/52d794b1-1afa-4ee1-868e-abf89dc7a91f-oauth-serving-cert\") pod \"console-9c4f7966-xxjsg\" (UID: \"52d794b1-1afa-4ee1-868e-abf89dc7a91f\") " pod="openshift-console/console-9c4f7966-xxjsg" Jul 22 02:45:23.250948 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:23.250809 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/52d794b1-1afa-4ee1-868e-abf89dc7a91f-service-ca\") pod \"console-9c4f7966-xxjsg\" (UID: \"52d794b1-1afa-4ee1-868e-abf89dc7a91f\") " pod="openshift-console/console-9c4f7966-xxjsg" Jul 22 02:45:23.250948 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:23.250843 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kvwzn\" (UniqueName: \"kubernetes.io/projected/52d794b1-1afa-4ee1-868e-abf89dc7a91f-kube-api-access-kvwzn\") pod \"console-9c4f7966-xxjsg\" (UID: \"52d794b1-1afa-4ee1-868e-abf89dc7a91f\") " pod="openshift-console/console-9c4f7966-xxjsg" Jul 22 02:45:23.250948 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:23.250894 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/52d794b1-1afa-4ee1-868e-abf89dc7a91f-console-config\") pod \"console-9c4f7966-xxjsg\" (UID: \"52d794b1-1afa-4ee1-868e-abf89dc7a91f\") " pod="openshift-console/console-9c4f7966-xxjsg" Jul 22 02:45:23.251524 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:23.251498 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/52d794b1-1afa-4ee1-868e-abf89dc7a91f-service-ca\") pod \"console-9c4f7966-xxjsg\" (UID: \"52d794b1-1afa-4ee1-868e-abf89dc7a91f\") " pod="openshift-console/console-9c4f7966-xxjsg" Jul 22 02:45:23.251722 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:23.251700 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/52d794b1-1afa-4ee1-868e-abf89dc7a91f-console-config\") pod \"console-9c4f7966-xxjsg\" (UID: \"52d794b1-1afa-4ee1-868e-abf89dc7a91f\") " pod="openshift-console/console-9c4f7966-xxjsg" Jul 22 02:45:23.251805 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:23.251728 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/52d794b1-1afa-4ee1-868e-abf89dc7a91f-trusted-ca-bundle\") pod \"console-9c4f7966-xxjsg\" (UID: \"52d794b1-1afa-4ee1-868e-abf89dc7a91f\") " pod="openshift-console/console-9c4f7966-xxjsg" Jul 22 02:45:23.251892 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:23.251867 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/52d794b1-1afa-4ee1-868e-abf89dc7a91f-oauth-serving-cert\") pod \"console-9c4f7966-xxjsg\" (UID: \"52d794b1-1afa-4ee1-868e-abf89dc7a91f\") " pod="openshift-console/console-9c4f7966-xxjsg" Jul 22 02:45:23.253001 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:23.252983 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/52d794b1-1afa-4ee1-868e-abf89dc7a91f-console-oauth-config\") pod \"console-9c4f7966-xxjsg\" (UID: \"52d794b1-1afa-4ee1-868e-abf89dc7a91f\") " pod="openshift-console/console-9c4f7966-xxjsg" Jul 22 02:45:23.253341 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:23.253323 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/52d794b1-1afa-4ee1-868e-abf89dc7a91f-console-serving-cert\") pod \"console-9c4f7966-xxjsg\" (UID: \"52d794b1-1afa-4ee1-868e-abf89dc7a91f\") " pod="openshift-console/console-9c4f7966-xxjsg" Jul 22 02:45:23.260074 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:23.260051 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kvwzn\" (UniqueName: \"kubernetes.io/projected/52d794b1-1afa-4ee1-868e-abf89dc7a91f-kube-api-access-kvwzn\") pod \"console-9c4f7966-xxjsg\" (UID: \"52d794b1-1afa-4ee1-868e-abf89dc7a91f\") " pod="openshift-console/console-9c4f7966-xxjsg" Jul 22 02:45:23.380668 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:23.380634 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-9c4f7966-xxjsg" Jul 22 02:45:23.516299 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:23.516277 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-9c4f7966-xxjsg"] Jul 22 02:45:23.518036 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:45:23.517999 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52d794b1_1afa_4ee1_868e_abf89dc7a91f.slice/crio-391809b95face880ee0856106c3629e85bbf43d23ffaa386e7a1fb0f5d45f2b7 WatchSource:0}: Error finding container 391809b95face880ee0856106c3629e85bbf43d23ffaa386e7a1fb0f5d45f2b7: Status 404 returned error can't find the container with id 391809b95face880ee0856106c3629e85bbf43d23ffaa386e7a1fb0f5d45f2b7 Jul 22 02:45:24.053236 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:24.053199 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-dzltm" event={"ID":"08542239-2a2d-463c-8b80-c3d5b90651b4","Type":"ContainerStarted","Data":"f21043ba8a06c01a026f2ec9c51b8a3bab39fe9f0225e10046457ace04b54aad"} Jul 22 02:45:24.053413 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:24.053294 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-dzltm" Jul 22 02:45:24.054774 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:24.054748 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-9c4f7966-xxjsg" event={"ID":"52d794b1-1afa-4ee1-868e-abf89dc7a91f","Type":"ContainerStarted","Data":"d79f8a974277ed711821028238b944ef758d8867c4fe40dec6512daab5618a1d"} Jul 22 02:45:24.054905 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:24.054793 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-9c4f7966-xxjsg" event={"ID":"52d794b1-1afa-4ee1-868e-abf89dc7a91f","Type":"ContainerStarted","Data":"391809b95face880ee0856106c3629e85bbf43d23ffaa386e7a1fb0f5d45f2b7"} Jul 22 02:45:24.078917 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:24.078870 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-dzltm" podStartSLOduration=2.093402768 podStartE2EDuration="4.078857125s" totalImagesPullingTime="1.985454357s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 02:45:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:45:24.076986228 +0000 UTC m=+490.396161532" watchObservedRunningTime="2026-07-22 02:45:24.078857125 +0000 UTC m=+490.398032426" Jul 22 02:45:24.097991 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:24.097946 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-9c4f7966-xxjsg" podStartSLOduration=1.097936088 podStartE2EDuration="1.097936088s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 02:45:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:45:24.09727481 +0000 UTC m=+490.416450112" watchObservedRunningTime="2026-07-22 02:45:24.097936088 +0000 UTC m=+490.417111389" Jul 22 02:45:26.270716 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:26.270683 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dwkdk"] Jul 22 02:45:26.273924 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:26.273909 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dwkdk" Jul 22 02:45:26.276725 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:26.276705 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-4fvbz\"" Jul 22 02:45:26.285583 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:26.285539 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dwkdk"] Jul 22 02:45:26.374619 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:26.374593 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/be48e9fc-47f6-476a-9db2-75bc27919788-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-dwkdk\" (UID: \"be48e9fc-47f6-476a-9db2-75bc27919788\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dwkdk" Jul 22 02:45:26.374727 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:26.374644 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tqsb7\" (UniqueName: \"kubernetes.io/projected/be48e9fc-47f6-476a-9db2-75bc27919788-kube-api-access-tqsb7\") pod \"kuadrant-operator-controller-manager-8684c9ff49-dwkdk\" (UID: \"be48e9fc-47f6-476a-9db2-75bc27919788\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dwkdk" Jul 22 02:45:26.475898 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:26.475870 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tqsb7\" (UniqueName: \"kubernetes.io/projected/be48e9fc-47f6-476a-9db2-75bc27919788-kube-api-access-tqsb7\") pod \"kuadrant-operator-controller-manager-8684c9ff49-dwkdk\" (UID: \"be48e9fc-47f6-476a-9db2-75bc27919788\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dwkdk" Jul 22 02:45:26.475997 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:26.475922 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/be48e9fc-47f6-476a-9db2-75bc27919788-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-dwkdk\" (UID: \"be48e9fc-47f6-476a-9db2-75bc27919788\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dwkdk" Jul 22 02:45:26.476213 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:26.476199 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/be48e9fc-47f6-476a-9db2-75bc27919788-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-dwkdk\" (UID: \"be48e9fc-47f6-476a-9db2-75bc27919788\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dwkdk" Jul 22 02:45:26.487279 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:26.487258 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tqsb7\" (UniqueName: \"kubernetes.io/projected/be48e9fc-47f6-476a-9db2-75bc27919788-kube-api-access-tqsb7\") pod \"kuadrant-operator-controller-manager-8684c9ff49-dwkdk\" (UID: \"be48e9fc-47f6-476a-9db2-75bc27919788\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dwkdk" Jul 22 02:45:26.584278 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:26.584254 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dwkdk" Jul 22 02:45:26.729633 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:26.729611 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dwkdk"] Jul 22 02:45:26.731587 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:45:26.731540 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe48e9fc_47f6_476a_9db2_75bc27919788.slice/crio-5aaf2ed041a9f60c5f877b97fd5f1a7c334d32b79b11447016b824cf3b073540 WatchSource:0}: Error finding container 5aaf2ed041a9f60c5f877b97fd5f1a7c334d32b79b11447016b824cf3b073540: Status 404 returned error can't find the container with id 5aaf2ed041a9f60c5f877b97fd5f1a7c334d32b79b11447016b824cf3b073540 Jul 22 02:45:27.070400 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:27.070315 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dwkdk" event={"ID":"be48e9fc-47f6-476a-9db2-75bc27919788","Type":"ContainerStarted","Data":"5aaf2ed041a9f60c5f877b97fd5f1a7c334d32b79b11447016b824cf3b073540"} Jul 22 02:45:30.474163 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:30.474100 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-v2m7g"] Jul 22 02:45:30.478224 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:30.478203 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-v2m7g" Jul 22 02:45:30.482019 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:30.482001 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-478s7\"" Jul 22 02:45:30.492679 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:30.492655 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-v2m7g"] Jul 22 02:45:30.613453 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:30.613414 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s558r\" (UniqueName: \"kubernetes.io/projected/9d127701-d2c6-4681-b40e-66ed377d5ef3-kube-api-access-s558r\") pod \"authorino-operator-75cc8d5cb7-v2m7g\" (UID: \"9d127701-d2c6-4681-b40e-66ed377d5ef3\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-v2m7g" Jul 22 02:45:30.714793 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:30.714751 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s558r\" (UniqueName: \"kubernetes.io/projected/9d127701-d2c6-4681-b40e-66ed377d5ef3-kube-api-access-s558r\") pod \"authorino-operator-75cc8d5cb7-v2m7g\" (UID: \"9d127701-d2c6-4681-b40e-66ed377d5ef3\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-v2m7g" Jul 22 02:45:30.733085 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:30.733009 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s558r\" (UniqueName: \"kubernetes.io/projected/9d127701-d2c6-4681-b40e-66ed377d5ef3-kube-api-access-s558r\") pod \"authorino-operator-75cc8d5cb7-v2m7g\" (UID: \"9d127701-d2c6-4681-b40e-66ed377d5ef3\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-v2m7g" Jul 22 02:45:30.789941 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:30.789907 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-v2m7g" Jul 22 02:45:31.679534 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:31.679505 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-v2m7g"] Jul 22 02:45:31.681181 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:45:31.681157 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d127701_d2c6_4681_b40e_66ed377d5ef3.slice/crio-1069cf0f38d271309b6ed666a37c4add516a20bc5cd03911603882b4bb8855b3 WatchSource:0}: Error finding container 1069cf0f38d271309b6ed666a37c4add516a20bc5cd03911603882b4bb8855b3: Status 404 returned error can't find the container with id 1069cf0f38d271309b6ed666a37c4add516a20bc5cd03911603882b4bb8855b3 Jul 22 02:45:32.097791 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:32.097755 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-v2m7g" event={"ID":"9d127701-d2c6-4681-b40e-66ed377d5ef3","Type":"ContainerStarted","Data":"1069cf0f38d271309b6ed666a37c4add516a20bc5cd03911603882b4bb8855b3"} Jul 22 02:45:32.099212 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:32.099190 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dwkdk" event={"ID":"be48e9fc-47f6-476a-9db2-75bc27919788","Type":"ContainerStarted","Data":"882abeaf8ee752ae09729822a5875a87a9cefa8f70797b1a793c72873bacd7c7"} Jul 22 02:45:32.099334 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:32.099318 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dwkdk" Jul 22 02:45:32.159414 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:32.159359 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dwkdk" podStartSLOduration=1.294127323 podStartE2EDuration="6.159342463s" totalImagesPullingTime="4.86521514s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 02:45:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:45:32.158806091 +0000 UTC m=+498.477981392" watchObservedRunningTime="2026-07-22 02:45:32.159342463 +0000 UTC m=+498.478517764" Jul 22 02:45:33.380930 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:33.380894 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-9c4f7966-xxjsg" Jul 22 02:45:33.381368 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:33.381032 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-9c4f7966-xxjsg" Jul 22 02:45:33.386480 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:33.386452 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-9c4f7966-xxjsg" Jul 22 02:45:33.394167 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:33.394129 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-jgxzw"] Jul 22 02:45:33.399518 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:33.399496 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-jgxzw" Jul 22 02:45:33.403529 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:33.403481 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-lvlxr\"" Jul 22 02:45:33.411467 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:33.411441 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-jgxzw"] Jul 22 02:45:33.542295 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:33.542260 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qc8lj\" (UniqueName: \"kubernetes.io/projected/340ba516-b472-446f-94a4-1cc708d4d13a-kube-api-access-qc8lj\") pod \"limitador-operator-controller-manager-859d7b78bd-jgxzw\" (UID: \"340ba516-b472-446f-94a4-1cc708d4d13a\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-jgxzw" Jul 22 02:45:33.642874 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:33.642794 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qc8lj\" (UniqueName: \"kubernetes.io/projected/340ba516-b472-446f-94a4-1cc708d4d13a-kube-api-access-qc8lj\") pod \"limitador-operator-controller-manager-859d7b78bd-jgxzw\" (UID: \"340ba516-b472-446f-94a4-1cc708d4d13a\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-jgxzw" Jul 22 02:45:33.652675 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:33.652647 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qc8lj\" (UniqueName: \"kubernetes.io/projected/340ba516-b472-446f-94a4-1cc708d4d13a-kube-api-access-qc8lj\") pod \"limitador-operator-controller-manager-859d7b78bd-jgxzw\" (UID: \"340ba516-b472-446f-94a4-1cc708d4d13a\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-jgxzw" Jul 22 02:45:33.713786 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:33.713752 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-jgxzw" Jul 22 02:45:33.982459 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:33.982436 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-jgxzw"] Jul 22 02:45:33.984402 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:45:33.984377 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod340ba516_b472_446f_94a4_1cc708d4d13a.slice/crio-cb8fe2028e4a365fbf454ce3509676039ec311746abaf18860a2c8695c55d8a6 WatchSource:0}: Error finding container cb8fe2028e4a365fbf454ce3509676039ec311746abaf18860a2c8695c55d8a6: Status 404 returned error can't find the container with id cb8fe2028e4a365fbf454ce3509676039ec311746abaf18860a2c8695c55d8a6 Jul 22 02:45:34.112530 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:34.112490 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-v2m7g" event={"ID":"9d127701-d2c6-4681-b40e-66ed377d5ef3","Type":"ContainerStarted","Data":"8bd2b6908f550f782e4e3f095b380bcc12a9fcdfa9b837c1c026ddf8adb307d0"} Jul 22 02:45:34.112742 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:34.112707 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-v2m7g" Jul 22 02:45:34.114144 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:34.114116 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-jgxzw" event={"ID":"340ba516-b472-446f-94a4-1cc708d4d13a","Type":"ContainerStarted","Data":"cb8fe2028e4a365fbf454ce3509676039ec311746abaf18860a2c8695c55d8a6"} Jul 22 02:45:34.118458 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:34.118438 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-9c4f7966-xxjsg" Jul 22 02:45:34.132911 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:34.132871 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-75cc8d5cb7-v2m7g" podStartSLOduration=1.9109341930000001 podStartE2EDuration="4.132857771s" totalImagesPullingTime="2.221923578s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 02:45:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:45:34.13197944 +0000 UTC m=+500.451154743" watchObservedRunningTime="2026-07-22 02:45:34.132857771 +0000 UTC m=+500.452033071" Jul 22 02:45:34.188626 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:34.188532 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-567848f679-z2t2x"] Jul 22 02:45:34.770251 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:34.770216 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-ttwjf"] Jul 22 02:45:34.773736 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:34.773713 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-ttwjf" Jul 22 02:45:34.776550 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:34.776479 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Jul 22 02:45:34.776550 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:34.776530 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-crzfq\"" Jul 22 02:45:34.777009 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:34.776996 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Jul 22 02:45:34.784464 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:34.784438 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-ttwjf"] Jul 22 02:45:34.855267 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:34.855232 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9vmls\" (UniqueName: \"kubernetes.io/projected/3979efe4-f663-44f9-ad69-c243fb7ea8fe-kube-api-access-9vmls\") pod \"kuadrant-console-plugin-66b5774985-ttwjf\" (UID: \"3979efe4-f663-44f9-ad69-c243fb7ea8fe\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-ttwjf" Jul 22 02:45:34.855439 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:34.855293 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/3979efe4-f663-44f9-ad69-c243fb7ea8fe-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-ttwjf\" (UID: \"3979efe4-f663-44f9-ad69-c243fb7ea8fe\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-ttwjf" Jul 22 02:45:34.855439 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:34.855316 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/3979efe4-f663-44f9-ad69-c243fb7ea8fe-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-ttwjf\" (UID: \"3979efe4-f663-44f9-ad69-c243fb7ea8fe\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-ttwjf" Jul 22 02:45:34.956391 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:34.956331 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9vmls\" (UniqueName: \"kubernetes.io/projected/3979efe4-f663-44f9-ad69-c243fb7ea8fe-kube-api-access-9vmls\") pod \"kuadrant-console-plugin-66b5774985-ttwjf\" (UID: \"3979efe4-f663-44f9-ad69-c243fb7ea8fe\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-ttwjf" Jul 22 02:45:34.956709 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:34.956429 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/3979efe4-f663-44f9-ad69-c243fb7ea8fe-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-ttwjf\" (UID: \"3979efe4-f663-44f9-ad69-c243fb7ea8fe\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-ttwjf" Jul 22 02:45:34.956709 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:34.956457 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/3979efe4-f663-44f9-ad69-c243fb7ea8fe-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-ttwjf\" (UID: \"3979efe4-f663-44f9-ad69-c243fb7ea8fe\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-ttwjf" Jul 22 02:45:34.957274 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:34.957242 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/3979efe4-f663-44f9-ad69-c243fb7ea8fe-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-ttwjf\" (UID: \"3979efe4-f663-44f9-ad69-c243fb7ea8fe\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-ttwjf" Jul 22 02:45:34.959172 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:34.959151 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/3979efe4-f663-44f9-ad69-c243fb7ea8fe-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-ttwjf\" (UID: \"3979efe4-f663-44f9-ad69-c243fb7ea8fe\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-ttwjf" Jul 22 02:45:34.967400 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:34.967374 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9vmls\" (UniqueName: \"kubernetes.io/projected/3979efe4-f663-44f9-ad69-c243fb7ea8fe-kube-api-access-9vmls\") pod \"kuadrant-console-plugin-66b5774985-ttwjf\" (UID: \"3979efe4-f663-44f9-ad69-c243fb7ea8fe\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-ttwjf" Jul 22 02:45:35.061862 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:35.061784 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-dzltm" Jul 22 02:45:35.085161 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:35.085125 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-ttwjf" Jul 22 02:45:35.252605 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:35.252581 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-ttwjf"] Jul 22 02:45:35.528257 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:45:35.528223 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3979efe4_f663_44f9_ad69_c243fb7ea8fe.slice/crio-cb5f86120ac75faec644501c948fad00f9c72ad872542ee5af49607b77a659b8 WatchSource:0}: Error finding container cb5f86120ac75faec644501c948fad00f9c72ad872542ee5af49607b77a659b8: Status 404 returned error can't find the container with id cb5f86120ac75faec644501c948fad00f9c72ad872542ee5af49607b77a659b8 Jul 22 02:45:36.126695 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:36.126664 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-jgxzw" event={"ID":"340ba516-b472-446f-94a4-1cc708d4d13a","Type":"ContainerStarted","Data":"7b5ae8fbe389011c156cb0fd4732da3573f155f7cdea133f9705ff29b8e7be26"} Jul 22 02:45:36.127138 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:36.126722 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-jgxzw" Jul 22 02:45:36.127870 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:36.127849 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-ttwjf" event={"ID":"3979efe4-f663-44f9-ad69-c243fb7ea8fe","Type":"ContainerStarted","Data":"cb5f86120ac75faec644501c948fad00f9c72ad872542ee5af49607b77a659b8"} Jul 22 02:45:36.170273 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:36.170216 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-jgxzw" podStartSLOduration=1.576262504 podStartE2EDuration="3.170193143s" totalImagesPullingTime="1.593930639s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 02:45:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:45:36.168032348 +0000 UTC m=+502.487207647" watchObservedRunningTime="2026-07-22 02:45:36.170193143 +0000 UTC m=+502.489368446" Jul 22 02:45:43.108250 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:43.108218 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dwkdk" Jul 22 02:45:45.123743 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:45.123709 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-v2m7g" Jul 22 02:45:45.219630 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:45.219591 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dwkdk"] Jul 22 02:45:45.219890 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:45.219794 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dwkdk" podUID="be48e9fc-47f6-476a-9db2-75bc27919788" containerName="manager" containerID="cri-o://882abeaf8ee752ae09729822a5875a87a9cefa8f70797b1a793c72873bacd7c7" gracePeriod=2 Jul 22 02:45:45.228440 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:45.228406 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dwkdk"] Jul 22 02:45:45.231606 ip-10-0-133-31 kubenswrapper[2578]: E0722 02:45:45.231548 2578 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"be48e9fc-47f6-476a-9db2-75bc27919788\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"kuadrant-system\"/\"kuadrant-operator-controller-manager-8684c9ff49-dwkdk\": pods \"kuadrant-operator-controller-manager-8684c9ff49-dwkdk\" not found" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dwkdk" Jul 22 02:45:45.242274 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:45.242253 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-jgxzw"] Jul 22 02:45:45.242503 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:45.242473 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-jgxzw" podUID="340ba516-b472-446f-94a4-1cc708d4d13a" containerName="manager" containerID="cri-o://7b5ae8fbe389011c156cb0fd4732da3573f155f7cdea133f9705ff29b8e7be26" gracePeriod=2 Jul 22 02:45:45.244639 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:45.244621 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-jgxzw" Jul 22 02:45:45.263892 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:45.263830 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-jgxzw"] Jul 22 02:45:45.282384 ip-10-0-133-31 kubenswrapper[2578]: E0722 02:45:45.282346 2578 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"340ba516-b472-446f-94a4-1cc708d4d13a\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"kuadrant-system\"/\"limitador-operator-controller-manager-859d7b78bd-jgxzw\": pods \"limitador-operator-controller-manager-859d7b78bd-jgxzw\" not found" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-jgxzw" Jul 22 02:45:45.285057 ip-10-0-133-31 kubenswrapper[2578]: E0722 02:45:45.285016 2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-dwkdk\" is forbidden: User \"system:node:ip-10-0-133-31.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-31.ec2.internal' and this object" podUID="be48e9fc-47f6-476a-9db2-75bc27919788" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dwkdk" Jul 22 02:45:45.287491 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:45.287466 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-pjpdf"] Jul 22 02:45:45.288186 ip-10-0-133-31 kubenswrapper[2578]: E0722 02:45:45.287891 2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-dwkdk\" is forbidden: User \"system:node:ip-10-0-133-31.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-31.ec2.internal' and this object" podUID="be48e9fc-47f6-476a-9db2-75bc27919788" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dwkdk" Jul 22 02:45:45.288403 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:45.288349 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="be48e9fc-47f6-476a-9db2-75bc27919788" containerName="manager" Jul 22 02:45:45.288403 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:45.288372 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="be48e9fc-47f6-476a-9db2-75bc27919788" containerName="manager" Jul 22 02:45:45.288403 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:45.288399 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="340ba516-b472-446f-94a4-1cc708d4d13a" containerName="manager" Jul 22 02:45:45.288597 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:45.288407 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="340ba516-b472-446f-94a4-1cc708d4d13a" containerName="manager" Jul 22 02:45:45.288597 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:45.288506 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="be48e9fc-47f6-476a-9db2-75bc27919788" containerName="manager" Jul 22 02:45:45.288597 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:45.288521 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="340ba516-b472-446f-94a4-1cc708d4d13a" containerName="manager" Jul 22 02:45:45.291161 ip-10-0-133-31 kubenswrapper[2578]: E0722 02:45:45.291131 2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-jgxzw\" is forbidden: User \"system:node:ip-10-0-133-31.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-31.ec2.internal' and this object" podUID="340ba516-b472-446f-94a4-1cc708d4d13a" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-jgxzw" Jul 22 02:45:45.291902 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:45.291881 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-pjpdf" Jul 22 02:45:45.294942 ip-10-0-133-31 kubenswrapper[2578]: E0722 02:45:45.294914 2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-dwkdk\" is forbidden: User \"system:node:ip-10-0-133-31.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-31.ec2.internal' and this object" podUID="be48e9fc-47f6-476a-9db2-75bc27919788" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dwkdk" Jul 22 02:45:45.311830 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:45.311806 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-pjpdf"] Jul 22 02:45:45.331754 ip-10-0-133-31 kubenswrapper[2578]: E0722 02:45:45.331726 2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-jgxzw\" is forbidden: User \"system:node:ip-10-0-133-31.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-31.ec2.internal' and this object" podUID="340ba516-b472-446f-94a4-1cc708d4d13a" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-jgxzw" Jul 22 02:45:45.457517 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:45.457433 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2fxmp\" (UniqueName: \"kubernetes.io/projected/44dc2356-c6f8-4a3a-b81e-c0e84ee68812-kube-api-access-2fxmp\") pod \"limitador-operator-controller-manager-859d7b78bd-pjpdf\" (UID: \"44dc2356-c6f8-4a3a-b81e-c0e84ee68812\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-pjpdf" Jul 22 02:45:45.558319 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:45.558275 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2fxmp\" (UniqueName: \"kubernetes.io/projected/44dc2356-c6f8-4a3a-b81e-c0e84ee68812-kube-api-access-2fxmp\") pod \"limitador-operator-controller-manager-859d7b78bd-pjpdf\" (UID: \"44dc2356-c6f8-4a3a-b81e-c0e84ee68812\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-pjpdf" Jul 22 02:45:45.569554 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:45.569524 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2fxmp\" (UniqueName: \"kubernetes.io/projected/44dc2356-c6f8-4a3a-b81e-c0e84ee68812-kube-api-access-2fxmp\") pod \"limitador-operator-controller-manager-859d7b78bd-pjpdf\" (UID: \"44dc2356-c6f8-4a3a-b81e-c0e84ee68812\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-pjpdf" Jul 22 02:45:45.646661 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:45.646628 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-pjpdf" Jul 22 02:45:46.236473 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:46.236443 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-822mm"] Jul 22 02:45:46.247178 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:46.247149 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-822mm" Jul 22 02:45:46.256025 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:46.255930 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-822mm"] Jul 22 02:45:46.367375 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:46.366897 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9633d4bc-4a36-46c7-af40-d1e630c2b586-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-822mm\" (UID: \"9633d4bc-4a36-46c7-af40-d1e630c2b586\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-822mm" Jul 22 02:45:46.367375 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:46.366975 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ftnvn\" (UniqueName: \"kubernetes.io/projected/9633d4bc-4a36-46c7-af40-d1e630c2b586-kube-api-access-ftnvn\") pod \"kuadrant-operator-controller-manager-86878c769f-822mm\" (UID: \"9633d4bc-4a36-46c7-af40-d1e630c2b586\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-822mm" Jul 22 02:45:46.467852 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:46.467813 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ftnvn\" (UniqueName: \"kubernetes.io/projected/9633d4bc-4a36-46c7-af40-d1e630c2b586-kube-api-access-ftnvn\") pod \"kuadrant-operator-controller-manager-86878c769f-822mm\" (UID: \"9633d4bc-4a36-46c7-af40-d1e630c2b586\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-822mm" Jul 22 02:45:46.467995 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:46.467923 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9633d4bc-4a36-46c7-af40-d1e630c2b586-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-822mm\" (UID: \"9633d4bc-4a36-46c7-af40-d1e630c2b586\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-822mm" Jul 22 02:45:46.468374 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:46.468351 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9633d4bc-4a36-46c7-af40-d1e630c2b586-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-822mm\" (UID: \"9633d4bc-4a36-46c7-af40-d1e630c2b586\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-822mm" Jul 22 02:45:46.483500 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:46.483469 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ftnvn\" (UniqueName: \"kubernetes.io/projected/9633d4bc-4a36-46c7-af40-d1e630c2b586-kube-api-access-ftnvn\") pod \"kuadrant-operator-controller-manager-86878c769f-822mm\" (UID: \"9633d4bc-4a36-46c7-af40-d1e630c2b586\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-822mm" Jul 22 02:45:46.562124 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:46.562053 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-822mm" Jul 22 02:45:57.144474 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:57.144378 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dwkdk" Jul 22 02:45:57.148073 ip-10-0-133-31 kubenswrapper[2578]: E0722 02:45:57.148007 2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-dwkdk\" is forbidden: User \"system:node:ip-10-0-133-31.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-31.ec2.internal' and this object" podUID="be48e9fc-47f6-476a-9db2-75bc27919788" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dwkdk" Jul 22 02:45:57.156853 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:57.156827 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-jgxzw" Jul 22 02:45:57.162539 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:57.162518 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tqsb7\" (UniqueName: \"kubernetes.io/projected/be48e9fc-47f6-476a-9db2-75bc27919788-kube-api-access-tqsb7\") pod \"be48e9fc-47f6-476a-9db2-75bc27919788\" (UID: \"be48e9fc-47f6-476a-9db2-75bc27919788\") " Jul 22 02:45:57.162660 ip-10-0-133-31 kubenswrapper[2578]: E0722 02:45:57.162595 2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-dwkdk\" is forbidden: User \"system:node:ip-10-0-133-31.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-31.ec2.internal' and this object" podUID="be48e9fc-47f6-476a-9db2-75bc27919788" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dwkdk" Jul 22 02:45:57.162660 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:57.162625 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qc8lj\" (UniqueName: \"kubernetes.io/projected/340ba516-b472-446f-94a4-1cc708d4d13a-kube-api-access-qc8lj\") pod \"340ba516-b472-446f-94a4-1cc708d4d13a\" (UID: \"340ba516-b472-446f-94a4-1cc708d4d13a\") " Jul 22 02:45:57.162660 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:57.162653 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/be48e9fc-47f6-476a-9db2-75bc27919788-extensions-socket-volume\") pod \"be48e9fc-47f6-476a-9db2-75bc27919788\" (UID: \"be48e9fc-47f6-476a-9db2-75bc27919788\") " Jul 22 02:45:57.163171 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:57.163150 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be48e9fc-47f6-476a-9db2-75bc27919788-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "be48e9fc-47f6-476a-9db2-75bc27919788" (UID: "be48e9fc-47f6-476a-9db2-75bc27919788"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 02:45:57.164739 ip-10-0-133-31 kubenswrapper[2578]: E0722 02:45:57.164706 2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-jgxzw\" is forbidden: User \"system:node:ip-10-0-133-31.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-31.ec2.internal' and this object" podUID="340ba516-b472-446f-94a4-1cc708d4d13a" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-jgxzw" Jul 22 02:45:57.165215 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:57.165188 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be48e9fc-47f6-476a-9db2-75bc27919788-kube-api-access-tqsb7" (OuterVolumeSpecName: "kube-api-access-tqsb7") pod "be48e9fc-47f6-476a-9db2-75bc27919788" (UID: "be48e9fc-47f6-476a-9db2-75bc27919788"). InnerVolumeSpecName "kube-api-access-tqsb7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 02:45:57.165280 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:57.165226 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/340ba516-b472-446f-94a4-1cc708d4d13a-kube-api-access-qc8lj" (OuterVolumeSpecName: "kube-api-access-qc8lj") pod "340ba516-b472-446f-94a4-1cc708d4d13a" (UID: "340ba516-b472-446f-94a4-1cc708d4d13a"). InnerVolumeSpecName "kube-api-access-qc8lj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 02:45:57.176863 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:57.176842 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-822mm"] Jul 22 02:45:57.177818 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:45:57.177788 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9633d4bc_4a36_46c7_af40_d1e630c2b586.slice/crio-7fb26138a1f2d4b9800827a4e18f2ba189d936327978537d713e960f005159da WatchSource:0}: Error finding container 7fb26138a1f2d4b9800827a4e18f2ba189d936327978537d713e960f005159da: Status 404 returned error can't find the container with id 7fb26138a1f2d4b9800827a4e18f2ba189d936327978537d713e960f005159da Jul 22 02:45:57.192121 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:57.192099 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-pjpdf"] Jul 22 02:45:57.192657 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:45:57.192639 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44dc2356_c6f8_4a3a_b81e_c0e84ee68812.slice/crio-7d5e0a3e6fd633a23219f539fa4c5e6ce2f3700b42968f26107317a69a284231 WatchSource:0}: Error finding container 7d5e0a3e6fd633a23219f539fa4c5e6ce2f3700b42968f26107317a69a284231: Status 404 returned error can't find the container with id 7d5e0a3e6fd633a23219f539fa4c5e6ce2f3700b42968f26107317a69a284231 Jul 22 02:45:57.254403 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:57.254371 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-822mm" event={"ID":"9633d4bc-4a36-46c7-af40-d1e630c2b586","Type":"ContainerStarted","Data":"8a8384604ea9cccafe781b39c4a4712d15540cbcc270b7ef649a83c7b49557e5"} Jul 22 02:45:57.254508 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:57.254422 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-822mm" event={"ID":"9633d4bc-4a36-46c7-af40-d1e630c2b586","Type":"ContainerStarted","Data":"7fb26138a1f2d4b9800827a4e18f2ba189d936327978537d713e960f005159da"} Jul 22 02:45:57.254575 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:57.254543 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-822mm" Jul 22 02:45:57.255912 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:57.255884 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-ttwjf" event={"ID":"3979efe4-f663-44f9-ad69-c243fb7ea8fe","Type":"ContainerStarted","Data":"9eb13035e4ee95dc7654b24215bc9166b7fd41eb8ab92cef58caa937b53affdd"} Jul 22 02:45:57.257069 ip-10-0-133-31 kubenswrapper[2578]: E0722 02:45:57.257044 2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-dwkdk\" is forbidden: User \"system:node:ip-10-0-133-31.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-31.ec2.internal' and this object" podUID="be48e9fc-47f6-476a-9db2-75bc27919788" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dwkdk" Jul 22 02:45:57.257311 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:57.257295 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-pjpdf" event={"ID":"44dc2356-c6f8-4a3a-b81e-c0e84ee68812","Type":"ContainerStarted","Data":"9a1c87bb15e21dd76892d44075cb564ee6230c19d0f68dadb843927cfc2ca1f1"} Jul 22 02:45:57.257387 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:57.257338 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-pjpdf" event={"ID":"44dc2356-c6f8-4a3a-b81e-c0e84ee68812","Type":"ContainerStarted","Data":"7d5e0a3e6fd633a23219f539fa4c5e6ce2f3700b42968f26107317a69a284231"} Jul 22 02:45:57.257387 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:57.257368 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-pjpdf" Jul 22 02:45:57.258405 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:57.258385 2578 generic.go:356] "Generic (PLEG): container finished" podID="340ba516-b472-446f-94a4-1cc708d4d13a" containerID="7b5ae8fbe389011c156cb0fd4732da3573f155f7cdea133f9705ff29b8e7be26" exitCode=0 Jul 22 02:45:57.258479 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:57.258417 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-jgxzw" Jul 22 02:45:57.258537 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:57.258480 2578 scope.go:117] "RemoveContainer" containerID="7b5ae8fbe389011c156cb0fd4732da3573f155f7cdea133f9705ff29b8e7be26" Jul 22 02:45:57.259457 ip-10-0-133-31 kubenswrapper[2578]: E0722 02:45:57.259349 2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-jgxzw\" is forbidden: User \"system:node:ip-10-0-133-31.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-31.ec2.internal' and this object" podUID="340ba516-b472-446f-94a4-1cc708d4d13a" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-jgxzw" Jul 22 02:45:57.259801 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:57.259739 2578 generic.go:356] "Generic (PLEG): container finished" podID="be48e9fc-47f6-476a-9db2-75bc27919788" containerID="882abeaf8ee752ae09729822a5875a87a9cefa8f70797b1a793c72873bacd7c7" exitCode=0 Jul 22 02:45:57.259801 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:57.259780 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dwkdk" Jul 22 02:45:57.263667 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:57.263282 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qc8lj\" (UniqueName: \"kubernetes.io/projected/340ba516-b472-446f-94a4-1cc708d4d13a-kube-api-access-qc8lj\") on node \"ip-10-0-133-31.ec2.internal\" DevicePath \"\"" Jul 22 02:45:57.263667 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:57.263307 2578 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/be48e9fc-47f6-476a-9db2-75bc27919788-extensions-socket-volume\") on node \"ip-10-0-133-31.ec2.internal\" DevicePath \"\"" Jul 22 02:45:57.263667 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:57.263475 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tqsb7\" (UniqueName: \"kubernetes.io/projected/be48e9fc-47f6-476a-9db2-75bc27919788-kube-api-access-tqsb7\") on node \"ip-10-0-133-31.ec2.internal\" DevicePath \"\"" Jul 22 02:45:57.267868 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:57.267851 2578 scope.go:117] "RemoveContainer" containerID="7b5ae8fbe389011c156cb0fd4732da3573f155f7cdea133f9705ff29b8e7be26" Jul 22 02:45:57.268124 ip-10-0-133-31 kubenswrapper[2578]: E0722 02:45:57.268106 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b5ae8fbe389011c156cb0fd4732da3573f155f7cdea133f9705ff29b8e7be26\": container with ID starting with 7b5ae8fbe389011c156cb0fd4732da3573f155f7cdea133f9705ff29b8e7be26 not found: ID does not exist" containerID="7b5ae8fbe389011c156cb0fd4732da3573f155f7cdea133f9705ff29b8e7be26" Jul 22 02:45:57.268203 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:57.268134 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b5ae8fbe389011c156cb0fd4732da3573f155f7cdea133f9705ff29b8e7be26"} err="failed to get container status \"7b5ae8fbe389011c156cb0fd4732da3573f155f7cdea133f9705ff29b8e7be26\": rpc error: code = NotFound desc = could not find container \"7b5ae8fbe389011c156cb0fd4732da3573f155f7cdea133f9705ff29b8e7be26\": container with ID starting with 7b5ae8fbe389011c156cb0fd4732da3573f155f7cdea133f9705ff29b8e7be26 not found: ID does not exist" Jul 22 02:45:57.268203 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:57.268155 2578 scope.go:117] "RemoveContainer" containerID="882abeaf8ee752ae09729822a5875a87a9cefa8f70797b1a793c72873bacd7c7" Jul 22 02:45:57.275627 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:57.275607 2578 scope.go:117] "RemoveContainer" containerID="882abeaf8ee752ae09729822a5875a87a9cefa8f70797b1a793c72873bacd7c7" Jul 22 02:45:57.275851 ip-10-0-133-31 kubenswrapper[2578]: E0722 02:45:57.275836 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"882abeaf8ee752ae09729822a5875a87a9cefa8f70797b1a793c72873bacd7c7\": container with ID starting with 882abeaf8ee752ae09729822a5875a87a9cefa8f70797b1a793c72873bacd7c7 not found: ID does not exist" containerID="882abeaf8ee752ae09729822a5875a87a9cefa8f70797b1a793c72873bacd7c7" Jul 22 02:45:57.275907 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:57.275856 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"882abeaf8ee752ae09729822a5875a87a9cefa8f70797b1a793c72873bacd7c7"} err="failed to get container status \"882abeaf8ee752ae09729822a5875a87a9cefa8f70797b1a793c72873bacd7c7\": rpc error: code = NotFound desc = could not find container \"882abeaf8ee752ae09729822a5875a87a9cefa8f70797b1a793c72873bacd7c7\": container with ID starting with 882abeaf8ee752ae09729822a5875a87a9cefa8f70797b1a793c72873bacd7c7 not found: ID does not exist" Jul 22 02:45:57.280982 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:57.280949 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-822mm" podStartSLOduration=11.280938382 podStartE2EDuration="11.280938382s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 02:45:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:45:57.280164611 +0000 UTC m=+523.599339911" watchObservedRunningTime="2026-07-22 02:45:57.280938382 +0000 UTC m=+523.600113661" Jul 22 02:45:57.301689 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:57.301638 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-ttwjf" podStartSLOduration=1.7425751630000001 podStartE2EDuration="23.301617601s" totalImagesPullingTime="21.559042438s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 02:45:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:45:57.299753944 +0000 UTC m=+523.618929244" watchObservedRunningTime="2026-07-22 02:45:57.301617601 +0000 UTC m=+523.620792902" Jul 22 02:45:57.302242 ip-10-0-133-31 kubenswrapper[2578]: E0722 02:45:57.302217 2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-dwkdk\" is forbidden: User \"system:node:ip-10-0-133-31.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-31.ec2.internal' and this object" podUID="be48e9fc-47f6-476a-9db2-75bc27919788" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dwkdk" Jul 22 02:45:57.332068 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:57.332016 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-pjpdf" podStartSLOduration=12.332000745 podStartE2EDuration="12.332000745s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 02:45:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:45:57.331452794 +0000 UTC m=+523.650628107" watchObservedRunningTime="2026-07-22 02:45:57.332000745 +0000 UTC m=+523.651176046" Jul 22 02:45:57.333896 ip-10-0-133-31 kubenswrapper[2578]: E0722 02:45:57.333868 2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-jgxzw\" is forbidden: User \"system:node:ip-10-0-133-31.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-31.ec2.internal' and this object" podUID="340ba516-b472-446f-94a4-1cc708d4d13a" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-jgxzw" Jul 22 02:45:58.142692 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:58.142658 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="340ba516-b472-446f-94a4-1cc708d4d13a" path="/var/lib/kubelet/pods/340ba516-b472-446f-94a4-1cc708d4d13a/volumes" Jul 22 02:45:58.143029 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:58.143016 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="be48e9fc-47f6-476a-9db2-75bc27919788" path="/var/lib/kubelet/pods/be48e9fc-47f6-476a-9db2-75bc27919788/volumes" Jul 22 02:45:59.210812 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:59.210742 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-567848f679-z2t2x" podUID="b5c79625-adb9-4d46-b9d9-7278cd5e310d" containerName="console" containerID="cri-o://f32889816e2dcc2f02f6deac9eed480ca1c5d0b0940a8a56207b7dc9ad967952" gracePeriod=15 Jul 22 02:45:59.469045 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:59.468984 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-567848f679-z2t2x" Jul 22 02:45:59.480917 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:59.480891 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b5c79625-adb9-4d46-b9d9-7278cd5e310d-trusted-ca-bundle\") pod \"b5c79625-adb9-4d46-b9d9-7278cd5e310d\" (UID: \"b5c79625-adb9-4d46-b9d9-7278cd5e310d\") " Jul 22 02:45:59.481016 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:59.480919 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b5c79625-adb9-4d46-b9d9-7278cd5e310d-service-ca\") pod \"b5c79625-adb9-4d46-b9d9-7278cd5e310d\" (UID: \"b5c79625-adb9-4d46-b9d9-7278cd5e310d\") " Jul 22 02:45:59.481016 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:59.480942 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b5c79625-adb9-4d46-b9d9-7278cd5e310d-oauth-serving-cert\") pod \"b5c79625-adb9-4d46-b9d9-7278cd5e310d\" (UID: \"b5c79625-adb9-4d46-b9d9-7278cd5e310d\") " Jul 22 02:45:59.481016 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:59.480967 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xj2h9\" (UniqueName: \"kubernetes.io/projected/b5c79625-adb9-4d46-b9d9-7278cd5e310d-kube-api-access-xj2h9\") pod \"b5c79625-adb9-4d46-b9d9-7278cd5e310d\" (UID: \"b5c79625-adb9-4d46-b9d9-7278cd5e310d\") " Jul 22 02:45:59.481142 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:59.481077 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b5c79625-adb9-4d46-b9d9-7278cd5e310d-console-config\") pod \"b5c79625-adb9-4d46-b9d9-7278cd5e310d\" (UID: \"b5c79625-adb9-4d46-b9d9-7278cd5e310d\") " Jul 22 02:45:59.481142 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:59.481126 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b5c79625-adb9-4d46-b9d9-7278cd5e310d-console-oauth-config\") pod \"b5c79625-adb9-4d46-b9d9-7278cd5e310d\" (UID: \"b5c79625-adb9-4d46-b9d9-7278cd5e310d\") " Jul 22 02:45:59.481232 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:59.481167 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b5c79625-adb9-4d46-b9d9-7278cd5e310d-console-serving-cert\") pod \"b5c79625-adb9-4d46-b9d9-7278cd5e310d\" (UID: \"b5c79625-adb9-4d46-b9d9-7278cd5e310d\") " Jul 22 02:45:59.481301 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:59.481274 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5c79625-adb9-4d46-b9d9-7278cd5e310d-service-ca" (OuterVolumeSpecName: "service-ca") pod "b5c79625-adb9-4d46-b9d9-7278cd5e310d" (UID: "b5c79625-adb9-4d46-b9d9-7278cd5e310d"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 02:45:59.481357 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:59.481325 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5c79625-adb9-4d46-b9d9-7278cd5e310d-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "b5c79625-adb9-4d46-b9d9-7278cd5e310d" (UID: "b5c79625-adb9-4d46-b9d9-7278cd5e310d"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 02:45:59.481357 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:59.481337 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5c79625-adb9-4d46-b9d9-7278cd5e310d-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "b5c79625-adb9-4d46-b9d9-7278cd5e310d" (UID: "b5c79625-adb9-4d46-b9d9-7278cd5e310d"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 02:45:59.481458 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:59.481383 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5c79625-adb9-4d46-b9d9-7278cd5e310d-console-config" (OuterVolumeSpecName: "console-config") pod "b5c79625-adb9-4d46-b9d9-7278cd5e310d" (UID: "b5c79625-adb9-4d46-b9d9-7278cd5e310d"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 02:45:59.481508 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:59.481457 2578 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b5c79625-adb9-4d46-b9d9-7278cd5e310d-trusted-ca-bundle\") on node \"ip-10-0-133-31.ec2.internal\" DevicePath \"\"" Jul 22 02:45:59.481508 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:59.481473 2578 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b5c79625-adb9-4d46-b9d9-7278cd5e310d-service-ca\") on node \"ip-10-0-133-31.ec2.internal\" DevicePath \"\"" Jul 22 02:45:59.481508 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:59.481492 2578 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b5c79625-adb9-4d46-b9d9-7278cd5e310d-oauth-serving-cert\") on node \"ip-10-0-133-31.ec2.internal\" DevicePath \"\"" Jul 22 02:45:59.481656 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:59.481508 2578 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b5c79625-adb9-4d46-b9d9-7278cd5e310d-console-config\") on node \"ip-10-0-133-31.ec2.internal\" DevicePath \"\"" Jul 22 02:45:59.483198 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:59.483174 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b5c79625-adb9-4d46-b9d9-7278cd5e310d-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "b5c79625-adb9-4d46-b9d9-7278cd5e310d" (UID: "b5c79625-adb9-4d46-b9d9-7278cd5e310d"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 02:45:59.483275 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:59.483190 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b5c79625-adb9-4d46-b9d9-7278cd5e310d-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "b5c79625-adb9-4d46-b9d9-7278cd5e310d" (UID: "b5c79625-adb9-4d46-b9d9-7278cd5e310d"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 02:45:59.483275 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:59.483252 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5c79625-adb9-4d46-b9d9-7278cd5e310d-kube-api-access-xj2h9" (OuterVolumeSpecName: "kube-api-access-xj2h9") pod "b5c79625-adb9-4d46-b9d9-7278cd5e310d" (UID: "b5c79625-adb9-4d46-b9d9-7278cd5e310d"). InnerVolumeSpecName "kube-api-access-xj2h9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 02:45:59.582924 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:59.582888 2578 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b5c79625-adb9-4d46-b9d9-7278cd5e310d-console-oauth-config\") on node \"ip-10-0-133-31.ec2.internal\" DevicePath \"\"" Jul 22 02:45:59.582924 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:59.582917 2578 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b5c79625-adb9-4d46-b9d9-7278cd5e310d-console-serving-cert\") on node \"ip-10-0-133-31.ec2.internal\" DevicePath \"\"" Jul 22 02:45:59.582924 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:45:59.582929 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xj2h9\" (UniqueName: \"kubernetes.io/projected/b5c79625-adb9-4d46-b9d9-7278cd5e310d-kube-api-access-xj2h9\") on node \"ip-10-0-133-31.ec2.internal\" DevicePath \"\"" Jul 22 02:46:00.278908 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:46:00.278871 2578 generic.go:356] "Generic (PLEG): container finished" podID="b5c79625-adb9-4d46-b9d9-7278cd5e310d" containerID="f32889816e2dcc2f02f6deac9eed480ca1c5d0b0940a8a56207b7dc9ad967952" exitCode=0 Jul 22 02:46:00.279352 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:46:00.278942 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-567848f679-z2t2x" Jul 22 02:46:00.279352 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:46:00.278950 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-567848f679-z2t2x" event={"ID":"b5c79625-adb9-4d46-b9d9-7278cd5e310d","Type":"ContainerDied","Data":"f32889816e2dcc2f02f6deac9eed480ca1c5d0b0940a8a56207b7dc9ad967952"} Jul 22 02:46:00.279352 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:46:00.279012 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-567848f679-z2t2x" event={"ID":"b5c79625-adb9-4d46-b9d9-7278cd5e310d","Type":"ContainerDied","Data":"a14127f9c7452c0c92c1ed5177785034e98421898e992eabf7a6ef015db69d8d"} Jul 22 02:46:00.279352 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:46:00.279043 2578 scope.go:117] "RemoveContainer" containerID="f32889816e2dcc2f02f6deac9eed480ca1c5d0b0940a8a56207b7dc9ad967952" Jul 22 02:46:00.287515 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:46:00.287498 2578 scope.go:117] "RemoveContainer" containerID="f32889816e2dcc2f02f6deac9eed480ca1c5d0b0940a8a56207b7dc9ad967952" Jul 22 02:46:00.287800 ip-10-0-133-31 kubenswrapper[2578]: E0722 02:46:00.287781 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f32889816e2dcc2f02f6deac9eed480ca1c5d0b0940a8a56207b7dc9ad967952\": container with ID starting with f32889816e2dcc2f02f6deac9eed480ca1c5d0b0940a8a56207b7dc9ad967952 not found: ID does not exist" containerID="f32889816e2dcc2f02f6deac9eed480ca1c5d0b0940a8a56207b7dc9ad967952" Jul 22 02:46:00.287871 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:46:00.287808 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f32889816e2dcc2f02f6deac9eed480ca1c5d0b0940a8a56207b7dc9ad967952"} err="failed to get container status \"f32889816e2dcc2f02f6deac9eed480ca1c5d0b0940a8a56207b7dc9ad967952\": rpc error: code = NotFound desc = could not find container \"f32889816e2dcc2f02f6deac9eed480ca1c5d0b0940a8a56207b7dc9ad967952\": container with ID starting with f32889816e2dcc2f02f6deac9eed480ca1c5d0b0940a8a56207b7dc9ad967952 not found: ID does not exist" Jul 22 02:46:00.312388 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:46:00.312363 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-567848f679-z2t2x"] Jul 22 02:46:00.327774 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:46:00.327751 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-567848f679-z2t2x"] Jul 22 02:46:02.142036 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:46:02.141992 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b5c79625-adb9-4d46-b9d9-7278cd5e310d" path="/var/lib/kubelet/pods/b5c79625-adb9-4d46-b9d9-7278cd5e310d/volumes" Jul 22 02:46:08.270103 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:46:08.270020 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-822mm" Jul 22 02:46:08.270467 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:46:08.270434 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-pjpdf" Jul 22 02:46:24.718838 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:46:24.718805 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-n8zpf"] Jul 22 02:46:24.719279 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:46:24.719188 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b5c79625-adb9-4d46-b9d9-7278cd5e310d" containerName="console" Jul 22 02:46:24.719279 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:46:24.719205 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="b5c79625-adb9-4d46-b9d9-7278cd5e310d" containerName="console" Jul 22 02:46:24.719279 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:46:24.719272 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="b5c79625-adb9-4d46-b9d9-7278cd5e310d" containerName="console" Jul 22 02:46:24.739486 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:46:24.739461 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-n8zpf"] Jul 22 02:46:24.739657 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:46:24.739614 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-n8zpf" Jul 22 02:46:24.742588 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:46:24.742547 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-ckkvm\"" Jul 22 02:46:24.783884 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:46:24.783858 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1021ed34-ba20-48c2-a67f-6bd304cfd0cb-credential-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-n8zpf\" (UID: \"1021ed34-ba20-48c2-a67f-6bd304cfd0cb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-n8zpf" Jul 22 02:46:24.783990 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:46:24.783895 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1021ed34-ba20-48c2-a67f-6bd304cfd0cb-istio-envoy\") pod \"maas-default-gateway-openshift-default-56654dcf7-n8zpf\" (UID: \"1021ed34-ba20-48c2-a67f-6bd304cfd0cb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-n8zpf" Jul 22 02:46:24.783990 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:46:24.783921 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1021ed34-ba20-48c2-a67f-6bd304cfd0cb-istio-podinfo\") pod \"maas-default-gateway-openshift-default-56654dcf7-n8zpf\" (UID: \"1021ed34-ba20-48c2-a67f-6bd304cfd0cb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-n8zpf" Jul 22 02:46:24.784106 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:46:24.783997 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1021ed34-ba20-48c2-a67f-6bd304cfd0cb-workload-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-n8zpf\" (UID: \"1021ed34-ba20-48c2-a67f-6bd304cfd0cb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-n8zpf" Jul 22 02:46:24.784106 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:46:24.784079 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kscd7\" (UniqueName: \"kubernetes.io/projected/1021ed34-ba20-48c2-a67f-6bd304cfd0cb-kube-api-access-kscd7\") pod \"maas-default-gateway-openshift-default-56654dcf7-n8zpf\" (UID: \"1021ed34-ba20-48c2-a67f-6bd304cfd0cb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-n8zpf" Jul 22 02:46:24.784194 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:46:24.784113 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1021ed34-ba20-48c2-a67f-6bd304cfd0cb-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-56654dcf7-n8zpf\" (UID: \"1021ed34-ba20-48c2-a67f-6bd304cfd0cb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-n8zpf" Jul 22 02:46:24.784233 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:46:24.784188 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1021ed34-ba20-48c2-a67f-6bd304cfd0cb-istio-data\") pod \"maas-default-gateway-openshift-default-56654dcf7-n8zpf\" (UID: \"1021ed34-ba20-48c2-a67f-6bd304cfd0cb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-n8zpf" Jul 22 02:46:24.784233 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:46:24.784221 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1021ed34-ba20-48c2-a67f-6bd304cfd0cb-workload-certs\") pod \"maas-default-gateway-openshift-default-56654dcf7-n8zpf\" (UID: \"1021ed34-ba20-48c2-a67f-6bd304cfd0cb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-n8zpf" Jul 22 02:46:24.784303 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:46:24.784262 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1021ed34-ba20-48c2-a67f-6bd304cfd0cb-istio-token\") pod \"maas-default-gateway-openshift-default-56654dcf7-n8zpf\" (UID: \"1021ed34-ba20-48c2-a67f-6bd304cfd0cb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-n8zpf" Jul 22 02:46:24.884729 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:46:24.884703 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kscd7\" (UniqueName: \"kubernetes.io/projected/1021ed34-ba20-48c2-a67f-6bd304cfd0cb-kube-api-access-kscd7\") pod \"maas-default-gateway-openshift-default-56654dcf7-n8zpf\" (UID: \"1021ed34-ba20-48c2-a67f-6bd304cfd0cb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-n8zpf" Jul 22 02:46:24.884842 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:46:24.884737 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1021ed34-ba20-48c2-a67f-6bd304cfd0cb-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-56654dcf7-n8zpf\" (UID: \"1021ed34-ba20-48c2-a67f-6bd304cfd0cb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-n8zpf" Jul 22 02:46:24.884842 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:46:24.884772 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1021ed34-ba20-48c2-a67f-6bd304cfd0cb-istio-data\") pod \"maas-default-gateway-openshift-default-56654dcf7-n8zpf\" (UID: \"1021ed34-ba20-48c2-a67f-6bd304cfd0cb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-n8zpf" Jul 22 02:46:24.884971 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:46:24.884948 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1021ed34-ba20-48c2-a67f-6bd304cfd0cb-workload-certs\") pod \"maas-default-gateway-openshift-default-56654dcf7-n8zpf\" (UID: \"1021ed34-ba20-48c2-a67f-6bd304cfd0cb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-n8zpf" Jul 22 02:46:24.885036 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:46:24.884988 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1021ed34-ba20-48c2-a67f-6bd304cfd0cb-istio-token\") pod \"maas-default-gateway-openshift-default-56654dcf7-n8zpf\" (UID: \"1021ed34-ba20-48c2-a67f-6bd304cfd0cb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-n8zpf" Jul 22 02:46:24.885100 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:46:24.885053 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1021ed34-ba20-48c2-a67f-6bd304cfd0cb-credential-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-n8zpf\" (UID: \"1021ed34-ba20-48c2-a67f-6bd304cfd0cb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-n8zpf" Jul 22 02:46:24.885100 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:46:24.885083 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1021ed34-ba20-48c2-a67f-6bd304cfd0cb-istio-envoy\") pod \"maas-default-gateway-openshift-default-56654dcf7-n8zpf\" (UID: \"1021ed34-ba20-48c2-a67f-6bd304cfd0cb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-n8zpf" Jul 22 02:46:24.885203 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:46:24.885108 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1021ed34-ba20-48c2-a67f-6bd304cfd0cb-istio-podinfo\") pod \"maas-default-gateway-openshift-default-56654dcf7-n8zpf\" (UID: \"1021ed34-ba20-48c2-a67f-6bd304cfd0cb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-n8zpf" Jul 22 02:46:24.885203 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:46:24.885116 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1021ed34-ba20-48c2-a67f-6bd304cfd0cb-istio-data\") pod \"maas-default-gateway-openshift-default-56654dcf7-n8zpf\" (UID: \"1021ed34-ba20-48c2-a67f-6bd304cfd0cb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-n8zpf" Jul 22 02:46:24.885203 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:46:24.885184 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1021ed34-ba20-48c2-a67f-6bd304cfd0cb-workload-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-n8zpf\" (UID: \"1021ed34-ba20-48c2-a67f-6bd304cfd0cb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-n8zpf" Jul 22 02:46:24.885390 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:46:24.885369 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1021ed34-ba20-48c2-a67f-6bd304cfd0cb-workload-certs\") pod \"maas-default-gateway-openshift-default-56654dcf7-n8zpf\" (UID: \"1021ed34-ba20-48c2-a67f-6bd304cfd0cb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-n8zpf" Jul 22 02:46:24.885501 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:46:24.885479 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1021ed34-ba20-48c2-a67f-6bd304cfd0cb-credential-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-n8zpf\" (UID: \"1021ed34-ba20-48c2-a67f-6bd304cfd0cb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-n8zpf" Jul 22 02:46:24.885582 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:46:24.885510 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1021ed34-ba20-48c2-a67f-6bd304cfd0cb-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-56654dcf7-n8zpf\" (UID: \"1021ed34-ba20-48c2-a67f-6bd304cfd0cb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-n8zpf" Jul 22 02:46:24.885582 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:46:24.885549 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1021ed34-ba20-48c2-a67f-6bd304cfd0cb-workload-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-n8zpf\" (UID: \"1021ed34-ba20-48c2-a67f-6bd304cfd0cb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-n8zpf" Jul 22 02:46:24.887448 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:46:24.887427 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1021ed34-ba20-48c2-a67f-6bd304cfd0cb-istio-envoy\") pod \"maas-default-gateway-openshift-default-56654dcf7-n8zpf\" (UID: \"1021ed34-ba20-48c2-a67f-6bd304cfd0cb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-n8zpf" Jul 22 02:46:24.887780 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:46:24.887759 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1021ed34-ba20-48c2-a67f-6bd304cfd0cb-istio-podinfo\") pod \"maas-default-gateway-openshift-default-56654dcf7-n8zpf\" (UID: \"1021ed34-ba20-48c2-a67f-6bd304cfd0cb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-n8zpf" Jul 22 02:46:24.893980 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:46:24.893944 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kscd7\" (UniqueName: \"kubernetes.io/projected/1021ed34-ba20-48c2-a67f-6bd304cfd0cb-kube-api-access-kscd7\") pod \"maas-default-gateway-openshift-default-56654dcf7-n8zpf\" (UID: \"1021ed34-ba20-48c2-a67f-6bd304cfd0cb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-n8zpf" Jul 22 02:46:24.894212 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:46:24.894196 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1021ed34-ba20-48c2-a67f-6bd304cfd0cb-istio-token\") pod \"maas-default-gateway-openshift-default-56654dcf7-n8zpf\" (UID: \"1021ed34-ba20-48c2-a67f-6bd304cfd0cb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-n8zpf" Jul 22 02:46:25.053720 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:46:25.053650 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-n8zpf" Jul 22 02:46:25.192556 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:46:25.192527 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-n8zpf"] Jul 22 02:46:25.194248 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:46:25.194216 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1021ed34_ba20_48c2_a67f_6bd304cfd0cb.slice/crio-8faff90396b89a79cedc2e0cf9afa8b85c066a6d3da0b89b5aa406c9bc0c1591 WatchSource:0}: Error finding container 8faff90396b89a79cedc2e0cf9afa8b85c066a6d3da0b89b5aa406c9bc0c1591: Status 404 returned error can't find the container with id 8faff90396b89a79cedc2e0cf9afa8b85c066a6d3da0b89b5aa406c9bc0c1591 Jul 22 02:46:25.196329 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:46:25.196295 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 22 02:46:25.196451 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:46:25.196381 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 22 02:46:25.196451 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:46:25.196426 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 22 02:46:25.414070 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:46:25.414030 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-n8zpf" event={"ID":"1021ed34-ba20-48c2-a67f-6bd304cfd0cb","Type":"ContainerStarted","Data":"c69f6890c0ac44fca80cfcb2e72e9d88bdbb2f6087826c6ee303d89c31bdc638"} Jul 22 02:46:25.414221 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:46:25.414105 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-n8zpf" event={"ID":"1021ed34-ba20-48c2-a67f-6bd304cfd0cb","Type":"ContainerStarted","Data":"8faff90396b89a79cedc2e0cf9afa8b85c066a6d3da0b89b5aa406c9bc0c1591"} Jul 22 02:46:25.434901 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:46:25.434859 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-n8zpf" podStartSLOduration=1.43484396 podStartE2EDuration="1.43484396s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 02:46:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:46:25.433309694 +0000 UTC m=+551.752484994" watchObservedRunningTime="2026-07-22 02:46:25.43484396 +0000 UTC m=+551.754019260" Jul 22 02:46:26.054805 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:46:26.054774 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-n8zpf" Jul 22 02:46:26.059358 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:46:26.059335 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-n8zpf" Jul 22 02:46:26.419593 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:46:26.419549 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-n8zpf" Jul 22 02:46:26.420338 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:46:26.420322 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-n8zpf" Jul 22 02:47:14.055701 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:47:14.055671 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-prfql_fd7bf923-8d16-476c-a8d0-a3e6fd645061/ovn-acl-logging/0.log" Jul 22 02:47:14.058531 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:47:14.058510 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-prfql_fd7bf923-8d16-476c-a8d0-a3e6fd645061/ovn-acl-logging/0.log" Jul 22 02:47:52.854657 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:47:52.854579 2578 generic.go:356] "Generic (PLEG): container finished" podID="1021ed34-ba20-48c2-a67f-6bd304cfd0cb" containerID="c69f6890c0ac44fca80cfcb2e72e9d88bdbb2f6087826c6ee303d89c31bdc638" exitCode=0 Jul 22 02:47:52.854657 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:47:52.854614 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-n8zpf" event={"ID":"1021ed34-ba20-48c2-a67f-6bd304cfd0cb","Type":"ContainerDied","Data":"c69f6890c0ac44fca80cfcb2e72e9d88bdbb2f6087826c6ee303d89c31bdc638"} Jul 22 02:47:52.855055 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:47:52.854914 2578 scope.go:117] "RemoveContainer" containerID="c69f6890c0ac44fca80cfcb2e72e9d88bdbb2f6087826c6ee303d89c31bdc638" Jul 22 02:47:52.855806 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:47:52.855780 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 22 02:47:52.855866 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:47:52.855843 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 22 02:47:52.855901 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:47:52.855873 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 22 02:47:53.861725 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:47:53.861681 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-n8zpf" event={"ID":"1021ed34-ba20-48c2-a67f-6bd304cfd0cb","Type":"ContainerStarted","Data":"e38ca87881f4700210c945bb5c8696ae99f5fcb1f0fa107a95e2c0d3eae2bf8d"} Jul 22 02:47:54.054714 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:47:54.054677 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-n8zpf" Jul 22 02:47:54.055694 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:47:54.055672 2578 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-n8zpf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body= Jul 22 02:47:54.055815 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:47:54.055714 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-n8zpf" podUID="1021ed34-ba20-48c2-a67f-6bd304cfd0cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" Jul 22 02:47:55.054636 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:47:55.054595 2578 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-n8zpf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body= Jul 22 02:47:55.055044 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:47:55.054658 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-n8zpf" podUID="1021ed34-ba20-48c2-a67f-6bd304cfd0cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" Jul 22 02:47:56.054183 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:47:56.054150 2578 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-n8zpf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body= Jul 22 02:47:56.054371 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:47:56.054202 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-n8zpf" podUID="1021ed34-ba20-48c2-a67f-6bd304cfd0cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" Jul 22 02:47:56.420159 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:47:56.420125 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-n8zpf" Jul 22 02:47:57.054258 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:47:57.054223 2578 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-n8zpf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body= Jul 22 02:47:57.054430 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:47:57.054277 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-n8zpf" podUID="1021ed34-ba20-48c2-a67f-6bd304cfd0cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" Jul 22 02:47:58.058169 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:47:58.058138 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-n8zpf" Jul 22 02:47:58.059039 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:47:58.059022 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-n8zpf" Jul 22 02:48:08.487305 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:48:08.487271 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-fws8d"] Jul 22 02:48:08.489379 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:48:08.489360 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fws8d" Jul 22 02:48:08.496205 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:48:08.495950 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 22 02:48:08.496641 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:48:08.496619 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Jul 22 02:48:08.497342 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:48:08.497324 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-xsxlk\"" Jul 22 02:48:08.497415 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:48:08.497365 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 22 02:48:08.530429 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:48:08.530399 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-fws8d"] Jul 22 02:48:08.564199 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:48:08.564172 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4d6vf\" (UniqueName: \"kubernetes.io/projected/c7548ae4-2b73-49a3-9d98-421a214bb792-kube-api-access-4d6vf\") pod \"e2e-distinct-simulated-kserve-59975d69f9-fws8d\" (UID: \"c7548ae4-2b73-49a3-9d98-421a214bb792\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fws8d" Jul 22 02:48:08.564304 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:48:08.564210 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c7548ae4-2b73-49a3-9d98-421a214bb792-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-fws8d\" (UID: \"c7548ae4-2b73-49a3-9d98-421a214bb792\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fws8d" Jul 22 02:48:08.564304 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:48:08.564232 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c7548ae4-2b73-49a3-9d98-421a214bb792-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-fws8d\" (UID: \"c7548ae4-2b73-49a3-9d98-421a214bb792\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fws8d" Jul 22 02:48:08.564304 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:48:08.564270 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c7548ae4-2b73-49a3-9d98-421a214bb792-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-fws8d\" (UID: \"c7548ae4-2b73-49a3-9d98-421a214bb792\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fws8d" Jul 22 02:48:08.564304 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:48:08.564296 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c7548ae4-2b73-49a3-9d98-421a214bb792-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-fws8d\" (UID: \"c7548ae4-2b73-49a3-9d98-421a214bb792\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fws8d" Jul 22 02:48:08.564452 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:48:08.564323 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c7548ae4-2b73-49a3-9d98-421a214bb792-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-fws8d\" (UID: \"c7548ae4-2b73-49a3-9d98-421a214bb792\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fws8d" Jul 22 02:48:08.664975 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:48:08.664936 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c7548ae4-2b73-49a3-9d98-421a214bb792-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-fws8d\" (UID: \"c7548ae4-2b73-49a3-9d98-421a214bb792\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fws8d" Jul 22 02:48:08.664975 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:48:08.664982 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c7548ae4-2b73-49a3-9d98-421a214bb792-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-fws8d\" (UID: \"c7548ae4-2b73-49a3-9d98-421a214bb792\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fws8d" Jul 22 02:48:08.665223 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:48:08.665004 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c7548ae4-2b73-49a3-9d98-421a214bb792-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-fws8d\" (UID: \"c7548ae4-2b73-49a3-9d98-421a214bb792\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fws8d" Jul 22 02:48:08.665223 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:48:08.665034 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c7548ae4-2b73-49a3-9d98-421a214bb792-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-fws8d\" (UID: \"c7548ae4-2b73-49a3-9d98-421a214bb792\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fws8d" Jul 22 02:48:08.665223 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:48:08.665075 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c7548ae4-2b73-49a3-9d98-421a214bb792-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-fws8d\" (UID: \"c7548ae4-2b73-49a3-9d98-421a214bb792\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fws8d" Jul 22 02:48:08.665223 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:48:08.665126 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4d6vf\" (UniqueName: \"kubernetes.io/projected/c7548ae4-2b73-49a3-9d98-421a214bb792-kube-api-access-4d6vf\") pod \"e2e-distinct-simulated-kserve-59975d69f9-fws8d\" (UID: \"c7548ae4-2b73-49a3-9d98-421a214bb792\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fws8d" Jul 22 02:48:08.665546 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:48:08.665516 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c7548ae4-2b73-49a3-9d98-421a214bb792-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-fws8d\" (UID: \"c7548ae4-2b73-49a3-9d98-421a214bb792\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fws8d" Jul 22 02:48:08.665673 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:48:08.665554 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c7548ae4-2b73-49a3-9d98-421a214bb792-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-fws8d\" (UID: \"c7548ae4-2b73-49a3-9d98-421a214bb792\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fws8d" Jul 22 02:48:08.665900 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:48:08.665614 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c7548ae4-2b73-49a3-9d98-421a214bb792-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-fws8d\" (UID: \"c7548ae4-2b73-49a3-9d98-421a214bb792\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fws8d" Jul 22 02:48:08.667865 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:48:08.667839 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c7548ae4-2b73-49a3-9d98-421a214bb792-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-fws8d\" (UID: \"c7548ae4-2b73-49a3-9d98-421a214bb792\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fws8d" Jul 22 02:48:08.668034 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:48:08.668017 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c7548ae4-2b73-49a3-9d98-421a214bb792-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-fws8d\" (UID: \"c7548ae4-2b73-49a3-9d98-421a214bb792\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fws8d" Jul 22 02:48:08.681152 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:48:08.681128 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4d6vf\" (UniqueName: \"kubernetes.io/projected/c7548ae4-2b73-49a3-9d98-421a214bb792-kube-api-access-4d6vf\") pod \"e2e-distinct-simulated-kserve-59975d69f9-fws8d\" (UID: \"c7548ae4-2b73-49a3-9d98-421a214bb792\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fws8d" Jul 22 02:48:08.799611 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:48:08.799501 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fws8d" Jul 22 02:48:08.956929 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:48:08.956829 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-fws8d"] Jul 22 02:48:08.958972 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:48:08.958942 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7548ae4_2b73_49a3_9d98_421a214bb792.slice/crio-482734cdc534246ed7e7f94e4871dfa962370497af776944b3432aaa0665d41f WatchSource:0}: Error finding container 482734cdc534246ed7e7f94e4871dfa962370497af776944b3432aaa0665d41f: Status 404 returned error can't find the container with id 482734cdc534246ed7e7f94e4871dfa962370497af776944b3432aaa0665d41f Jul 22 02:48:08.960314 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:48:08.960295 2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 22 02:48:09.944248 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:48:09.944214 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fws8d" event={"ID":"c7548ae4-2b73-49a3-9d98-421a214bb792","Type":"ContainerStarted","Data":"482734cdc534246ed7e7f94e4871dfa962370497af776944b3432aaa0665d41f"} Jul 22 02:48:10.950737 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:48:10.950703 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fws8d" event={"ID":"c7548ae4-2b73-49a3-9d98-421a214bb792","Type":"ContainerStarted","Data":"b593f02db804e96150fa177750e28a6f99172e602b3fcdceedb8551aa6b4406f"} Jul 22 02:48:10.989889 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:48:10.989830 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fws8d" podStartSLOduration=2.00865871 podStartE2EDuration="2.989815379s" totalImagesPullingTime="981.156669ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 02:48:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:48:10.98617036 +0000 UTC m=+657.305345670" watchObservedRunningTime="2026-07-22 02:48:10.989815379 +0000 UTC m=+657.308990679" Jul 22 02:48:18.800475 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:48:18.800437 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fws8d" Jul 22 02:48:18.800888 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:48:18.800497 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fws8d" Jul 22 02:48:18.812822 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:48:18.812794 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fws8d" Jul 22 02:48:19.004226 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:48:19.004196 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fws8d" Jul 22 02:48:33.769936 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:48:33.769904 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fztprq"] Jul 22 02:48:33.773940 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:48:33.773922 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fztprq" Jul 22 02:48:33.786671 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:48:33.786650 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Jul 22 02:48:33.795448 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:48:33.795425 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fztprq"] Jul 22 02:48:33.884376 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:48:33.884341 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/62dc2edb-a885-4562-8b80-597c1b43926c-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fztprq\" (UID: \"62dc2edb-a885-4562-8b80-597c1b43926c\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fztprq" Jul 22 02:48:33.884596 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:48:33.884396 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/62dc2edb-a885-4562-8b80-597c1b43926c-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fztprq\" (UID: \"62dc2edb-a885-4562-8b80-597c1b43926c\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fztprq" Jul 22 02:48:33.884596 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:48:33.884464 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/62dc2edb-a885-4562-8b80-597c1b43926c-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fztprq\" (UID: \"62dc2edb-a885-4562-8b80-597c1b43926c\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fztprq" Jul 22 02:48:33.884681 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:48:33.884615 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ccvsc\" (UniqueName: \"kubernetes.io/projected/62dc2edb-a885-4562-8b80-597c1b43926c-kube-api-access-ccvsc\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fztprq\" (UID: \"62dc2edb-a885-4562-8b80-597c1b43926c\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fztprq" Jul 22 02:48:33.884681 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:48:33.884652 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/62dc2edb-a885-4562-8b80-597c1b43926c-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fztprq\" (UID: \"62dc2edb-a885-4562-8b80-597c1b43926c\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fztprq" Jul 22 02:48:33.884750 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:48:33.884716 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/62dc2edb-a885-4562-8b80-597c1b43926c-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fztprq\" (UID: \"62dc2edb-a885-4562-8b80-597c1b43926c\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fztprq" Jul 22 02:48:33.985680 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:48:33.985643 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ccvsc\" (UniqueName: \"kubernetes.io/projected/62dc2edb-a885-4562-8b80-597c1b43926c-kube-api-access-ccvsc\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fztprq\" (UID: \"62dc2edb-a885-4562-8b80-597c1b43926c\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fztprq" Jul 22 02:48:33.985680 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:48:33.985680 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/62dc2edb-a885-4562-8b80-597c1b43926c-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fztprq\" (UID: \"62dc2edb-a885-4562-8b80-597c1b43926c\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fztprq" Jul 22 02:48:33.985969 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:48:33.985719 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/62dc2edb-a885-4562-8b80-597c1b43926c-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fztprq\" (UID: \"62dc2edb-a885-4562-8b80-597c1b43926c\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fztprq" Jul 22 02:48:33.985969 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:48:33.985888 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/62dc2edb-a885-4562-8b80-597c1b43926c-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fztprq\" (UID: \"62dc2edb-a885-4562-8b80-597c1b43926c\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fztprq" Jul 22 02:48:33.986083 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:48:33.985968 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/62dc2edb-a885-4562-8b80-597c1b43926c-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fztprq\" (UID: \"62dc2edb-a885-4562-8b80-597c1b43926c\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fztprq" Jul 22 02:48:33.986083 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:48:33.986002 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/62dc2edb-a885-4562-8b80-597c1b43926c-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fztprq\" (UID: \"62dc2edb-a885-4562-8b80-597c1b43926c\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fztprq" Jul 22 02:48:33.986083 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:48:33.986056 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/62dc2edb-a885-4562-8b80-597c1b43926c-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fztprq\" (UID: \"62dc2edb-a885-4562-8b80-597c1b43926c\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fztprq" Jul 22 02:48:33.986191 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:48:33.986082 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/62dc2edb-a885-4562-8b80-597c1b43926c-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fztprq\" (UID: \"62dc2edb-a885-4562-8b80-597c1b43926c\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fztprq" Jul 22 02:48:33.986191 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:48:33.986161 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/62dc2edb-a885-4562-8b80-597c1b43926c-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fztprq\" (UID: \"62dc2edb-a885-4562-8b80-597c1b43926c\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fztprq" Jul 22 02:48:33.988271 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:48:33.988240 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/62dc2edb-a885-4562-8b80-597c1b43926c-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fztprq\" (UID: \"62dc2edb-a885-4562-8b80-597c1b43926c\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fztprq" Jul 22 02:48:33.988497 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:48:33.988478 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/62dc2edb-a885-4562-8b80-597c1b43926c-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fztprq\" (UID: \"62dc2edb-a885-4562-8b80-597c1b43926c\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fztprq" Jul 22 02:48:33.995353 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:48:33.995328 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ccvsc\" (UniqueName: \"kubernetes.io/projected/62dc2edb-a885-4562-8b80-597c1b43926c-kube-api-access-ccvsc\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fztprq\" (UID: \"62dc2edb-a885-4562-8b80-597c1b43926c\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fztprq" Jul 22 02:48:34.083731 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:48:34.083703 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fztprq" Jul 22 02:48:34.226819 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:48:34.226786 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fztprq"] Jul 22 02:48:34.228521 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:48:34.228466 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62dc2edb_a885_4562_8b80_597c1b43926c.slice/crio-ee8f2ebcce1ec00de56c91de15a097a86336f0cc79a16b1c6ff239cd1c0a0409 WatchSource:0}: Error finding container ee8f2ebcce1ec00de56c91de15a097a86336f0cc79a16b1c6ff239cd1c0a0409: Status 404 returned error can't find the container with id ee8f2ebcce1ec00de56c91de15a097a86336f0cc79a16b1c6ff239cd1c0a0409 Jul 22 02:48:35.079577 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:48:35.079534 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fztprq" event={"ID":"62dc2edb-a885-4562-8b80-597c1b43926c","Type":"ContainerStarted","Data":"f995bfb4f66b5637ebe406105fb59b8dc4fe79fa15f72560c9547d0c3a4f59d7"} Jul 22 02:48:35.079967 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:48:35.079603 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fztprq" event={"ID":"62dc2edb-a885-4562-8b80-597c1b43926c","Type":"ContainerStarted","Data":"ee8f2ebcce1ec00de56c91de15a097a86336f0cc79a16b1c6ff239cd1c0a0409"} Jul 22 02:48:35.105644 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:48:35.105595 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fztprq" podStartSLOduration=1.9581014140000002 podStartE2EDuration="2.105580335s" totalImagesPullingTime="147.478921ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 02:48:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:48:35.103851011 +0000 UTC m=+681.423026310" watchObservedRunningTime="2026-07-22 02:48:35.105580335 +0000 UTC m=+681.424755663" Jul 22 02:48:44.083831 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:48:44.083789 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fztprq" Jul 22 02:48:44.084265 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:48:44.084155 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fztprq" Jul 22 02:48:44.096277 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:48:44.096249 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fztprq" Jul 22 02:48:44.142217 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:48:44.142187 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fztprq" Jul 22 02:48:53.436855 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:48:53.436821 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-bjwkm"] Jul 22 02:48:53.440358 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:48:53.440342 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-bjwkm" Jul 22 02:48:53.444096 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:48:53.444069 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" Jul 22 02:48:53.463332 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:48:53.463309 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-bjwkm"] Jul 22 02:48:53.568148 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:48:53.568117 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9f685069-4532-491e-a7f7-e11cf0816f72-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-bjwkm\" (UID: \"9f685069-4532-491e-a7f7-e11cf0816f72\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-bjwkm" Jul 22 02:48:53.568338 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:48:53.568161 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9f685069-4532-491e-a7f7-e11cf0816f72-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-bjwkm\" (UID: \"9f685069-4532-491e-a7f7-e11cf0816f72\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-bjwkm" Jul 22 02:48:53.568338 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:48:53.568200 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f685069-4532-491e-a7f7-e11cf0816f72-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-bjwkm\" (UID: \"9f685069-4532-491e-a7f7-e11cf0816f72\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-bjwkm" Jul 22 02:48:53.568338 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:48:53.568225 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9f685069-4532-491e-a7f7-e11cf0816f72-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-bjwkm\" (UID: \"9f685069-4532-491e-a7f7-e11cf0816f72\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-bjwkm" Jul 22 02:48:53.568338 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:48:53.568248 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9f685069-4532-491e-a7f7-e11cf0816f72-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-bjwkm\" (UID: \"9f685069-4532-491e-a7f7-e11cf0816f72\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-bjwkm" Jul 22 02:48:53.568338 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:48:53.568312 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4g9sg\" (UniqueName: \"kubernetes.io/projected/9f685069-4532-491e-a7f7-e11cf0816f72-kube-api-access-4g9sg\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-bjwkm\" (UID: \"9f685069-4532-491e-a7f7-e11cf0816f72\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-bjwkm" Jul 22 02:48:53.668914 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:48:53.668884 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9f685069-4532-491e-a7f7-e11cf0816f72-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-bjwkm\" (UID: \"9f685069-4532-491e-a7f7-e11cf0816f72\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-bjwkm" Jul 22 02:48:53.669090 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:48:53.668928 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9f685069-4532-491e-a7f7-e11cf0816f72-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-bjwkm\" (UID: \"9f685069-4532-491e-a7f7-e11cf0816f72\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-bjwkm" Jul 22 02:48:53.669090 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:48:53.668966 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f685069-4532-491e-a7f7-e11cf0816f72-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-bjwkm\" (UID: \"9f685069-4532-491e-a7f7-e11cf0816f72\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-bjwkm" Jul 22 02:48:53.669090 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:48:53.669012 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9f685069-4532-491e-a7f7-e11cf0816f72-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-bjwkm\" (UID: \"9f685069-4532-491e-a7f7-e11cf0816f72\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-bjwkm" Jul 22 02:48:53.669253 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:48:53.669129 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9f685069-4532-491e-a7f7-e11cf0816f72-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-bjwkm\" (UID: \"9f685069-4532-491e-a7f7-e11cf0816f72\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-bjwkm" Jul 22 02:48:53.669253 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:48:53.669210 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4g9sg\" (UniqueName: \"kubernetes.io/projected/9f685069-4532-491e-a7f7-e11cf0816f72-kube-api-access-4g9sg\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-bjwkm\" (UID: \"9f685069-4532-491e-a7f7-e11cf0816f72\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-bjwkm" Jul 22 02:48:53.669428 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:48:53.669407 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f685069-4532-491e-a7f7-e11cf0816f72-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-bjwkm\" (UID: \"9f685069-4532-491e-a7f7-e11cf0816f72\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-bjwkm" Jul 22 02:48:53.669514 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:48:53.669442 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9f685069-4532-491e-a7f7-e11cf0816f72-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-bjwkm\" (UID: \"9f685069-4532-491e-a7f7-e11cf0816f72\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-bjwkm" Jul 22 02:48:53.669880 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:48:53.669813 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9f685069-4532-491e-a7f7-e11cf0816f72-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-bjwkm\" (UID: \"9f685069-4532-491e-a7f7-e11cf0816f72\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-bjwkm" Jul 22 02:48:53.671397 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:48:53.671372 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9f685069-4532-491e-a7f7-e11cf0816f72-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-bjwkm\" (UID: \"9f685069-4532-491e-a7f7-e11cf0816f72\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-bjwkm" Jul 22 02:48:53.671645 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:48:53.671625 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9f685069-4532-491e-a7f7-e11cf0816f72-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-bjwkm\" (UID: \"9f685069-4532-491e-a7f7-e11cf0816f72\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-bjwkm" Jul 22 02:48:53.686000 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:48:53.685973 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4g9sg\" (UniqueName: \"kubernetes.io/projected/9f685069-4532-491e-a7f7-e11cf0816f72-kube-api-access-4g9sg\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-bjwkm\" (UID: \"9f685069-4532-491e-a7f7-e11cf0816f72\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-bjwkm" Jul 22 02:48:53.750843 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:48:53.750782 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-bjwkm" Jul 22 02:48:53.901235 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:48:53.901207 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9f685069_4532_491e_a7f7_e11cf0816f72.slice/crio-99163eac5b9b2d319208b99edd422c70a66da4d1b80f21a825d2c6579fdd9e2c WatchSource:0}: Error finding container 99163eac5b9b2d319208b99edd422c70a66da4d1b80f21a825d2c6579fdd9e2c: Status 404 returned error can't find the container with id 99163eac5b9b2d319208b99edd422c70a66da4d1b80f21a825d2c6579fdd9e2c Jul 22 02:48:53.901498 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:48:53.901479 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-bjwkm"] Jul 22 02:48:54.175912 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:48:54.175882 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-bjwkm" event={"ID":"9f685069-4532-491e-a7f7-e11cf0816f72","Type":"ContainerStarted","Data":"38420609ca4140a020485d7761ced5d176c720b396a34f14b08c01b7dd82f921"} Jul 22 02:48:54.176013 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:48:54.175920 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-bjwkm" event={"ID":"9f685069-4532-491e-a7f7-e11cf0816f72","Type":"ContainerStarted","Data":"99163eac5b9b2d319208b99edd422c70a66da4d1b80f21a825d2c6579fdd9e2c"} Jul 22 02:48:54.209017 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:48:54.208973 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-bjwkm" podStartSLOduration=1.030100015 podStartE2EDuration="1.208957354s" totalImagesPullingTime="178.857339ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 02:48:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:48:54.207456821 +0000 UTC m=+700.526632131" watchObservedRunningTime="2026-07-22 02:48:54.208957354 +0000 UTC m=+700.528132655" Jul 22 02:49:03.751826 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:49:03.751792 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-bjwkm" Jul 22 02:49:03.752205 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:49:03.752061 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-bjwkm" Jul 22 02:49:03.764441 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:49:03.764413 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-bjwkm" Jul 22 02:49:04.238553 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:49:04.238516 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-bjwkm" Jul 22 02:52:14.088842 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:52:14.088809 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-prfql_fd7bf923-8d16-476c-a8d0-a3e6fd645061/ovn-acl-logging/0.log" Jul 22 02:52:14.090920 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:52:14.090893 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-prfql_fd7bf923-8d16-476c-a8d0-a3e6fd645061/ovn-acl-logging/0.log" Jul 22 02:57:05.062897 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:57:05.062828 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-822mm"] Jul 22 02:57:05.063408 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:57:05.063001 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-822mm" podUID="9633d4bc-4a36-46c7-af40-d1e630c2b586" containerName="manager" containerID="cri-o://8a8384604ea9cccafe781b39c4a4712d15540cbcc270b7ef649a83c7b49557e5" gracePeriod=10 Jul 22 02:57:06.924884 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:57:06.924858 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-822mm" Jul 22 02:57:06.944951 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:57:06.944924 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9633d4bc-4a36-46c7-af40-d1e630c2b586-extensions-socket-volume\") pod \"9633d4bc-4a36-46c7-af40-d1e630c2b586\" (UID: \"9633d4bc-4a36-46c7-af40-d1e630c2b586\") " Jul 22 02:57:06.945074 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:57:06.945016 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ftnvn\" (UniqueName: \"kubernetes.io/projected/9633d4bc-4a36-46c7-af40-d1e630c2b586-kube-api-access-ftnvn\") pod \"9633d4bc-4a36-46c7-af40-d1e630c2b586\" (UID: \"9633d4bc-4a36-46c7-af40-d1e630c2b586\") " Jul 22 02:57:06.945212 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:57:06.945191 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9633d4bc-4a36-46c7-af40-d1e630c2b586-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "9633d4bc-4a36-46c7-af40-d1e630c2b586" (UID: "9633d4bc-4a36-46c7-af40-d1e630c2b586"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 02:57:06.946975 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:57:06.946940 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9633d4bc-4a36-46c7-af40-d1e630c2b586-kube-api-access-ftnvn" (OuterVolumeSpecName: "kube-api-access-ftnvn") pod "9633d4bc-4a36-46c7-af40-d1e630c2b586" (UID: "9633d4bc-4a36-46c7-af40-d1e630c2b586"). InnerVolumeSpecName "kube-api-access-ftnvn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 02:57:07.046274 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:57:07.046241 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ftnvn\" (UniqueName: \"kubernetes.io/projected/9633d4bc-4a36-46c7-af40-d1e630c2b586-kube-api-access-ftnvn\") on node \"ip-10-0-133-31.ec2.internal\" DevicePath \"\"" Jul 22 02:57:07.046274 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:57:07.046268 2578 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9633d4bc-4a36-46c7-af40-d1e630c2b586-extensions-socket-volume\") on node \"ip-10-0-133-31.ec2.internal\" DevicePath \"\"" Jul 22 02:57:07.716063 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:57:07.716026 2578 generic.go:356] "Generic (PLEG): container finished" podID="9633d4bc-4a36-46c7-af40-d1e630c2b586" containerID="8a8384604ea9cccafe781b39c4a4712d15540cbcc270b7ef649a83c7b49557e5" exitCode=0 Jul 22 02:57:07.716215 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:57:07.716071 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-822mm" Jul 22 02:57:07.716215 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:57:07.716110 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-822mm" event={"ID":"9633d4bc-4a36-46c7-af40-d1e630c2b586","Type":"ContainerDied","Data":"8a8384604ea9cccafe781b39c4a4712d15540cbcc270b7ef649a83c7b49557e5"} Jul 22 02:57:07.716215 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:57:07.716164 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-822mm" event={"ID":"9633d4bc-4a36-46c7-af40-d1e630c2b586","Type":"ContainerDied","Data":"7fb26138a1f2d4b9800827a4e18f2ba189d936327978537d713e960f005159da"} Jul 22 02:57:07.716215 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:57:07.716185 2578 scope.go:117] "RemoveContainer" containerID="8a8384604ea9cccafe781b39c4a4712d15540cbcc270b7ef649a83c7b49557e5" Jul 22 02:57:07.725892 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:57:07.725875 2578 scope.go:117] "RemoveContainer" containerID="8a8384604ea9cccafe781b39c4a4712d15540cbcc270b7ef649a83c7b49557e5" Jul 22 02:57:07.726138 ip-10-0-133-31 kubenswrapper[2578]: E0722 02:57:07.726115 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a8384604ea9cccafe781b39c4a4712d15540cbcc270b7ef649a83c7b49557e5\": container with ID starting with 8a8384604ea9cccafe781b39c4a4712d15540cbcc270b7ef649a83c7b49557e5 not found: ID does not exist" containerID="8a8384604ea9cccafe781b39c4a4712d15540cbcc270b7ef649a83c7b49557e5" Jul 22 02:57:07.726202 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:57:07.726151 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a8384604ea9cccafe781b39c4a4712d15540cbcc270b7ef649a83c7b49557e5"} err="failed to get container status \"8a8384604ea9cccafe781b39c4a4712d15540cbcc270b7ef649a83c7b49557e5\": rpc error: code = NotFound desc = could not find container \"8a8384604ea9cccafe781b39c4a4712d15540cbcc270b7ef649a83c7b49557e5\": container with ID starting with 8a8384604ea9cccafe781b39c4a4712d15540cbcc270b7ef649a83c7b49557e5 not found: ID does not exist" Jul 22 02:57:07.739384 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:57:07.739362 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-822mm"] Jul 22 02:57:07.745059 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:57:07.745036 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-822mm"] Jul 22 02:57:08.142624 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:57:08.142598 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9633d4bc-4a36-46c7-af40-d1e630c2b586" path="/var/lib/kubelet/pods/9633d4bc-4a36-46c7-af40-d1e630c2b586/volumes" Jul 22 02:57:14.125799 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:57:14.125774 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-prfql_fd7bf923-8d16-476c-a8d0-a3e6fd645061/ovn-acl-logging/0.log" Jul 22 02:57:14.126326 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:57:14.125775 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-prfql_fd7bf923-8d16-476c-a8d0-a3e6fd645061/ovn-acl-logging/0.log" Jul 22 02:58:06.209694 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:58:06.209611 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-7md5z"] Jul 22 02:58:06.212086 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:58:06.209967 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9633d4bc-4a36-46c7-af40-d1e630c2b586" containerName="manager" Jul 22 02:58:06.212086 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:58:06.209979 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="9633d4bc-4a36-46c7-af40-d1e630c2b586" containerName="manager" Jul 22 02:58:06.212086 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:58:06.210066 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="9633d4bc-4a36-46c7-af40-d1e630c2b586" containerName="manager" Jul 22 02:58:06.212964 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:58:06.212949 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-7md5z" Jul 22 02:58:06.216742 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:58:06.216711 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-4fvbz\"" Jul 22 02:58:06.220029 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:58:06.219995 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9jw5v\" (UniqueName: \"kubernetes.io/projected/e01f4154-c4a7-4994-9685-8111f41adff0-kube-api-access-9jw5v\") pod \"kuadrant-operator-controller-manager-86878c769f-7md5z\" (UID: \"e01f4154-c4a7-4994-9685-8111f41adff0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-7md5z" Jul 22 02:58:06.220109 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:58:06.220090 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e01f4154-c4a7-4994-9685-8111f41adff0-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-7md5z\" (UID: \"e01f4154-c4a7-4994-9685-8111f41adff0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-7md5z" Jul 22 02:58:06.234305 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:58:06.234279 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-7md5z"] Jul 22 02:58:06.321214 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:58:06.321184 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9jw5v\" (UniqueName: \"kubernetes.io/projected/e01f4154-c4a7-4994-9685-8111f41adff0-kube-api-access-9jw5v\") pod \"kuadrant-operator-controller-manager-86878c769f-7md5z\" (UID: \"e01f4154-c4a7-4994-9685-8111f41adff0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-7md5z" Jul 22 02:58:06.321343 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:58:06.321289 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e01f4154-c4a7-4994-9685-8111f41adff0-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-7md5z\" (UID: \"e01f4154-c4a7-4994-9685-8111f41adff0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-7md5z" Jul 22 02:58:06.321701 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:58:06.321680 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e01f4154-c4a7-4994-9685-8111f41adff0-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-7md5z\" (UID: \"e01f4154-c4a7-4994-9685-8111f41adff0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-7md5z" Jul 22 02:58:06.331331 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:58:06.331309 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9jw5v\" (UniqueName: \"kubernetes.io/projected/e01f4154-c4a7-4994-9685-8111f41adff0-kube-api-access-9jw5v\") pod \"kuadrant-operator-controller-manager-86878c769f-7md5z\" (UID: \"e01f4154-c4a7-4994-9685-8111f41adff0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-7md5z" Jul 22 02:58:06.522784 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:58:06.522731 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-7md5z" Jul 22 02:58:06.655590 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:58:06.655556 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-7md5z"] Jul 22 02:58:06.657344 ip-10-0-133-31 kubenswrapper[2578]: W0722 02:58:06.657311 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode01f4154_c4a7_4994_9685_8111f41adff0.slice/crio-77ef0224e2824f9341ca64bbb12b011725a2c4b9bd02774a21f9c01cb09d6854 WatchSource:0}: Error finding container 77ef0224e2824f9341ca64bbb12b011725a2c4b9bd02774a21f9c01cb09d6854: Status 404 returned error can't find the container with id 77ef0224e2824f9341ca64bbb12b011725a2c4b9bd02774a21f9c01cb09d6854 Jul 22 02:58:07.011150 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:58:07.011113 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-7md5z" event={"ID":"e01f4154-c4a7-4994-9685-8111f41adff0","Type":"ContainerStarted","Data":"7c3ff6014a99503d91cb25e178643312fb0f3a0a6eee3f6dbd9ccb5768f9d92f"} Jul 22 02:58:07.011333 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:58:07.011206 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-7md5z" event={"ID":"e01f4154-c4a7-4994-9685-8111f41adff0","Type":"ContainerStarted","Data":"77ef0224e2824f9341ca64bbb12b011725a2c4b9bd02774a21f9c01cb09d6854"} Jul 22 02:58:07.011333 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:58:07.011306 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-7md5z" Jul 22 02:58:07.051236 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:58:07.051184 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-7md5z" podStartSLOduration=1.051170038 podStartE2EDuration="1.051170038s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 02:58:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:58:07.050031361 +0000 UTC m=+1253.369206660" watchObservedRunningTime="2026-07-22 02:58:07.051170038 +0000 UTC m=+1253.370345340" Jul 22 02:58:18.018749 ip-10-0-133-31 kubenswrapper[2578]: I0722 02:58:18.018715 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-7md5z" Jul 22 03:00:41.264573 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:00:41.264538 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-499f8b96-kserve-6569f545f5-zmlpt"] Jul 22 03:00:41.268073 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:00:41.268057 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-499f8b96-kserve-6569f545f5-zmlpt" Jul 22 03:00:41.271124 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:00:41.271106 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-499f8b96-kserve-self-signed-certs\"" Jul 22 03:00:41.277799 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:00:41.277776 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-499f8b96-kserve-6569f545f5-zmlpt"] Jul 22 03:00:41.398648 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:00:41.398621 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gtzbb\" (UniqueName: \"kubernetes.io/projected/6b2589a1-aec4-433c-89a5-5513923e1e95-kube-api-access-gtzbb\") pod \"e2e-same-id-a-499f8b96-kserve-6569f545f5-zmlpt\" (UID: \"6b2589a1-aec4-433c-89a5-5513923e1e95\") " pod="llm/e2e-same-id-a-499f8b96-kserve-6569f545f5-zmlpt" Jul 22 03:00:41.398787 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:00:41.398682 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6b2589a1-aec4-433c-89a5-5513923e1e95-home\") pod \"e2e-same-id-a-499f8b96-kserve-6569f545f5-zmlpt\" (UID: \"6b2589a1-aec4-433c-89a5-5513923e1e95\") " pod="llm/e2e-same-id-a-499f8b96-kserve-6569f545f5-zmlpt" Jul 22 03:00:41.398787 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:00:41.398725 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6b2589a1-aec4-433c-89a5-5513923e1e95-tls-certs\") pod \"e2e-same-id-a-499f8b96-kserve-6569f545f5-zmlpt\" (UID: \"6b2589a1-aec4-433c-89a5-5513923e1e95\") " pod="llm/e2e-same-id-a-499f8b96-kserve-6569f545f5-zmlpt" Jul 22 03:00:41.398787 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:00:41.398749 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6b2589a1-aec4-433c-89a5-5513923e1e95-tmp-dir\") pod \"e2e-same-id-a-499f8b96-kserve-6569f545f5-zmlpt\" (UID: \"6b2589a1-aec4-433c-89a5-5513923e1e95\") " pod="llm/e2e-same-id-a-499f8b96-kserve-6569f545f5-zmlpt" Jul 22 03:00:41.398787 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:00:41.398767 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6b2589a1-aec4-433c-89a5-5513923e1e95-dshm\") pod \"e2e-same-id-a-499f8b96-kserve-6569f545f5-zmlpt\" (UID: \"6b2589a1-aec4-433c-89a5-5513923e1e95\") " pod="llm/e2e-same-id-a-499f8b96-kserve-6569f545f5-zmlpt" Jul 22 03:00:41.398787 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:00:41.398787 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6b2589a1-aec4-433c-89a5-5513923e1e95-model-cache\") pod \"e2e-same-id-a-499f8b96-kserve-6569f545f5-zmlpt\" (UID: \"6b2589a1-aec4-433c-89a5-5513923e1e95\") " pod="llm/e2e-same-id-a-499f8b96-kserve-6569f545f5-zmlpt" Jul 22 03:00:41.500213 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:00:41.500181 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6b2589a1-aec4-433c-89a5-5513923e1e95-tmp-dir\") pod \"e2e-same-id-a-499f8b96-kserve-6569f545f5-zmlpt\" (UID: \"6b2589a1-aec4-433c-89a5-5513923e1e95\") " pod="llm/e2e-same-id-a-499f8b96-kserve-6569f545f5-zmlpt" Jul 22 03:00:41.500405 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:00:41.500232 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6b2589a1-aec4-433c-89a5-5513923e1e95-dshm\") pod \"e2e-same-id-a-499f8b96-kserve-6569f545f5-zmlpt\" (UID: \"6b2589a1-aec4-433c-89a5-5513923e1e95\") " pod="llm/e2e-same-id-a-499f8b96-kserve-6569f545f5-zmlpt" Jul 22 03:00:41.500405 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:00:41.500263 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6b2589a1-aec4-433c-89a5-5513923e1e95-model-cache\") pod \"e2e-same-id-a-499f8b96-kserve-6569f545f5-zmlpt\" (UID: \"6b2589a1-aec4-433c-89a5-5513923e1e95\") " pod="llm/e2e-same-id-a-499f8b96-kserve-6569f545f5-zmlpt" Jul 22 03:00:41.500405 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:00:41.500326 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gtzbb\" (UniqueName: \"kubernetes.io/projected/6b2589a1-aec4-433c-89a5-5513923e1e95-kube-api-access-gtzbb\") pod \"e2e-same-id-a-499f8b96-kserve-6569f545f5-zmlpt\" (UID: \"6b2589a1-aec4-433c-89a5-5513923e1e95\") " pod="llm/e2e-same-id-a-499f8b96-kserve-6569f545f5-zmlpt" Jul 22 03:00:41.500405 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:00:41.500399 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6b2589a1-aec4-433c-89a5-5513923e1e95-home\") pod \"e2e-same-id-a-499f8b96-kserve-6569f545f5-zmlpt\" (UID: \"6b2589a1-aec4-433c-89a5-5513923e1e95\") " pod="llm/e2e-same-id-a-499f8b96-kserve-6569f545f5-zmlpt" Jul 22 03:00:41.500672 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:00:41.500437 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6b2589a1-aec4-433c-89a5-5513923e1e95-tls-certs\") pod \"e2e-same-id-a-499f8b96-kserve-6569f545f5-zmlpt\" (UID: \"6b2589a1-aec4-433c-89a5-5513923e1e95\") " pod="llm/e2e-same-id-a-499f8b96-kserve-6569f545f5-zmlpt" Jul 22 03:00:41.500735 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:00:41.500692 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6b2589a1-aec4-433c-89a5-5513923e1e95-tmp-dir\") pod \"e2e-same-id-a-499f8b96-kserve-6569f545f5-zmlpt\" (UID: \"6b2589a1-aec4-433c-89a5-5513923e1e95\") " pod="llm/e2e-same-id-a-499f8b96-kserve-6569f545f5-zmlpt" Jul 22 03:00:41.500795 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:00:41.500760 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6b2589a1-aec4-433c-89a5-5513923e1e95-model-cache\") pod \"e2e-same-id-a-499f8b96-kserve-6569f545f5-zmlpt\" (UID: \"6b2589a1-aec4-433c-89a5-5513923e1e95\") " pod="llm/e2e-same-id-a-499f8b96-kserve-6569f545f5-zmlpt" Jul 22 03:00:41.500850 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:00:41.500772 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6b2589a1-aec4-433c-89a5-5513923e1e95-home\") pod \"e2e-same-id-a-499f8b96-kserve-6569f545f5-zmlpt\" (UID: \"6b2589a1-aec4-433c-89a5-5513923e1e95\") " pod="llm/e2e-same-id-a-499f8b96-kserve-6569f545f5-zmlpt" Jul 22 03:00:41.502643 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:00:41.502624 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6b2589a1-aec4-433c-89a5-5513923e1e95-dshm\") pod \"e2e-same-id-a-499f8b96-kserve-6569f545f5-zmlpt\" (UID: \"6b2589a1-aec4-433c-89a5-5513923e1e95\") " pod="llm/e2e-same-id-a-499f8b96-kserve-6569f545f5-zmlpt" Jul 22 03:00:41.502967 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:00:41.502951 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6b2589a1-aec4-433c-89a5-5513923e1e95-tls-certs\") pod \"e2e-same-id-a-499f8b96-kserve-6569f545f5-zmlpt\" (UID: \"6b2589a1-aec4-433c-89a5-5513923e1e95\") " pod="llm/e2e-same-id-a-499f8b96-kserve-6569f545f5-zmlpt" Jul 22 03:00:41.511039 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:00:41.511020 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gtzbb\" (UniqueName: \"kubernetes.io/projected/6b2589a1-aec4-433c-89a5-5513923e1e95-kube-api-access-gtzbb\") pod \"e2e-same-id-a-499f8b96-kserve-6569f545f5-zmlpt\" (UID: \"6b2589a1-aec4-433c-89a5-5513923e1e95\") " pod="llm/e2e-same-id-a-499f8b96-kserve-6569f545f5-zmlpt" Jul 22 03:00:41.578281 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:00:41.578232 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-499f8b96-kserve-6569f545f5-zmlpt" Jul 22 03:00:41.714799 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:00:41.714776 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-499f8b96-kserve-6569f545f5-zmlpt"] Jul 22 03:00:41.716537 ip-10-0-133-31 kubenswrapper[2578]: W0722 03:00:41.716515 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6b2589a1_aec4_433c_89a5_5513923e1e95.slice/crio-fca5c332324bba3b5ccb9238363e5a31c9c3f4e935e66ebbb76cc12071d0ca79 WatchSource:0}: Error finding container fca5c332324bba3b5ccb9238363e5a31c9c3f4e935e66ebbb76cc12071d0ca79: Status 404 returned error can't find the container with id fca5c332324bba3b5ccb9238363e5a31c9c3f4e935e66ebbb76cc12071d0ca79 Jul 22 03:00:41.792549 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:00:41.792510 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-499f8b96-kserve-6569f545f5-zmlpt" event={"ID":"6b2589a1-aec4-433c-89a5-5513923e1e95","Type":"ContainerStarted","Data":"ea9c0e9de4a109ac73d574eddf00eb5375593e316519be6f6a4218f28f6a6338"} Jul 22 03:00:41.792707 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:00:41.792591 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-499f8b96-kserve-6569f545f5-zmlpt" event={"ID":"6b2589a1-aec4-433c-89a5-5513923e1e95","Type":"ContainerStarted","Data":"fca5c332324bba3b5ccb9238363e5a31c9c3f4e935e66ebbb76cc12071d0ca79"} Jul 22 03:00:41.814142 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:00:41.814094 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-a-499f8b96-kserve-6569f545f5-zmlpt" podStartSLOduration=0.814079207 podStartE2EDuration="814.079207ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 03:00:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 03:00:41.811509943 +0000 UTC m=+1408.130685243" watchObservedRunningTime="2026-07-22 03:00:41.814079207 +0000 UTC m=+1408.133254531" Jul 22 03:00:51.578685 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:00:51.578644 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-499f8b96-kserve-6569f545f5-zmlpt" Jul 22 03:00:51.579291 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:00:51.578705 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-499f8b96-kserve-6569f545f5-zmlpt" Jul 22 03:00:51.590901 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:00:51.590877 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-499f8b96-kserve-6569f545f5-zmlpt" Jul 22 03:00:51.854393 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:00:51.854366 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-499f8b96-kserve-6569f545f5-zmlpt" Jul 22 03:01:38.127796 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:01:38.127761 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-499f8b96-kserve-6569f545f5-zmlpt"] Jul 22 03:01:53.148082 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:01:53.148041 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-a-499f8b96-kserve-6569f545f5-zmlpt" podUID="6b2589a1-aec4-433c-89a5-5513923e1e95" containerName="main" containerID="cri-o://ea9c0e9de4a109ac73d574eddf00eb5375593e316519be6f6a4218f28f6a6338" gracePeriod=45 Jul 22 03:01:53.406685 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:01:53.406622 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-499f8b96-kserve-6569f545f5-zmlpt" Jul 22 03:01:53.506156 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:01:53.506126 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6b2589a1-aec4-433c-89a5-5513923e1e95-home\") pod \"6b2589a1-aec4-433c-89a5-5513923e1e95\" (UID: \"6b2589a1-aec4-433c-89a5-5513923e1e95\") " Jul 22 03:01:53.506341 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:01:53.506166 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gtzbb\" (UniqueName: \"kubernetes.io/projected/6b2589a1-aec4-433c-89a5-5513923e1e95-kube-api-access-gtzbb\") pod \"6b2589a1-aec4-433c-89a5-5513923e1e95\" (UID: \"6b2589a1-aec4-433c-89a5-5513923e1e95\") " Jul 22 03:01:53.506341 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:01:53.506211 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6b2589a1-aec4-433c-89a5-5513923e1e95-model-cache\") pod \"6b2589a1-aec4-433c-89a5-5513923e1e95\" (UID: \"6b2589a1-aec4-433c-89a5-5513923e1e95\") " Jul 22 03:01:53.506341 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:01:53.506249 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6b2589a1-aec4-433c-89a5-5513923e1e95-dshm\") pod \"6b2589a1-aec4-433c-89a5-5513923e1e95\" (UID: \"6b2589a1-aec4-433c-89a5-5513923e1e95\") " Jul 22 03:01:53.506341 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:01:53.506281 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6b2589a1-aec4-433c-89a5-5513923e1e95-tmp-dir\") pod \"6b2589a1-aec4-433c-89a5-5513923e1e95\" (UID: \"6b2589a1-aec4-433c-89a5-5513923e1e95\") " Jul 22 03:01:53.506341 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:01:53.506314 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6b2589a1-aec4-433c-89a5-5513923e1e95-tls-certs\") pod \"6b2589a1-aec4-433c-89a5-5513923e1e95\" (UID: \"6b2589a1-aec4-433c-89a5-5513923e1e95\") " Jul 22 03:01:53.506614 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:01:53.506387 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b2589a1-aec4-433c-89a5-5513923e1e95-home" (OuterVolumeSpecName: "home") pod "6b2589a1-aec4-433c-89a5-5513923e1e95" (UID: "6b2589a1-aec4-433c-89a5-5513923e1e95"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 03:01:53.506614 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:01:53.506502 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b2589a1-aec4-433c-89a5-5513923e1e95-model-cache" (OuterVolumeSpecName: "model-cache") pod "6b2589a1-aec4-433c-89a5-5513923e1e95" (UID: "6b2589a1-aec4-433c-89a5-5513923e1e95"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 03:01:53.506614 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:01:53.506520 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b2589a1-aec4-433c-89a5-5513923e1e95-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6b2589a1-aec4-433c-89a5-5513923e1e95" (UID: "6b2589a1-aec4-433c-89a5-5513923e1e95"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 03:01:53.506763 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:01:53.506725 2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6b2589a1-aec4-433c-89a5-5513923e1e95-model-cache\") on node \"ip-10-0-133-31.ec2.internal\" DevicePath \"\"" Jul 22 03:01:53.506763 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:01:53.506741 2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6b2589a1-aec4-433c-89a5-5513923e1e95-tmp-dir\") on node \"ip-10-0-133-31.ec2.internal\" DevicePath \"\"" Jul 22 03:01:53.506763 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:01:53.506748 2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6b2589a1-aec4-433c-89a5-5513923e1e95-home\") on node \"ip-10-0-133-31.ec2.internal\" DevicePath \"\"" Jul 22 03:01:53.508519 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:01:53.508496 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6b2589a1-aec4-433c-89a5-5513923e1e95-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6b2589a1-aec4-433c-89a5-5513923e1e95" (UID: "6b2589a1-aec4-433c-89a5-5513923e1e95"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 03:01:53.508686 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:01:53.508670 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b2589a1-aec4-433c-89a5-5513923e1e95-dshm" (OuterVolumeSpecName: "dshm") pod "6b2589a1-aec4-433c-89a5-5513923e1e95" (UID: "6b2589a1-aec4-433c-89a5-5513923e1e95"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 03:01:53.508759 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:01:53.508722 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b2589a1-aec4-433c-89a5-5513923e1e95-kube-api-access-gtzbb" (OuterVolumeSpecName: "kube-api-access-gtzbb") pod "6b2589a1-aec4-433c-89a5-5513923e1e95" (UID: "6b2589a1-aec4-433c-89a5-5513923e1e95"). InnerVolumeSpecName "kube-api-access-gtzbb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 03:01:53.607282 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:01:53.607256 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gtzbb\" (UniqueName: \"kubernetes.io/projected/6b2589a1-aec4-433c-89a5-5513923e1e95-kube-api-access-gtzbb\") on node \"ip-10-0-133-31.ec2.internal\" DevicePath \"\"" Jul 22 03:01:53.607282 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:01:53.607280 2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6b2589a1-aec4-433c-89a5-5513923e1e95-dshm\") on node \"ip-10-0-133-31.ec2.internal\" DevicePath \"\"" Jul 22 03:01:53.607424 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:01:53.607290 2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6b2589a1-aec4-433c-89a5-5513923e1e95-tls-certs\") on node \"ip-10-0-133-31.ec2.internal\" DevicePath \"\"" Jul 22 03:01:54.155860 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:01:54.155835 2578 generic.go:356] "Generic (PLEG): container finished" podID="6b2589a1-aec4-433c-89a5-5513923e1e95" containerID="ea9c0e9de4a109ac73d574eddf00eb5375593e316519be6f6a4218f28f6a6338" exitCode=0 Jul 22 03:01:54.156253 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:01:54.155876 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-499f8b96-kserve-6569f545f5-zmlpt" Jul 22 03:01:54.156253 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:01:54.155887 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-499f8b96-kserve-6569f545f5-zmlpt" event={"ID":"6b2589a1-aec4-433c-89a5-5513923e1e95","Type":"ContainerDied","Data":"ea9c0e9de4a109ac73d574eddf00eb5375593e316519be6f6a4218f28f6a6338"} Jul 22 03:01:54.156253 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:01:54.155908 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-499f8b96-kserve-6569f545f5-zmlpt" event={"ID":"6b2589a1-aec4-433c-89a5-5513923e1e95","Type":"ContainerDied","Data":"fca5c332324bba3b5ccb9238363e5a31c9c3f4e935e66ebbb76cc12071d0ca79"} Jul 22 03:01:54.156253 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:01:54.155924 2578 scope.go:117] "RemoveContainer" containerID="ea9c0e9de4a109ac73d574eddf00eb5375593e316519be6f6a4218f28f6a6338" Jul 22 03:01:54.164499 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:01:54.164481 2578 scope.go:117] "RemoveContainer" containerID="ea9c0e9de4a109ac73d574eddf00eb5375593e316519be6f6a4218f28f6a6338" Jul 22 03:01:54.164760 ip-10-0-133-31 kubenswrapper[2578]: E0722 03:01:54.164741 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea9c0e9de4a109ac73d574eddf00eb5375593e316519be6f6a4218f28f6a6338\": container with ID starting with ea9c0e9de4a109ac73d574eddf00eb5375593e316519be6f6a4218f28f6a6338 not found: ID does not exist" containerID="ea9c0e9de4a109ac73d574eddf00eb5375593e316519be6f6a4218f28f6a6338" Jul 22 03:01:54.164819 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:01:54.164767 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea9c0e9de4a109ac73d574eddf00eb5375593e316519be6f6a4218f28f6a6338"} err="failed to get container status \"ea9c0e9de4a109ac73d574eddf00eb5375593e316519be6f6a4218f28f6a6338\": rpc error: code = NotFound desc = could not find container \"ea9c0e9de4a109ac73d574eddf00eb5375593e316519be6f6a4218f28f6a6338\": container with ID starting with ea9c0e9de4a109ac73d574eddf00eb5375593e316519be6f6a4218f28f6a6338 not found: ID does not exist" Jul 22 03:01:54.176960 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:01:54.176929 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-499f8b96-kserve-6569f545f5-zmlpt"] Jul 22 03:01:54.181517 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:01:54.181496 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-499f8b96-kserve-6569f545f5-zmlpt"] Jul 22 03:01:56.142326 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:01:56.142293 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6b2589a1-aec4-433c-89a5-5513923e1e95" path="/var/lib/kubelet/pods/6b2589a1-aec4-433c-89a5-5513923e1e95/volumes" Jul 22 03:02:14.162175 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:02:14.162149 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-prfql_fd7bf923-8d16-476c-a8d0-a3e6fd645061/ovn-acl-logging/0.log" Jul 22 03:02:14.164256 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:02:14.164241 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-prfql_fd7bf923-8d16-476c-a8d0-a3e6fd645061/ovn-acl-logging/0.log" Jul 22 03:05:15.707674 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:05:15.707640 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-c22b602d-openshift-default-9c57744b9-vpxcj"] Jul 22 03:05:15.709678 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:05:15.708026 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6b2589a1-aec4-433c-89a5-5513923e1e95" containerName="main" Jul 22 03:05:15.709678 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:05:15.708041 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="6b2589a1-aec4-433c-89a5-5513923e1e95" containerName="main" Jul 22 03:05:15.709678 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:05:15.708095 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="6b2589a1-aec4-433c-89a5-5513923e1e95" containerName="main" Jul 22 03:05:15.711089 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:05:15.711074 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-c22b602d-openshift-default-9c57744b9-vpxcj" Jul 22 03:05:15.713920 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:05:15.713898 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-c22b602d-openshift-default-dockercfg-hqcp9\"" Jul 22 03:05:15.739805 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:05:15.739782 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-c22b602d-openshift-default-9c57744b9-vpxcj"] Jul 22 03:05:15.759441 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:05:15.759415 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/783af08b-92ba-423e-b5d4-25c6ad01d2ad-istio-token\") pod \"e2e-ait-c22b602d-openshift-default-9c57744b9-vpxcj\" (UID: \"783af08b-92ba-423e-b5d4-25c6ad01d2ad\") " pod="openshift-ingress/e2e-ait-c22b602d-openshift-default-9c57744b9-vpxcj" Jul 22 03:05:15.759546 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:05:15.759465 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/783af08b-92ba-423e-b5d4-25c6ad01d2ad-workload-socket\") pod \"e2e-ait-c22b602d-openshift-default-9c57744b9-vpxcj\" (UID: \"783af08b-92ba-423e-b5d4-25c6ad01d2ad\") " pod="openshift-ingress/e2e-ait-c22b602d-openshift-default-9c57744b9-vpxcj" Jul 22 03:05:15.759546 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:05:15.759482 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wrcrq\" (UniqueName: \"kubernetes.io/projected/783af08b-92ba-423e-b5d4-25c6ad01d2ad-kube-api-access-wrcrq\") pod \"e2e-ait-c22b602d-openshift-default-9c57744b9-vpxcj\" (UID: \"783af08b-92ba-423e-b5d4-25c6ad01d2ad\") " pod="openshift-ingress/e2e-ait-c22b602d-openshift-default-9c57744b9-vpxcj" Jul 22 03:05:15.759638 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:05:15.759589 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/783af08b-92ba-423e-b5d4-25c6ad01d2ad-workload-certs\") pod \"e2e-ait-c22b602d-openshift-default-9c57744b9-vpxcj\" (UID: \"783af08b-92ba-423e-b5d4-25c6ad01d2ad\") " pod="openshift-ingress/e2e-ait-c22b602d-openshift-default-9c57744b9-vpxcj" Jul 22 03:05:15.759679 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:05:15.759662 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/783af08b-92ba-423e-b5d4-25c6ad01d2ad-istio-envoy\") pod \"e2e-ait-c22b602d-openshift-default-9c57744b9-vpxcj\" (UID: \"783af08b-92ba-423e-b5d4-25c6ad01d2ad\") " pod="openshift-ingress/e2e-ait-c22b602d-openshift-default-9c57744b9-vpxcj" Jul 22 03:05:15.759720 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:05:15.759700 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/783af08b-92ba-423e-b5d4-25c6ad01d2ad-istiod-ca-cert\") pod \"e2e-ait-c22b602d-openshift-default-9c57744b9-vpxcj\" (UID: \"783af08b-92ba-423e-b5d4-25c6ad01d2ad\") " pod="openshift-ingress/e2e-ait-c22b602d-openshift-default-9c57744b9-vpxcj" Jul 22 03:05:15.759759 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:05:15.759729 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/783af08b-92ba-423e-b5d4-25c6ad01d2ad-istio-data\") pod \"e2e-ait-c22b602d-openshift-default-9c57744b9-vpxcj\" (UID: \"783af08b-92ba-423e-b5d4-25c6ad01d2ad\") " pod="openshift-ingress/e2e-ait-c22b602d-openshift-default-9c57744b9-vpxcj" Jul 22 03:05:15.759797 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:05:15.759763 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/783af08b-92ba-423e-b5d4-25c6ad01d2ad-credential-socket\") pod \"e2e-ait-c22b602d-openshift-default-9c57744b9-vpxcj\" (UID: \"783af08b-92ba-423e-b5d4-25c6ad01d2ad\") " pod="openshift-ingress/e2e-ait-c22b602d-openshift-default-9c57744b9-vpxcj" Jul 22 03:05:15.759797 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:05:15.759791 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/783af08b-92ba-423e-b5d4-25c6ad01d2ad-istio-podinfo\") pod \"e2e-ait-c22b602d-openshift-default-9c57744b9-vpxcj\" (UID: \"783af08b-92ba-423e-b5d4-25c6ad01d2ad\") " pod="openshift-ingress/e2e-ait-c22b602d-openshift-default-9c57744b9-vpxcj" Jul 22 03:05:15.860757 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:05:15.860725 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/783af08b-92ba-423e-b5d4-25c6ad01d2ad-credential-socket\") pod \"e2e-ait-c22b602d-openshift-default-9c57744b9-vpxcj\" (UID: \"783af08b-92ba-423e-b5d4-25c6ad01d2ad\") " pod="openshift-ingress/e2e-ait-c22b602d-openshift-default-9c57744b9-vpxcj" Jul 22 03:05:15.860757 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:05:15.860760 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/783af08b-92ba-423e-b5d4-25c6ad01d2ad-istio-podinfo\") pod \"e2e-ait-c22b602d-openshift-default-9c57744b9-vpxcj\" (UID: \"783af08b-92ba-423e-b5d4-25c6ad01d2ad\") " pod="openshift-ingress/e2e-ait-c22b602d-openshift-default-9c57744b9-vpxcj" Jul 22 03:05:15.860966 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:05:15.860792 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/783af08b-92ba-423e-b5d4-25c6ad01d2ad-istio-token\") pod \"e2e-ait-c22b602d-openshift-default-9c57744b9-vpxcj\" (UID: \"783af08b-92ba-423e-b5d4-25c6ad01d2ad\") " pod="openshift-ingress/e2e-ait-c22b602d-openshift-default-9c57744b9-vpxcj" Jul 22 03:05:15.860966 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:05:15.860914 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/783af08b-92ba-423e-b5d4-25c6ad01d2ad-workload-socket\") pod \"e2e-ait-c22b602d-openshift-default-9c57744b9-vpxcj\" (UID: \"783af08b-92ba-423e-b5d4-25c6ad01d2ad\") " pod="openshift-ingress/e2e-ait-c22b602d-openshift-default-9c57744b9-vpxcj" Jul 22 03:05:15.860966 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:05:15.860942 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wrcrq\" (UniqueName: \"kubernetes.io/projected/783af08b-92ba-423e-b5d4-25c6ad01d2ad-kube-api-access-wrcrq\") pod \"e2e-ait-c22b602d-openshift-default-9c57744b9-vpxcj\" (UID: \"783af08b-92ba-423e-b5d4-25c6ad01d2ad\") " pod="openshift-ingress/e2e-ait-c22b602d-openshift-default-9c57744b9-vpxcj" Jul 22 03:05:15.861143 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:05:15.861101 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/783af08b-92ba-423e-b5d4-25c6ad01d2ad-workload-certs\") pod \"e2e-ait-c22b602d-openshift-default-9c57744b9-vpxcj\" (UID: \"783af08b-92ba-423e-b5d4-25c6ad01d2ad\") " pod="openshift-ingress/e2e-ait-c22b602d-openshift-default-9c57744b9-vpxcj" Jul 22 03:05:15.861207 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:05:15.861180 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/783af08b-92ba-423e-b5d4-25c6ad01d2ad-credential-socket\") pod \"e2e-ait-c22b602d-openshift-default-9c57744b9-vpxcj\" (UID: \"783af08b-92ba-423e-b5d4-25c6ad01d2ad\") " pod="openshift-ingress/e2e-ait-c22b602d-openshift-default-9c57744b9-vpxcj" Jul 22 03:05:15.861268 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:05:15.861207 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/783af08b-92ba-423e-b5d4-25c6ad01d2ad-istio-envoy\") pod \"e2e-ait-c22b602d-openshift-default-9c57744b9-vpxcj\" (UID: \"783af08b-92ba-423e-b5d4-25c6ad01d2ad\") " pod="openshift-ingress/e2e-ait-c22b602d-openshift-default-9c57744b9-vpxcj" Jul 22 03:05:15.861268 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:05:15.861244 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/783af08b-92ba-423e-b5d4-25c6ad01d2ad-istiod-ca-cert\") pod \"e2e-ait-c22b602d-openshift-default-9c57744b9-vpxcj\" (UID: \"783af08b-92ba-423e-b5d4-25c6ad01d2ad\") " pod="openshift-ingress/e2e-ait-c22b602d-openshift-default-9c57744b9-vpxcj" Jul 22 03:05:15.861364 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:05:15.861270 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/783af08b-92ba-423e-b5d4-25c6ad01d2ad-istio-data\") pod \"e2e-ait-c22b602d-openshift-default-9c57744b9-vpxcj\" (UID: \"783af08b-92ba-423e-b5d4-25c6ad01d2ad\") " pod="openshift-ingress/e2e-ait-c22b602d-openshift-default-9c57744b9-vpxcj" Jul 22 03:05:15.861364 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:05:15.861276 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/783af08b-92ba-423e-b5d4-25c6ad01d2ad-workload-socket\") pod \"e2e-ait-c22b602d-openshift-default-9c57744b9-vpxcj\" (UID: \"783af08b-92ba-423e-b5d4-25c6ad01d2ad\") " pod="openshift-ingress/e2e-ait-c22b602d-openshift-default-9c57744b9-vpxcj" Jul 22 03:05:15.861514 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:05:15.861493 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/783af08b-92ba-423e-b5d4-25c6ad01d2ad-workload-certs\") pod \"e2e-ait-c22b602d-openshift-default-9c57744b9-vpxcj\" (UID: \"783af08b-92ba-423e-b5d4-25c6ad01d2ad\") " pod="openshift-ingress/e2e-ait-c22b602d-openshift-default-9c57744b9-vpxcj" Jul 22 03:05:15.861635 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:05:15.861612 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/783af08b-92ba-423e-b5d4-25c6ad01d2ad-istio-data\") pod \"e2e-ait-c22b602d-openshift-default-9c57744b9-vpxcj\" (UID: \"783af08b-92ba-423e-b5d4-25c6ad01d2ad\") " pod="openshift-ingress/e2e-ait-c22b602d-openshift-default-9c57744b9-vpxcj" Jul 22 03:05:15.861871 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:05:15.861855 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/783af08b-92ba-423e-b5d4-25c6ad01d2ad-istiod-ca-cert\") pod \"e2e-ait-c22b602d-openshift-default-9c57744b9-vpxcj\" (UID: \"783af08b-92ba-423e-b5d4-25c6ad01d2ad\") " pod="openshift-ingress/e2e-ait-c22b602d-openshift-default-9c57744b9-vpxcj" Jul 22 03:05:15.863331 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:05:15.863310 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/783af08b-92ba-423e-b5d4-25c6ad01d2ad-istio-envoy\") pod \"e2e-ait-c22b602d-openshift-default-9c57744b9-vpxcj\" (UID: \"783af08b-92ba-423e-b5d4-25c6ad01d2ad\") " pod="openshift-ingress/e2e-ait-c22b602d-openshift-default-9c57744b9-vpxcj" Jul 22 03:05:15.863541 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:05:15.863520 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/783af08b-92ba-423e-b5d4-25c6ad01d2ad-istio-podinfo\") pod \"e2e-ait-c22b602d-openshift-default-9c57744b9-vpxcj\" (UID: \"783af08b-92ba-423e-b5d4-25c6ad01d2ad\") " pod="openshift-ingress/e2e-ait-c22b602d-openshift-default-9c57744b9-vpxcj" Jul 22 03:05:15.878756 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:05:15.878733 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/783af08b-92ba-423e-b5d4-25c6ad01d2ad-istio-token\") pod \"e2e-ait-c22b602d-openshift-default-9c57744b9-vpxcj\" (UID: \"783af08b-92ba-423e-b5d4-25c6ad01d2ad\") " pod="openshift-ingress/e2e-ait-c22b602d-openshift-default-9c57744b9-vpxcj" Jul 22 03:05:15.879208 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:05:15.879193 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wrcrq\" (UniqueName: \"kubernetes.io/projected/783af08b-92ba-423e-b5d4-25c6ad01d2ad-kube-api-access-wrcrq\") pod \"e2e-ait-c22b602d-openshift-default-9c57744b9-vpxcj\" (UID: \"783af08b-92ba-423e-b5d4-25c6ad01d2ad\") " pod="openshift-ingress/e2e-ait-c22b602d-openshift-default-9c57744b9-vpxcj" Jul 22 03:05:16.021508 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:05:16.021426 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-c22b602d-openshift-default-9c57744b9-vpxcj" Jul 22 03:05:16.165748 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:05:16.165726 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-c22b602d-openshift-default-9c57744b9-vpxcj"] Jul 22 03:05:16.167610 ip-10-0-133-31 kubenswrapper[2578]: W0722 03:05:16.167583 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod783af08b_92ba_423e_b5d4_25c6ad01d2ad.slice/crio-6556dfc847c9dbe6248c9ea36990bee68224d881f07fb87dea9b093c191fa310 WatchSource:0}: Error finding container 6556dfc847c9dbe6248c9ea36990bee68224d881f07fb87dea9b093c191fa310: Status 404 returned error can't find the container with id 6556dfc847c9dbe6248c9ea36990bee68224d881f07fb87dea9b093c191fa310 Jul 22 03:05:16.169938 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:05:16.169908 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 22 03:05:16.170017 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:05:16.169975 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 22 03:05:16.170017 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:05:16.170008 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 22 03:05:16.189277 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:05:16.189202 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-c22b602d-openshift-default-9c57744b9-vpxcj" event={"ID":"783af08b-92ba-423e-b5d4-25c6ad01d2ad","Type":"ContainerStarted","Data":"6556dfc847c9dbe6248c9ea36990bee68224d881f07fb87dea9b093c191fa310"} Jul 22 03:05:17.196502 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:05:17.196462 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-c22b602d-openshift-default-9c57744b9-vpxcj" event={"ID":"783af08b-92ba-423e-b5d4-25c6ad01d2ad","Type":"ContainerStarted","Data":"ae9f7aebc3949eb21e3cc4c18135239d9a73076cc3eee2d2ff6392eed157d985"} Jul 22 03:05:17.227477 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:05:17.227432 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-c22b602d-openshift-default-9c57744b9-vpxcj" podStartSLOduration=2.227418278 podStartE2EDuration="2.227418278s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 03:05:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 03:05:17.225543574 +0000 UTC m=+1683.544718874" watchObservedRunningTime="2026-07-22 03:05:17.227418278 +0000 UTC m=+1683.546593578" Jul 22 03:05:18.022263 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:05:18.022231 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-c22b602d-openshift-default-9c57744b9-vpxcj" Jul 22 03:05:18.026857 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:05:18.026837 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-c22b602d-openshift-default-9c57744b9-vpxcj" Jul 22 03:05:18.207895 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:05:18.202450 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-c22b602d-openshift-default-9c57744b9-vpxcj" Jul 22 03:05:18.208667 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:05:18.208647 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-c22b602d-openshift-default-9c57744b9-vpxcj" Jul 22 03:05:31.268508 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:05:31.268407 2578 generic.go:356] "Generic (PLEG): container finished" podID="783af08b-92ba-423e-b5d4-25c6ad01d2ad" containerID="ae9f7aebc3949eb21e3cc4c18135239d9a73076cc3eee2d2ff6392eed157d985" exitCode=0 Jul 22 03:05:31.268508 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:05:31.268474 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-c22b602d-openshift-default-9c57744b9-vpxcj" event={"ID":"783af08b-92ba-423e-b5d4-25c6ad01d2ad","Type":"ContainerDied","Data":"ae9f7aebc3949eb21e3cc4c18135239d9a73076cc3eee2d2ff6392eed157d985"} Jul 22 03:05:31.268946 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:05:31.268857 2578 scope.go:117] "RemoveContainer" containerID="ae9f7aebc3949eb21e3cc4c18135239d9a73076cc3eee2d2ff6392eed157d985" Jul 22 03:05:31.269777 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:05:31.269755 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 22 03:05:31.269831 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:05:31.269821 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 22 03:05:31.269867 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:05:31.269851 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 22 03:05:32.275502 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:05:32.275472 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-c22b602d-openshift-default-9c57744b9-vpxcj" event={"ID":"783af08b-92ba-423e-b5d4-25c6ad01d2ad","Type":"ContainerStarted","Data":"2ed426ad73cacbe68203e15ed63362fb59ad3176b1eab946cbe978fccf73a491"} Jul 22 03:05:33.021838 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:05:33.021801 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-c22b602d-openshift-default-9c57744b9-vpxcj" Jul 22 03:05:33.022817 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:05:33.022790 2578 patch_prober.go:28] interesting pod/e2e-ait-c22b602d-openshift-default-9c57744b9-vpxcj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body= Jul 22 03:05:33.022939 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:05:33.022852 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-c22b602d-openshift-default-9c57744b9-vpxcj" podUID="783af08b-92ba-423e-b5d4-25c6ad01d2ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" Jul 22 03:05:33.201818 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:05:33.201777 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-c22b602d-openshift-default-9c57744b9-vpxcj" Jul 22 03:05:34.022823 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:05:34.022785 2578 patch_prober.go:28] interesting pod/e2e-ait-c22b602d-openshift-default-9c57744b9-vpxcj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body= Jul 22 03:05:34.023211 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:05:34.022835 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-c22b602d-openshift-default-9c57744b9-vpxcj" podUID="783af08b-92ba-423e-b5d4-25c6ad01d2ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" Jul 22 03:05:35.022809 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:05:35.022727 2578 patch_prober.go:28] interesting pod/e2e-ait-c22b602d-openshift-default-9c57744b9-vpxcj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body= Jul 22 03:05:35.022809 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:05:35.022783 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-c22b602d-openshift-default-9c57744b9-vpxcj" podUID="783af08b-92ba-423e-b5d4-25c6ad01d2ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" Jul 22 03:05:36.022577 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:05:36.022522 2578 patch_prober.go:28] interesting pod/e2e-ait-c22b602d-openshift-default-9c57744b9-vpxcj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body= Jul 22 03:05:36.022751 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:05:36.022621 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-c22b602d-openshift-default-9c57744b9-vpxcj" podUID="783af08b-92ba-423e-b5d4-25c6ad01d2ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" Jul 22 03:05:37.025507 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:05:37.025482 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-c22b602d-openshift-default-9c57744b9-vpxcj" Jul 22 03:05:37.026526 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:05:37.026507 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-c22b602d-openshift-default-9c57744b9-vpxcj" Jul 22 03:05:59.330106 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:05:59.330070 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-c22b602d-openshift-default-9c57744b9-vpxcj"] Jul 22 03:05:59.330652 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:05:59.330312 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-c22b602d-openshift-default-9c57744b9-vpxcj" podUID="783af08b-92ba-423e-b5d4-25c6ad01d2ad" containerName="istio-proxy" containerID="cri-o://2ed426ad73cacbe68203e15ed63362fb59ad3176b1eab946cbe978fccf73a491" gracePeriod=30 Jul 22 03:06:00.395785 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:06:00.395755 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-7c53861a-gw-openshift-default-8b7cc865f-95kxb"] Jul 22 03:06:00.399097 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:06:00.399077 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-7c53861a-gw-openshift-default-8b7cc865f-95kxb" Jul 22 03:06:00.402765 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:06:00.402742 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-7c53861a-gw-openshift-default-dockercfg-khk4w\"" Jul 22 03:06:00.415790 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:06:00.415764 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-7c53861a-gw-openshift-default-8b7cc865f-95kxb"] Jul 22 03:06:00.474947 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:06:00.474914 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f04bcb1-b73d-4f1a-b60c-aefcd88bd212-istiod-ca-cert\") pod \"e2e-migrate-7c53861a-gw-openshift-default-8b7cc865f-95kxb\" (UID: \"3f04bcb1-b73d-4f1a-b60c-aefcd88bd212\") " pod="openshift-ingress/e2e-migrate-7c53861a-gw-openshift-default-8b7cc865f-95kxb" Jul 22 03:06:00.475119 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:06:00.474958 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f04bcb1-b73d-4f1a-b60c-aefcd88bd212-istio-envoy\") pod \"e2e-migrate-7c53861a-gw-openshift-default-8b7cc865f-95kxb\" (UID: \"3f04bcb1-b73d-4f1a-b60c-aefcd88bd212\") " pod="openshift-ingress/e2e-migrate-7c53861a-gw-openshift-default-8b7cc865f-95kxb" Jul 22 03:06:00.475119 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:06:00.474988 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f04bcb1-b73d-4f1a-b60c-aefcd88bd212-workload-certs\") pod \"e2e-migrate-7c53861a-gw-openshift-default-8b7cc865f-95kxb\" (UID: \"3f04bcb1-b73d-4f1a-b60c-aefcd88bd212\") " pod="openshift-ingress/e2e-migrate-7c53861a-gw-openshift-default-8b7cc865f-95kxb" Jul 22 03:06:00.475119 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:06:00.475050 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f04bcb1-b73d-4f1a-b60c-aefcd88bd212-istio-data\") pod \"e2e-migrate-7c53861a-gw-openshift-default-8b7cc865f-95kxb\" (UID: \"3f04bcb1-b73d-4f1a-b60c-aefcd88bd212\") " pod="openshift-ingress/e2e-migrate-7c53861a-gw-openshift-default-8b7cc865f-95kxb" Jul 22 03:06:00.475119 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:06:00.475105 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f04bcb1-b73d-4f1a-b60c-aefcd88bd212-istio-podinfo\") pod \"e2e-migrate-7c53861a-gw-openshift-default-8b7cc865f-95kxb\" (UID: \"3f04bcb1-b73d-4f1a-b60c-aefcd88bd212\") " pod="openshift-ingress/e2e-migrate-7c53861a-gw-openshift-default-8b7cc865f-95kxb" Jul 22 03:06:00.475257 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:06:00.475147 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f04bcb1-b73d-4f1a-b60c-aefcd88bd212-credential-socket\") pod \"e2e-migrate-7c53861a-gw-openshift-default-8b7cc865f-95kxb\" (UID: \"3f04bcb1-b73d-4f1a-b60c-aefcd88bd212\") " pod="openshift-ingress/e2e-migrate-7c53861a-gw-openshift-default-8b7cc865f-95kxb" Jul 22 03:06:00.475257 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:06:00.475164 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f04bcb1-b73d-4f1a-b60c-aefcd88bd212-istio-token\") pod \"e2e-migrate-7c53861a-gw-openshift-default-8b7cc865f-95kxb\" (UID: \"3f04bcb1-b73d-4f1a-b60c-aefcd88bd212\") " pod="openshift-ingress/e2e-migrate-7c53861a-gw-openshift-default-8b7cc865f-95kxb" Jul 22 03:06:00.475257 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:06:00.475204 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f04bcb1-b73d-4f1a-b60c-aefcd88bd212-workload-socket\") pod \"e2e-migrate-7c53861a-gw-openshift-default-8b7cc865f-95kxb\" (UID: \"3f04bcb1-b73d-4f1a-b60c-aefcd88bd212\") " pod="openshift-ingress/e2e-migrate-7c53861a-gw-openshift-default-8b7cc865f-95kxb" Jul 22 03:06:00.475257 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:06:00.475220 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2wlq8\" (UniqueName: \"kubernetes.io/projected/3f04bcb1-b73d-4f1a-b60c-aefcd88bd212-kube-api-access-2wlq8\") pod \"e2e-migrate-7c53861a-gw-openshift-default-8b7cc865f-95kxb\" (UID: \"3f04bcb1-b73d-4f1a-b60c-aefcd88bd212\") " pod="openshift-ingress/e2e-migrate-7c53861a-gw-openshift-default-8b7cc865f-95kxb" Jul 22 03:06:00.576458 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:06:00.576424 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f04bcb1-b73d-4f1a-b60c-aefcd88bd212-credential-socket\") pod \"e2e-migrate-7c53861a-gw-openshift-default-8b7cc865f-95kxb\" (UID: \"3f04bcb1-b73d-4f1a-b60c-aefcd88bd212\") " pod="openshift-ingress/e2e-migrate-7c53861a-gw-openshift-default-8b7cc865f-95kxb" Jul 22 03:06:00.576458 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:06:00.576458 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f04bcb1-b73d-4f1a-b60c-aefcd88bd212-istio-token\") pod \"e2e-migrate-7c53861a-gw-openshift-default-8b7cc865f-95kxb\" (UID: \"3f04bcb1-b73d-4f1a-b60c-aefcd88bd212\") " pod="openshift-ingress/e2e-migrate-7c53861a-gw-openshift-default-8b7cc865f-95kxb" Jul 22 03:06:00.576753 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:06:00.576479 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f04bcb1-b73d-4f1a-b60c-aefcd88bd212-workload-socket\") pod \"e2e-migrate-7c53861a-gw-openshift-default-8b7cc865f-95kxb\" (UID: \"3f04bcb1-b73d-4f1a-b60c-aefcd88bd212\") " pod="openshift-ingress/e2e-migrate-7c53861a-gw-openshift-default-8b7cc865f-95kxb" Jul 22 03:06:00.576753 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:06:00.576500 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2wlq8\" (UniqueName: \"kubernetes.io/projected/3f04bcb1-b73d-4f1a-b60c-aefcd88bd212-kube-api-access-2wlq8\") pod \"e2e-migrate-7c53861a-gw-openshift-default-8b7cc865f-95kxb\" (UID: \"3f04bcb1-b73d-4f1a-b60c-aefcd88bd212\") " pod="openshift-ingress/e2e-migrate-7c53861a-gw-openshift-default-8b7cc865f-95kxb" Jul 22 03:06:00.576753 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:06:00.576613 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f04bcb1-b73d-4f1a-b60c-aefcd88bd212-istiod-ca-cert\") pod \"e2e-migrate-7c53861a-gw-openshift-default-8b7cc865f-95kxb\" (UID: \"3f04bcb1-b73d-4f1a-b60c-aefcd88bd212\") " pod="openshift-ingress/e2e-migrate-7c53861a-gw-openshift-default-8b7cc865f-95kxb" Jul 22 03:06:00.576753 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:06:00.576661 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f04bcb1-b73d-4f1a-b60c-aefcd88bd212-istio-envoy\") pod \"e2e-migrate-7c53861a-gw-openshift-default-8b7cc865f-95kxb\" (UID: \"3f04bcb1-b73d-4f1a-b60c-aefcd88bd212\") " pod="openshift-ingress/e2e-migrate-7c53861a-gw-openshift-default-8b7cc865f-95kxb" Jul 22 03:06:00.576753 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:06:00.576709 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f04bcb1-b73d-4f1a-b60c-aefcd88bd212-workload-certs\") pod \"e2e-migrate-7c53861a-gw-openshift-default-8b7cc865f-95kxb\" (UID: \"3f04bcb1-b73d-4f1a-b60c-aefcd88bd212\") " pod="openshift-ingress/e2e-migrate-7c53861a-gw-openshift-default-8b7cc865f-95kxb" Jul 22 03:06:00.576753 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:06:00.576743 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f04bcb1-b73d-4f1a-b60c-aefcd88bd212-istio-data\") pod \"e2e-migrate-7c53861a-gw-openshift-default-8b7cc865f-95kxb\" (UID: \"3f04bcb1-b73d-4f1a-b60c-aefcd88bd212\") " pod="openshift-ingress/e2e-migrate-7c53861a-gw-openshift-default-8b7cc865f-95kxb" Jul 22 03:06:00.577022 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:06:00.576775 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f04bcb1-b73d-4f1a-b60c-aefcd88bd212-istio-podinfo\") pod \"e2e-migrate-7c53861a-gw-openshift-default-8b7cc865f-95kxb\" (UID: \"3f04bcb1-b73d-4f1a-b60c-aefcd88bd212\") " pod="openshift-ingress/e2e-migrate-7c53861a-gw-openshift-default-8b7cc865f-95kxb" Jul 22 03:06:00.577022 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:06:00.576871 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f04bcb1-b73d-4f1a-b60c-aefcd88bd212-credential-socket\") pod \"e2e-migrate-7c53861a-gw-openshift-default-8b7cc865f-95kxb\" (UID: \"3f04bcb1-b73d-4f1a-b60c-aefcd88bd212\") " pod="openshift-ingress/e2e-migrate-7c53861a-gw-openshift-default-8b7cc865f-95kxb" Jul 22 03:06:00.577168 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:06:00.577144 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f04bcb1-b73d-4f1a-b60c-aefcd88bd212-workload-socket\") pod \"e2e-migrate-7c53861a-gw-openshift-default-8b7cc865f-95kxb\" (UID: \"3f04bcb1-b73d-4f1a-b60c-aefcd88bd212\") " pod="openshift-ingress/e2e-migrate-7c53861a-gw-openshift-default-8b7cc865f-95kxb" Jul 22 03:06:00.577932 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:06:00.577160 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f04bcb1-b73d-4f1a-b60c-aefcd88bd212-workload-certs\") pod \"e2e-migrate-7c53861a-gw-openshift-default-8b7cc865f-95kxb\" (UID: \"3f04bcb1-b73d-4f1a-b60c-aefcd88bd212\") " pod="openshift-ingress/e2e-migrate-7c53861a-gw-openshift-default-8b7cc865f-95kxb" Jul 22 03:06:00.577932 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:06:00.577450 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f04bcb1-b73d-4f1a-b60c-aefcd88bd212-istio-data\") pod \"e2e-migrate-7c53861a-gw-openshift-default-8b7cc865f-95kxb\" (UID: \"3f04bcb1-b73d-4f1a-b60c-aefcd88bd212\") " pod="openshift-ingress/e2e-migrate-7c53861a-gw-openshift-default-8b7cc865f-95kxb" Jul 22 03:06:00.578097 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:06:00.577781 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f04bcb1-b73d-4f1a-b60c-aefcd88bd212-istiod-ca-cert\") pod \"e2e-migrate-7c53861a-gw-openshift-default-8b7cc865f-95kxb\" (UID: \"3f04bcb1-b73d-4f1a-b60c-aefcd88bd212\") " pod="openshift-ingress/e2e-migrate-7c53861a-gw-openshift-default-8b7cc865f-95kxb" Jul 22 03:06:00.579174 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:06:00.579154 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f04bcb1-b73d-4f1a-b60c-aefcd88bd212-istio-envoy\") pod \"e2e-migrate-7c53861a-gw-openshift-default-8b7cc865f-95kxb\" (UID: \"3f04bcb1-b73d-4f1a-b60c-aefcd88bd212\") " pod="openshift-ingress/e2e-migrate-7c53861a-gw-openshift-default-8b7cc865f-95kxb" Jul 22 03:06:00.579440 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:06:00.579425 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f04bcb1-b73d-4f1a-b60c-aefcd88bd212-istio-podinfo\") pod \"e2e-migrate-7c53861a-gw-openshift-default-8b7cc865f-95kxb\" (UID: \"3f04bcb1-b73d-4f1a-b60c-aefcd88bd212\") " pod="openshift-ingress/e2e-migrate-7c53861a-gw-openshift-default-8b7cc865f-95kxb" Jul 22 03:06:00.585755 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:06:00.585731 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2wlq8\" (UniqueName: \"kubernetes.io/projected/3f04bcb1-b73d-4f1a-b60c-aefcd88bd212-kube-api-access-2wlq8\") pod \"e2e-migrate-7c53861a-gw-openshift-default-8b7cc865f-95kxb\" (UID: \"3f04bcb1-b73d-4f1a-b60c-aefcd88bd212\") " pod="openshift-ingress/e2e-migrate-7c53861a-gw-openshift-default-8b7cc865f-95kxb" Jul 22 03:06:00.585834 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:06:00.585821 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f04bcb1-b73d-4f1a-b60c-aefcd88bd212-istio-token\") pod \"e2e-migrate-7c53861a-gw-openshift-default-8b7cc865f-95kxb\" (UID: \"3f04bcb1-b73d-4f1a-b60c-aefcd88bd212\") " pod="openshift-ingress/e2e-migrate-7c53861a-gw-openshift-default-8b7cc865f-95kxb" Jul 22 03:06:00.710198 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:06:00.710117 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-7c53861a-gw-openshift-default-8b7cc865f-95kxb" Jul 22 03:06:00.852119 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:06:00.852088 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-7c53861a-gw-openshift-default-8b7cc865f-95kxb"] Jul 22 03:06:00.852544 ip-10-0-133-31 kubenswrapper[2578]: W0722 03:06:00.852517 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f04bcb1_b73d_4f1a_b60c_aefcd88bd212.slice/crio-ea58ddf482dd4e8f704019debbaeeebfcec898313705e2eadb2604ab81e8236c WatchSource:0}: Error finding container ea58ddf482dd4e8f704019debbaeeebfcec898313705e2eadb2604ab81e8236c: Status 404 returned error can't find the container with id ea58ddf482dd4e8f704019debbaeeebfcec898313705e2eadb2604ab81e8236c Jul 22 03:06:00.854753 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:06:00.854722 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 22 03:06:00.854828 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:06:00.854789 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 22 03:06:00.854828 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:06:00.854817 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 22 03:06:01.434021 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:06:01.433982 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-7c53861a-gw-openshift-default-8b7cc865f-95kxb" event={"ID":"3f04bcb1-b73d-4f1a-b60c-aefcd88bd212","Type":"ContainerStarted","Data":"5f18007b0921ff2cc218a80d020caa02f327de97922fc9ef05f750e215e97bd9"} Jul 22 03:06:01.434397 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:06:01.434040 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-7c53861a-gw-openshift-default-8b7cc865f-95kxb" event={"ID":"3f04bcb1-b73d-4f1a-b60c-aefcd88bd212","Type":"ContainerStarted","Data":"ea58ddf482dd4e8f704019debbaeeebfcec898313705e2eadb2604ab81e8236c"} Jul 22 03:06:01.711239 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:06:01.711152 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-7c53861a-gw-openshift-default-8b7cc865f-95kxb" Jul 22 03:06:01.716157 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:06:01.716133 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-7c53861a-gw-openshift-default-8b7cc865f-95kxb" Jul 22 03:06:01.750382 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:06:01.750328 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-7c53861a-gw-openshift-default-8b7cc865f-95kxb" podStartSLOduration=1.750313504 podStartE2EDuration="1.750313504s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 03:06:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 03:06:01.45800814 +0000 UTC m=+1727.777183441" watchObservedRunningTime="2026-07-22 03:06:01.750313504 +0000 UTC m=+1728.069488804" Jul 22 03:06:02.438817 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:06:02.438787 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-7c53861a-gw-openshift-default-8b7cc865f-95kxb" Jul 22 03:06:02.439902 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:06:02.439886 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-7c53861a-gw-openshift-default-8b7cc865f-95kxb" Jul 22 03:06:04.450676 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:06:04.450647 2578 generic.go:356] "Generic (PLEG): container finished" podID="783af08b-92ba-423e-b5d4-25c6ad01d2ad" containerID="2ed426ad73cacbe68203e15ed63362fb59ad3176b1eab946cbe978fccf73a491" exitCode=0 Jul 22 03:06:04.450986 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:06:04.450670 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-c22b602d-openshift-default-9c57744b9-vpxcj" event={"ID":"783af08b-92ba-423e-b5d4-25c6ad01d2ad","Type":"ContainerDied","Data":"2ed426ad73cacbe68203e15ed63362fb59ad3176b1eab946cbe978fccf73a491"} Jul 22 03:06:04.450986 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:06:04.450729 2578 scope.go:117] "RemoveContainer" containerID="ae9f7aebc3949eb21e3cc4c18135239d9a73076cc3eee2d2ff6392eed157d985" Jul 22 03:06:04.599266 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:06:04.599205 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-c22b602d-openshift-default-9c57744b9-vpxcj" Jul 22 03:06:04.716288 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:06:04.716258 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/783af08b-92ba-423e-b5d4-25c6ad01d2ad-istiod-ca-cert\") pod \"783af08b-92ba-423e-b5d4-25c6ad01d2ad\" (UID: \"783af08b-92ba-423e-b5d4-25c6ad01d2ad\") " Jul 22 03:06:04.716448 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:06:04.716298 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/783af08b-92ba-423e-b5d4-25c6ad01d2ad-istio-token\") pod \"783af08b-92ba-423e-b5d4-25c6ad01d2ad\" (UID: \"783af08b-92ba-423e-b5d4-25c6ad01d2ad\") " Jul 22 03:06:04.716448 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:06:04.716328 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/783af08b-92ba-423e-b5d4-25c6ad01d2ad-workload-certs\") pod \"783af08b-92ba-423e-b5d4-25c6ad01d2ad\" (UID: \"783af08b-92ba-423e-b5d4-25c6ad01d2ad\") " Jul 22 03:06:04.716448 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:06:04.716361 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/783af08b-92ba-423e-b5d4-25c6ad01d2ad-workload-socket\") pod \"783af08b-92ba-423e-b5d4-25c6ad01d2ad\" (UID: \"783af08b-92ba-423e-b5d4-25c6ad01d2ad\") " Jul 22 03:06:04.716448 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:06:04.716400 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wrcrq\" (UniqueName: \"kubernetes.io/projected/783af08b-92ba-423e-b5d4-25c6ad01d2ad-kube-api-access-wrcrq\") pod \"783af08b-92ba-423e-b5d4-25c6ad01d2ad\" (UID: \"783af08b-92ba-423e-b5d4-25c6ad01d2ad\") " Jul 22 03:06:04.716448 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:06:04.716424 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/783af08b-92ba-423e-b5d4-25c6ad01d2ad-istio-podinfo\") pod \"783af08b-92ba-423e-b5d4-25c6ad01d2ad\" (UID: \"783af08b-92ba-423e-b5d4-25c6ad01d2ad\") " Jul 22 03:06:04.716748 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:06:04.716514 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/783af08b-92ba-423e-b5d4-25c6ad01d2ad-istio-envoy\") pod \"783af08b-92ba-423e-b5d4-25c6ad01d2ad\" (UID: \"783af08b-92ba-423e-b5d4-25c6ad01d2ad\") " Jul 22 03:06:04.716748 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:06:04.716541 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/783af08b-92ba-423e-b5d4-25c6ad01d2ad-istio-data\") pod \"783af08b-92ba-423e-b5d4-25c6ad01d2ad\" (UID: \"783af08b-92ba-423e-b5d4-25c6ad01d2ad\") " Jul 22 03:06:04.716919 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:06:04.716769 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/783af08b-92ba-423e-b5d4-25c6ad01d2ad-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "783af08b-92ba-423e-b5d4-25c6ad01d2ad" (UID: "783af08b-92ba-423e-b5d4-25c6ad01d2ad"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 03:06:04.716919 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:06:04.716785 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/783af08b-92ba-423e-b5d4-25c6ad01d2ad-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "783af08b-92ba-423e-b5d4-25c6ad01d2ad" (UID: "783af08b-92ba-423e-b5d4-25c6ad01d2ad"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 03:06:04.716919 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:06:04.716827 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/783af08b-92ba-423e-b5d4-25c6ad01d2ad-credential-socket\") pod \"783af08b-92ba-423e-b5d4-25c6ad01d2ad\" (UID: \"783af08b-92ba-423e-b5d4-25c6ad01d2ad\") " Jul 22 03:06:04.717072 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:06:04.717010 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/783af08b-92ba-423e-b5d4-25c6ad01d2ad-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "783af08b-92ba-423e-b5d4-25c6ad01d2ad" (UID: "783af08b-92ba-423e-b5d4-25c6ad01d2ad"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 03:06:04.717298 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:06:04.717093 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/783af08b-92ba-423e-b5d4-25c6ad01d2ad-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "783af08b-92ba-423e-b5d4-25c6ad01d2ad" (UID: "783af08b-92ba-423e-b5d4-25c6ad01d2ad"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 03:06:04.717411 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:06:04.717387 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/783af08b-92ba-423e-b5d4-25c6ad01d2ad-istio-data" (OuterVolumeSpecName: "istio-data") pod "783af08b-92ba-423e-b5d4-25c6ad01d2ad" (UID: "783af08b-92ba-423e-b5d4-25c6ad01d2ad"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 03:06:04.717521 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:06:04.717448 2578 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/783af08b-92ba-423e-b5d4-25c6ad01d2ad-credential-socket\") on node \"ip-10-0-133-31.ec2.internal\" DevicePath \"\"" Jul 22 03:06:04.717521 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:06:04.717470 2578 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/783af08b-92ba-423e-b5d4-25c6ad01d2ad-istiod-ca-cert\") on node \"ip-10-0-133-31.ec2.internal\" DevicePath \"\"" Jul 22 03:06:04.717649 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:06:04.717527 2578 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/783af08b-92ba-423e-b5d4-25c6ad01d2ad-workload-certs\") on node \"ip-10-0-133-31.ec2.internal\" DevicePath \"\"" Jul 22 03:06:04.717649 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:06:04.717548 2578 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/783af08b-92ba-423e-b5d4-25c6ad01d2ad-workload-socket\") on node \"ip-10-0-133-31.ec2.internal\" DevicePath \"\"" Jul 22 03:06:04.718824 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:06:04.718803 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/783af08b-92ba-423e-b5d4-25c6ad01d2ad-istio-token" (OuterVolumeSpecName: "istio-token") pod "783af08b-92ba-423e-b5d4-25c6ad01d2ad" (UID: "783af08b-92ba-423e-b5d4-25c6ad01d2ad"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 03:06:04.718920 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:06:04.718799 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/783af08b-92ba-423e-b5d4-25c6ad01d2ad-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "783af08b-92ba-423e-b5d4-25c6ad01d2ad" (UID: "783af08b-92ba-423e-b5d4-25c6ad01d2ad"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 03:06:04.719263 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:06:04.719240 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/783af08b-92ba-423e-b5d4-25c6ad01d2ad-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "783af08b-92ba-423e-b5d4-25c6ad01d2ad" (UID: "783af08b-92ba-423e-b5d4-25c6ad01d2ad"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 22 03:06:04.719394 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:06:04.719246 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/783af08b-92ba-423e-b5d4-25c6ad01d2ad-kube-api-access-wrcrq" (OuterVolumeSpecName: "kube-api-access-wrcrq") pod "783af08b-92ba-423e-b5d4-25c6ad01d2ad" (UID: "783af08b-92ba-423e-b5d4-25c6ad01d2ad"). InnerVolumeSpecName "kube-api-access-wrcrq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 03:06:04.818818 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:06:04.818790 2578 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/783af08b-92ba-423e-b5d4-25c6ad01d2ad-istio-envoy\") on node \"ip-10-0-133-31.ec2.internal\" DevicePath \"\"" Jul 22 03:06:04.818818 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:06:04.818813 2578 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/783af08b-92ba-423e-b5d4-25c6ad01d2ad-istio-data\") on node \"ip-10-0-133-31.ec2.internal\" DevicePath \"\"" Jul 22 03:06:04.818818 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:06:04.818822 2578 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/783af08b-92ba-423e-b5d4-25c6ad01d2ad-istio-token\") on node \"ip-10-0-133-31.ec2.internal\" DevicePath \"\"" Jul 22 03:06:04.818987 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:06:04.818831 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wrcrq\" (UniqueName: \"kubernetes.io/projected/783af08b-92ba-423e-b5d4-25c6ad01d2ad-kube-api-access-wrcrq\") on node \"ip-10-0-133-31.ec2.internal\" DevicePath \"\"" Jul 22 03:06:04.818987 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:06:04.818841 2578 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/783af08b-92ba-423e-b5d4-25c6ad01d2ad-istio-podinfo\") on node \"ip-10-0-133-31.ec2.internal\" DevicePath \"\"" Jul 22 03:06:05.457051 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:06:05.457021 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-c22b602d-openshift-default-9c57744b9-vpxcj" Jul 22 03:06:05.457441 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:06:05.457063 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-c22b602d-openshift-default-9c57744b9-vpxcj" event={"ID":"783af08b-92ba-423e-b5d4-25c6ad01d2ad","Type":"ContainerDied","Data":"6556dfc847c9dbe6248c9ea36990bee68224d881f07fb87dea9b093c191fa310"} Jul 22 03:06:05.457441 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:06:05.457125 2578 scope.go:117] "RemoveContainer" containerID="2ed426ad73cacbe68203e15ed63362fb59ad3176b1eab946cbe978fccf73a491" Jul 22 03:06:05.492047 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:06:05.492018 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-c22b602d-openshift-default-9c57744b9-vpxcj"] Jul 22 03:06:05.495137 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:06:05.495115 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-c22b602d-openshift-default-9c57744b9-vpxcj"] Jul 22 03:06:06.142312 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:06:06.142277 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="783af08b-92ba-423e-b5d4-25c6ad01d2ad" path="/var/lib/kubelet/pods/783af08b-92ba-423e-b5d4-25c6ad01d2ad/volumes" Jul 22 03:06:17.440247 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:06:17.440214 2578 patch_prober.go:28] interesting pod/e2e-migrate-7c53861a-gw-openshift-default-8b7cc865f-95kxb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body= Jul 22 03:06:17.440681 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:06:17.440266 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-migrate-7c53861a-gw-openshift-default-8b7cc865f-95kxb" podUID="3f04bcb1-b73d-4f1a-b60c-aefcd88bd212" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" Jul 22 03:06:17.524116 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:06:17.524084 2578 generic.go:356] "Generic (PLEG): container finished" podID="3f04bcb1-b73d-4f1a-b60c-aefcd88bd212" containerID="5f18007b0921ff2cc218a80d020caa02f327de97922fc9ef05f750e215e97bd9" exitCode=0 Jul 22 03:06:17.524275 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:06:17.524163 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-7c53861a-gw-openshift-default-8b7cc865f-95kxb" event={"ID":"3f04bcb1-b73d-4f1a-b60c-aefcd88bd212","Type":"ContainerDied","Data":"5f18007b0921ff2cc218a80d020caa02f327de97922fc9ef05f750e215e97bd9"} Jul 22 03:06:17.524584 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:06:17.524552 2578 scope.go:117] "RemoveContainer" containerID="5f18007b0921ff2cc218a80d020caa02f327de97922fc9ef05f750e215e97bd9" Jul 22 03:06:17.525534 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:06:17.525509 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 22 03:06:17.525624 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:06:17.525607 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 22 03:06:17.525663 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:06:17.525647 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 22 03:06:18.531281 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:06:18.531249 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-7c53861a-gw-openshift-default-8b7cc865f-95kxb" event={"ID":"3f04bcb1-b73d-4f1a-b60c-aefcd88bd212","Type":"ContainerStarted","Data":"9141409f3eae9574f01ac61297b4ee7b8d009bd5642e6506fd486edee1a8b383"} Jul 22 03:06:18.710765 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:06:18.710728 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-7c53861a-gw-openshift-default-8b7cc865f-95kxb" Jul 22 03:06:18.711056 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:06:18.711026 2578 patch_prober.go:28] interesting pod/e2e-migrate-7c53861a-gw-openshift-default-8b7cc865f-95kxb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body= Jul 22 03:06:18.711168 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:06:18.711087 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-7c53861a-gw-openshift-default-8b7cc865f-95kxb" podUID="3f04bcb1-b73d-4f1a-b60c-aefcd88bd212" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" Jul 22 03:06:19.711403 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:06:19.711366 2578 patch_prober.go:28] interesting pod/e2e-migrate-7c53861a-gw-openshift-default-8b7cc865f-95kxb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body= Jul 22 03:06:19.712009 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:06:19.711975 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-7c53861a-gw-openshift-default-8b7cc865f-95kxb" podUID="3f04bcb1-b73d-4f1a-b60c-aefcd88bd212" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" Jul 22 03:06:20.710832 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:06:20.710796 2578 patch_prober.go:28] interesting pod/e2e-migrate-7c53861a-gw-openshift-default-8b7cc865f-95kxb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body= Jul 22 03:06:20.711109 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:06:20.711080 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-7c53861a-gw-openshift-default-8b7cc865f-95kxb" podUID="3f04bcb1-b73d-4f1a-b60c-aefcd88bd212" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" Jul 22 03:06:21.711235 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:06:21.711188 2578 patch_prober.go:28] interesting pod/e2e-migrate-7c53861a-gw-openshift-default-8b7cc865f-95kxb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body= Jul 22 03:06:21.711769 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:06:21.711259 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-7c53861a-gw-openshift-default-8b7cc865f-95kxb" podUID="3f04bcb1-b73d-4f1a-b60c-aefcd88bd212" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" Jul 22 03:06:22.714220 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:06:22.714191 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-7c53861a-gw-openshift-default-8b7cc865f-95kxb" Jul 22 03:06:22.714697 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:06:22.714456 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-7c53861a-gw-openshift-default-8b7cc865f-95kxb" Jul 22 03:06:22.715216 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:06:22.715196 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-7c53861a-gw-openshift-default-8b7cc865f-95kxb" Jul 22 03:06:43.320581 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:06:43.320538 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-7c53861a-gw-openshift-default-8b7cc865f-95kxb"] Jul 22 03:06:43.320980 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:06:43.320746 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-7c53861a-gw-openshift-default-8b7cc865f-95kxb" podUID="3f04bcb1-b73d-4f1a-b60c-aefcd88bd212" containerName="istio-proxy" containerID="cri-o://9141409f3eae9574f01ac61297b4ee7b8d009bd5642e6506fd486edee1a8b383" gracePeriod=30 Jul 22 03:06:44.210594 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:06:44.210542 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-b3e829f9-openshift-default-b987cc7d7-m7z4v"] Jul 22 03:06:44.210924 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:06:44.210910 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="783af08b-92ba-423e-b5d4-25c6ad01d2ad" containerName="istio-proxy" Jul 22 03:06:44.210969 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:06:44.210926 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="783af08b-92ba-423e-b5d4-25c6ad01d2ad" containerName="istio-proxy" Jul 22 03:06:44.211003 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:06:44.210995 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="783af08b-92ba-423e-b5d4-25c6ad01d2ad" containerName="istio-proxy" Jul 22 03:06:44.212972 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:06:44.212956 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-b3e829f9-openshift-default-b987cc7d7-m7z4v" Jul 22 03:06:44.216229 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:06:44.216206 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-b3e829f9-openshift-default-dockercfg-drqz5\"" Jul 22 03:06:44.239996 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:06:44.239973 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-b3e829f9-openshift-default-b987cc7d7-m7z4v"] Jul 22 03:06:44.274188 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:06:44.274164 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de03c673-5a6c-4183-bec8-ee4302e65e47-istiod-ca-cert\") pod \"e2e-ait-b3e829f9-openshift-default-b987cc7d7-m7z4v\" (UID: \"de03c673-5a6c-4183-bec8-ee4302e65e47\") " pod="openshift-ingress/e2e-ait-b3e829f9-openshift-default-b987cc7d7-m7z4v" Jul 22 03:06:44.274315 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:06:44.274199 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de03c673-5a6c-4183-bec8-ee4302e65e47-workload-certs\") pod \"e2e-ait-b3e829f9-openshift-default-b987cc7d7-m7z4v\" (UID: \"de03c673-5a6c-4183-bec8-ee4302e65e47\") " pod="openshift-ingress/e2e-ait-b3e829f9-openshift-default-b987cc7d7-m7z4v" Jul 22 03:06:44.274315 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:06:44.274231 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de03c673-5a6c-4183-bec8-ee4302e65e47-credential-socket\") pod \"e2e-ait-b3e829f9-openshift-default-b987cc7d7-m7z4v\" (UID: \"de03c673-5a6c-4183-bec8-ee4302e65e47\") " pod="openshift-ingress/e2e-ait-b3e829f9-openshift-default-b987cc7d7-m7z4v" Jul 22 03:06:44.274423 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:06:44.274353 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de03c673-5a6c-4183-bec8-ee4302e65e47-istio-podinfo\") pod \"e2e-ait-b3e829f9-openshift-default-b987cc7d7-m7z4v\" (UID: \"de03c673-5a6c-4183-bec8-ee4302e65e47\") " pod="openshift-ingress/e2e-ait-b3e829f9-openshift-default-b987cc7d7-m7z4v" Jul 22 03:06:44.274423 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:06:44.274416 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de03c673-5a6c-4183-bec8-ee4302e65e47-workload-socket\") pod \"e2e-ait-b3e829f9-openshift-default-b987cc7d7-m7z4v\" (UID: \"de03c673-5a6c-4183-bec8-ee4302e65e47\") " pod="openshift-ingress/e2e-ait-b3e829f9-openshift-default-b987cc7d7-m7z4v" Jul 22 03:06:44.274529 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:06:44.274476 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de03c673-5a6c-4183-bec8-ee4302e65e47-istio-data\") pod \"e2e-ait-b3e829f9-openshift-default-b987cc7d7-m7z4v\" (UID: \"de03c673-5a6c-4183-bec8-ee4302e65e47\") " pod="openshift-ingress/e2e-ait-b3e829f9-openshift-default-b987cc7d7-m7z4v" Jul 22 03:06:44.274529 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:06:44.274522 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de03c673-5a6c-4183-bec8-ee4302e65e47-istio-envoy\") pod \"e2e-ait-b3e829f9-openshift-default-b987cc7d7-m7z4v\" (UID: \"de03c673-5a6c-4183-bec8-ee4302e65e47\") " pod="openshift-ingress/e2e-ait-b3e829f9-openshift-default-b987cc7d7-m7z4v" Jul 22 03:06:44.274708 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:06:44.274556 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de03c673-5a6c-4183-bec8-ee4302e65e47-istio-token\") pod \"e2e-ait-b3e829f9-openshift-default-b987cc7d7-m7z4v\" (UID: \"de03c673-5a6c-4183-bec8-ee4302e65e47\") " pod="openshift-ingress/e2e-ait-b3e829f9-openshift-default-b987cc7d7-m7z4v" Jul 22 03:06:44.274708 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:06:44.274670 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4252k\" (UniqueName: \"kubernetes.io/projected/de03c673-5a6c-4183-bec8-ee4302e65e47-kube-api-access-4252k\") pod \"e2e-ait-b3e829f9-openshift-default-b987cc7d7-m7z4v\" (UID: \"de03c673-5a6c-4183-bec8-ee4302e65e47\") " pod="openshift-ingress/e2e-ait-b3e829f9-openshift-default-b987cc7d7-m7z4v" Jul 22 03:06:44.376019 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:06:44.375987 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de03c673-5a6c-4183-bec8-ee4302e65e47-istio-podinfo\") pod \"e2e-ait-b3e829f9-openshift-default-b987cc7d7-m7z4v\" (UID: \"de03c673-5a6c-4183-bec8-ee4302e65e47\") " pod="openshift-ingress/e2e-ait-b3e829f9-openshift-default-b987cc7d7-m7z4v" Jul 22 03:06:44.376486 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:06:44.376057 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de03c673-5a6c-4183-bec8-ee4302e65e47-workload-socket\") pod \"e2e-ait-b3e829f9-openshift-default-b987cc7d7-m7z4v\" (UID: \"de03c673-5a6c-4183-bec8-ee4302e65e47\") " pod="openshift-ingress/e2e-ait-b3e829f9-openshift-default-b987cc7d7-m7z4v" Jul 22 03:06:44.376486 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:06:44.376094 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de03c673-5a6c-4183-bec8-ee4302e65e47-istio-data\") pod \"e2e-ait-b3e829f9-openshift-default-b987cc7d7-m7z4v\" (UID: \"de03c673-5a6c-4183-bec8-ee4302e65e47\") " pod="openshift-ingress/e2e-ait-b3e829f9-openshift-default-b987cc7d7-m7z4v" Jul 22 03:06:44.376486 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:06:44.376126 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de03c673-5a6c-4183-bec8-ee4302e65e47-istio-envoy\") pod \"e2e-ait-b3e829f9-openshift-default-b987cc7d7-m7z4v\" (UID: \"de03c673-5a6c-4183-bec8-ee4302e65e47\") " pod="openshift-ingress/e2e-ait-b3e829f9-openshift-default-b987cc7d7-m7z4v" Jul 22 03:06:44.376486 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:06:44.376159 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de03c673-5a6c-4183-bec8-ee4302e65e47-istio-token\") pod \"e2e-ait-b3e829f9-openshift-default-b987cc7d7-m7z4v\" (UID: \"de03c673-5a6c-4183-bec8-ee4302e65e47\") " pod="openshift-ingress/e2e-ait-b3e829f9-openshift-default-b987cc7d7-m7z4v" Jul 22 03:06:44.376486 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:06:44.376183 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4252k\" (UniqueName: \"kubernetes.io/projected/de03c673-5a6c-4183-bec8-ee4302e65e47-kube-api-access-4252k\") pod \"e2e-ait-b3e829f9-openshift-default-b987cc7d7-m7z4v\" (UID: \"de03c673-5a6c-4183-bec8-ee4302e65e47\") " pod="openshift-ingress/e2e-ait-b3e829f9-openshift-default-b987cc7d7-m7z4v" Jul 22 03:06:44.376486 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:06:44.376466 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de03c673-5a6c-4183-bec8-ee4302e65e47-istiod-ca-cert\") pod \"e2e-ait-b3e829f9-openshift-default-b987cc7d7-m7z4v\" (UID: \"de03c673-5a6c-4183-bec8-ee4302e65e47\") " pod="openshift-ingress/e2e-ait-b3e829f9-openshift-default-b987cc7d7-m7z4v" Jul 22 03:06:44.376847 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:06:44.376514 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de03c673-5a6c-4183-bec8-ee4302e65e47-workload-certs\") pod \"e2e-ait-b3e829f9-openshift-default-b987cc7d7-m7z4v\" (UID: \"de03c673-5a6c-4183-bec8-ee4302e65e47\") " pod="openshift-ingress/e2e-ait-b3e829f9-openshift-default-b987cc7d7-m7z4v" Jul 22 03:06:44.376847 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:06:44.376550 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de03c673-5a6c-4183-bec8-ee4302e65e47-credential-socket\") pod \"e2e-ait-b3e829f9-openshift-default-b987cc7d7-m7z4v\" (UID: \"de03c673-5a6c-4183-bec8-ee4302e65e47\") " pod="openshift-ingress/e2e-ait-b3e829f9-openshift-default-b987cc7d7-m7z4v" Jul 22 03:06:44.376847 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:06:44.376630 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de03c673-5a6c-4183-bec8-ee4302e65e47-workload-socket\") pod \"e2e-ait-b3e829f9-openshift-default-b987cc7d7-m7z4v\" (UID: \"de03c673-5a6c-4183-bec8-ee4302e65e47\") " pod="openshift-ingress/e2e-ait-b3e829f9-openshift-default-b987cc7d7-m7z4v" Jul 22 03:06:44.376847 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:06:44.376775 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de03c673-5a6c-4183-bec8-ee4302e65e47-istio-data\") pod \"e2e-ait-b3e829f9-openshift-default-b987cc7d7-m7z4v\" (UID: \"de03c673-5a6c-4183-bec8-ee4302e65e47\") " pod="openshift-ingress/e2e-ait-b3e829f9-openshift-default-b987cc7d7-m7z4v" Jul 22 03:06:44.377047 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:06:44.376853 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de03c673-5a6c-4183-bec8-ee4302e65e47-workload-certs\") pod \"e2e-ait-b3e829f9-openshift-default-b987cc7d7-m7z4v\" (UID: \"de03c673-5a6c-4183-bec8-ee4302e65e47\") " pod="openshift-ingress/e2e-ait-b3e829f9-openshift-default-b987cc7d7-m7z4v" Jul 22 03:06:44.377047 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:06:44.376891 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de03c673-5a6c-4183-bec8-ee4302e65e47-credential-socket\") pod \"e2e-ait-b3e829f9-openshift-default-b987cc7d7-m7z4v\" (UID: \"de03c673-5a6c-4183-bec8-ee4302e65e47\") " pod="openshift-ingress/e2e-ait-b3e829f9-openshift-default-b987cc7d7-m7z4v" Jul 22 03:06:44.377220 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:06:44.377201 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de03c673-5a6c-4183-bec8-ee4302e65e47-istiod-ca-cert\") pod \"e2e-ait-b3e829f9-openshift-default-b987cc7d7-m7z4v\" (UID: \"de03c673-5a6c-4183-bec8-ee4302e65e47\") " pod="openshift-ingress/e2e-ait-b3e829f9-openshift-default-b987cc7d7-m7z4v" Jul 22 03:06:44.378756 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:06:44.378735 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de03c673-5a6c-4183-bec8-ee4302e65e47-istio-envoy\") pod \"e2e-ait-b3e829f9-openshift-default-b987cc7d7-m7z4v\" (UID: \"de03c673-5a6c-4183-bec8-ee4302e65e47\") " pod="openshift-ingress/e2e-ait-b3e829f9-openshift-default-b987cc7d7-m7z4v" Jul 22 03:06:44.378953 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:06:44.378936 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de03c673-5a6c-4183-bec8-ee4302e65e47-istio-podinfo\") pod \"e2e-ait-b3e829f9-openshift-default-b987cc7d7-m7z4v\" (UID: \"de03c673-5a6c-4183-bec8-ee4302e65e47\") " pod="openshift-ingress/e2e-ait-b3e829f9-openshift-default-b987cc7d7-m7z4v" Jul 22 03:06:44.394169 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:06:44.394141 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de03c673-5a6c-4183-bec8-ee4302e65e47-istio-token\") pod \"e2e-ait-b3e829f9-openshift-default-b987cc7d7-m7z4v\" (UID: \"de03c673-5a6c-4183-bec8-ee4302e65e47\") " pod="openshift-ingress/e2e-ait-b3e829f9-openshift-default-b987cc7d7-m7z4v" Jul 22 03:06:44.396805 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:06:44.396776 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4252k\" (UniqueName: \"kubernetes.io/projected/de03c673-5a6c-4183-bec8-ee4302e65e47-kube-api-access-4252k\") pod \"e2e-ait-b3e829f9-openshift-default-b987cc7d7-m7z4v\" (UID: \"de03c673-5a6c-4183-bec8-ee4302e65e47\") " pod="openshift-ingress/e2e-ait-b3e829f9-openshift-default-b987cc7d7-m7z4v" Jul 22 03:06:44.523398 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:06:44.523312 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-b3e829f9-openshift-default-b987cc7d7-m7z4v" Jul 22 03:06:44.681913 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:06:44.681886 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-b3e829f9-openshift-default-b987cc7d7-m7z4v"] Jul 22 03:06:44.682791 ip-10-0-133-31 kubenswrapper[2578]: W0722 03:06:44.682768 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde03c673_5a6c_4183_bec8_ee4302e65e47.slice/crio-0fb0ba71725e978c9ad4c31f2d8b9cc295fe87c134ebfdc5fb06f273c75ee5e6 WatchSource:0}: Error finding container 0fb0ba71725e978c9ad4c31f2d8b9cc295fe87c134ebfdc5fb06f273c75ee5e6: Status 404 returned error can't find the container with id 0fb0ba71725e978c9ad4c31f2d8b9cc295fe87c134ebfdc5fb06f273c75ee5e6 Jul 22 03:06:44.684632 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:06:44.684605 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 22 03:06:44.684696 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:06:44.684673 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 22 03:06:44.684740 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:06:44.684703 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 22 03:06:45.671874 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:06:45.671832 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b3e829f9-openshift-default-b987cc7d7-m7z4v" event={"ID":"de03c673-5a6c-4183-bec8-ee4302e65e47","Type":"ContainerStarted","Data":"8d54d702fadd5561ad5a4b17dad37c23037c42193eca4821740da410260c2148"} Jul 22 03:06:45.672329 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:06:45.671926 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b3e829f9-openshift-default-b987cc7d7-m7z4v" event={"ID":"de03c673-5a6c-4183-bec8-ee4302e65e47","Type":"ContainerStarted","Data":"0fb0ba71725e978c9ad4c31f2d8b9cc295fe87c134ebfdc5fb06f273c75ee5e6"} Jul 22 03:06:45.730323 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:06:45.730279 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-b3e829f9-openshift-default-b987cc7d7-m7z4v" podStartSLOduration=1.730261149 podStartE2EDuration="1.730261149s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 03:06:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 03:06:45.728388446 +0000 UTC m=+1772.047563746" watchObservedRunningTime="2026-07-22 03:06:45.730261149 +0000 UTC m=+1772.049436451" Jul 22 03:06:46.523552 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:06:46.523516 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-b3e829f9-openshift-default-b987cc7d7-m7z4v" Jul 22 03:06:46.528271 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:06:46.528250 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-b3e829f9-openshift-default-b987cc7d7-m7z4v" Jul 22 03:06:46.676953 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:06:46.676924 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-b3e829f9-openshift-default-b987cc7d7-m7z4v" Jul 22 03:06:46.677825 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:06:46.677804 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-b3e829f9-openshift-default-b987cc7d7-m7z4v" Jul 22 03:06:48.583234 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:06:48.583172 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-7c53861a-gw-openshift-default-8b7cc865f-95kxb" Jul 22 03:06:48.618276 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:06:48.618214 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2wlq8\" (UniqueName: \"kubernetes.io/projected/3f04bcb1-b73d-4f1a-b60c-aefcd88bd212-kube-api-access-2wlq8\") pod \"3f04bcb1-b73d-4f1a-b60c-aefcd88bd212\" (UID: \"3f04bcb1-b73d-4f1a-b60c-aefcd88bd212\") " Jul 22 03:06:48.618276 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:06:48.618259 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f04bcb1-b73d-4f1a-b60c-aefcd88bd212-workload-certs\") pod \"3f04bcb1-b73d-4f1a-b60c-aefcd88bd212\" (UID: \"3f04bcb1-b73d-4f1a-b60c-aefcd88bd212\") " Jul 22 03:06:48.618467 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:06:48.618293 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f04bcb1-b73d-4f1a-b60c-aefcd88bd212-istio-data\") pod \"3f04bcb1-b73d-4f1a-b60c-aefcd88bd212\" (UID: \"3f04bcb1-b73d-4f1a-b60c-aefcd88bd212\") " Jul 22 03:06:48.618615 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:06:48.618585 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f04bcb1-b73d-4f1a-b60c-aefcd88bd212-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "3f04bcb1-b73d-4f1a-b60c-aefcd88bd212" (UID: "3f04bcb1-b73d-4f1a-b60c-aefcd88bd212"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 03:06:48.618710 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:06:48.618620 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f04bcb1-b73d-4f1a-b60c-aefcd88bd212-istio-token\") pod \"3f04bcb1-b73d-4f1a-b60c-aefcd88bd212\" (UID: \"3f04bcb1-b73d-4f1a-b60c-aefcd88bd212\") " Jul 22 03:06:48.618710 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:06:48.618672 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f04bcb1-b73d-4f1a-b60c-aefcd88bd212-credential-socket\") pod \"3f04bcb1-b73d-4f1a-b60c-aefcd88bd212\" (UID: \"3f04bcb1-b73d-4f1a-b60c-aefcd88bd212\") " Jul 22 03:06:48.618815 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:06:48.618715 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f04bcb1-b73d-4f1a-b60c-aefcd88bd212-workload-socket\") pod \"3f04bcb1-b73d-4f1a-b60c-aefcd88bd212\" (UID: \"3f04bcb1-b73d-4f1a-b60c-aefcd88bd212\") " Jul 22 03:06:48.618815 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:06:48.618742 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f04bcb1-b73d-4f1a-b60c-aefcd88bd212-istio-envoy\") pod \"3f04bcb1-b73d-4f1a-b60c-aefcd88bd212\" (UID: \"3f04bcb1-b73d-4f1a-b60c-aefcd88bd212\") " Jul 22 03:06:48.618815 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:06:48.618794 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f04bcb1-b73d-4f1a-b60c-aefcd88bd212-istio-podinfo\") pod \"3f04bcb1-b73d-4f1a-b60c-aefcd88bd212\" (UID: \"3f04bcb1-b73d-4f1a-b60c-aefcd88bd212\") " Jul 22 03:06:48.618957 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:06:48.618854 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f04bcb1-b73d-4f1a-b60c-aefcd88bd212-istiod-ca-cert\") pod \"3f04bcb1-b73d-4f1a-b60c-aefcd88bd212\" (UID: \"3f04bcb1-b73d-4f1a-b60c-aefcd88bd212\") " Jul 22 03:06:48.619357 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:06:48.619148 2578 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f04bcb1-b73d-4f1a-b60c-aefcd88bd212-workload-certs\") on node \"ip-10-0-133-31.ec2.internal\" DevicePath \"\"" Jul 22 03:06:48.619357 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:06:48.619143 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f04bcb1-b73d-4f1a-b60c-aefcd88bd212-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "3f04bcb1-b73d-4f1a-b60c-aefcd88bd212" (UID: "3f04bcb1-b73d-4f1a-b60c-aefcd88bd212"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 03:06:48.619357 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:06:48.619195 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f04bcb1-b73d-4f1a-b60c-aefcd88bd212-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "3f04bcb1-b73d-4f1a-b60c-aefcd88bd212" (UID: "3f04bcb1-b73d-4f1a-b60c-aefcd88bd212"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 03:06:48.619600 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:06:48.619367 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f04bcb1-b73d-4f1a-b60c-aefcd88bd212-istio-data" (OuterVolumeSpecName: "istio-data") pod "3f04bcb1-b73d-4f1a-b60c-aefcd88bd212" (UID: "3f04bcb1-b73d-4f1a-b60c-aefcd88bd212"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 03:06:48.619600 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:06:48.619450 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f04bcb1-b73d-4f1a-b60c-aefcd88bd212-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "3f04bcb1-b73d-4f1a-b60c-aefcd88bd212" (UID: "3f04bcb1-b73d-4f1a-b60c-aefcd88bd212"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 03:06:48.621613 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:06:48.621233 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f04bcb1-b73d-4f1a-b60c-aefcd88bd212-kube-api-access-2wlq8" (OuterVolumeSpecName: "kube-api-access-2wlq8") pod "3f04bcb1-b73d-4f1a-b60c-aefcd88bd212" (UID: "3f04bcb1-b73d-4f1a-b60c-aefcd88bd212"). InnerVolumeSpecName "kube-api-access-2wlq8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 03:06:48.621613 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:06:48.621231 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f04bcb1-b73d-4f1a-b60c-aefcd88bd212-istio-token" (OuterVolumeSpecName: "istio-token") pod "3f04bcb1-b73d-4f1a-b60c-aefcd88bd212" (UID: "3f04bcb1-b73d-4f1a-b60c-aefcd88bd212"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 03:06:48.621613 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:06:48.621532 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3f04bcb1-b73d-4f1a-b60c-aefcd88bd212-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "3f04bcb1-b73d-4f1a-b60c-aefcd88bd212" (UID: "3f04bcb1-b73d-4f1a-b60c-aefcd88bd212"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 22 03:06:48.621832 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:06:48.621637 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f04bcb1-b73d-4f1a-b60c-aefcd88bd212-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "3f04bcb1-b73d-4f1a-b60c-aefcd88bd212" (UID: "3f04bcb1-b73d-4f1a-b60c-aefcd88bd212"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 03:06:48.689391 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:06:48.689361 2578 generic.go:356] "Generic (PLEG): container finished" podID="3f04bcb1-b73d-4f1a-b60c-aefcd88bd212" containerID="9141409f3eae9574f01ac61297b4ee7b8d009bd5642e6506fd486edee1a8b383" exitCode=0 Jul 22 03:06:48.689726 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:06:48.689432 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-7c53861a-gw-openshift-default-8b7cc865f-95kxb" Jul 22 03:06:48.689726 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:06:48.689461 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-7c53861a-gw-openshift-default-8b7cc865f-95kxb" event={"ID":"3f04bcb1-b73d-4f1a-b60c-aefcd88bd212","Type":"ContainerDied","Data":"9141409f3eae9574f01ac61297b4ee7b8d009bd5642e6506fd486edee1a8b383"} Jul 22 03:06:48.689726 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:06:48.689537 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-7c53861a-gw-openshift-default-8b7cc865f-95kxb" event={"ID":"3f04bcb1-b73d-4f1a-b60c-aefcd88bd212","Type":"ContainerDied","Data":"ea58ddf482dd4e8f704019debbaeeebfcec898313705e2eadb2604ab81e8236c"} Jul 22 03:06:48.689726 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:06:48.689616 2578 scope.go:117] "RemoveContainer" containerID="9141409f3eae9574f01ac61297b4ee7b8d009bd5642e6506fd486edee1a8b383" Jul 22 03:06:48.701852 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:06:48.701837 2578 scope.go:117] "RemoveContainer" containerID="5f18007b0921ff2cc218a80d020caa02f327de97922fc9ef05f750e215e97bd9" Jul 22 03:06:48.709819 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:06:48.709803 2578 scope.go:117] "RemoveContainer" containerID="9141409f3eae9574f01ac61297b4ee7b8d009bd5642e6506fd486edee1a8b383" Jul 22 03:06:48.710067 ip-10-0-133-31 kubenswrapper[2578]: E0722 03:06:48.710051 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9141409f3eae9574f01ac61297b4ee7b8d009bd5642e6506fd486edee1a8b383\": container with ID starting with 9141409f3eae9574f01ac61297b4ee7b8d009bd5642e6506fd486edee1a8b383 not found: ID does not exist" containerID="9141409f3eae9574f01ac61297b4ee7b8d009bd5642e6506fd486edee1a8b383" Jul 22 03:06:48.710118 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:06:48.710074 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9141409f3eae9574f01ac61297b4ee7b8d009bd5642e6506fd486edee1a8b383"} err="failed to get container status \"9141409f3eae9574f01ac61297b4ee7b8d009bd5642e6506fd486edee1a8b383\": rpc error: code = NotFound desc = could not find container \"9141409f3eae9574f01ac61297b4ee7b8d009bd5642e6506fd486edee1a8b383\": container with ID starting with 9141409f3eae9574f01ac61297b4ee7b8d009bd5642e6506fd486edee1a8b383 not found: ID does not exist" Jul 22 03:06:48.710118 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:06:48.710088 2578 scope.go:117] "RemoveContainer" containerID="5f18007b0921ff2cc218a80d020caa02f327de97922fc9ef05f750e215e97bd9" Jul 22 03:06:48.710280 ip-10-0-133-31 kubenswrapper[2578]: E0722 03:06:48.710267 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f18007b0921ff2cc218a80d020caa02f327de97922fc9ef05f750e215e97bd9\": container with ID starting with 5f18007b0921ff2cc218a80d020caa02f327de97922fc9ef05f750e215e97bd9 not found: ID does not exist" containerID="5f18007b0921ff2cc218a80d020caa02f327de97922fc9ef05f750e215e97bd9" Jul 22 03:06:48.710319 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:06:48.710284 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f18007b0921ff2cc218a80d020caa02f327de97922fc9ef05f750e215e97bd9"} err="failed to get container status \"5f18007b0921ff2cc218a80d020caa02f327de97922fc9ef05f750e215e97bd9\": rpc error: code = NotFound desc = could not find container \"5f18007b0921ff2cc218a80d020caa02f327de97922fc9ef05f750e215e97bd9\": container with ID starting with 5f18007b0921ff2cc218a80d020caa02f327de97922fc9ef05f750e215e97bd9 not found: ID does not exist" Jul 22 03:06:48.720155 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:06:48.720133 2578 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f04bcb1-b73d-4f1a-b60c-aefcd88bd212-istio-data\") on node \"ip-10-0-133-31.ec2.internal\" DevicePath \"\"" Jul 22 03:06:48.720220 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:06:48.720159 2578 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f04bcb1-b73d-4f1a-b60c-aefcd88bd212-istio-token\") on node \"ip-10-0-133-31.ec2.internal\" DevicePath \"\"" Jul 22 03:06:48.720220 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:06:48.720172 2578 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f04bcb1-b73d-4f1a-b60c-aefcd88bd212-credential-socket\") on node \"ip-10-0-133-31.ec2.internal\" DevicePath \"\"" Jul 22 03:06:48.720220 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:06:48.720185 2578 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f04bcb1-b73d-4f1a-b60c-aefcd88bd212-workload-socket\") on node \"ip-10-0-133-31.ec2.internal\" DevicePath \"\"" Jul 22 03:06:48.720220 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:06:48.720198 2578 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f04bcb1-b73d-4f1a-b60c-aefcd88bd212-istio-envoy\") on node \"ip-10-0-133-31.ec2.internal\" DevicePath \"\"" Jul 22 03:06:48.720220 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:06:48.720211 2578 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f04bcb1-b73d-4f1a-b60c-aefcd88bd212-istio-podinfo\") on node \"ip-10-0-133-31.ec2.internal\" DevicePath \"\"" Jul 22 03:06:48.720382 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:06:48.720221 2578 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f04bcb1-b73d-4f1a-b60c-aefcd88bd212-istiod-ca-cert\") on node \"ip-10-0-133-31.ec2.internal\" DevicePath \"\"" Jul 22 03:06:48.720382 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:06:48.720233 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2wlq8\" (UniqueName: \"kubernetes.io/projected/3f04bcb1-b73d-4f1a-b60c-aefcd88bd212-kube-api-access-2wlq8\") on node \"ip-10-0-133-31.ec2.internal\" DevicePath \"\"" Jul 22 03:06:48.721518 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:06:48.721497 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-7c53861a-gw-openshift-default-8b7cc865f-95kxb"] Jul 22 03:06:48.733811 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:06:48.733791 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-7c53861a-gw-openshift-default-8b7cc865f-95kxb"] Jul 22 03:06:50.146677 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:06:50.146643 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3f04bcb1-b73d-4f1a-b60c-aefcd88bd212" path="/var/lib/kubelet/pods/3f04bcb1-b73d-4f1a-b60c-aefcd88bd212/volumes" Jul 22 03:06:59.754783 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:06:59.754698 2578 generic.go:356] "Generic (PLEG): container finished" podID="de03c673-5a6c-4183-bec8-ee4302e65e47" containerID="8d54d702fadd5561ad5a4b17dad37c23037c42193eca4821740da410260c2148" exitCode=0 Jul 22 03:06:59.754783 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:06:59.754761 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b3e829f9-openshift-default-b987cc7d7-m7z4v" event={"ID":"de03c673-5a6c-4183-bec8-ee4302e65e47","Type":"ContainerDied","Data":"8d54d702fadd5561ad5a4b17dad37c23037c42193eca4821740da410260c2148"} Jul 22 03:06:59.755185 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:06:59.755126 2578 scope.go:117] "RemoveContainer" containerID="8d54d702fadd5561ad5a4b17dad37c23037c42193eca4821740da410260c2148" Jul 22 03:06:59.756071 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:06:59.756036 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 22 03:06:59.756175 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:06:59.756104 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 22 03:06:59.756175 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:06:59.756135 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 22 03:07:00.762083 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:07:00.762052 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b3e829f9-openshift-default-b987cc7d7-m7z4v" event={"ID":"de03c673-5a6c-4183-bec8-ee4302e65e47","Type":"ContainerStarted","Data":"b8df0af859da71d715eb3a63585aaea120c842a5ac04e4df2e8aada4f6a4a0eb"} Jul 22 03:07:01.524245 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:07:01.524209 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-b3e829f9-openshift-default-b987cc7d7-m7z4v" Jul 22 03:07:01.525419 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:07:01.525390 2578 patch_prober.go:28] interesting pod/e2e-ait-b3e829f9-openshift-default-b987cc7d7-m7z4v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body= Jul 22 03:07:01.525546 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:07:01.525452 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-b3e829f9-openshift-default-b987cc7d7-m7z4v" podUID="de03c673-5a6c-4183-bec8-ee4302e65e47" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" Jul 22 03:07:01.677244 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:07:01.677216 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-b3e829f9-openshift-default-b987cc7d7-m7z4v" Jul 22 03:07:02.524088 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:07:02.524046 2578 patch_prober.go:28] interesting pod/e2e-ait-b3e829f9-openshift-default-b987cc7d7-m7z4v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body= Jul 22 03:07:02.524514 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:07:02.524118 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-b3e829f9-openshift-default-b987cc7d7-m7z4v" podUID="de03c673-5a6c-4183-bec8-ee4302e65e47" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" Jul 22 03:07:03.524008 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:07:03.523976 2578 patch_prober.go:28] interesting pod/e2e-ait-b3e829f9-openshift-default-b987cc7d7-m7z4v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body= Jul 22 03:07:03.524182 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:07:03.524030 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-b3e829f9-openshift-default-b987cc7d7-m7z4v" podUID="de03c673-5a6c-4183-bec8-ee4302e65e47" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" Jul 22 03:07:04.527827 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:07:04.527798 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-b3e829f9-openshift-default-b987cc7d7-m7z4v" Jul 22 03:07:04.529055 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:07:04.529031 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-b3e829f9-openshift-default-b987cc7d7-m7z4v" Jul 22 03:07:14.192826 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:07:14.192797 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-prfql_fd7bf923-8d16-476c-a8d0-a3e6fd645061/ovn-acl-logging/0.log" Jul 22 03:07:14.196665 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:07:14.196648 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-prfql_fd7bf923-8d16-476c-a8d0-a3e6fd645061/ovn-acl-logging/0.log" Jul 22 03:07:29.207526 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:07:29.207495 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-b3e829f9-openshift-default-b987cc7d7-m7z4v"] Jul 22 03:07:29.207995 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:07:29.207766 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-b3e829f9-openshift-default-b987cc7d7-m7z4v" podUID="de03c673-5a6c-4183-bec8-ee4302e65e47" containerName="istio-proxy" containerID="cri-o://b8df0af859da71d715eb3a63585aaea120c842a5ac04e4df2e8aada4f6a4a0eb" gracePeriod=30 Jul 22 03:07:29.962847 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:07:29.962806 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-f3673a9a-openshift-default-57859ff45f-b5whn"] Jul 22 03:07:29.963372 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:07:29.963349 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3f04bcb1-b73d-4f1a-b60c-aefcd88bd212" containerName="istio-proxy" Jul 22 03:07:29.963372 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:07:29.963373 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="3f04bcb1-b73d-4f1a-b60c-aefcd88bd212" containerName="istio-proxy" Jul 22 03:07:29.963543 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:07:29.963389 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3f04bcb1-b73d-4f1a-b60c-aefcd88bd212" containerName="istio-proxy" Jul 22 03:07:29.963543 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:07:29.963397 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="3f04bcb1-b73d-4f1a-b60c-aefcd88bd212" containerName="istio-proxy" Jul 22 03:07:29.963543 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:07:29.963425 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="783af08b-92ba-423e-b5d4-25c6ad01d2ad" containerName="istio-proxy" Jul 22 03:07:29.963543 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:07:29.963431 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="783af08b-92ba-423e-b5d4-25c6ad01d2ad" containerName="istio-proxy" Jul 22 03:07:29.963543 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:07:29.963500 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="3f04bcb1-b73d-4f1a-b60c-aefcd88bd212" containerName="istio-proxy" Jul 22 03:07:29.963543 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:07:29.963507 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="3f04bcb1-b73d-4f1a-b60c-aefcd88bd212" containerName="istio-proxy" Jul 22 03:07:29.963543 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:07:29.963515 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="783af08b-92ba-423e-b5d4-25c6ad01d2ad" containerName="istio-proxy" Jul 22 03:07:29.966713 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:07:29.966696 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-f3673a9a-openshift-default-57859ff45f-b5whn" Jul 22 03:07:29.969760 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:07:29.969730 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-f3673a9a-openshift-default-dockercfg-qjcxs\"" Jul 22 03:07:29.970689 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:07:29.970671 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee6e4d28-0b3d-4c36-92a1-6b2f366a6748-istio-data\") pod \"e2e-derive-f3673a9a-openshift-default-57859ff45f-b5whn\" (UID: \"ee6e4d28-0b3d-4c36-92a1-6b2f366a6748\") " pod="openshift-ingress/e2e-derive-f3673a9a-openshift-default-57859ff45f-b5whn" Jul 22 03:07:29.970759 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:07:29.970702 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee6e4d28-0b3d-4c36-92a1-6b2f366a6748-istio-podinfo\") pod \"e2e-derive-f3673a9a-openshift-default-57859ff45f-b5whn\" (UID: \"ee6e4d28-0b3d-4c36-92a1-6b2f366a6748\") " pod="openshift-ingress/e2e-derive-f3673a9a-openshift-default-57859ff45f-b5whn" Jul 22 03:07:29.970759 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:07:29.970722 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee6e4d28-0b3d-4c36-92a1-6b2f366a6748-istio-token\") pod \"e2e-derive-f3673a9a-openshift-default-57859ff45f-b5whn\" (UID: \"ee6e4d28-0b3d-4c36-92a1-6b2f366a6748\") " pod="openshift-ingress/e2e-derive-f3673a9a-openshift-default-57859ff45f-b5whn" Jul 22 03:07:29.970843 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:07:29.970794 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee6e4d28-0b3d-4c36-92a1-6b2f366a6748-workload-certs\") pod \"e2e-derive-f3673a9a-openshift-default-57859ff45f-b5whn\" (UID: \"ee6e4d28-0b3d-4c36-92a1-6b2f366a6748\") " pod="openshift-ingress/e2e-derive-f3673a9a-openshift-default-57859ff45f-b5whn" Jul 22 03:07:29.970843 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:07:29.970827 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee6e4d28-0b3d-4c36-92a1-6b2f366a6748-istio-envoy\") pod \"e2e-derive-f3673a9a-openshift-default-57859ff45f-b5whn\" (UID: \"ee6e4d28-0b3d-4c36-92a1-6b2f366a6748\") " pod="openshift-ingress/e2e-derive-f3673a9a-openshift-default-57859ff45f-b5whn" Jul 22 03:07:29.970916 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:07:29.970855 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8lp9g\" (UniqueName: \"kubernetes.io/projected/ee6e4d28-0b3d-4c36-92a1-6b2f366a6748-kube-api-access-8lp9g\") pod \"e2e-derive-f3673a9a-openshift-default-57859ff45f-b5whn\" (UID: \"ee6e4d28-0b3d-4c36-92a1-6b2f366a6748\") " pod="openshift-ingress/e2e-derive-f3673a9a-openshift-default-57859ff45f-b5whn" Jul 22 03:07:29.970916 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:07:29.970900 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee6e4d28-0b3d-4c36-92a1-6b2f366a6748-credential-socket\") pod \"e2e-derive-f3673a9a-openshift-default-57859ff45f-b5whn\" (UID: \"ee6e4d28-0b3d-4c36-92a1-6b2f366a6748\") " pod="openshift-ingress/e2e-derive-f3673a9a-openshift-default-57859ff45f-b5whn" Jul 22 03:07:29.971003 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:07:29.970985 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee6e4d28-0b3d-4c36-92a1-6b2f366a6748-workload-socket\") pod \"e2e-derive-f3673a9a-openshift-default-57859ff45f-b5whn\" (UID: \"ee6e4d28-0b3d-4c36-92a1-6b2f366a6748\") " pod="openshift-ingress/e2e-derive-f3673a9a-openshift-default-57859ff45f-b5whn" Jul 22 03:07:29.971038 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:07:29.971021 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee6e4d28-0b3d-4c36-92a1-6b2f366a6748-istiod-ca-cert\") pod \"e2e-derive-f3673a9a-openshift-default-57859ff45f-b5whn\" (UID: \"ee6e4d28-0b3d-4c36-92a1-6b2f366a6748\") " pod="openshift-ingress/e2e-derive-f3673a9a-openshift-default-57859ff45f-b5whn" Jul 22 03:07:29.982395 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:07:29.982374 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-f3673a9a-openshift-default-57859ff45f-b5whn"] Jul 22 03:07:30.071988 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:07:30.071956 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee6e4d28-0b3d-4c36-92a1-6b2f366a6748-credential-socket\") pod \"e2e-derive-f3673a9a-openshift-default-57859ff45f-b5whn\" (UID: \"ee6e4d28-0b3d-4c36-92a1-6b2f366a6748\") " pod="openshift-ingress/e2e-derive-f3673a9a-openshift-default-57859ff45f-b5whn" Jul 22 03:07:30.072183 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:07:30.072026 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee6e4d28-0b3d-4c36-92a1-6b2f366a6748-workload-socket\") pod \"e2e-derive-f3673a9a-openshift-default-57859ff45f-b5whn\" (UID: \"ee6e4d28-0b3d-4c36-92a1-6b2f366a6748\") " pod="openshift-ingress/e2e-derive-f3673a9a-openshift-default-57859ff45f-b5whn" Jul 22 03:07:30.072183 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:07:30.072062 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee6e4d28-0b3d-4c36-92a1-6b2f366a6748-istiod-ca-cert\") pod \"e2e-derive-f3673a9a-openshift-default-57859ff45f-b5whn\" (UID: \"ee6e4d28-0b3d-4c36-92a1-6b2f366a6748\") " pod="openshift-ingress/e2e-derive-f3673a9a-openshift-default-57859ff45f-b5whn" Jul 22 03:07:30.072183 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:07:30.072104 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee6e4d28-0b3d-4c36-92a1-6b2f366a6748-istio-data\") pod \"e2e-derive-f3673a9a-openshift-default-57859ff45f-b5whn\" (UID: \"ee6e4d28-0b3d-4c36-92a1-6b2f366a6748\") " pod="openshift-ingress/e2e-derive-f3673a9a-openshift-default-57859ff45f-b5whn" Jul 22 03:07:30.072419 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:07:30.072389 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee6e4d28-0b3d-4c36-92a1-6b2f366a6748-credential-socket\") pod \"e2e-derive-f3673a9a-openshift-default-57859ff45f-b5whn\" (UID: \"ee6e4d28-0b3d-4c36-92a1-6b2f366a6748\") " pod="openshift-ingress/e2e-derive-f3673a9a-openshift-default-57859ff45f-b5whn" Jul 22 03:07:30.072606 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:07:30.072402 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee6e4d28-0b3d-4c36-92a1-6b2f366a6748-istio-data\") pod \"e2e-derive-f3673a9a-openshift-default-57859ff45f-b5whn\" (UID: \"ee6e4d28-0b3d-4c36-92a1-6b2f366a6748\") " pod="openshift-ingress/e2e-derive-f3673a9a-openshift-default-57859ff45f-b5whn" Jul 22 03:07:30.072606 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:07:30.072433 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee6e4d28-0b3d-4c36-92a1-6b2f366a6748-istio-podinfo\") pod \"e2e-derive-f3673a9a-openshift-default-57859ff45f-b5whn\" (UID: \"ee6e4d28-0b3d-4c36-92a1-6b2f366a6748\") " pod="openshift-ingress/e2e-derive-f3673a9a-openshift-default-57859ff45f-b5whn" Jul 22 03:07:30.072728 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:07:30.072643 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee6e4d28-0b3d-4c36-92a1-6b2f366a6748-istio-token\") pod \"e2e-derive-f3673a9a-openshift-default-57859ff45f-b5whn\" (UID: \"ee6e4d28-0b3d-4c36-92a1-6b2f366a6748\") " pod="openshift-ingress/e2e-derive-f3673a9a-openshift-default-57859ff45f-b5whn" Jul 22 03:07:30.072728 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:07:30.072717 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee6e4d28-0b3d-4c36-92a1-6b2f366a6748-workload-certs\") pod \"e2e-derive-f3673a9a-openshift-default-57859ff45f-b5whn\" (UID: \"ee6e4d28-0b3d-4c36-92a1-6b2f366a6748\") " pod="openshift-ingress/e2e-derive-f3673a9a-openshift-default-57859ff45f-b5whn" Jul 22 03:07:30.072824 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:07:30.072457 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee6e4d28-0b3d-4c36-92a1-6b2f366a6748-workload-socket\") pod \"e2e-derive-f3673a9a-openshift-default-57859ff45f-b5whn\" (UID: \"ee6e4d28-0b3d-4c36-92a1-6b2f366a6748\") " pod="openshift-ingress/e2e-derive-f3673a9a-openshift-default-57859ff45f-b5whn" Jul 22 03:07:30.072824 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:07:30.072758 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee6e4d28-0b3d-4c36-92a1-6b2f366a6748-istio-envoy\") pod \"e2e-derive-f3673a9a-openshift-default-57859ff45f-b5whn\" (UID: \"ee6e4d28-0b3d-4c36-92a1-6b2f366a6748\") " pod="openshift-ingress/e2e-derive-f3673a9a-openshift-default-57859ff45f-b5whn" Jul 22 03:07:30.072824 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:07:30.072790 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8lp9g\" (UniqueName: \"kubernetes.io/projected/ee6e4d28-0b3d-4c36-92a1-6b2f366a6748-kube-api-access-8lp9g\") pod \"e2e-derive-f3673a9a-openshift-default-57859ff45f-b5whn\" (UID: \"ee6e4d28-0b3d-4c36-92a1-6b2f366a6748\") " pod="openshift-ingress/e2e-derive-f3673a9a-openshift-default-57859ff45f-b5whn" Jul 22 03:07:30.072824 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:07:30.072814 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee6e4d28-0b3d-4c36-92a1-6b2f366a6748-istiod-ca-cert\") pod \"e2e-derive-f3673a9a-openshift-default-57859ff45f-b5whn\" (UID: \"ee6e4d28-0b3d-4c36-92a1-6b2f366a6748\") " pod="openshift-ingress/e2e-derive-f3673a9a-openshift-default-57859ff45f-b5whn" Jul 22 03:07:30.073317 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:07:30.073296 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee6e4d28-0b3d-4c36-92a1-6b2f366a6748-workload-certs\") pod \"e2e-derive-f3673a9a-openshift-default-57859ff45f-b5whn\" (UID: \"ee6e4d28-0b3d-4c36-92a1-6b2f366a6748\") " pod="openshift-ingress/e2e-derive-f3673a9a-openshift-default-57859ff45f-b5whn" Jul 22 03:07:30.075132 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:07:30.075052 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee6e4d28-0b3d-4c36-92a1-6b2f366a6748-istio-podinfo\") pod \"e2e-derive-f3673a9a-openshift-default-57859ff45f-b5whn\" (UID: \"ee6e4d28-0b3d-4c36-92a1-6b2f366a6748\") " pod="openshift-ingress/e2e-derive-f3673a9a-openshift-default-57859ff45f-b5whn" Jul 22 03:07:30.075270 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:07:30.075253 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee6e4d28-0b3d-4c36-92a1-6b2f366a6748-istio-envoy\") pod \"e2e-derive-f3673a9a-openshift-default-57859ff45f-b5whn\" (UID: \"ee6e4d28-0b3d-4c36-92a1-6b2f366a6748\") " pod="openshift-ingress/e2e-derive-f3673a9a-openshift-default-57859ff45f-b5whn" Jul 22 03:07:30.081216 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:07:30.081196 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee6e4d28-0b3d-4c36-92a1-6b2f366a6748-istio-token\") pod \"e2e-derive-f3673a9a-openshift-default-57859ff45f-b5whn\" (UID: \"ee6e4d28-0b3d-4c36-92a1-6b2f366a6748\") " pod="openshift-ingress/e2e-derive-f3673a9a-openshift-default-57859ff45f-b5whn" Jul 22 03:07:30.081477 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:07:30.081457 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8lp9g\" (UniqueName: \"kubernetes.io/projected/ee6e4d28-0b3d-4c36-92a1-6b2f366a6748-kube-api-access-8lp9g\") pod \"e2e-derive-f3673a9a-openshift-default-57859ff45f-b5whn\" (UID: \"ee6e4d28-0b3d-4c36-92a1-6b2f366a6748\") " pod="openshift-ingress/e2e-derive-f3673a9a-openshift-default-57859ff45f-b5whn" Jul 22 03:07:30.278488 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:07:30.278413 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-f3673a9a-openshift-default-57859ff45f-b5whn" Jul 22 03:07:30.431743 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:07:30.431706 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-f3673a9a-openshift-default-57859ff45f-b5whn"] Jul 22 03:07:30.434532 ip-10-0-133-31 kubenswrapper[2578]: W0722 03:07:30.432894 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee6e4d28_0b3d_4c36_92a1_6b2f366a6748.slice/crio-7bedcee5298016a6e733d3f36617db79fbeebfebd3f13d3bba62f624b0ecc397 WatchSource:0}: Error finding container 7bedcee5298016a6e733d3f36617db79fbeebfebd3f13d3bba62f624b0ecc397: Status 404 returned error can't find the container with id 7bedcee5298016a6e733d3f36617db79fbeebfebd3f13d3bba62f624b0ecc397 Jul 22 03:07:30.436764 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:07:30.436728 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 22 03:07:30.436860 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:07:30.436805 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 22 03:07:30.436860 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:07:30.436843 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 22 03:07:30.917197 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:07:30.917158 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-f3673a9a-openshift-default-57859ff45f-b5whn" event={"ID":"ee6e4d28-0b3d-4c36-92a1-6b2f366a6748","Type":"ContainerStarted","Data":"e16be3a9aea92822470921253402b06519427a2e845f9b1ad75909b2ac00e903"} Jul 22 03:07:30.917374 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:07:30.917243 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-f3673a9a-openshift-default-57859ff45f-b5whn" event={"ID":"ee6e4d28-0b3d-4c36-92a1-6b2f366a6748","Type":"ContainerStarted","Data":"7bedcee5298016a6e733d3f36617db79fbeebfebd3f13d3bba62f624b0ecc397"} Jul 22 03:07:30.942922 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:07:30.942876 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-f3673a9a-openshift-default-57859ff45f-b5whn" podStartSLOduration=1.942862783 podStartE2EDuration="1.942862783s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 03:07:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 03:07:30.940715195 +0000 UTC m=+1817.259890495" watchObservedRunningTime="2026-07-22 03:07:30.942862783 +0000 UTC m=+1817.262038082" Jul 22 03:07:31.279167 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:07:31.279088 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-f3673a9a-openshift-default-57859ff45f-b5whn" Jul 22 03:07:31.283789 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:07:31.283761 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-f3673a9a-openshift-default-57859ff45f-b5whn" Jul 22 03:07:31.922609 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:07:31.922580 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-f3673a9a-openshift-default-57859ff45f-b5whn" Jul 22 03:07:31.923528 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:07:31.923507 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-f3673a9a-openshift-default-57859ff45f-b5whn" Jul 22 03:07:34.462876 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:07:34.462849 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-b3e829f9-openshift-default-b987cc7d7-m7z4v" Jul 22 03:07:34.514219 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:07:34.514144 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de03c673-5a6c-4183-bec8-ee4302e65e47-istio-envoy\") pod \"de03c673-5a6c-4183-bec8-ee4302e65e47\" (UID: \"de03c673-5a6c-4183-bec8-ee4302e65e47\") " Jul 22 03:07:34.514219 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:07:34.514207 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de03c673-5a6c-4183-bec8-ee4302e65e47-istio-podinfo\") pod \"de03c673-5a6c-4183-bec8-ee4302e65e47\" (UID: \"de03c673-5a6c-4183-bec8-ee4302e65e47\") " Jul 22 03:07:34.514413 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:07:34.514260 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de03c673-5a6c-4183-bec8-ee4302e65e47-workload-certs\") pod \"de03c673-5a6c-4183-bec8-ee4302e65e47\" (UID: \"de03c673-5a6c-4183-bec8-ee4302e65e47\") " Jul 22 03:07:34.514413 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:07:34.514284 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de03c673-5a6c-4183-bec8-ee4302e65e47-istio-data\") pod \"de03c673-5a6c-4183-bec8-ee4302e65e47\" (UID: \"de03c673-5a6c-4183-bec8-ee4302e65e47\") " Jul 22 03:07:34.514413 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:07:34.514308 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de03c673-5a6c-4183-bec8-ee4302e65e47-workload-socket\") pod \"de03c673-5a6c-4183-bec8-ee4302e65e47\" (UID: \"de03c673-5a6c-4183-bec8-ee4302e65e47\") " Jul 22 03:07:34.514413 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:07:34.514389 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de03c673-5a6c-4183-bec8-ee4302e65e47-istiod-ca-cert\") pod \"de03c673-5a6c-4183-bec8-ee4302e65e47\" (UID: \"de03c673-5a6c-4183-bec8-ee4302e65e47\") " Jul 22 03:07:34.514647 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:07:34.514430 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de03c673-5a6c-4183-bec8-ee4302e65e47-credential-socket\") pod \"de03c673-5a6c-4183-bec8-ee4302e65e47\" (UID: \"de03c673-5a6c-4183-bec8-ee4302e65e47\") " Jul 22 03:07:34.514647 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:07:34.514461 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de03c673-5a6c-4183-bec8-ee4302e65e47-istio-token\") pod \"de03c673-5a6c-4183-bec8-ee4302e65e47\" (UID: \"de03c673-5a6c-4183-bec8-ee4302e65e47\") " Jul 22 03:07:34.514647 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:07:34.514485 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4252k\" (UniqueName: \"kubernetes.io/projected/de03c673-5a6c-4183-bec8-ee4302e65e47-kube-api-access-4252k\") pod \"de03c673-5a6c-4183-bec8-ee4302e65e47\" (UID: \"de03c673-5a6c-4183-bec8-ee4302e65e47\") " Jul 22 03:07:34.514647 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:07:34.514497 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de03c673-5a6c-4183-bec8-ee4302e65e47-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "de03c673-5a6c-4183-bec8-ee4302e65e47" (UID: "de03c673-5a6c-4183-bec8-ee4302e65e47"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 03:07:34.514851 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:07:34.514639 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de03c673-5a6c-4183-bec8-ee4302e65e47-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "de03c673-5a6c-4183-bec8-ee4302e65e47" (UID: "de03c673-5a6c-4183-bec8-ee4302e65e47"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 03:07:34.514851 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:07:34.514788 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de03c673-5a6c-4183-bec8-ee4302e65e47-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "de03c673-5a6c-4183-bec8-ee4302e65e47" (UID: "de03c673-5a6c-4183-bec8-ee4302e65e47"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 03:07:34.514851 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:07:34.514814 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de03c673-5a6c-4183-bec8-ee4302e65e47-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "de03c673-5a6c-4183-bec8-ee4302e65e47" (UID: "de03c673-5a6c-4183-bec8-ee4302e65e47"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 03:07:34.515102 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:07:34.515078 2578 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de03c673-5a6c-4183-bec8-ee4302e65e47-workload-certs\") on node \"ip-10-0-133-31.ec2.internal\" DevicePath \"\"" Jul 22 03:07:34.515208 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:07:34.515107 2578 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de03c673-5a6c-4183-bec8-ee4302e65e47-workload-socket\") on node \"ip-10-0-133-31.ec2.internal\" DevicePath \"\"" Jul 22 03:07:34.515208 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:07:34.515122 2578 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de03c673-5a6c-4183-bec8-ee4302e65e47-istiod-ca-cert\") on node \"ip-10-0-133-31.ec2.internal\" DevicePath \"\"" Jul 22 03:07:34.515208 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:07:34.515138 2578 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de03c673-5a6c-4183-bec8-ee4302e65e47-credential-socket\") on node \"ip-10-0-133-31.ec2.internal\" DevicePath \"\"" Jul 22 03:07:34.515452 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:07:34.515429 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de03c673-5a6c-4183-bec8-ee4302e65e47-istio-data" (OuterVolumeSpecName: "istio-data") pod "de03c673-5a6c-4183-bec8-ee4302e65e47" (UID: "de03c673-5a6c-4183-bec8-ee4302e65e47"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 03:07:34.516381 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:07:34.516363 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/de03c673-5a6c-4183-bec8-ee4302e65e47-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "de03c673-5a6c-4183-bec8-ee4302e65e47" (UID: "de03c673-5a6c-4183-bec8-ee4302e65e47"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 22 03:07:34.516523 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:07:34.516509 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de03c673-5a6c-4183-bec8-ee4302e65e47-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "de03c673-5a6c-4183-bec8-ee4302e65e47" (UID: "de03c673-5a6c-4183-bec8-ee4302e65e47"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 03:07:34.517140 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:07:34.517116 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de03c673-5a6c-4183-bec8-ee4302e65e47-kube-api-access-4252k" (OuterVolumeSpecName: "kube-api-access-4252k") pod "de03c673-5a6c-4183-bec8-ee4302e65e47" (UID: "de03c673-5a6c-4183-bec8-ee4302e65e47"). InnerVolumeSpecName "kube-api-access-4252k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 03:07:34.517140 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:07:34.517123 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de03c673-5a6c-4183-bec8-ee4302e65e47-istio-token" (OuterVolumeSpecName: "istio-token") pod "de03c673-5a6c-4183-bec8-ee4302e65e47" (UID: "de03c673-5a6c-4183-bec8-ee4302e65e47"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 03:07:34.616477 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:07:34.616438 2578 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de03c673-5a6c-4183-bec8-ee4302e65e47-istio-token\") on node \"ip-10-0-133-31.ec2.internal\" DevicePath \"\"" Jul 22 03:07:34.616477 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:07:34.616475 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4252k\" (UniqueName: \"kubernetes.io/projected/de03c673-5a6c-4183-bec8-ee4302e65e47-kube-api-access-4252k\") on node \"ip-10-0-133-31.ec2.internal\" DevicePath \"\"" Jul 22 03:07:34.616477 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:07:34.616486 2578 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de03c673-5a6c-4183-bec8-ee4302e65e47-istio-envoy\") on node \"ip-10-0-133-31.ec2.internal\" DevicePath \"\"" Jul 22 03:07:34.616477 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:07:34.616495 2578 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de03c673-5a6c-4183-bec8-ee4302e65e47-istio-podinfo\") on node \"ip-10-0-133-31.ec2.internal\" DevicePath \"\"" Jul 22 03:07:34.616757 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:07:34.616503 2578 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de03c673-5a6c-4183-bec8-ee4302e65e47-istio-data\") on node \"ip-10-0-133-31.ec2.internal\" DevicePath \"\"" Jul 22 03:07:34.941525 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:07:34.941488 2578 generic.go:356] "Generic (PLEG): container finished" podID="de03c673-5a6c-4183-bec8-ee4302e65e47" containerID="b8df0af859da71d715eb3a63585aaea120c842a5ac04e4df2e8aada4f6a4a0eb" exitCode=0 Jul 22 03:07:34.941690 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:07:34.941540 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-b3e829f9-openshift-default-b987cc7d7-m7z4v" Jul 22 03:07:34.941690 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:07:34.941580 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b3e829f9-openshift-default-b987cc7d7-m7z4v" event={"ID":"de03c673-5a6c-4183-bec8-ee4302e65e47","Type":"ContainerDied","Data":"b8df0af859da71d715eb3a63585aaea120c842a5ac04e4df2e8aada4f6a4a0eb"} Jul 22 03:07:34.941690 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:07:34.941642 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b3e829f9-openshift-default-b987cc7d7-m7z4v" event={"ID":"de03c673-5a6c-4183-bec8-ee4302e65e47","Type":"ContainerDied","Data":"0fb0ba71725e978c9ad4c31f2d8b9cc295fe87c134ebfdc5fb06f273c75ee5e6"} Jul 22 03:07:34.941690 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:07:34.941666 2578 scope.go:117] "RemoveContainer" containerID="b8df0af859da71d715eb3a63585aaea120c842a5ac04e4df2e8aada4f6a4a0eb" Jul 22 03:07:34.951354 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:07:34.951339 2578 scope.go:117] "RemoveContainer" containerID="8d54d702fadd5561ad5a4b17dad37c23037c42193eca4821740da410260c2148" Jul 22 03:07:34.959106 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:07:34.959087 2578 scope.go:117] "RemoveContainer" containerID="b8df0af859da71d715eb3a63585aaea120c842a5ac04e4df2e8aada4f6a4a0eb" Jul 22 03:07:34.959351 ip-10-0-133-31 kubenswrapper[2578]: E0722 03:07:34.959331 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b8df0af859da71d715eb3a63585aaea120c842a5ac04e4df2e8aada4f6a4a0eb\": container with ID starting with b8df0af859da71d715eb3a63585aaea120c842a5ac04e4df2e8aada4f6a4a0eb not found: ID does not exist" containerID="b8df0af859da71d715eb3a63585aaea120c842a5ac04e4df2e8aada4f6a4a0eb" Jul 22 03:07:34.959447 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:07:34.959359 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b8df0af859da71d715eb3a63585aaea120c842a5ac04e4df2e8aada4f6a4a0eb"} err="failed to get container status \"b8df0af859da71d715eb3a63585aaea120c842a5ac04e4df2e8aada4f6a4a0eb\": rpc error: code = NotFound desc = could not find container \"b8df0af859da71d715eb3a63585aaea120c842a5ac04e4df2e8aada4f6a4a0eb\": container with ID starting with b8df0af859da71d715eb3a63585aaea120c842a5ac04e4df2e8aada4f6a4a0eb not found: ID does not exist" Jul 22 03:07:34.959447 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:07:34.959376 2578 scope.go:117] "RemoveContainer" containerID="8d54d702fadd5561ad5a4b17dad37c23037c42193eca4821740da410260c2148" Jul 22 03:07:34.959621 ip-10-0-133-31 kubenswrapper[2578]: E0722 03:07:34.959605 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8d54d702fadd5561ad5a4b17dad37c23037c42193eca4821740da410260c2148\": container with ID starting with 8d54d702fadd5561ad5a4b17dad37c23037c42193eca4821740da410260c2148 not found: ID does not exist" containerID="8d54d702fadd5561ad5a4b17dad37c23037c42193eca4821740da410260c2148" Jul 22 03:07:34.959667 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:07:34.959626 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d54d702fadd5561ad5a4b17dad37c23037c42193eca4821740da410260c2148"} err="failed to get container status \"8d54d702fadd5561ad5a4b17dad37c23037c42193eca4821740da410260c2148\": rpc error: code = NotFound desc = could not find container \"8d54d702fadd5561ad5a4b17dad37c23037c42193eca4821740da410260c2148\": container with ID starting with 8d54d702fadd5561ad5a4b17dad37c23037c42193eca4821740da410260c2148 not found: ID does not exist" Jul 22 03:07:34.966836 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:07:34.966817 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-b3e829f9-openshift-default-b987cc7d7-m7z4v"] Jul 22 03:07:34.970845 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:07:34.970828 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-b3e829f9-openshift-default-b987cc7d7-m7z4v"] Jul 22 03:07:36.146967 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:07:36.146933 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="de03c673-5a6c-4183-bec8-ee4302e65e47" path="/var/lib/kubelet/pods/de03c673-5a6c-4183-bec8-ee4302e65e47/volumes" Jul 22 03:07:43.992299 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:07:43.992265 2578 generic.go:356] "Generic (PLEG): container finished" podID="ee6e4d28-0b3d-4c36-92a1-6b2f366a6748" containerID="e16be3a9aea92822470921253402b06519427a2e845f9b1ad75909b2ac00e903" exitCode=0 Jul 22 03:07:43.992770 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:07:43.992339 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-f3673a9a-openshift-default-57859ff45f-b5whn" event={"ID":"ee6e4d28-0b3d-4c36-92a1-6b2f366a6748","Type":"ContainerDied","Data":"e16be3a9aea92822470921253402b06519427a2e845f9b1ad75909b2ac00e903"} Jul 22 03:07:43.992908 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:07:43.992890 2578 scope.go:117] "RemoveContainer" containerID="e16be3a9aea92822470921253402b06519427a2e845f9b1ad75909b2ac00e903" Jul 22 03:07:43.994018 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:07:43.993987 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 22 03:07:43.994107 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:07:43.994092 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 22 03:07:43.994156 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:07:43.994139 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 22 03:07:45.004132 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:07:45.004084 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-f3673a9a-openshift-default-57859ff45f-b5whn" event={"ID":"ee6e4d28-0b3d-4c36-92a1-6b2f366a6748","Type":"ContainerStarted","Data":"b31f0f90ba9ec1a7ebf2a2b0fa49a3f88e23432e5927e18afc2f626c9ed7b13e"} Jul 22 03:07:45.279167 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:07:45.279052 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-f3673a9a-openshift-default-57859ff45f-b5whn" Jul 22 03:07:45.279801 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:07:45.279775 2578 patch_prober.go:28] interesting pod/e2e-derive-f3673a9a-openshift-default-57859ff45f-b5whn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body= Jul 22 03:07:45.279922 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:07:45.279854 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-f3673a9a-openshift-default-57859ff45f-b5whn" podUID="ee6e4d28-0b3d-4c36-92a1-6b2f366a6748" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" Jul 22 03:07:46.279299 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:07:46.279266 2578 patch_prober.go:28] interesting pod/e2e-derive-f3673a9a-openshift-default-57859ff45f-b5whn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body= Jul 22 03:07:46.279698 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:07:46.279318 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-f3673a9a-openshift-default-57859ff45f-b5whn" podUID="ee6e4d28-0b3d-4c36-92a1-6b2f366a6748" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" Jul 22 03:07:46.922940 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:07:46.922894 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-f3673a9a-openshift-default-57859ff45f-b5whn" Jul 22 03:07:47.279133 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:07:47.278964 2578 patch_prober.go:28] interesting pod/e2e-derive-f3673a9a-openshift-default-57859ff45f-b5whn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body= Jul 22 03:07:47.279133 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:07:47.279013 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-f3673a9a-openshift-default-57859ff45f-b5whn" podUID="ee6e4d28-0b3d-4c36-92a1-6b2f366a6748" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" Jul 22 03:07:48.278749 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:07:48.278720 2578 patch_prober.go:28] interesting pod/e2e-derive-f3673a9a-openshift-default-57859ff45f-b5whn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body= Jul 22 03:07:48.279214 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:07:48.278777 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-f3673a9a-openshift-default-57859ff45f-b5whn" podUID="ee6e4d28-0b3d-4c36-92a1-6b2f366a6748" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" Jul 22 03:07:49.282695 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:07:49.282662 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-f3673a9a-openshift-default-57859ff45f-b5whn" Jul 22 03:07:49.283625 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:07:49.283602 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-f3673a9a-openshift-default-57859ff45f-b5whn" Jul 22 03:08:12.369739 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:08:12.369709 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-f3673a9a-openshift-default-57859ff45f-b5whn"] Jul 22 03:08:12.370219 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:08:12.369899 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-f3673a9a-openshift-default-57859ff45f-b5whn" podUID="ee6e4d28-0b3d-4c36-92a1-6b2f366a6748" containerName="istio-proxy" containerID="cri-o://b31f0f90ba9ec1a7ebf2a2b0fa49a3f88e23432e5927e18afc2f626c9ed7b13e" gracePeriod=30 Jul 22 03:08:17.626956 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:08:17.626927 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-f3673a9a-openshift-default-57859ff45f-b5whn" Jul 22 03:08:17.692767 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:08:17.692695 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee6e4d28-0b3d-4c36-92a1-6b2f366a6748-istio-data\") pod \"ee6e4d28-0b3d-4c36-92a1-6b2f366a6748\" (UID: \"ee6e4d28-0b3d-4c36-92a1-6b2f366a6748\") " Jul 22 03:08:17.692767 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:08:17.692730 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee6e4d28-0b3d-4c36-92a1-6b2f366a6748-credential-socket\") pod \"ee6e4d28-0b3d-4c36-92a1-6b2f366a6748\" (UID: \"ee6e4d28-0b3d-4c36-92a1-6b2f366a6748\") " Jul 22 03:08:17.692767 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:08:17.692763 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee6e4d28-0b3d-4c36-92a1-6b2f366a6748-istio-podinfo\") pod \"ee6e4d28-0b3d-4c36-92a1-6b2f366a6748\" (UID: \"ee6e4d28-0b3d-4c36-92a1-6b2f366a6748\") " Jul 22 03:08:17.693030 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:08:17.692799 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee6e4d28-0b3d-4c36-92a1-6b2f366a6748-istio-envoy\") pod \"ee6e4d28-0b3d-4c36-92a1-6b2f366a6748\" (UID: \"ee6e4d28-0b3d-4c36-92a1-6b2f366a6748\") " Jul 22 03:08:17.693030 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:08:17.692819 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee6e4d28-0b3d-4c36-92a1-6b2f366a6748-istio-token\") pod \"ee6e4d28-0b3d-4c36-92a1-6b2f366a6748\" (UID: \"ee6e4d28-0b3d-4c36-92a1-6b2f366a6748\") " Jul 22 03:08:17.693030 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:08:17.692864 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee6e4d28-0b3d-4c36-92a1-6b2f366a6748-istiod-ca-cert\") pod \"ee6e4d28-0b3d-4c36-92a1-6b2f366a6748\" (UID: \"ee6e4d28-0b3d-4c36-92a1-6b2f366a6748\") " Jul 22 03:08:17.693030 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:08:17.693009 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee6e4d28-0b3d-4c36-92a1-6b2f366a6748-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "ee6e4d28-0b3d-4c36-92a1-6b2f366a6748" (UID: "ee6e4d28-0b3d-4c36-92a1-6b2f366a6748"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 03:08:17.693238 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:08:17.693117 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee6e4d28-0b3d-4c36-92a1-6b2f366a6748-workload-socket\") pod \"ee6e4d28-0b3d-4c36-92a1-6b2f366a6748\" (UID: \"ee6e4d28-0b3d-4c36-92a1-6b2f366a6748\") " Jul 22 03:08:17.693238 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:08:17.693180 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8lp9g\" (UniqueName: \"kubernetes.io/projected/ee6e4d28-0b3d-4c36-92a1-6b2f366a6748-kube-api-access-8lp9g\") pod \"ee6e4d28-0b3d-4c36-92a1-6b2f366a6748\" (UID: \"ee6e4d28-0b3d-4c36-92a1-6b2f366a6748\") " Jul 22 03:08:17.693238 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:08:17.693196 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee6e4d28-0b3d-4c36-92a1-6b2f366a6748-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "ee6e4d28-0b3d-4c36-92a1-6b2f366a6748" (UID: "ee6e4d28-0b3d-4c36-92a1-6b2f366a6748"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 03:08:17.693238 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:08:17.693222 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee6e4d28-0b3d-4c36-92a1-6b2f366a6748-workload-certs\") pod \"ee6e4d28-0b3d-4c36-92a1-6b2f366a6748\" (UID: \"ee6e4d28-0b3d-4c36-92a1-6b2f366a6748\") " Jul 22 03:08:17.693480 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:08:17.693454 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee6e4d28-0b3d-4c36-92a1-6b2f366a6748-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "ee6e4d28-0b3d-4c36-92a1-6b2f366a6748" (UID: "ee6e4d28-0b3d-4c36-92a1-6b2f366a6748"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 03:08:17.693660 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:08:17.693590 2578 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee6e4d28-0b3d-4c36-92a1-6b2f366a6748-credential-socket\") on node \"ip-10-0-133-31.ec2.internal\" DevicePath \"\"" Jul 22 03:08:17.693869 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:08:17.693839 2578 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee6e4d28-0b3d-4c36-92a1-6b2f366a6748-istiod-ca-cert\") on node \"ip-10-0-133-31.ec2.internal\" DevicePath \"\"" Jul 22 03:08:17.694021 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:08:17.693704 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee6e4d28-0b3d-4c36-92a1-6b2f366a6748-istio-data" (OuterVolumeSpecName: "istio-data") pod "ee6e4d28-0b3d-4c36-92a1-6b2f366a6748" (UID: "ee6e4d28-0b3d-4c36-92a1-6b2f366a6748"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 03:08:17.694400 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:08:17.694231 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee6e4d28-0b3d-4c36-92a1-6b2f366a6748-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "ee6e4d28-0b3d-4c36-92a1-6b2f366a6748" (UID: "ee6e4d28-0b3d-4c36-92a1-6b2f366a6748"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 03:08:17.696006 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:08:17.695955 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ee6e4d28-0b3d-4c36-92a1-6b2f366a6748-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "ee6e4d28-0b3d-4c36-92a1-6b2f366a6748" (UID: "ee6e4d28-0b3d-4c36-92a1-6b2f366a6748"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 22 03:08:17.696164 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:08:17.696059 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee6e4d28-0b3d-4c36-92a1-6b2f366a6748-kube-api-access-8lp9g" (OuterVolumeSpecName: "kube-api-access-8lp9g") pod "ee6e4d28-0b3d-4c36-92a1-6b2f366a6748" (UID: "ee6e4d28-0b3d-4c36-92a1-6b2f366a6748"). InnerVolumeSpecName "kube-api-access-8lp9g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 03:08:17.696164 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:08:17.696120 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee6e4d28-0b3d-4c36-92a1-6b2f366a6748-istio-token" (OuterVolumeSpecName: "istio-token") pod "ee6e4d28-0b3d-4c36-92a1-6b2f366a6748" (UID: "ee6e4d28-0b3d-4c36-92a1-6b2f366a6748"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 03:08:17.696251 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:08:17.696172 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee6e4d28-0b3d-4c36-92a1-6b2f366a6748-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "ee6e4d28-0b3d-4c36-92a1-6b2f366a6748" (UID: "ee6e4d28-0b3d-4c36-92a1-6b2f366a6748"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 03:08:17.794394 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:08:17.794368 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8lp9g\" (UniqueName: \"kubernetes.io/projected/ee6e4d28-0b3d-4c36-92a1-6b2f366a6748-kube-api-access-8lp9g\") on node \"ip-10-0-133-31.ec2.internal\" DevicePath \"\"" Jul 22 03:08:17.794394 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:08:17.794391 2578 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee6e4d28-0b3d-4c36-92a1-6b2f366a6748-workload-certs\") on node \"ip-10-0-133-31.ec2.internal\" DevicePath \"\"" Jul 22 03:08:17.794519 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:08:17.794400 2578 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee6e4d28-0b3d-4c36-92a1-6b2f366a6748-istio-data\") on node \"ip-10-0-133-31.ec2.internal\" DevicePath \"\"" Jul 22 03:08:17.794519 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:08:17.794407 2578 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee6e4d28-0b3d-4c36-92a1-6b2f366a6748-istio-podinfo\") on node \"ip-10-0-133-31.ec2.internal\" DevicePath \"\"" Jul 22 03:08:17.794519 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:08:17.794416 2578 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee6e4d28-0b3d-4c36-92a1-6b2f366a6748-istio-envoy\") on node \"ip-10-0-133-31.ec2.internal\" DevicePath \"\"" Jul 22 03:08:17.794519 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:08:17.794423 2578 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee6e4d28-0b3d-4c36-92a1-6b2f366a6748-istio-token\") on node \"ip-10-0-133-31.ec2.internal\" DevicePath \"\"" Jul 22 03:08:17.794519 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:08:17.794430 2578 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee6e4d28-0b3d-4c36-92a1-6b2f366a6748-workload-socket\") on node \"ip-10-0-133-31.ec2.internal\" DevicePath \"\"" Jul 22 03:08:18.180977 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:08:18.180947 2578 generic.go:356] "Generic (PLEG): container finished" podID="ee6e4d28-0b3d-4c36-92a1-6b2f366a6748" containerID="b31f0f90ba9ec1a7ebf2a2b0fa49a3f88e23432e5927e18afc2f626c9ed7b13e" exitCode=0 Jul 22 03:08:18.181091 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:08:18.180995 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-f3673a9a-openshift-default-57859ff45f-b5whn" Jul 22 03:08:18.181091 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:08:18.181025 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-f3673a9a-openshift-default-57859ff45f-b5whn" event={"ID":"ee6e4d28-0b3d-4c36-92a1-6b2f366a6748","Type":"ContainerDied","Data":"b31f0f90ba9ec1a7ebf2a2b0fa49a3f88e23432e5927e18afc2f626c9ed7b13e"} Jul 22 03:08:18.181091 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:08:18.181077 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-f3673a9a-openshift-default-57859ff45f-b5whn" event={"ID":"ee6e4d28-0b3d-4c36-92a1-6b2f366a6748","Type":"ContainerDied","Data":"7bedcee5298016a6e733d3f36617db79fbeebfebd3f13d3bba62f624b0ecc397"} Jul 22 03:08:18.181194 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:08:18.181107 2578 scope.go:117] "RemoveContainer" containerID="b31f0f90ba9ec1a7ebf2a2b0fa49a3f88e23432e5927e18afc2f626c9ed7b13e" Jul 22 03:08:18.190196 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:08:18.190180 2578 scope.go:117] "RemoveContainer" containerID="e16be3a9aea92822470921253402b06519427a2e845f9b1ad75909b2ac00e903" Jul 22 03:08:18.197988 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:08:18.197952 2578 scope.go:117] "RemoveContainer" containerID="b31f0f90ba9ec1a7ebf2a2b0fa49a3f88e23432e5927e18afc2f626c9ed7b13e" Jul 22 03:08:18.198358 ip-10-0-133-31 kubenswrapper[2578]: E0722 03:08:18.198322 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b31f0f90ba9ec1a7ebf2a2b0fa49a3f88e23432e5927e18afc2f626c9ed7b13e\": container with ID starting with b31f0f90ba9ec1a7ebf2a2b0fa49a3f88e23432e5927e18afc2f626c9ed7b13e not found: ID does not exist" containerID="b31f0f90ba9ec1a7ebf2a2b0fa49a3f88e23432e5927e18afc2f626c9ed7b13e" Jul 22 03:08:18.198473 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:08:18.198367 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b31f0f90ba9ec1a7ebf2a2b0fa49a3f88e23432e5927e18afc2f626c9ed7b13e"} err="failed to get container status \"b31f0f90ba9ec1a7ebf2a2b0fa49a3f88e23432e5927e18afc2f626c9ed7b13e\": rpc error: code = NotFound desc = could not find container \"b31f0f90ba9ec1a7ebf2a2b0fa49a3f88e23432e5927e18afc2f626c9ed7b13e\": container with ID starting with b31f0f90ba9ec1a7ebf2a2b0fa49a3f88e23432e5927e18afc2f626c9ed7b13e not found: ID does not exist" Jul 22 03:08:18.198473 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:08:18.198389 2578 scope.go:117] "RemoveContainer" containerID="e16be3a9aea92822470921253402b06519427a2e845f9b1ad75909b2ac00e903" Jul 22 03:08:18.198815 ip-10-0-133-31 kubenswrapper[2578]: E0722 03:08:18.198787 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e16be3a9aea92822470921253402b06519427a2e845f9b1ad75909b2ac00e903\": container with ID starting with e16be3a9aea92822470921253402b06519427a2e845f9b1ad75909b2ac00e903 not found: ID does not exist" containerID="e16be3a9aea92822470921253402b06519427a2e845f9b1ad75909b2ac00e903" Jul 22 03:08:18.198916 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:08:18.198823 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e16be3a9aea92822470921253402b06519427a2e845f9b1ad75909b2ac00e903"} err="failed to get container status \"e16be3a9aea92822470921253402b06519427a2e845f9b1ad75909b2ac00e903\": rpc error: code = NotFound desc = could not find container \"e16be3a9aea92822470921253402b06519427a2e845f9b1ad75909b2ac00e903\": container with ID starting with e16be3a9aea92822470921253402b06519427a2e845f9b1ad75909b2ac00e903 not found: ID does not exist" Jul 22 03:08:18.201314 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:08:18.201294 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-f3673a9a-openshift-default-57859ff45f-b5whn"] Jul 22 03:08:18.207458 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:08:18.207438 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-f3673a9a-openshift-default-57859ff45f-b5whn"] Jul 22 03:08:20.141973 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:08:20.141941 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ee6e4d28-0b3d-4c36-92a1-6b2f366a6748" path="/var/lib/kubelet/pods/ee6e4d28-0b3d-4c36-92a1-6b2f366a6748/volumes" Jul 22 03:09:53.719663 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:09:53.719630 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-30037ff3-openshift-default-7d85bbf5b7-w8297"] Jul 22 03:09:53.720119 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:09:53.719974 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="de03c673-5a6c-4183-bec8-ee4302e65e47" containerName="istio-proxy" Jul 22 03:09:53.720119 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:09:53.719985 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="de03c673-5a6c-4183-bec8-ee4302e65e47" containerName="istio-proxy" Jul 22 03:09:53.720119 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:09:53.719997 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ee6e4d28-0b3d-4c36-92a1-6b2f366a6748" containerName="istio-proxy" Jul 22 03:09:53.720119 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:09:53.720002 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="ee6e4d28-0b3d-4c36-92a1-6b2f366a6748" containerName="istio-proxy" Jul 22 03:09:53.720119 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:09:53.720066 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="de03c673-5a6c-4183-bec8-ee4302e65e47" containerName="istio-proxy" Jul 22 03:09:53.720119 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:09:53.720078 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="ee6e4d28-0b3d-4c36-92a1-6b2f366a6748" containerName="istio-proxy" Jul 22 03:09:53.722965 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:09:53.722947 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-30037ff3-openshift-default-7d85bbf5b7-w8297" Jul 22 03:09:53.726864 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:09:53.726840 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-30037ff3-openshift-default-dockercfg-kl99j\"" Jul 22 03:09:53.750592 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:09:53.750549 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-30037ff3-openshift-default-7d85bbf5b7-w8297"] Jul 22 03:09:53.835139 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:09:53.835102 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d246310-a408-4cee-811a-9181b88b14d7-istio-envoy\") pod \"e2e-mt-30037ff3-openshift-default-7d85bbf5b7-w8297\" (UID: \"9d246310-a408-4cee-811a-9181b88b14d7\") " pod="openshift-ingress/e2e-mt-30037ff3-openshift-default-7d85bbf5b7-w8297" Jul 22 03:09:53.835335 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:09:53.835142 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d246310-a408-4cee-811a-9181b88b14d7-istio-token\") pod \"e2e-mt-30037ff3-openshift-default-7d85bbf5b7-w8297\" (UID: \"9d246310-a408-4cee-811a-9181b88b14d7\") " pod="openshift-ingress/e2e-mt-30037ff3-openshift-default-7d85bbf5b7-w8297" Jul 22 03:09:53.835335 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:09:53.835170 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d246310-a408-4cee-811a-9181b88b14d7-workload-certs\") pod \"e2e-mt-30037ff3-openshift-default-7d85bbf5b7-w8297\" (UID: \"9d246310-a408-4cee-811a-9181b88b14d7\") " pod="openshift-ingress/e2e-mt-30037ff3-openshift-default-7d85bbf5b7-w8297" Jul 22 03:09:53.835335 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:09:53.835245 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d246310-a408-4cee-811a-9181b88b14d7-credential-socket\") pod \"e2e-mt-30037ff3-openshift-default-7d85bbf5b7-w8297\" (UID: \"9d246310-a408-4cee-811a-9181b88b14d7\") " pod="openshift-ingress/e2e-mt-30037ff3-openshift-default-7d85bbf5b7-w8297" Jul 22 03:09:53.835335 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:09:53.835271 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-75rw8\" (UniqueName: \"kubernetes.io/projected/9d246310-a408-4cee-811a-9181b88b14d7-kube-api-access-75rw8\") pod \"e2e-mt-30037ff3-openshift-default-7d85bbf5b7-w8297\" (UID: \"9d246310-a408-4cee-811a-9181b88b14d7\") " pod="openshift-ingress/e2e-mt-30037ff3-openshift-default-7d85bbf5b7-w8297" Jul 22 03:09:53.835335 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:09:53.835301 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d246310-a408-4cee-811a-9181b88b14d7-istiod-ca-cert\") pod \"e2e-mt-30037ff3-openshift-default-7d85bbf5b7-w8297\" (UID: \"9d246310-a408-4cee-811a-9181b88b14d7\") " pod="openshift-ingress/e2e-mt-30037ff3-openshift-default-7d85bbf5b7-w8297" Jul 22 03:09:53.835619 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:09:53.835372 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d246310-a408-4cee-811a-9181b88b14d7-istio-data\") pod \"e2e-mt-30037ff3-openshift-default-7d85bbf5b7-w8297\" (UID: \"9d246310-a408-4cee-811a-9181b88b14d7\") " pod="openshift-ingress/e2e-mt-30037ff3-openshift-default-7d85bbf5b7-w8297" Jul 22 03:09:53.835619 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:09:53.835414 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d246310-a408-4cee-811a-9181b88b14d7-workload-socket\") pod \"e2e-mt-30037ff3-openshift-default-7d85bbf5b7-w8297\" (UID: \"9d246310-a408-4cee-811a-9181b88b14d7\") " pod="openshift-ingress/e2e-mt-30037ff3-openshift-default-7d85bbf5b7-w8297" Jul 22 03:09:53.835619 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:09:53.835441 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d246310-a408-4cee-811a-9181b88b14d7-istio-podinfo\") pod \"e2e-mt-30037ff3-openshift-default-7d85bbf5b7-w8297\" (UID: \"9d246310-a408-4cee-811a-9181b88b14d7\") " pod="openshift-ingress/e2e-mt-30037ff3-openshift-default-7d85bbf5b7-w8297" Jul 22 03:09:53.935897 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:09:53.935865 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d246310-a408-4cee-811a-9181b88b14d7-istiod-ca-cert\") pod \"e2e-mt-30037ff3-openshift-default-7d85bbf5b7-w8297\" (UID: \"9d246310-a408-4cee-811a-9181b88b14d7\") " pod="openshift-ingress/e2e-mt-30037ff3-openshift-default-7d85bbf5b7-w8297" Jul 22 03:09:53.936063 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:09:53.935916 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d246310-a408-4cee-811a-9181b88b14d7-istio-data\") pod \"e2e-mt-30037ff3-openshift-default-7d85bbf5b7-w8297\" (UID: \"9d246310-a408-4cee-811a-9181b88b14d7\") " pod="openshift-ingress/e2e-mt-30037ff3-openshift-default-7d85bbf5b7-w8297" Jul 22 03:09:53.936063 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:09:53.935940 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d246310-a408-4cee-811a-9181b88b14d7-workload-socket\") pod \"e2e-mt-30037ff3-openshift-default-7d85bbf5b7-w8297\" (UID: \"9d246310-a408-4cee-811a-9181b88b14d7\") " pod="openshift-ingress/e2e-mt-30037ff3-openshift-default-7d85bbf5b7-w8297" Jul 22 03:09:53.936063 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:09:53.935961 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d246310-a408-4cee-811a-9181b88b14d7-istio-podinfo\") pod \"e2e-mt-30037ff3-openshift-default-7d85bbf5b7-w8297\" (UID: \"9d246310-a408-4cee-811a-9181b88b14d7\") " pod="openshift-ingress/e2e-mt-30037ff3-openshift-default-7d85bbf5b7-w8297" Jul 22 03:09:53.936063 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:09:53.936021 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d246310-a408-4cee-811a-9181b88b14d7-istio-envoy\") pod \"e2e-mt-30037ff3-openshift-default-7d85bbf5b7-w8297\" (UID: \"9d246310-a408-4cee-811a-9181b88b14d7\") " pod="openshift-ingress/e2e-mt-30037ff3-openshift-default-7d85bbf5b7-w8297" Jul 22 03:09:53.936063 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:09:53.936040 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d246310-a408-4cee-811a-9181b88b14d7-istio-token\") pod \"e2e-mt-30037ff3-openshift-default-7d85bbf5b7-w8297\" (UID: \"9d246310-a408-4cee-811a-9181b88b14d7\") " pod="openshift-ingress/e2e-mt-30037ff3-openshift-default-7d85bbf5b7-w8297" Jul 22 03:09:53.936063 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:09:53.936062 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d246310-a408-4cee-811a-9181b88b14d7-workload-certs\") pod \"e2e-mt-30037ff3-openshift-default-7d85bbf5b7-w8297\" (UID: \"9d246310-a408-4cee-811a-9181b88b14d7\") " pod="openshift-ingress/e2e-mt-30037ff3-openshift-default-7d85bbf5b7-w8297" Jul 22 03:09:53.936284 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:09:53.936107 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d246310-a408-4cee-811a-9181b88b14d7-credential-socket\") pod \"e2e-mt-30037ff3-openshift-default-7d85bbf5b7-w8297\" (UID: \"9d246310-a408-4cee-811a-9181b88b14d7\") " pod="openshift-ingress/e2e-mt-30037ff3-openshift-default-7d85bbf5b7-w8297" Jul 22 03:09:53.936284 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:09:53.936134 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-75rw8\" (UniqueName: \"kubernetes.io/projected/9d246310-a408-4cee-811a-9181b88b14d7-kube-api-access-75rw8\") pod \"e2e-mt-30037ff3-openshift-default-7d85bbf5b7-w8297\" (UID: \"9d246310-a408-4cee-811a-9181b88b14d7\") " pod="openshift-ingress/e2e-mt-30037ff3-openshift-default-7d85bbf5b7-w8297" Jul 22 03:09:53.936344 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:09:53.936331 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d246310-a408-4cee-811a-9181b88b14d7-workload-socket\") pod \"e2e-mt-30037ff3-openshift-default-7d85bbf5b7-w8297\" (UID: \"9d246310-a408-4cee-811a-9181b88b14d7\") " pod="openshift-ingress/e2e-mt-30037ff3-openshift-default-7d85bbf5b7-w8297" Jul 22 03:09:53.936593 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:09:53.936549 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d246310-a408-4cee-811a-9181b88b14d7-istiod-ca-cert\") pod \"e2e-mt-30037ff3-openshift-default-7d85bbf5b7-w8297\" (UID: \"9d246310-a408-4cee-811a-9181b88b14d7\") " pod="openshift-ingress/e2e-mt-30037ff3-openshift-default-7d85bbf5b7-w8297" Jul 22 03:09:53.936729 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:09:53.936710 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d246310-a408-4cee-811a-9181b88b14d7-istio-data\") pod \"e2e-mt-30037ff3-openshift-default-7d85bbf5b7-w8297\" (UID: \"9d246310-a408-4cee-811a-9181b88b14d7\") " pod="openshift-ingress/e2e-mt-30037ff3-openshift-default-7d85bbf5b7-w8297" Jul 22 03:09:53.936782 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:09:53.936732 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d246310-a408-4cee-811a-9181b88b14d7-workload-certs\") pod \"e2e-mt-30037ff3-openshift-default-7d85bbf5b7-w8297\" (UID: \"9d246310-a408-4cee-811a-9181b88b14d7\") " pod="openshift-ingress/e2e-mt-30037ff3-openshift-default-7d85bbf5b7-w8297" Jul 22 03:09:53.937076 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:09:53.937055 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d246310-a408-4cee-811a-9181b88b14d7-credential-socket\") pod \"e2e-mt-30037ff3-openshift-default-7d85bbf5b7-w8297\" (UID: \"9d246310-a408-4cee-811a-9181b88b14d7\") " pod="openshift-ingress/e2e-mt-30037ff3-openshift-default-7d85bbf5b7-w8297" Jul 22 03:09:53.938506 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:09:53.938486 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d246310-a408-4cee-811a-9181b88b14d7-istio-envoy\") pod \"e2e-mt-30037ff3-openshift-default-7d85bbf5b7-w8297\" (UID: \"9d246310-a408-4cee-811a-9181b88b14d7\") " pod="openshift-ingress/e2e-mt-30037ff3-openshift-default-7d85bbf5b7-w8297" Jul 22 03:09:53.938652 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:09:53.938524 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d246310-a408-4cee-811a-9181b88b14d7-istio-podinfo\") pod \"e2e-mt-30037ff3-openshift-default-7d85bbf5b7-w8297\" (UID: \"9d246310-a408-4cee-811a-9181b88b14d7\") " pod="openshift-ingress/e2e-mt-30037ff3-openshift-default-7d85bbf5b7-w8297" Jul 22 03:09:53.945363 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:09:53.945342 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-75rw8\" (UniqueName: \"kubernetes.io/projected/9d246310-a408-4cee-811a-9181b88b14d7-kube-api-access-75rw8\") pod \"e2e-mt-30037ff3-openshift-default-7d85bbf5b7-w8297\" (UID: \"9d246310-a408-4cee-811a-9181b88b14d7\") " pod="openshift-ingress/e2e-mt-30037ff3-openshift-default-7d85bbf5b7-w8297" Jul 22 03:09:53.945712 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:09:53.945696 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d246310-a408-4cee-811a-9181b88b14d7-istio-token\") pod \"e2e-mt-30037ff3-openshift-default-7d85bbf5b7-w8297\" (UID: \"9d246310-a408-4cee-811a-9181b88b14d7\") " pod="openshift-ingress/e2e-mt-30037ff3-openshift-default-7d85bbf5b7-w8297" Jul 22 03:09:54.033151 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:09:54.033087 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-30037ff3-openshift-default-7d85bbf5b7-w8297" Jul 22 03:09:54.181840 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:09:54.181812 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-30037ff3-openshift-default-7d85bbf5b7-w8297"] Jul 22 03:09:54.184354 ip-10-0-133-31 kubenswrapper[2578]: W0722 03:09:54.184327 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d246310_a408_4cee_811a_9181b88b14d7.slice/crio-def0404513e8a157c20d441f1ac1a5d9685e2739042d214fa8a1751a50222ee3 WatchSource:0}: Error finding container def0404513e8a157c20d441f1ac1a5d9685e2739042d214fa8a1751a50222ee3: Status 404 returned error can't find the container with id def0404513e8a157c20d441f1ac1a5d9685e2739042d214fa8a1751a50222ee3 Jul 22 03:09:54.186376 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:09:54.186347 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 22 03:09:54.186455 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:09:54.186405 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 22 03:09:54.186455 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:09:54.186433 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 22 03:09:54.684963 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:09:54.684925 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-30037ff3-openshift-default-7d85bbf5b7-w8297" event={"ID":"9d246310-a408-4cee-811a-9181b88b14d7","Type":"ContainerStarted","Data":"238d34063681a8cc5b0ecea24f87006299bc1cd0efb05664dc2c2663b2686f27"} Jul 22 03:09:54.685145 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:09:54.684977 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-30037ff3-openshift-default-7d85bbf5b7-w8297" event={"ID":"9d246310-a408-4cee-811a-9181b88b14d7","Type":"ContainerStarted","Data":"def0404513e8a157c20d441f1ac1a5d9685e2739042d214fa8a1751a50222ee3"} Jul 22 03:09:54.710673 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:09:54.710627 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-30037ff3-openshift-default-7d85bbf5b7-w8297" podStartSLOduration=1.710612319 podStartE2EDuration="1.710612319s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 03:09:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 03:09:54.708843908 +0000 UTC m=+1961.028019207" watchObservedRunningTime="2026-07-22 03:09:54.710612319 +0000 UTC m=+1961.029787701" Jul 22 03:09:55.034148 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:09:55.034056 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-30037ff3-openshift-default-7d85bbf5b7-w8297" Jul 22 03:09:55.039001 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:09:55.038977 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-30037ff3-openshift-default-7d85bbf5b7-w8297" Jul 22 03:09:55.691933 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:09:55.691899 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-30037ff3-openshift-default-7d85bbf5b7-w8297" Jul 22 03:09:55.692873 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:09:55.692850 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-30037ff3-openshift-default-7d85bbf5b7-w8297" Jul 22 03:10:09.768842 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:10:09.768804 2578 generic.go:356] "Generic (PLEG): container finished" podID="9d246310-a408-4cee-811a-9181b88b14d7" containerID="238d34063681a8cc5b0ecea24f87006299bc1cd0efb05664dc2c2663b2686f27" exitCode=0 Jul 22 03:10:09.769270 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:10:09.768879 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-30037ff3-openshift-default-7d85bbf5b7-w8297" event={"ID":"9d246310-a408-4cee-811a-9181b88b14d7","Type":"ContainerDied","Data":"238d34063681a8cc5b0ecea24f87006299bc1cd0efb05664dc2c2663b2686f27"} Jul 22 03:10:09.769317 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:10:09.769290 2578 scope.go:117] "RemoveContainer" containerID="238d34063681a8cc5b0ecea24f87006299bc1cd0efb05664dc2c2663b2686f27" Jul 22 03:10:09.770248 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:10:09.770221 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 22 03:10:09.770341 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:10:09.770286 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 22 03:10:09.770341 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:10:09.770315 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 22 03:10:10.692018 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:10:10.691979 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-30037ff3-openshift-default-7d85bbf5b7-w8297" Jul 22 03:10:10.775450 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:10:10.775415 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-30037ff3-openshift-default-7d85bbf5b7-w8297" event={"ID":"9d246310-a408-4cee-811a-9181b88b14d7","Type":"ContainerStarted","Data":"6586fc0f0cfe574dee0c24f136e5986a15eae17c6047c6be75e5e11950ec876f"} Jul 22 03:10:11.034036 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:10:11.033943 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-30037ff3-openshift-default-7d85bbf5b7-w8297" Jul 22 03:10:11.035310 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:10:11.035285 2578 patch_prober.go:28] interesting pod/e2e-mt-30037ff3-openshift-default-7d85bbf5b7-w8297 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body= Jul 22 03:10:11.035435 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:10:11.035337 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-30037ff3-openshift-default-7d85bbf5b7-w8297" podUID="9d246310-a408-4cee-811a-9181b88b14d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" Jul 22 03:10:12.034383 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:10:12.034347 2578 patch_prober.go:28] interesting pod/e2e-mt-30037ff3-openshift-default-7d85bbf5b7-w8297 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body= Jul 22 03:10:12.034899 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:10:12.034410 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-30037ff3-openshift-default-7d85bbf5b7-w8297" podUID="9d246310-a408-4cee-811a-9181b88b14d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" Jul 22 03:10:13.034381 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:10:13.034340 2578 patch_prober.go:28] interesting pod/e2e-mt-30037ff3-openshift-default-7d85bbf5b7-w8297 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body= Jul 22 03:10:13.034594 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:10:13.034396 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-30037ff3-openshift-default-7d85bbf5b7-w8297" podUID="9d246310-a408-4cee-811a-9181b88b14d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" Jul 22 03:10:14.039426 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:10:14.039381 2578 patch_prober.go:28] interesting pod/e2e-mt-30037ff3-openshift-default-7d85bbf5b7-w8297 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body= Jul 22 03:10:14.039923 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:10:14.039454 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-30037ff3-openshift-default-7d85bbf5b7-w8297" podUID="9d246310-a408-4cee-811a-9181b88b14d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" Jul 22 03:10:15.037763 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:10:15.037733 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-30037ff3-openshift-default-7d85bbf5b7-w8297" Jul 22 03:10:15.038014 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:10:15.037993 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-30037ff3-openshift-default-7d85bbf5b7-w8297" Jul 22 03:10:15.038728 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:10:15.038707 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-30037ff3-openshift-default-7d85bbf5b7-w8297" Jul 22 03:10:30.395210 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:10:30.395176 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-30037ff3-openshift-default-7d85bbf5b7-w8297"] Jul 22 03:10:30.395721 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:10:30.395437 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-30037ff3-openshift-default-7d85bbf5b7-w8297" podUID="9d246310-a408-4cee-811a-9181b88b14d7" containerName="istio-proxy" containerID="cri-o://6586fc0f0cfe574dee0c24f136e5986a15eae17c6047c6be75e5e11950ec876f" gracePeriod=30 Jul 22 03:10:35.651791 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:10:35.651764 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-30037ff3-openshift-default-7d85bbf5b7-w8297" Jul 22 03:10:35.721456 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:10:35.721376 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d246310-a408-4cee-811a-9181b88b14d7-istio-envoy\") pod \"9d246310-a408-4cee-811a-9181b88b14d7\" (UID: \"9d246310-a408-4cee-811a-9181b88b14d7\") " Jul 22 03:10:35.721456 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:10:35.721411 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d246310-a408-4cee-811a-9181b88b14d7-istio-token\") pod \"9d246310-a408-4cee-811a-9181b88b14d7\" (UID: \"9d246310-a408-4cee-811a-9181b88b14d7\") " Jul 22 03:10:35.721456 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:10:35.721433 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-75rw8\" (UniqueName: \"kubernetes.io/projected/9d246310-a408-4cee-811a-9181b88b14d7-kube-api-access-75rw8\") pod \"9d246310-a408-4cee-811a-9181b88b14d7\" (UID: \"9d246310-a408-4cee-811a-9181b88b14d7\") " Jul 22 03:10:35.721456 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:10:35.721454 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d246310-a408-4cee-811a-9181b88b14d7-istio-podinfo\") pod \"9d246310-a408-4cee-811a-9181b88b14d7\" (UID: \"9d246310-a408-4cee-811a-9181b88b14d7\") " Jul 22 03:10:35.721845 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:10:35.721496 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d246310-a408-4cee-811a-9181b88b14d7-istiod-ca-cert\") pod \"9d246310-a408-4cee-811a-9181b88b14d7\" (UID: \"9d246310-a408-4cee-811a-9181b88b14d7\") " Jul 22 03:10:35.721845 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:10:35.721519 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d246310-a408-4cee-811a-9181b88b14d7-workload-certs\") pod \"9d246310-a408-4cee-811a-9181b88b14d7\" (UID: \"9d246310-a408-4cee-811a-9181b88b14d7\") " Jul 22 03:10:35.721845 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:10:35.721547 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d246310-a408-4cee-811a-9181b88b14d7-istio-data\") pod \"9d246310-a408-4cee-811a-9181b88b14d7\" (UID: \"9d246310-a408-4cee-811a-9181b88b14d7\") " Jul 22 03:10:35.722061 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:10:35.721865 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d246310-a408-4cee-811a-9181b88b14d7-credential-socket\") pod \"9d246310-a408-4cee-811a-9181b88b14d7\" (UID: \"9d246310-a408-4cee-811a-9181b88b14d7\") " Jul 22 03:10:35.722061 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:10:35.721946 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d246310-a408-4cee-811a-9181b88b14d7-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "9d246310-a408-4cee-811a-9181b88b14d7" (UID: "9d246310-a408-4cee-811a-9181b88b14d7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 03:10:35.722061 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:10:35.721954 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d246310-a408-4cee-811a-9181b88b14d7-workload-socket\") pod \"9d246310-a408-4cee-811a-9181b88b14d7\" (UID: \"9d246310-a408-4cee-811a-9181b88b14d7\") " Jul 22 03:10:35.722414 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:10:35.722138 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d246310-a408-4cee-811a-9181b88b14d7-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "9d246310-a408-4cee-811a-9181b88b14d7" (UID: "9d246310-a408-4cee-811a-9181b88b14d7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 03:10:35.722504 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:10:35.722481 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d246310-a408-4cee-811a-9181b88b14d7-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "9d246310-a408-4cee-811a-9181b88b14d7" (UID: "9d246310-a408-4cee-811a-9181b88b14d7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 03:10:35.722585 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:10:35.722546 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d246310-a408-4cee-811a-9181b88b14d7-istio-data" (OuterVolumeSpecName: "istio-data") pod "9d246310-a408-4cee-811a-9181b88b14d7" (UID: "9d246310-a408-4cee-811a-9181b88b14d7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 03:10:35.722651 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:10:35.722586 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d246310-a408-4cee-811a-9181b88b14d7-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "9d246310-a408-4cee-811a-9181b88b14d7" (UID: "9d246310-a408-4cee-811a-9181b88b14d7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 03:10:35.722651 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:10:35.722590 2578 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d246310-a408-4cee-811a-9181b88b14d7-istiod-ca-cert\") on node \"ip-10-0-133-31.ec2.internal\" DevicePath \"\"" Jul 22 03:10:35.722651 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:10:35.722624 2578 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d246310-a408-4cee-811a-9181b88b14d7-workload-certs\") on node \"ip-10-0-133-31.ec2.internal\" DevicePath \"\"" Jul 22 03:10:35.723930 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:10:35.723900 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d246310-a408-4cee-811a-9181b88b14d7-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "9d246310-a408-4cee-811a-9181b88b14d7" (UID: "9d246310-a408-4cee-811a-9181b88b14d7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 03:10:35.723930 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:10:35.723899 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d246310-a408-4cee-811a-9181b88b14d7-istio-token" (OuterVolumeSpecName: "istio-token") pod "9d246310-a408-4cee-811a-9181b88b14d7" (UID: "9d246310-a408-4cee-811a-9181b88b14d7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 03:10:35.724078 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:10:35.723931 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d246310-a408-4cee-811a-9181b88b14d7-kube-api-access-75rw8" (OuterVolumeSpecName: "kube-api-access-75rw8") pod "9d246310-a408-4cee-811a-9181b88b14d7" (UID: "9d246310-a408-4cee-811a-9181b88b14d7"). InnerVolumeSpecName "kube-api-access-75rw8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 03:10:35.724078 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:10:35.723950 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9d246310-a408-4cee-811a-9181b88b14d7-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "9d246310-a408-4cee-811a-9181b88b14d7" (UID: "9d246310-a408-4cee-811a-9181b88b14d7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 22 03:10:35.823504 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:10:35.823468 2578 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d246310-a408-4cee-811a-9181b88b14d7-istio-data\") on node \"ip-10-0-133-31.ec2.internal\" DevicePath \"\"" Jul 22 03:10:35.823504 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:10:35.823497 2578 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d246310-a408-4cee-811a-9181b88b14d7-credential-socket\") on node \"ip-10-0-133-31.ec2.internal\" DevicePath \"\"" Jul 22 03:10:35.823504 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:10:35.823507 2578 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d246310-a408-4cee-811a-9181b88b14d7-workload-socket\") on node \"ip-10-0-133-31.ec2.internal\" DevicePath \"\"" Jul 22 03:10:35.823504 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:10:35.823517 2578 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d246310-a408-4cee-811a-9181b88b14d7-istio-envoy\") on node \"ip-10-0-133-31.ec2.internal\" DevicePath \"\"" Jul 22 03:10:35.823846 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:10:35.823526 2578 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d246310-a408-4cee-811a-9181b88b14d7-istio-token\") on node \"ip-10-0-133-31.ec2.internal\" DevicePath \"\"" Jul 22 03:10:35.823846 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:10:35.823533 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-75rw8\" (UniqueName: \"kubernetes.io/projected/9d246310-a408-4cee-811a-9181b88b14d7-kube-api-access-75rw8\") on node \"ip-10-0-133-31.ec2.internal\" DevicePath \"\"" Jul 22 03:10:35.823846 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:10:35.823541 2578 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d246310-a408-4cee-811a-9181b88b14d7-istio-podinfo\") on node \"ip-10-0-133-31.ec2.internal\" DevicePath \"\"" Jul 22 03:10:35.914801 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:10:35.914764 2578 generic.go:356] "Generic (PLEG): container finished" podID="9d246310-a408-4cee-811a-9181b88b14d7" containerID="6586fc0f0cfe574dee0c24f136e5986a15eae17c6047c6be75e5e11950ec876f" exitCode=0 Jul 22 03:10:35.914958 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:10:35.914833 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-30037ff3-openshift-default-7d85bbf5b7-w8297" event={"ID":"9d246310-a408-4cee-811a-9181b88b14d7","Type":"ContainerDied","Data":"6586fc0f0cfe574dee0c24f136e5986a15eae17c6047c6be75e5e11950ec876f"} Jul 22 03:10:35.914958 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:10:35.914913 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-30037ff3-openshift-default-7d85bbf5b7-w8297" event={"ID":"9d246310-a408-4cee-811a-9181b88b14d7","Type":"ContainerDied","Data":"def0404513e8a157c20d441f1ac1a5d9685e2739042d214fa8a1751a50222ee3"} Jul 22 03:10:35.914958 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:10:35.914841 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-30037ff3-openshift-default-7d85bbf5b7-w8297" Jul 22 03:10:35.914958 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:10:35.914936 2578 scope.go:117] "RemoveContainer" containerID="6586fc0f0cfe574dee0c24f136e5986a15eae17c6047c6be75e5e11950ec876f" Jul 22 03:10:35.924522 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:10:35.924498 2578 scope.go:117] "RemoveContainer" containerID="238d34063681a8cc5b0ecea24f87006299bc1cd0efb05664dc2c2663b2686f27" Jul 22 03:10:35.932595 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:10:35.932577 2578 scope.go:117] "RemoveContainer" containerID="6586fc0f0cfe574dee0c24f136e5986a15eae17c6047c6be75e5e11950ec876f" Jul 22 03:10:35.932810 ip-10-0-133-31 kubenswrapper[2578]: E0722 03:10:35.932795 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6586fc0f0cfe574dee0c24f136e5986a15eae17c6047c6be75e5e11950ec876f\": container with ID starting with 6586fc0f0cfe574dee0c24f136e5986a15eae17c6047c6be75e5e11950ec876f not found: ID does not exist" containerID="6586fc0f0cfe574dee0c24f136e5986a15eae17c6047c6be75e5e11950ec876f" Jul 22 03:10:35.932868 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:10:35.932818 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6586fc0f0cfe574dee0c24f136e5986a15eae17c6047c6be75e5e11950ec876f"} err="failed to get container status \"6586fc0f0cfe574dee0c24f136e5986a15eae17c6047c6be75e5e11950ec876f\": rpc error: code = NotFound desc = could not find container \"6586fc0f0cfe574dee0c24f136e5986a15eae17c6047c6be75e5e11950ec876f\": container with ID starting with 6586fc0f0cfe574dee0c24f136e5986a15eae17c6047c6be75e5e11950ec876f not found: ID does not exist" Jul 22 03:10:35.932868 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:10:35.932832 2578 scope.go:117] "RemoveContainer" containerID="238d34063681a8cc5b0ecea24f87006299bc1cd0efb05664dc2c2663b2686f27" Jul 22 03:10:35.933058 ip-10-0-133-31 kubenswrapper[2578]: E0722 03:10:35.933039 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"238d34063681a8cc5b0ecea24f87006299bc1cd0efb05664dc2c2663b2686f27\": container with ID starting with 238d34063681a8cc5b0ecea24f87006299bc1cd0efb05664dc2c2663b2686f27 not found: ID does not exist" containerID="238d34063681a8cc5b0ecea24f87006299bc1cd0efb05664dc2c2663b2686f27" Jul 22 03:10:35.933103 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:10:35.933066 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"238d34063681a8cc5b0ecea24f87006299bc1cd0efb05664dc2c2663b2686f27"} err="failed to get container status \"238d34063681a8cc5b0ecea24f87006299bc1cd0efb05664dc2c2663b2686f27\": rpc error: code = NotFound desc = could not find container \"238d34063681a8cc5b0ecea24f87006299bc1cd0efb05664dc2c2663b2686f27\": container with ID starting with 238d34063681a8cc5b0ecea24f87006299bc1cd0efb05664dc2c2663b2686f27 not found: ID does not exist" Jul 22 03:10:35.937897 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:10:35.937874 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-30037ff3-openshift-default-7d85bbf5b7-w8297"] Jul 22 03:10:35.941487 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:10:35.941466 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-30037ff3-openshift-default-7d85bbf5b7-w8297"] Jul 22 03:10:36.141606 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:10:36.141573 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9d246310-a408-4cee-811a-9181b88b14d7" path="/var/lib/kubelet/pods/9d246310-a408-4cee-811a-9181b88b14d7/volumes" Jul 22 03:10:50.405869 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:10:50.405833 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-e1c7132e-openshift-default-588794d9f9-v679l"] Jul 22 03:10:50.406285 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:10:50.406251 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9d246310-a408-4cee-811a-9181b88b14d7" containerName="istio-proxy" Jul 22 03:10:50.406285 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:10:50.406264 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="9d246310-a408-4cee-811a-9181b88b14d7" containerName="istio-proxy" Jul 22 03:10:50.406285 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:10:50.406275 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="de03c673-5a6c-4183-bec8-ee4302e65e47" containerName="istio-proxy" Jul 22 03:10:50.406285 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:10:50.406280 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="de03c673-5a6c-4183-bec8-ee4302e65e47" containerName="istio-proxy" Jul 22 03:10:50.406435 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:10:50.406291 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9d246310-a408-4cee-811a-9181b88b14d7" containerName="istio-proxy" Jul 22 03:10:50.406435 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:10:50.406300 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="9d246310-a408-4cee-811a-9181b88b14d7" containerName="istio-proxy" Jul 22 03:10:50.406435 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:10:50.406312 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ee6e4d28-0b3d-4c36-92a1-6b2f366a6748" containerName="istio-proxy" Jul 22 03:10:50.406435 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:10:50.406317 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="ee6e4d28-0b3d-4c36-92a1-6b2f366a6748" containerName="istio-proxy" Jul 22 03:10:50.406435 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:10:50.406383 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="ee6e4d28-0b3d-4c36-92a1-6b2f366a6748" containerName="istio-proxy" Jul 22 03:10:50.406435 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:10:50.406392 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="9d246310-a408-4cee-811a-9181b88b14d7" containerName="istio-proxy" Jul 22 03:10:50.406435 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:10:50.406398 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="de03c673-5a6c-4183-bec8-ee4302e65e47" containerName="istio-proxy" Jul 22 03:10:50.406435 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:10:50.406402 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="9d246310-a408-4cee-811a-9181b88b14d7" containerName="istio-proxy" Jul 22 03:10:50.409248 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:10:50.409233 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-e1c7132e-openshift-default-588794d9f9-v679l" Jul 22 03:10:50.412143 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:10:50.412124 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-e1c7132e-openshift-default-dockercfg-5t5br\"" Jul 22 03:10:50.424539 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:10:50.424518 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-e1c7132e-openshift-default-588794d9f9-v679l"] Jul 22 03:10:50.444043 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:10:50.444023 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f18ffc96-ad50-42d7-af4a-0cdddf90fe5a-istio-podinfo\") pod \"e2e-mt-e1c7132e-openshift-default-588794d9f9-v679l\" (UID: \"f18ffc96-ad50-42d7-af4a-0cdddf90fe5a\") " pod="openshift-ingress/e2e-mt-e1c7132e-openshift-default-588794d9f9-v679l" Jul 22 03:10:50.444166 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:10:50.444049 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f18ffc96-ad50-42d7-af4a-0cdddf90fe5a-istio-data\") pod \"e2e-mt-e1c7132e-openshift-default-588794d9f9-v679l\" (UID: \"f18ffc96-ad50-42d7-af4a-0cdddf90fe5a\") " pod="openshift-ingress/e2e-mt-e1c7132e-openshift-default-588794d9f9-v679l" Jul 22 03:10:50.444166 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:10:50.444069 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f18ffc96-ad50-42d7-af4a-0cdddf90fe5a-istiod-ca-cert\") pod \"e2e-mt-e1c7132e-openshift-default-588794d9f9-v679l\" (UID: \"f18ffc96-ad50-42d7-af4a-0cdddf90fe5a\") " pod="openshift-ingress/e2e-mt-e1c7132e-openshift-default-588794d9f9-v679l" Jul 22 03:10:50.444166 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:10:50.444102 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f18ffc96-ad50-42d7-af4a-0cdddf90fe5a-istio-envoy\") pod \"e2e-mt-e1c7132e-openshift-default-588794d9f9-v679l\" (UID: \"f18ffc96-ad50-42d7-af4a-0cdddf90fe5a\") " pod="openshift-ingress/e2e-mt-e1c7132e-openshift-default-588794d9f9-v679l" Jul 22 03:10:50.444331 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:10:50.444200 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jrl8d\" (UniqueName: \"kubernetes.io/projected/f18ffc96-ad50-42d7-af4a-0cdddf90fe5a-kube-api-access-jrl8d\") pod \"e2e-mt-e1c7132e-openshift-default-588794d9f9-v679l\" (UID: \"f18ffc96-ad50-42d7-af4a-0cdddf90fe5a\") " pod="openshift-ingress/e2e-mt-e1c7132e-openshift-default-588794d9f9-v679l" Jul 22 03:10:50.444331 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:10:50.444237 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f18ffc96-ad50-42d7-af4a-0cdddf90fe5a-workload-certs\") pod \"e2e-mt-e1c7132e-openshift-default-588794d9f9-v679l\" (UID: \"f18ffc96-ad50-42d7-af4a-0cdddf90fe5a\") " pod="openshift-ingress/e2e-mt-e1c7132e-openshift-default-588794d9f9-v679l" Jul 22 03:10:50.444331 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:10:50.444319 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f18ffc96-ad50-42d7-af4a-0cdddf90fe5a-workload-socket\") pod \"e2e-mt-e1c7132e-openshift-default-588794d9f9-v679l\" (UID: \"f18ffc96-ad50-42d7-af4a-0cdddf90fe5a\") " pod="openshift-ingress/e2e-mt-e1c7132e-openshift-default-588794d9f9-v679l" Jul 22 03:10:50.444482 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:10:50.444348 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f18ffc96-ad50-42d7-af4a-0cdddf90fe5a-istio-token\") pod \"e2e-mt-e1c7132e-openshift-default-588794d9f9-v679l\" (UID: \"f18ffc96-ad50-42d7-af4a-0cdddf90fe5a\") " pod="openshift-ingress/e2e-mt-e1c7132e-openshift-default-588794d9f9-v679l" Jul 22 03:10:50.444482 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:10:50.444382 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f18ffc96-ad50-42d7-af4a-0cdddf90fe5a-credential-socket\") pod \"e2e-mt-e1c7132e-openshift-default-588794d9f9-v679l\" (UID: \"f18ffc96-ad50-42d7-af4a-0cdddf90fe5a\") " pod="openshift-ingress/e2e-mt-e1c7132e-openshift-default-588794d9f9-v679l" Jul 22 03:10:50.545453 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:10:50.545416 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jrl8d\" (UniqueName: \"kubernetes.io/projected/f18ffc96-ad50-42d7-af4a-0cdddf90fe5a-kube-api-access-jrl8d\") pod \"e2e-mt-e1c7132e-openshift-default-588794d9f9-v679l\" (UID: \"f18ffc96-ad50-42d7-af4a-0cdddf90fe5a\") " pod="openshift-ingress/e2e-mt-e1c7132e-openshift-default-588794d9f9-v679l" Jul 22 03:10:50.545453 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:10:50.545456 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f18ffc96-ad50-42d7-af4a-0cdddf90fe5a-workload-certs\") pod \"e2e-mt-e1c7132e-openshift-default-588794d9f9-v679l\" (UID: \"f18ffc96-ad50-42d7-af4a-0cdddf90fe5a\") " pod="openshift-ingress/e2e-mt-e1c7132e-openshift-default-588794d9f9-v679l" Jul 22 03:10:50.545727 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:10:50.545496 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f18ffc96-ad50-42d7-af4a-0cdddf90fe5a-workload-socket\") pod \"e2e-mt-e1c7132e-openshift-default-588794d9f9-v679l\" (UID: \"f18ffc96-ad50-42d7-af4a-0cdddf90fe5a\") " pod="openshift-ingress/e2e-mt-e1c7132e-openshift-default-588794d9f9-v679l" Jul 22 03:10:50.545727 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:10:50.545511 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f18ffc96-ad50-42d7-af4a-0cdddf90fe5a-istio-token\") pod \"e2e-mt-e1c7132e-openshift-default-588794d9f9-v679l\" (UID: \"f18ffc96-ad50-42d7-af4a-0cdddf90fe5a\") " pod="openshift-ingress/e2e-mt-e1c7132e-openshift-default-588794d9f9-v679l" Jul 22 03:10:50.545727 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:10:50.545533 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f18ffc96-ad50-42d7-af4a-0cdddf90fe5a-credential-socket\") pod \"e2e-mt-e1c7132e-openshift-default-588794d9f9-v679l\" (UID: \"f18ffc96-ad50-42d7-af4a-0cdddf90fe5a\") " pod="openshift-ingress/e2e-mt-e1c7132e-openshift-default-588794d9f9-v679l" Jul 22 03:10:50.545727 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:10:50.545553 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f18ffc96-ad50-42d7-af4a-0cdddf90fe5a-istio-podinfo\") pod \"e2e-mt-e1c7132e-openshift-default-588794d9f9-v679l\" (UID: \"f18ffc96-ad50-42d7-af4a-0cdddf90fe5a\") " pod="openshift-ingress/e2e-mt-e1c7132e-openshift-default-588794d9f9-v679l" Jul 22 03:10:50.545727 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:10:50.545604 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f18ffc96-ad50-42d7-af4a-0cdddf90fe5a-istio-data\") pod \"e2e-mt-e1c7132e-openshift-default-588794d9f9-v679l\" (UID: \"f18ffc96-ad50-42d7-af4a-0cdddf90fe5a\") " pod="openshift-ingress/e2e-mt-e1c7132e-openshift-default-588794d9f9-v679l" Jul 22 03:10:50.545727 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:10:50.545626 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f18ffc96-ad50-42d7-af4a-0cdddf90fe5a-istiod-ca-cert\") pod \"e2e-mt-e1c7132e-openshift-default-588794d9f9-v679l\" (UID: \"f18ffc96-ad50-42d7-af4a-0cdddf90fe5a\") " pod="openshift-ingress/e2e-mt-e1c7132e-openshift-default-588794d9f9-v679l" Jul 22 03:10:50.545727 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:10:50.545675 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f18ffc96-ad50-42d7-af4a-0cdddf90fe5a-istio-envoy\") pod \"e2e-mt-e1c7132e-openshift-default-588794d9f9-v679l\" (UID: \"f18ffc96-ad50-42d7-af4a-0cdddf90fe5a\") " pod="openshift-ingress/e2e-mt-e1c7132e-openshift-default-588794d9f9-v679l" Jul 22 03:10:50.546084 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:10:50.545954 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f18ffc96-ad50-42d7-af4a-0cdddf90fe5a-workload-socket\") pod \"e2e-mt-e1c7132e-openshift-default-588794d9f9-v679l\" (UID: \"f18ffc96-ad50-42d7-af4a-0cdddf90fe5a\") " pod="openshift-ingress/e2e-mt-e1c7132e-openshift-default-588794d9f9-v679l" Jul 22 03:10:50.546084 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:10:50.546001 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f18ffc96-ad50-42d7-af4a-0cdddf90fe5a-workload-certs\") pod \"e2e-mt-e1c7132e-openshift-default-588794d9f9-v679l\" (UID: \"f18ffc96-ad50-42d7-af4a-0cdddf90fe5a\") " pod="openshift-ingress/e2e-mt-e1c7132e-openshift-default-588794d9f9-v679l" Jul 22 03:10:50.546240 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:10:50.546200 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f18ffc96-ad50-42d7-af4a-0cdddf90fe5a-credential-socket\") pod \"e2e-mt-e1c7132e-openshift-default-588794d9f9-v679l\" (UID: \"f18ffc96-ad50-42d7-af4a-0cdddf90fe5a\") " pod="openshift-ingress/e2e-mt-e1c7132e-openshift-default-588794d9f9-v679l" Jul 22 03:10:50.546592 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:10:50.546548 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f18ffc96-ad50-42d7-af4a-0cdddf90fe5a-istio-data\") pod \"e2e-mt-e1c7132e-openshift-default-588794d9f9-v679l\" (UID: \"f18ffc96-ad50-42d7-af4a-0cdddf90fe5a\") " pod="openshift-ingress/e2e-mt-e1c7132e-openshift-default-588794d9f9-v679l" Jul 22 03:10:50.547023 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:10:50.547001 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f18ffc96-ad50-42d7-af4a-0cdddf90fe5a-istiod-ca-cert\") pod \"e2e-mt-e1c7132e-openshift-default-588794d9f9-v679l\" (UID: \"f18ffc96-ad50-42d7-af4a-0cdddf90fe5a\") " pod="openshift-ingress/e2e-mt-e1c7132e-openshift-default-588794d9f9-v679l" Jul 22 03:10:50.548128 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:10:50.548104 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f18ffc96-ad50-42d7-af4a-0cdddf90fe5a-istio-envoy\") pod \"e2e-mt-e1c7132e-openshift-default-588794d9f9-v679l\" (UID: \"f18ffc96-ad50-42d7-af4a-0cdddf90fe5a\") " pod="openshift-ingress/e2e-mt-e1c7132e-openshift-default-588794d9f9-v679l" Jul 22 03:10:50.548340 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:10:50.548319 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f18ffc96-ad50-42d7-af4a-0cdddf90fe5a-istio-podinfo\") pod \"e2e-mt-e1c7132e-openshift-default-588794d9f9-v679l\" (UID: \"f18ffc96-ad50-42d7-af4a-0cdddf90fe5a\") " pod="openshift-ingress/e2e-mt-e1c7132e-openshift-default-588794d9f9-v679l" Jul 22 03:10:50.554839 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:10:50.554810 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f18ffc96-ad50-42d7-af4a-0cdddf90fe5a-istio-token\") pod \"e2e-mt-e1c7132e-openshift-default-588794d9f9-v679l\" (UID: \"f18ffc96-ad50-42d7-af4a-0cdddf90fe5a\") " pod="openshift-ingress/e2e-mt-e1c7132e-openshift-default-588794d9f9-v679l" Jul 22 03:10:50.554982 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:10:50.554962 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jrl8d\" (UniqueName: \"kubernetes.io/projected/f18ffc96-ad50-42d7-af4a-0cdddf90fe5a-kube-api-access-jrl8d\") pod \"e2e-mt-e1c7132e-openshift-default-588794d9f9-v679l\" (UID: \"f18ffc96-ad50-42d7-af4a-0cdddf90fe5a\") " pod="openshift-ingress/e2e-mt-e1c7132e-openshift-default-588794d9f9-v679l" Jul 22 03:10:50.719060 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:10:50.718989 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-e1c7132e-openshift-default-588794d9f9-v679l" Jul 22 03:10:50.860607 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:10:50.860553 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-e1c7132e-openshift-default-588794d9f9-v679l"] Jul 22 03:10:50.861641 ip-10-0-133-31 kubenswrapper[2578]: W0722 03:10:50.861615 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf18ffc96_ad50_42d7_af4a_0cdddf90fe5a.slice/crio-7fad45c22d308312348773006f44d52c339b2a40add9d578ce8281742dfeddc7 WatchSource:0}: Error finding container 7fad45c22d308312348773006f44d52c339b2a40add9d578ce8281742dfeddc7: Status 404 returned error can't find the container with id 7fad45c22d308312348773006f44d52c339b2a40add9d578ce8281742dfeddc7 Jul 22 03:10:50.863873 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:10:50.863834 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 22 03:10:50.863948 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:10:50.863913 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 22 03:10:50.864000 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:10:50.863952 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 22 03:10:50.995305 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:10:50.995168 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e1c7132e-openshift-default-588794d9f9-v679l" event={"ID":"f18ffc96-ad50-42d7-af4a-0cdddf90fe5a","Type":"ContainerStarted","Data":"b6254b611f97536d1891d3fa1b9cbc51324595d067449189f7cf1168a2be528b"} Jul 22 03:10:50.995305 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:10:50.995244 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e1c7132e-openshift-default-588794d9f9-v679l" event={"ID":"f18ffc96-ad50-42d7-af4a-0cdddf90fe5a","Type":"ContainerStarted","Data":"7fad45c22d308312348773006f44d52c339b2a40add9d578ce8281742dfeddc7"} Jul 22 03:10:51.020823 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:10:51.020776 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-e1c7132e-openshift-default-588794d9f9-v679l" podStartSLOduration=1.020757271 podStartE2EDuration="1.020757271s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 03:10:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 03:10:51.018106981 +0000 UTC m=+2017.337282281" watchObservedRunningTime="2026-07-22 03:10:51.020757271 +0000 UTC m=+2017.339932572" Jul 22 03:10:51.719255 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:10:51.719226 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-e1c7132e-openshift-default-588794d9f9-v679l" Jul 22 03:10:51.724100 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:10:51.724076 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-e1c7132e-openshift-default-588794d9f9-v679l" Jul 22 03:10:52.003896 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:10:52.003816 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-e1c7132e-openshift-default-588794d9f9-v679l" Jul 22 03:10:52.004664 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:10:52.004644 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-e1c7132e-openshift-default-588794d9f9-v679l" Jul 22 03:11:05.075125 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:11:05.075092 2578 generic.go:356] "Generic (PLEG): container finished" podID="f18ffc96-ad50-42d7-af4a-0cdddf90fe5a" containerID="b6254b611f97536d1891d3fa1b9cbc51324595d067449189f7cf1168a2be528b" exitCode=0 Jul 22 03:11:05.075528 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:11:05.075163 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e1c7132e-openshift-default-588794d9f9-v679l" event={"ID":"f18ffc96-ad50-42d7-af4a-0cdddf90fe5a","Type":"ContainerDied","Data":"b6254b611f97536d1891d3fa1b9cbc51324595d067449189f7cf1168a2be528b"} Jul 22 03:11:05.075588 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:11:05.075533 2578 scope.go:117] "RemoveContainer" containerID="b6254b611f97536d1891d3fa1b9cbc51324595d067449189f7cf1168a2be528b" Jul 22 03:11:05.076658 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:11:05.076630 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 22 03:11:05.076732 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:11:05.076701 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 22 03:11:05.076773 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:11:05.076742 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 22 03:11:06.086760 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:11:06.086718 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e1c7132e-openshift-default-588794d9f9-v679l" event={"ID":"f18ffc96-ad50-42d7-af4a-0cdddf90fe5a","Type":"ContainerStarted","Data":"930d58ebb191f483811b3cbd9ad20945631c762b0992f6db568588ea018cbd75"} Jul 22 03:11:06.720235 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:11:06.720197 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-e1c7132e-openshift-default-588794d9f9-v679l" Jul 22 03:11:06.721537 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:11:06.721507 2578 patch_prober.go:28] interesting pod/e2e-mt-e1c7132e-openshift-default-588794d9f9-v679l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body= Jul 22 03:11:06.721681 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:11:06.721585 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e1c7132e-openshift-default-588794d9f9-v679l" podUID="f18ffc96-ad50-42d7-af4a-0cdddf90fe5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" Jul 22 03:11:07.004812 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:11:07.004713 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-e1c7132e-openshift-default-588794d9f9-v679l" Jul 22 03:11:07.719489 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:11:07.719451 2578 patch_prober.go:28] interesting pod/e2e-mt-e1c7132e-openshift-default-588794d9f9-v679l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body= Jul 22 03:11:07.719911 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:11:07.719526 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e1c7132e-openshift-default-588794d9f9-v679l" podUID="f18ffc96-ad50-42d7-af4a-0cdddf90fe5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" Jul 22 03:11:08.719710 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:11:08.719668 2578 patch_prober.go:28] interesting pod/e2e-mt-e1c7132e-openshift-default-588794d9f9-v679l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body= Jul 22 03:11:08.720108 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:11:08.719737 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e1c7132e-openshift-default-588794d9f9-v679l" podUID="f18ffc96-ad50-42d7-af4a-0cdddf90fe5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" Jul 22 03:11:09.719448 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:11:09.719410 2578 patch_prober.go:28] interesting pod/e2e-mt-e1c7132e-openshift-default-588794d9f9-v679l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body= Jul 22 03:11:09.719657 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:11:09.719469 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e1c7132e-openshift-default-588794d9f9-v679l" podUID="f18ffc96-ad50-42d7-af4a-0cdddf90fe5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" Jul 22 03:11:10.723793 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:11:10.723761 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-e1c7132e-openshift-default-588794d9f9-v679l" Jul 22 03:11:10.724716 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:11:10.724699 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-e1c7132e-openshift-default-588794d9f9-v679l" Jul 22 03:11:11.253211 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:11:11.253166 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-e1c7132e/e2e-lifecycle-model-e1c7132e-kserve-7dc4df997f-ccwmn"] Jul 22 03:11:11.256497 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:11:11.256466 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-e1c7132e/e2e-lifecycle-model-e1c7132e-kserve-7dc4df997f-ccwmn" Jul 22 03:11:11.260596 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:11:11.260551 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-e1c7132e\"/\"kube-root-ca.crt\"" Jul 22 03:11:11.260739 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:11:11.260555 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-e1c7132e\"/\"e2e-lifecycle-model-e1c7132e-kserve-self-signed-certs\"" Jul 22 03:11:11.260739 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:11:11.260607 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-e1c7132e\"/\"openshift-service-ca.crt\"" Jul 22 03:11:11.260739 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:11:11.260682 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-e1c7132e\"/\"default-dockercfg-72jrs\"" Jul 22 03:11:11.267240 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:11:11.267216 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-e1c7132e/e2e-lifecycle-model-e1c7132e-kserve-7dc4df997f-ccwmn"] Jul 22 03:11:11.343388 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:11:11.343358 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/220087b0-b847-485b-aff8-878c4080c014-tmp-dir\") pod \"e2e-lifecycle-model-e1c7132e-kserve-7dc4df997f-ccwmn\" (UID: \"220087b0-b847-485b-aff8-878c4080c014\") " pod="ai-tenant-e2e-mt-e1c7132e/e2e-lifecycle-model-e1c7132e-kserve-7dc4df997f-ccwmn" Jul 22 03:11:11.343622 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:11:11.343401 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/220087b0-b847-485b-aff8-878c4080c014-home\") pod \"e2e-lifecycle-model-e1c7132e-kserve-7dc4df997f-ccwmn\" (UID: \"220087b0-b847-485b-aff8-878c4080c014\") " pod="ai-tenant-e2e-mt-e1c7132e/e2e-lifecycle-model-e1c7132e-kserve-7dc4df997f-ccwmn" Jul 22 03:11:11.343622 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:11:11.343489 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-84zm5\" (UniqueName: \"kubernetes.io/projected/220087b0-b847-485b-aff8-878c4080c014-kube-api-access-84zm5\") pod \"e2e-lifecycle-model-e1c7132e-kserve-7dc4df997f-ccwmn\" (UID: \"220087b0-b847-485b-aff8-878c4080c014\") " pod="ai-tenant-e2e-mt-e1c7132e/e2e-lifecycle-model-e1c7132e-kserve-7dc4df997f-ccwmn" Jul 22 03:11:11.343622 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:11:11.343524 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/220087b0-b847-485b-aff8-878c4080c014-dshm\") pod \"e2e-lifecycle-model-e1c7132e-kserve-7dc4df997f-ccwmn\" (UID: \"220087b0-b847-485b-aff8-878c4080c014\") " pod="ai-tenant-e2e-mt-e1c7132e/e2e-lifecycle-model-e1c7132e-kserve-7dc4df997f-ccwmn" Jul 22 03:11:11.343622 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:11:11.343548 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/220087b0-b847-485b-aff8-878c4080c014-model-cache\") pod \"e2e-lifecycle-model-e1c7132e-kserve-7dc4df997f-ccwmn\" (UID: \"220087b0-b847-485b-aff8-878c4080c014\") " pod="ai-tenant-e2e-mt-e1c7132e/e2e-lifecycle-model-e1c7132e-kserve-7dc4df997f-ccwmn" Jul 22 03:11:11.343793 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:11:11.343646 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/220087b0-b847-485b-aff8-878c4080c014-tls-certs\") pod \"e2e-lifecycle-model-e1c7132e-kserve-7dc4df997f-ccwmn\" (UID: \"220087b0-b847-485b-aff8-878c4080c014\") " pod="ai-tenant-e2e-mt-e1c7132e/e2e-lifecycle-model-e1c7132e-kserve-7dc4df997f-ccwmn" Jul 22 03:11:11.444494 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:11:11.444448 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/220087b0-b847-485b-aff8-878c4080c014-model-cache\") pod \"e2e-lifecycle-model-e1c7132e-kserve-7dc4df997f-ccwmn\" (UID: \"220087b0-b847-485b-aff8-878c4080c014\") " pod="ai-tenant-e2e-mt-e1c7132e/e2e-lifecycle-model-e1c7132e-kserve-7dc4df997f-ccwmn" Jul 22 03:11:11.444494 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:11:11.444502 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/220087b0-b847-485b-aff8-878c4080c014-tls-certs\") pod \"e2e-lifecycle-model-e1c7132e-kserve-7dc4df997f-ccwmn\" (UID: \"220087b0-b847-485b-aff8-878c4080c014\") " pod="ai-tenant-e2e-mt-e1c7132e/e2e-lifecycle-model-e1c7132e-kserve-7dc4df997f-ccwmn" Jul 22 03:11:11.444797 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:11:11.444549 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/220087b0-b847-485b-aff8-878c4080c014-tmp-dir\") pod \"e2e-lifecycle-model-e1c7132e-kserve-7dc4df997f-ccwmn\" (UID: \"220087b0-b847-485b-aff8-878c4080c014\") " pod="ai-tenant-e2e-mt-e1c7132e/e2e-lifecycle-model-e1c7132e-kserve-7dc4df997f-ccwmn" Jul 22 03:11:11.444797 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:11:11.444611 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/220087b0-b847-485b-aff8-878c4080c014-home\") pod \"e2e-lifecycle-model-e1c7132e-kserve-7dc4df997f-ccwmn\" (UID: \"220087b0-b847-485b-aff8-878c4080c014\") " pod="ai-tenant-e2e-mt-e1c7132e/e2e-lifecycle-model-e1c7132e-kserve-7dc4df997f-ccwmn" Jul 22 03:11:11.444797 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:11:11.444671 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-84zm5\" (UniqueName: \"kubernetes.io/projected/220087b0-b847-485b-aff8-878c4080c014-kube-api-access-84zm5\") pod \"e2e-lifecycle-model-e1c7132e-kserve-7dc4df997f-ccwmn\" (UID: \"220087b0-b847-485b-aff8-878c4080c014\") " pod="ai-tenant-e2e-mt-e1c7132e/e2e-lifecycle-model-e1c7132e-kserve-7dc4df997f-ccwmn" Jul 22 03:11:11.444797 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:11:11.444694 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/220087b0-b847-485b-aff8-878c4080c014-dshm\") pod \"e2e-lifecycle-model-e1c7132e-kserve-7dc4df997f-ccwmn\" (UID: \"220087b0-b847-485b-aff8-878c4080c014\") " pod="ai-tenant-e2e-mt-e1c7132e/e2e-lifecycle-model-e1c7132e-kserve-7dc4df997f-ccwmn" Jul 22 03:11:11.444994 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:11:11.444879 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/220087b0-b847-485b-aff8-878c4080c014-model-cache\") pod \"e2e-lifecycle-model-e1c7132e-kserve-7dc4df997f-ccwmn\" (UID: \"220087b0-b847-485b-aff8-878c4080c014\") " pod="ai-tenant-e2e-mt-e1c7132e/e2e-lifecycle-model-e1c7132e-kserve-7dc4df997f-ccwmn" Jul 22 03:11:11.444994 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:11:11.444933 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/220087b0-b847-485b-aff8-878c4080c014-tmp-dir\") pod \"e2e-lifecycle-model-e1c7132e-kserve-7dc4df997f-ccwmn\" (UID: \"220087b0-b847-485b-aff8-878c4080c014\") " pod="ai-tenant-e2e-mt-e1c7132e/e2e-lifecycle-model-e1c7132e-kserve-7dc4df997f-ccwmn" Jul 22 03:11:11.445070 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:11:11.445034 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/220087b0-b847-485b-aff8-878c4080c014-home\") pod \"e2e-lifecycle-model-e1c7132e-kserve-7dc4df997f-ccwmn\" (UID: \"220087b0-b847-485b-aff8-878c4080c014\") " pod="ai-tenant-e2e-mt-e1c7132e/e2e-lifecycle-model-e1c7132e-kserve-7dc4df997f-ccwmn" Jul 22 03:11:11.446990 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:11:11.446968 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/220087b0-b847-485b-aff8-878c4080c014-dshm\") pod \"e2e-lifecycle-model-e1c7132e-kserve-7dc4df997f-ccwmn\" (UID: \"220087b0-b847-485b-aff8-878c4080c014\") " pod="ai-tenant-e2e-mt-e1c7132e/e2e-lifecycle-model-e1c7132e-kserve-7dc4df997f-ccwmn" Jul 22 03:11:11.447267 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:11:11.447245 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/220087b0-b847-485b-aff8-878c4080c014-tls-certs\") pod \"e2e-lifecycle-model-e1c7132e-kserve-7dc4df997f-ccwmn\" (UID: \"220087b0-b847-485b-aff8-878c4080c014\") " pod="ai-tenant-e2e-mt-e1c7132e/e2e-lifecycle-model-e1c7132e-kserve-7dc4df997f-ccwmn" Jul 22 03:11:11.455107 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:11:11.455068 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-84zm5\" (UniqueName: \"kubernetes.io/projected/220087b0-b847-485b-aff8-878c4080c014-kube-api-access-84zm5\") pod \"e2e-lifecycle-model-e1c7132e-kserve-7dc4df997f-ccwmn\" (UID: \"220087b0-b847-485b-aff8-878c4080c014\") " pod="ai-tenant-e2e-mt-e1c7132e/e2e-lifecycle-model-e1c7132e-kserve-7dc4df997f-ccwmn" Jul 22 03:11:11.566344 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:11:11.566258 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-e1c7132e/e2e-lifecycle-model-e1c7132e-kserve-7dc4df997f-ccwmn" Jul 22 03:11:11.703831 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:11:11.703781 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-e1c7132e/e2e-lifecycle-model-e1c7132e-kserve-7dc4df997f-ccwmn"] Jul 22 03:11:11.706067 ip-10-0-133-31 kubenswrapper[2578]: W0722 03:11:11.706042 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod220087b0_b847_485b_aff8_878c4080c014.slice/crio-d5f9291df1c638e8d71227c181c1402cb8ff6b6eb5ec384a8ec237c153e05d67 WatchSource:0}: Error finding container d5f9291df1c638e8d71227c181c1402cb8ff6b6eb5ec384a8ec237c153e05d67: Status 404 returned error can't find the container with id d5f9291df1c638e8d71227c181c1402cb8ff6b6eb5ec384a8ec237c153e05d67 Jul 22 03:11:12.127855 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:11:12.127816 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-e1c7132e/e2e-lifecycle-model-e1c7132e-kserve-7dc4df997f-ccwmn" event={"ID":"220087b0-b847-485b-aff8-878c4080c014","Type":"ContainerStarted","Data":"7301cbb9774bd24aa2ad073eba812584d4ebe7d9f3ccc4165f5bbaa922928bb2"} Jul 22 03:11:12.128446 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:11:12.127868 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-e1c7132e/e2e-lifecycle-model-e1c7132e-kserve-7dc4df997f-ccwmn" event={"ID":"220087b0-b847-485b-aff8-878c4080c014","Type":"ContainerStarted","Data":"d5f9291df1c638e8d71227c181c1402cb8ff6b6eb5ec384a8ec237c153e05d67"} Jul 22 03:11:12.148035 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:11:12.147966 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-e1c7132e/e2e-lifecycle-model-e1c7132e-kserve-7dc4df997f-ccwmn" podStartSLOduration=1.147951767 podStartE2EDuration="1.147951767s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 03:11:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 03:11:12.14675151 +0000 UTC m=+2038.465926808" watchObservedRunningTime="2026-07-22 03:11:12.147951767 +0000 UTC m=+2038.467127067" Jul 22 03:11:21.566963 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:11:21.566902 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-e1c7132e/e2e-lifecycle-model-e1c7132e-kserve-7dc4df997f-ccwmn" Jul 22 03:11:21.566963 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:11:21.566976 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-e1c7132e/e2e-lifecycle-model-e1c7132e-kserve-7dc4df997f-ccwmn" Jul 22 03:11:21.579200 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:11:21.579173 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-e1c7132e/e2e-lifecycle-model-e1c7132e-kserve-7dc4df997f-ccwmn" Jul 22 03:11:22.191486 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:11:22.191453 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-e1c7132e/e2e-lifecycle-model-e1c7132e-kserve-7dc4df997f-ccwmn" Jul 22 03:11:58.627980 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:11:58.627900 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-e1c7132e/e2e-lifecycle-model-e1c7132e-kserve-7dc4df997f-ccwmn"] Jul 22 03:12:13.648585 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:12:13.648507 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-e1c7132e/e2e-lifecycle-model-e1c7132e-kserve-7dc4df997f-ccwmn" podUID="220087b0-b847-485b-aff8-878c4080c014" containerName="main" containerID="cri-o://7301cbb9774bd24aa2ad073eba812584d4ebe7d9f3ccc4165f5bbaa922928bb2" gracePeriod=45 Jul 22 03:12:13.911209 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:12:13.911141 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-e1c7132e/e2e-lifecycle-model-e1c7132e-kserve-7dc4df997f-ccwmn" Jul 22 03:12:13.944187 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:12:13.944160 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/220087b0-b847-485b-aff8-878c4080c014-dshm\") pod \"220087b0-b847-485b-aff8-878c4080c014\" (UID: \"220087b0-b847-485b-aff8-878c4080c014\") " Jul 22 03:12:13.944370 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:12:13.944206 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/220087b0-b847-485b-aff8-878c4080c014-model-cache\") pod \"220087b0-b847-485b-aff8-878c4080c014\" (UID: \"220087b0-b847-485b-aff8-878c4080c014\") " Jul 22 03:12:13.944370 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:12:13.944287 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/220087b0-b847-485b-aff8-878c4080c014-tls-certs\") pod \"220087b0-b847-485b-aff8-878c4080c014\" (UID: \"220087b0-b847-485b-aff8-878c4080c014\") " Jul 22 03:12:13.944502 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:12:13.944374 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/220087b0-b847-485b-aff8-878c4080c014-tmp-dir\") pod \"220087b0-b847-485b-aff8-878c4080c014\" (UID: \"220087b0-b847-485b-aff8-878c4080c014\") " Jul 22 03:12:13.944502 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:12:13.944421 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-84zm5\" (UniqueName: \"kubernetes.io/projected/220087b0-b847-485b-aff8-878c4080c014-kube-api-access-84zm5\") pod \"220087b0-b847-485b-aff8-878c4080c014\" (UID: \"220087b0-b847-485b-aff8-878c4080c014\") " Jul 22 03:12:13.944502 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:12:13.944465 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/220087b0-b847-485b-aff8-878c4080c014-home\") pod \"220087b0-b847-485b-aff8-878c4080c014\" (UID: \"220087b0-b847-485b-aff8-878c4080c014\") " Jul 22 03:12:13.944700 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:12:13.944542 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/220087b0-b847-485b-aff8-878c4080c014-model-cache" (OuterVolumeSpecName: "model-cache") pod "220087b0-b847-485b-aff8-878c4080c014" (UID: "220087b0-b847-485b-aff8-878c4080c014"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 03:12:13.944755 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:12:13.944690 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/220087b0-b847-485b-aff8-878c4080c014-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "220087b0-b847-485b-aff8-878c4080c014" (UID: "220087b0-b847-485b-aff8-878c4080c014"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 03:12:13.944812 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:12:13.944788 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/220087b0-b847-485b-aff8-878c4080c014-home" (OuterVolumeSpecName: "home") pod "220087b0-b847-485b-aff8-878c4080c014" (UID: "220087b0-b847-485b-aff8-878c4080c014"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 03:12:13.944914 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:12:13.944891 2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/220087b0-b847-485b-aff8-878c4080c014-model-cache\") on node \"ip-10-0-133-31.ec2.internal\" DevicePath \"\"" Jul 22 03:12:13.944988 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:12:13.944920 2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/220087b0-b847-485b-aff8-878c4080c014-tmp-dir\") on node \"ip-10-0-133-31.ec2.internal\" DevicePath \"\"" Jul 22 03:12:13.944988 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:12:13.944933 2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/220087b0-b847-485b-aff8-878c4080c014-home\") on node \"ip-10-0-133-31.ec2.internal\" DevicePath \"\"" Jul 22 03:12:13.946503 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:12:13.946478 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/220087b0-b847-485b-aff8-878c4080c014-dshm" (OuterVolumeSpecName: "dshm") pod "220087b0-b847-485b-aff8-878c4080c014" (UID: "220087b0-b847-485b-aff8-878c4080c014"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 03:12:13.946784 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:12:13.946760 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/220087b0-b847-485b-aff8-878c4080c014-kube-api-access-84zm5" (OuterVolumeSpecName: "kube-api-access-84zm5") pod "220087b0-b847-485b-aff8-878c4080c014" (UID: "220087b0-b847-485b-aff8-878c4080c014"). InnerVolumeSpecName "kube-api-access-84zm5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 03:12:13.947046 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:12:13.947030 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/220087b0-b847-485b-aff8-878c4080c014-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "220087b0-b847-485b-aff8-878c4080c014" (UID: "220087b0-b847-485b-aff8-878c4080c014"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 03:12:14.046127 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:12:14.046093 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-84zm5\" (UniqueName: \"kubernetes.io/projected/220087b0-b847-485b-aff8-878c4080c014-kube-api-access-84zm5\") on node \"ip-10-0-133-31.ec2.internal\" DevicePath \"\"" Jul 22 03:12:14.046127 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:12:14.046121 2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/220087b0-b847-485b-aff8-878c4080c014-dshm\") on node \"ip-10-0-133-31.ec2.internal\" DevicePath \"\"" Jul 22 03:12:14.046127 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:12:14.046130 2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/220087b0-b847-485b-aff8-878c4080c014-tls-certs\") on node \"ip-10-0-133-31.ec2.internal\" DevicePath \"\"" Jul 22 03:12:14.226897 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:12:14.226795 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-prfql_fd7bf923-8d16-476c-a8d0-a3e6fd645061/ovn-acl-logging/0.log" Jul 22 03:12:14.236620 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:12:14.236588 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-prfql_fd7bf923-8d16-476c-a8d0-a3e6fd645061/ovn-acl-logging/0.log" Jul 22 03:12:14.459667 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:12:14.459630 2578 generic.go:356] "Generic (PLEG): container finished" podID="220087b0-b847-485b-aff8-878c4080c014" containerID="7301cbb9774bd24aa2ad073eba812584d4ebe7d9f3ccc4165f5bbaa922928bb2" exitCode=0 Jul 22 03:12:14.459857 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:12:14.459686 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-e1c7132e/e2e-lifecycle-model-e1c7132e-kserve-7dc4df997f-ccwmn" Jul 22 03:12:14.459857 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:12:14.459706 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-e1c7132e/e2e-lifecycle-model-e1c7132e-kserve-7dc4df997f-ccwmn" event={"ID":"220087b0-b847-485b-aff8-878c4080c014","Type":"ContainerDied","Data":"7301cbb9774bd24aa2ad073eba812584d4ebe7d9f3ccc4165f5bbaa922928bb2"} Jul 22 03:12:14.459857 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:12:14.459750 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-e1c7132e/e2e-lifecycle-model-e1c7132e-kserve-7dc4df997f-ccwmn" event={"ID":"220087b0-b847-485b-aff8-878c4080c014","Type":"ContainerDied","Data":"d5f9291df1c638e8d71227c181c1402cb8ff6b6eb5ec384a8ec237c153e05d67"} Jul 22 03:12:14.459857 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:12:14.459773 2578 scope.go:117] "RemoveContainer" containerID="7301cbb9774bd24aa2ad073eba812584d4ebe7d9f3ccc4165f5bbaa922928bb2" Jul 22 03:12:14.468951 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:12:14.468934 2578 scope.go:117] "RemoveContainer" containerID="7301cbb9774bd24aa2ad073eba812584d4ebe7d9f3ccc4165f5bbaa922928bb2" Jul 22 03:12:14.469201 ip-10-0-133-31 kubenswrapper[2578]: E0722 03:12:14.469181 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7301cbb9774bd24aa2ad073eba812584d4ebe7d9f3ccc4165f5bbaa922928bb2\": container with ID starting with 7301cbb9774bd24aa2ad073eba812584d4ebe7d9f3ccc4165f5bbaa922928bb2 not found: ID does not exist" containerID="7301cbb9774bd24aa2ad073eba812584d4ebe7d9f3ccc4165f5bbaa922928bb2" Jul 22 03:12:14.469266 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:12:14.469209 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7301cbb9774bd24aa2ad073eba812584d4ebe7d9f3ccc4165f5bbaa922928bb2"} err="failed to get container status \"7301cbb9774bd24aa2ad073eba812584d4ebe7d9f3ccc4165f5bbaa922928bb2\": rpc error: code = NotFound desc = could not find container \"7301cbb9774bd24aa2ad073eba812584d4ebe7d9f3ccc4165f5bbaa922928bb2\": container with ID starting with 7301cbb9774bd24aa2ad073eba812584d4ebe7d9f3ccc4165f5bbaa922928bb2 not found: ID does not exist" Jul 22 03:12:14.477480 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:12:14.477420 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-e1c7132e/e2e-lifecycle-model-e1c7132e-kserve-7dc4df997f-ccwmn"] Jul 22 03:12:14.482397 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:12:14.482375 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-e1c7132e/e2e-lifecycle-model-e1c7132e-kserve-7dc4df997f-ccwmn"] Jul 22 03:12:16.142247 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:12:16.142214 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="220087b0-b847-485b-aff8-878c4080c014" path="/var/lib/kubelet/pods/220087b0-b847-485b-aff8-878c4080c014/volumes" Jul 22 03:12:57.278008 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:12:57.277978 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-e1c7132e-openshift-default-588794d9f9-v679l"] Jul 22 03:12:57.278502 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:12:57.278200 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-e1c7132e-openshift-default-588794d9f9-v679l" podUID="f18ffc96-ad50-42d7-af4a-0cdddf90fe5a" containerName="istio-proxy" containerID="cri-o://930d58ebb191f483811b3cbd9ad20945631c762b0992f6db568588ea018cbd75" gracePeriod=30 Jul 22 03:13:02.536266 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:13:02.536200 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-e1c7132e-openshift-default-588794d9f9-v679l" Jul 22 03:13:02.586045 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:13:02.586010 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jrl8d\" (UniqueName: \"kubernetes.io/projected/f18ffc96-ad50-42d7-af4a-0cdddf90fe5a-kube-api-access-jrl8d\") pod \"f18ffc96-ad50-42d7-af4a-0cdddf90fe5a\" (UID: \"f18ffc96-ad50-42d7-af4a-0cdddf90fe5a\") " Jul 22 03:13:02.586045 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:13:02.586044 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f18ffc96-ad50-42d7-af4a-0cdddf90fe5a-istio-token\") pod \"f18ffc96-ad50-42d7-af4a-0cdddf90fe5a\" (UID: \"f18ffc96-ad50-42d7-af4a-0cdddf90fe5a\") " Jul 22 03:13:02.586304 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:13:02.586076 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f18ffc96-ad50-42d7-af4a-0cdddf90fe5a-istio-podinfo\") pod \"f18ffc96-ad50-42d7-af4a-0cdddf90fe5a\" (UID: \"f18ffc96-ad50-42d7-af4a-0cdddf90fe5a\") " Jul 22 03:13:02.586304 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:13:02.586117 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f18ffc96-ad50-42d7-af4a-0cdddf90fe5a-istiod-ca-cert\") pod \"f18ffc96-ad50-42d7-af4a-0cdddf90fe5a\" (UID: \"f18ffc96-ad50-42d7-af4a-0cdddf90fe5a\") " Jul 22 03:13:02.586304 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:13:02.586155 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f18ffc96-ad50-42d7-af4a-0cdddf90fe5a-credential-socket\") pod \"f18ffc96-ad50-42d7-af4a-0cdddf90fe5a\" (UID: \"f18ffc96-ad50-42d7-af4a-0cdddf90fe5a\") " Jul 22 03:13:02.586304 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:13:02.586194 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f18ffc96-ad50-42d7-af4a-0cdddf90fe5a-istio-envoy\") pod \"f18ffc96-ad50-42d7-af4a-0cdddf90fe5a\" (UID: \"f18ffc96-ad50-42d7-af4a-0cdddf90fe5a\") " Jul 22 03:13:02.586304 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:13:02.586236 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f18ffc96-ad50-42d7-af4a-0cdddf90fe5a-workload-certs\") pod \"f18ffc96-ad50-42d7-af4a-0cdddf90fe5a\" (UID: \"f18ffc96-ad50-42d7-af4a-0cdddf90fe5a\") " Jul 22 03:13:02.586304 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:13:02.586280 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f18ffc96-ad50-42d7-af4a-0cdddf90fe5a-workload-socket\") pod \"f18ffc96-ad50-42d7-af4a-0cdddf90fe5a\" (UID: \"f18ffc96-ad50-42d7-af4a-0cdddf90fe5a\") " Jul 22 03:13:02.586671 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:13:02.586310 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f18ffc96-ad50-42d7-af4a-0cdddf90fe5a-istio-data\") pod \"f18ffc96-ad50-42d7-af4a-0cdddf90fe5a\" (UID: \"f18ffc96-ad50-42d7-af4a-0cdddf90fe5a\") " Jul 22 03:13:02.586671 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:13:02.586491 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f18ffc96-ad50-42d7-af4a-0cdddf90fe5a-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "f18ffc96-ad50-42d7-af4a-0cdddf90fe5a" (UID: "f18ffc96-ad50-42d7-af4a-0cdddf90fe5a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 03:13:02.586788 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:13:02.586721 2578 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f18ffc96-ad50-42d7-af4a-0cdddf90fe5a-istiod-ca-cert\") on node \"ip-10-0-133-31.ec2.internal\" DevicePath \"\"" Jul 22 03:13:02.586788 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:13:02.586766 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f18ffc96-ad50-42d7-af4a-0cdddf90fe5a-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "f18ffc96-ad50-42d7-af4a-0cdddf90fe5a" (UID: "f18ffc96-ad50-42d7-af4a-0cdddf90fe5a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 03:13:02.587042 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:13:02.586994 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f18ffc96-ad50-42d7-af4a-0cdddf90fe5a-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "f18ffc96-ad50-42d7-af4a-0cdddf90fe5a" (UID: "f18ffc96-ad50-42d7-af4a-0cdddf90fe5a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 03:13:02.587042 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:13:02.587018 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f18ffc96-ad50-42d7-af4a-0cdddf90fe5a-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "f18ffc96-ad50-42d7-af4a-0cdddf90fe5a" (UID: "f18ffc96-ad50-42d7-af4a-0cdddf90fe5a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 03:13:02.587429 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:13:02.587404 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f18ffc96-ad50-42d7-af4a-0cdddf90fe5a-istio-data" (OuterVolumeSpecName: "istio-data") pod "f18ffc96-ad50-42d7-af4a-0cdddf90fe5a" (UID: "f18ffc96-ad50-42d7-af4a-0cdddf90fe5a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 03:13:02.588536 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:13:02.588513 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f18ffc96-ad50-42d7-af4a-0cdddf90fe5a-kube-api-access-jrl8d" (OuterVolumeSpecName: "kube-api-access-jrl8d") pod "f18ffc96-ad50-42d7-af4a-0cdddf90fe5a" (UID: "f18ffc96-ad50-42d7-af4a-0cdddf90fe5a"). InnerVolumeSpecName "kube-api-access-jrl8d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 03:13:02.590702 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:13:02.588770 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f18ffc96-ad50-42d7-af4a-0cdddf90fe5a-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "f18ffc96-ad50-42d7-af4a-0cdddf90fe5a" (UID: "f18ffc96-ad50-42d7-af4a-0cdddf90fe5a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 03:13:02.590702 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:13:02.588842 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f18ffc96-ad50-42d7-af4a-0cdddf90fe5a-istio-token" (OuterVolumeSpecName: "istio-token") pod "f18ffc96-ad50-42d7-af4a-0cdddf90fe5a" (UID: "f18ffc96-ad50-42d7-af4a-0cdddf90fe5a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 03:13:02.590702 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:13:02.588888 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f18ffc96-ad50-42d7-af4a-0cdddf90fe5a-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "f18ffc96-ad50-42d7-af4a-0cdddf90fe5a" (UID: "f18ffc96-ad50-42d7-af4a-0cdddf90fe5a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 22 03:13:02.687529 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:13:02.687491 2578 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f18ffc96-ad50-42d7-af4a-0cdddf90fe5a-credential-socket\") on node \"ip-10-0-133-31.ec2.internal\" DevicePath \"\"" Jul 22 03:13:02.687529 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:13:02.687521 2578 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f18ffc96-ad50-42d7-af4a-0cdddf90fe5a-istio-envoy\") on node \"ip-10-0-133-31.ec2.internal\" DevicePath \"\"" Jul 22 03:13:02.687529 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:13:02.687532 2578 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f18ffc96-ad50-42d7-af4a-0cdddf90fe5a-workload-certs\") on node \"ip-10-0-133-31.ec2.internal\" DevicePath \"\"" Jul 22 03:13:02.687529 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:13:02.687539 2578 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f18ffc96-ad50-42d7-af4a-0cdddf90fe5a-workload-socket\") on node \"ip-10-0-133-31.ec2.internal\" DevicePath \"\"" Jul 22 03:13:02.687823 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:13:02.687549 2578 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f18ffc96-ad50-42d7-af4a-0cdddf90fe5a-istio-data\") on node \"ip-10-0-133-31.ec2.internal\" DevicePath \"\"" Jul 22 03:13:02.687823 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:13:02.687596 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jrl8d\" (UniqueName: \"kubernetes.io/projected/f18ffc96-ad50-42d7-af4a-0cdddf90fe5a-kube-api-access-jrl8d\") on node \"ip-10-0-133-31.ec2.internal\" DevicePath \"\"" Jul 22 03:13:02.687823 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:13:02.687609 2578 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f18ffc96-ad50-42d7-af4a-0cdddf90fe5a-istio-token\") on node \"ip-10-0-133-31.ec2.internal\" DevicePath \"\"" Jul 22 03:13:02.687823 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:13:02.687620 2578 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f18ffc96-ad50-42d7-af4a-0cdddf90fe5a-istio-podinfo\") on node \"ip-10-0-133-31.ec2.internal\" DevicePath \"\"" Jul 22 03:13:02.724180 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:13:02.724140 2578 generic.go:356] "Generic (PLEG): container finished" podID="f18ffc96-ad50-42d7-af4a-0cdddf90fe5a" containerID="930d58ebb191f483811b3cbd9ad20945631c762b0992f6db568588ea018cbd75" exitCode=0 Jul 22 03:13:02.724336 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:13:02.724192 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-e1c7132e-openshift-default-588794d9f9-v679l" Jul 22 03:13:02.724336 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:13:02.724216 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e1c7132e-openshift-default-588794d9f9-v679l" event={"ID":"f18ffc96-ad50-42d7-af4a-0cdddf90fe5a","Type":"ContainerDied","Data":"930d58ebb191f483811b3cbd9ad20945631c762b0992f6db568588ea018cbd75"} Jul 22 03:13:02.724336 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:13:02.724266 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e1c7132e-openshift-default-588794d9f9-v679l" event={"ID":"f18ffc96-ad50-42d7-af4a-0cdddf90fe5a","Type":"ContainerDied","Data":"7fad45c22d308312348773006f44d52c339b2a40add9d578ce8281742dfeddc7"} Jul 22 03:13:02.724336 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:13:02.724290 2578 scope.go:117] "RemoveContainer" containerID="930d58ebb191f483811b3cbd9ad20945631c762b0992f6db568588ea018cbd75" Jul 22 03:13:02.733828 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:13:02.733812 2578 scope.go:117] "RemoveContainer" containerID="b6254b611f97536d1891d3fa1b9cbc51324595d067449189f7cf1168a2be528b" Jul 22 03:13:02.742133 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:13:02.742113 2578 scope.go:117] "RemoveContainer" containerID="930d58ebb191f483811b3cbd9ad20945631c762b0992f6db568588ea018cbd75" Jul 22 03:13:02.742406 ip-10-0-133-31 kubenswrapper[2578]: E0722 03:13:02.742386 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"930d58ebb191f483811b3cbd9ad20945631c762b0992f6db568588ea018cbd75\": container with ID starting with 930d58ebb191f483811b3cbd9ad20945631c762b0992f6db568588ea018cbd75 not found: ID does not exist" containerID="930d58ebb191f483811b3cbd9ad20945631c762b0992f6db568588ea018cbd75" Jul 22 03:13:02.742484 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:13:02.742419 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"930d58ebb191f483811b3cbd9ad20945631c762b0992f6db568588ea018cbd75"} err="failed to get container status \"930d58ebb191f483811b3cbd9ad20945631c762b0992f6db568588ea018cbd75\": rpc error: code = NotFound desc = could not find container \"930d58ebb191f483811b3cbd9ad20945631c762b0992f6db568588ea018cbd75\": container with ID starting with 930d58ebb191f483811b3cbd9ad20945631c762b0992f6db568588ea018cbd75 not found: ID does not exist" Jul 22 03:13:02.742484 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:13:02.742442 2578 scope.go:117] "RemoveContainer" containerID="b6254b611f97536d1891d3fa1b9cbc51324595d067449189f7cf1168a2be528b" Jul 22 03:13:02.742721 ip-10-0-133-31 kubenswrapper[2578]: E0722 03:13:02.742703 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b6254b611f97536d1891d3fa1b9cbc51324595d067449189f7cf1168a2be528b\": container with ID starting with b6254b611f97536d1891d3fa1b9cbc51324595d067449189f7cf1168a2be528b not found: ID does not exist" containerID="b6254b611f97536d1891d3fa1b9cbc51324595d067449189f7cf1168a2be528b" Jul 22 03:13:02.742773 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:13:02.742727 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6254b611f97536d1891d3fa1b9cbc51324595d067449189f7cf1168a2be528b"} err="failed to get container status \"b6254b611f97536d1891d3fa1b9cbc51324595d067449189f7cf1168a2be528b\": rpc error: code = NotFound desc = could not find container \"b6254b611f97536d1891d3fa1b9cbc51324595d067449189f7cf1168a2be528b\": container with ID starting with b6254b611f97536d1891d3fa1b9cbc51324595d067449189f7cf1168a2be528b not found: ID does not exist" Jul 22 03:13:02.748225 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:13:02.748203 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-e1c7132e-openshift-default-588794d9f9-v679l"] Jul 22 03:13:02.754656 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:13:02.754632 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-e1c7132e-openshift-default-588794d9f9-v679l"] Jul 22 03:13:04.142414 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:13:04.142376 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f18ffc96-ad50-42d7-af4a-0cdddf90fe5a" path="/var/lib/kubelet/pods/f18ffc96-ad50-42d7-af4a-0cdddf90fe5a/volumes" Jul 22 03:13:54.117479 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:13:54.117441 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-9ded6d-openshift-default-7f969bd769-l268z"] Jul 22 03:13:54.117996 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:13:54.117847 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f18ffc96-ad50-42d7-af4a-0cdddf90fe5a" containerName="istio-proxy" Jul 22 03:13:54.117996 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:13:54.117858 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="f18ffc96-ad50-42d7-af4a-0cdddf90fe5a" containerName="istio-proxy" Jul 22 03:13:54.117996 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:13:54.117868 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="220087b0-b847-485b-aff8-878c4080c014" containerName="main" Jul 22 03:13:54.117996 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:13:54.117874 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="220087b0-b847-485b-aff8-878c4080c014" containerName="main" Jul 22 03:13:54.117996 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:13:54.117880 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f18ffc96-ad50-42d7-af4a-0cdddf90fe5a" containerName="istio-proxy" Jul 22 03:13:54.117996 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:13:54.117885 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="f18ffc96-ad50-42d7-af4a-0cdddf90fe5a" containerName="istio-proxy" Jul 22 03:13:54.117996 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:13:54.117935 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="220087b0-b847-485b-aff8-878c4080c014" containerName="main" Jul 22 03:13:54.117996 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:13:54.117943 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="f18ffc96-ad50-42d7-af4a-0cdddf90fe5a" containerName="istio-proxy" Jul 22 03:13:54.120962 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:13:54.120945 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-9ded6d-openshift-default-7f969bd769-l268z" Jul 22 03:13:54.123868 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:13:54.123846 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-9ded6d-openshift-default-dockercfg-gxddx\"" Jul 22 03:13:54.144336 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:13:54.144310 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-9ded6d-openshift-default-7f969bd769-l268z"] Jul 22 03:13:54.237710 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:13:54.237675 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca96aeae-09aa-46f7-88e2-8a15233737b6-istio-data\") pod \"e2e-inf-a-9ded6d-openshift-default-7f969bd769-l268z\" (UID: \"ca96aeae-09aa-46f7-88e2-8a15233737b6\") " pod="openshift-ingress/e2e-inf-a-9ded6d-openshift-default-7f969bd769-l268z" Jul 22 03:13:54.237902 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:13:54.237720 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca96aeae-09aa-46f7-88e2-8a15233737b6-workload-certs\") pod \"e2e-inf-a-9ded6d-openshift-default-7f969bd769-l268z\" (UID: \"ca96aeae-09aa-46f7-88e2-8a15233737b6\") " pod="openshift-ingress/e2e-inf-a-9ded6d-openshift-default-7f969bd769-l268z" Jul 22 03:13:54.237902 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:13:54.237740 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca96aeae-09aa-46f7-88e2-8a15233737b6-workload-socket\") pod \"e2e-inf-a-9ded6d-openshift-default-7f969bd769-l268z\" (UID: \"ca96aeae-09aa-46f7-88e2-8a15233737b6\") " pod="openshift-ingress/e2e-inf-a-9ded6d-openshift-default-7f969bd769-l268z" Jul 22 03:13:54.237902 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:13:54.237789 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ca96aeae-09aa-46f7-88e2-8a15233737b6-istio-token\") pod \"e2e-inf-a-9ded6d-openshift-default-7f969bd769-l268z\" (UID: \"ca96aeae-09aa-46f7-88e2-8a15233737b6\") " pod="openshift-ingress/e2e-inf-a-9ded6d-openshift-default-7f969bd769-l268z" Jul 22 03:13:54.237902 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:13:54.237839 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lx8x6\" (UniqueName: \"kubernetes.io/projected/ca96aeae-09aa-46f7-88e2-8a15233737b6-kube-api-access-lx8x6\") pod \"e2e-inf-a-9ded6d-openshift-default-7f969bd769-l268z\" (UID: \"ca96aeae-09aa-46f7-88e2-8a15233737b6\") " pod="openshift-ingress/e2e-inf-a-9ded6d-openshift-default-7f969bd769-l268z" Jul 22 03:13:54.237902 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:13:54.237883 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca96aeae-09aa-46f7-88e2-8a15233737b6-istio-podinfo\") pod \"e2e-inf-a-9ded6d-openshift-default-7f969bd769-l268z\" (UID: \"ca96aeae-09aa-46f7-88e2-8a15233737b6\") " pod="openshift-ingress/e2e-inf-a-9ded6d-openshift-default-7f969bd769-l268z" Jul 22 03:13:54.238170 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:13:54.237969 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca96aeae-09aa-46f7-88e2-8a15233737b6-credential-socket\") pod \"e2e-inf-a-9ded6d-openshift-default-7f969bd769-l268z\" (UID: \"ca96aeae-09aa-46f7-88e2-8a15233737b6\") " pod="openshift-ingress/e2e-inf-a-9ded6d-openshift-default-7f969bd769-l268z" Jul 22 03:13:54.238170 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:13:54.238097 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca96aeae-09aa-46f7-88e2-8a15233737b6-istio-envoy\") pod \"e2e-inf-a-9ded6d-openshift-default-7f969bd769-l268z\" (UID: \"ca96aeae-09aa-46f7-88e2-8a15233737b6\") " pod="openshift-ingress/e2e-inf-a-9ded6d-openshift-default-7f969bd769-l268z" Jul 22 03:13:54.238170 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:13:54.238129 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca96aeae-09aa-46f7-88e2-8a15233737b6-istiod-ca-cert\") pod \"e2e-inf-a-9ded6d-openshift-default-7f969bd769-l268z\" (UID: \"ca96aeae-09aa-46f7-88e2-8a15233737b6\") " pod="openshift-ingress/e2e-inf-a-9ded6d-openshift-default-7f969bd769-l268z" Jul 22 03:13:54.338766 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:13:54.338732 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca96aeae-09aa-46f7-88e2-8a15233737b6-credential-socket\") pod \"e2e-inf-a-9ded6d-openshift-default-7f969bd769-l268z\" (UID: \"ca96aeae-09aa-46f7-88e2-8a15233737b6\") " pod="openshift-ingress/e2e-inf-a-9ded6d-openshift-default-7f969bd769-l268z" Jul 22 03:13:54.338766 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:13:54.338777 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca96aeae-09aa-46f7-88e2-8a15233737b6-istio-envoy\") pod \"e2e-inf-a-9ded6d-openshift-default-7f969bd769-l268z\" (UID: \"ca96aeae-09aa-46f7-88e2-8a15233737b6\") " pod="openshift-ingress/e2e-inf-a-9ded6d-openshift-default-7f969bd769-l268z" Jul 22 03:13:54.339031 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:13:54.338797 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca96aeae-09aa-46f7-88e2-8a15233737b6-istiod-ca-cert\") pod \"e2e-inf-a-9ded6d-openshift-default-7f969bd769-l268z\" (UID: \"ca96aeae-09aa-46f7-88e2-8a15233737b6\") " pod="openshift-ingress/e2e-inf-a-9ded6d-openshift-default-7f969bd769-l268z" Jul 22 03:13:54.339031 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:13:54.338845 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca96aeae-09aa-46f7-88e2-8a15233737b6-istio-data\") pod \"e2e-inf-a-9ded6d-openshift-default-7f969bd769-l268z\" (UID: \"ca96aeae-09aa-46f7-88e2-8a15233737b6\") " pod="openshift-ingress/e2e-inf-a-9ded6d-openshift-default-7f969bd769-l268z" Jul 22 03:13:54.339031 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:13:54.338891 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca96aeae-09aa-46f7-88e2-8a15233737b6-workload-certs\") pod \"e2e-inf-a-9ded6d-openshift-default-7f969bd769-l268z\" (UID: \"ca96aeae-09aa-46f7-88e2-8a15233737b6\") " pod="openshift-ingress/e2e-inf-a-9ded6d-openshift-default-7f969bd769-l268z" Jul 22 03:13:54.339031 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:13:54.338919 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca96aeae-09aa-46f7-88e2-8a15233737b6-workload-socket\") pod \"e2e-inf-a-9ded6d-openshift-default-7f969bd769-l268z\" (UID: \"ca96aeae-09aa-46f7-88e2-8a15233737b6\") " pod="openshift-ingress/e2e-inf-a-9ded6d-openshift-default-7f969bd769-l268z" Jul 22 03:13:54.339031 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:13:54.338957 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ca96aeae-09aa-46f7-88e2-8a15233737b6-istio-token\") pod \"e2e-inf-a-9ded6d-openshift-default-7f969bd769-l268z\" (UID: \"ca96aeae-09aa-46f7-88e2-8a15233737b6\") " pod="openshift-ingress/e2e-inf-a-9ded6d-openshift-default-7f969bd769-l268z" Jul 22 03:13:54.339031 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:13:54.338986 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lx8x6\" (UniqueName: \"kubernetes.io/projected/ca96aeae-09aa-46f7-88e2-8a15233737b6-kube-api-access-lx8x6\") pod \"e2e-inf-a-9ded6d-openshift-default-7f969bd769-l268z\" (UID: \"ca96aeae-09aa-46f7-88e2-8a15233737b6\") " pod="openshift-ingress/e2e-inf-a-9ded6d-openshift-default-7f969bd769-l268z" Jul 22 03:13:54.339282 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:13:54.339042 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca96aeae-09aa-46f7-88e2-8a15233737b6-istio-podinfo\") pod \"e2e-inf-a-9ded6d-openshift-default-7f969bd769-l268z\" (UID: \"ca96aeae-09aa-46f7-88e2-8a15233737b6\") " pod="openshift-ingress/e2e-inf-a-9ded6d-openshift-default-7f969bd769-l268z" Jul 22 03:13:54.339282 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:13:54.339209 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca96aeae-09aa-46f7-88e2-8a15233737b6-credential-socket\") pod \"e2e-inf-a-9ded6d-openshift-default-7f969bd769-l268z\" (UID: \"ca96aeae-09aa-46f7-88e2-8a15233737b6\") " pod="openshift-ingress/e2e-inf-a-9ded6d-openshift-default-7f969bd769-l268z" Jul 22 03:13:54.339369 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:13:54.339309 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca96aeae-09aa-46f7-88e2-8a15233737b6-workload-certs\") pod \"e2e-inf-a-9ded6d-openshift-default-7f969bd769-l268z\" (UID: \"ca96aeae-09aa-46f7-88e2-8a15233737b6\") " pod="openshift-ingress/e2e-inf-a-9ded6d-openshift-default-7f969bd769-l268z" Jul 22 03:13:54.339553 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:13:54.339526 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca96aeae-09aa-46f7-88e2-8a15233737b6-istiod-ca-cert\") pod \"e2e-inf-a-9ded6d-openshift-default-7f969bd769-l268z\" (UID: \"ca96aeae-09aa-46f7-88e2-8a15233737b6\") " pod="openshift-ingress/e2e-inf-a-9ded6d-openshift-default-7f969bd769-l268z" Jul 22 03:13:54.340158 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:13:54.339642 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca96aeae-09aa-46f7-88e2-8a15233737b6-istio-data\") pod \"e2e-inf-a-9ded6d-openshift-default-7f969bd769-l268z\" (UID: \"ca96aeae-09aa-46f7-88e2-8a15233737b6\") " pod="openshift-ingress/e2e-inf-a-9ded6d-openshift-default-7f969bd769-l268z" Jul 22 03:13:54.340158 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:13:54.339732 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca96aeae-09aa-46f7-88e2-8a15233737b6-workload-socket\") pod \"e2e-inf-a-9ded6d-openshift-default-7f969bd769-l268z\" (UID: \"ca96aeae-09aa-46f7-88e2-8a15233737b6\") " pod="openshift-ingress/e2e-inf-a-9ded6d-openshift-default-7f969bd769-l268z" Jul 22 03:13:54.341623 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:13:54.341602 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca96aeae-09aa-46f7-88e2-8a15233737b6-istio-envoy\") pod \"e2e-inf-a-9ded6d-openshift-default-7f969bd769-l268z\" (UID: \"ca96aeae-09aa-46f7-88e2-8a15233737b6\") " pod="openshift-ingress/e2e-inf-a-9ded6d-openshift-default-7f969bd769-l268z" Jul 22 03:13:54.341834 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:13:54.341815 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca96aeae-09aa-46f7-88e2-8a15233737b6-istio-podinfo\") pod \"e2e-inf-a-9ded6d-openshift-default-7f969bd769-l268z\" (UID: \"ca96aeae-09aa-46f7-88e2-8a15233737b6\") " pod="openshift-ingress/e2e-inf-a-9ded6d-openshift-default-7f969bd769-l268z" Jul 22 03:13:54.349960 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:13:54.349935 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ca96aeae-09aa-46f7-88e2-8a15233737b6-istio-token\") pod \"e2e-inf-a-9ded6d-openshift-default-7f969bd769-l268z\" (UID: \"ca96aeae-09aa-46f7-88e2-8a15233737b6\") " pod="openshift-ingress/e2e-inf-a-9ded6d-openshift-default-7f969bd769-l268z" Jul 22 03:13:54.350207 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:13:54.350168 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lx8x6\" (UniqueName: \"kubernetes.io/projected/ca96aeae-09aa-46f7-88e2-8a15233737b6-kube-api-access-lx8x6\") pod \"e2e-inf-a-9ded6d-openshift-default-7f969bd769-l268z\" (UID: \"ca96aeae-09aa-46f7-88e2-8a15233737b6\") " pod="openshift-ingress/e2e-inf-a-9ded6d-openshift-default-7f969bd769-l268z" Jul 22 03:13:54.432304 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:13:54.432206 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-9ded6d-openshift-default-7f969bd769-l268z" Jul 22 03:13:54.589582 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:13:54.589522 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-9ded6d-openshift-default-7f969bd769-l268z"] Jul 22 03:13:54.590529 ip-10-0-133-31 kubenswrapper[2578]: W0722 03:13:54.590501 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca96aeae_09aa_46f7_88e2_8a15233737b6.slice/crio-38f7fd9be2c3232684cf5e0f403c2052929a1b2558c9a3c7ebc6bfac721d34d8 WatchSource:0}: Error finding container 38f7fd9be2c3232684cf5e0f403c2052929a1b2558c9a3c7ebc6bfac721d34d8: Status 404 returned error can't find the container with id 38f7fd9be2c3232684cf5e0f403c2052929a1b2558c9a3c7ebc6bfac721d34d8 Jul 22 03:13:54.593018 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:13:54.592987 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 22 03:13:54.593137 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:13:54.593051 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 22 03:13:54.593137 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:13:54.593082 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 22 03:13:54.995328 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:13:54.995285 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-9ded6d-openshift-default-7f969bd769-l268z" event={"ID":"ca96aeae-09aa-46f7-88e2-8a15233737b6","Type":"ContainerStarted","Data":"21ab39aaa43de9950fcc75e94d63cb3f5139ab210d1901b75161496742e85598"} Jul 22 03:13:54.995620 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:13:54.995385 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-9ded6d-openshift-default-7f969bd769-l268z" event={"ID":"ca96aeae-09aa-46f7-88e2-8a15233737b6","Type":"ContainerStarted","Data":"38f7fd9be2c3232684cf5e0f403c2052929a1b2558c9a3c7ebc6bfac721d34d8"} Jul 22 03:13:55.020594 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:13:55.020521 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-9ded6d-openshift-default-7f969bd769-l268z" podStartSLOduration=1.020507139 podStartE2EDuration="1.020507139s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 03:13:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 03:13:55.017598362 +0000 UTC m=+2201.336773657" watchObservedRunningTime="2026-07-22 03:13:55.020507139 +0000 UTC m=+2201.339682515" Jul 22 03:13:55.433412 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:13:55.433376 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-9ded6d-openshift-default-7f969bd769-l268z" Jul 22 03:13:55.438665 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:13:55.438640 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-9ded6d-openshift-default-7f969bd769-l268z" Jul 22 03:13:56.000955 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:13:56.000923 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-9ded6d-openshift-default-7f969bd769-l268z" Jul 22 03:13:56.001989 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:13:56.001967 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-9ded6d-openshift-default-7f969bd769-l268z" Jul 22 03:14:02.269252 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:14:02.269219 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-fec108-openshift-default-584cbd6489-vbqmp"] Jul 22 03:14:02.269898 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:14:02.269672 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="f18ffc96-ad50-42d7-af4a-0cdddf90fe5a" containerName="istio-proxy" Jul 22 03:14:02.273026 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:14:02.273009 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-fec108-openshift-default-584cbd6489-vbqmp" Jul 22 03:14:02.276655 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:14:02.276631 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-fec108-openshift-default-dockercfg-j746t\"" Jul 22 03:14:02.301604 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:14:02.301575 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-fec108-openshift-default-584cbd6489-vbqmp"] Jul 22 03:14:02.316642 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:14:02.316616 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/232038b8-4749-4401-959f-d4456462eff3-istiod-ca-cert\") pod \"e2e-inf-b-fec108-openshift-default-584cbd6489-vbqmp\" (UID: \"232038b8-4749-4401-959f-d4456462eff3\") " pod="openshift-ingress/e2e-inf-b-fec108-openshift-default-584cbd6489-vbqmp" Jul 22 03:14:02.316790 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:14:02.316654 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/232038b8-4749-4401-959f-d4456462eff3-istio-token\") pod \"e2e-inf-b-fec108-openshift-default-584cbd6489-vbqmp\" (UID: \"232038b8-4749-4401-959f-d4456462eff3\") " pod="openshift-ingress/e2e-inf-b-fec108-openshift-default-584cbd6489-vbqmp" Jul 22 03:14:02.316790 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:14:02.316672 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6b5j6\" (UniqueName: \"kubernetes.io/projected/232038b8-4749-4401-959f-d4456462eff3-kube-api-access-6b5j6\") pod \"e2e-inf-b-fec108-openshift-default-584cbd6489-vbqmp\" (UID: \"232038b8-4749-4401-959f-d4456462eff3\") " pod="openshift-ingress/e2e-inf-b-fec108-openshift-default-584cbd6489-vbqmp" Jul 22 03:14:02.316901 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:14:02.316789 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/232038b8-4749-4401-959f-d4456462eff3-workload-certs\") pod \"e2e-inf-b-fec108-openshift-default-584cbd6489-vbqmp\" (UID: \"232038b8-4749-4401-959f-d4456462eff3\") " pod="openshift-ingress/e2e-inf-b-fec108-openshift-default-584cbd6489-vbqmp" Jul 22 03:14:02.316901 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:14:02.316833 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/232038b8-4749-4401-959f-d4456462eff3-credential-socket\") pod \"e2e-inf-b-fec108-openshift-default-584cbd6489-vbqmp\" (UID: \"232038b8-4749-4401-959f-d4456462eff3\") " pod="openshift-ingress/e2e-inf-b-fec108-openshift-default-584cbd6489-vbqmp" Jul 22 03:14:02.316901 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:14:02.316857 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/232038b8-4749-4401-959f-d4456462eff3-workload-socket\") pod \"e2e-inf-b-fec108-openshift-default-584cbd6489-vbqmp\" (UID: \"232038b8-4749-4401-959f-d4456462eff3\") " pod="openshift-ingress/e2e-inf-b-fec108-openshift-default-584cbd6489-vbqmp" Jul 22 03:14:02.316901 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:14:02.316872 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/232038b8-4749-4401-959f-d4456462eff3-istio-data\") pod \"e2e-inf-b-fec108-openshift-default-584cbd6489-vbqmp\" (UID: \"232038b8-4749-4401-959f-d4456462eff3\") " pod="openshift-ingress/e2e-inf-b-fec108-openshift-default-584cbd6489-vbqmp" Jul 22 03:14:02.316901 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:14:02.316890 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/232038b8-4749-4401-959f-d4456462eff3-istio-envoy\") pod \"e2e-inf-b-fec108-openshift-default-584cbd6489-vbqmp\" (UID: \"232038b8-4749-4401-959f-d4456462eff3\") " pod="openshift-ingress/e2e-inf-b-fec108-openshift-default-584cbd6489-vbqmp" Jul 22 03:14:02.317162 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:14:02.316975 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/232038b8-4749-4401-959f-d4456462eff3-istio-podinfo\") pod \"e2e-inf-b-fec108-openshift-default-584cbd6489-vbqmp\" (UID: \"232038b8-4749-4401-959f-d4456462eff3\") " pod="openshift-ingress/e2e-inf-b-fec108-openshift-default-584cbd6489-vbqmp" Jul 22 03:14:02.417919 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:14:02.417882 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/232038b8-4749-4401-959f-d4456462eff3-workload-certs\") pod \"e2e-inf-b-fec108-openshift-default-584cbd6489-vbqmp\" (UID: \"232038b8-4749-4401-959f-d4456462eff3\") " pod="openshift-ingress/e2e-inf-b-fec108-openshift-default-584cbd6489-vbqmp" Jul 22 03:14:02.418081 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:14:02.417927 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/232038b8-4749-4401-959f-d4456462eff3-credential-socket\") pod \"e2e-inf-b-fec108-openshift-default-584cbd6489-vbqmp\" (UID: \"232038b8-4749-4401-959f-d4456462eff3\") " pod="openshift-ingress/e2e-inf-b-fec108-openshift-default-584cbd6489-vbqmp" Jul 22 03:14:02.418081 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:14:02.417951 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/232038b8-4749-4401-959f-d4456462eff3-workload-socket\") pod \"e2e-inf-b-fec108-openshift-default-584cbd6489-vbqmp\" (UID: \"232038b8-4749-4401-959f-d4456462eff3\") " pod="openshift-ingress/e2e-inf-b-fec108-openshift-default-584cbd6489-vbqmp" Jul 22 03:14:02.418081 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:14:02.417967 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/232038b8-4749-4401-959f-d4456462eff3-istio-data\") pod \"e2e-inf-b-fec108-openshift-default-584cbd6489-vbqmp\" (UID: \"232038b8-4749-4401-959f-d4456462eff3\") " pod="openshift-ingress/e2e-inf-b-fec108-openshift-default-584cbd6489-vbqmp" Jul 22 03:14:02.418081 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:14:02.417986 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/232038b8-4749-4401-959f-d4456462eff3-istio-envoy\") pod \"e2e-inf-b-fec108-openshift-default-584cbd6489-vbqmp\" (UID: \"232038b8-4749-4401-959f-d4456462eff3\") " pod="openshift-ingress/e2e-inf-b-fec108-openshift-default-584cbd6489-vbqmp" Jul 22 03:14:02.418081 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:14:02.418026 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/232038b8-4749-4401-959f-d4456462eff3-istio-podinfo\") pod \"e2e-inf-b-fec108-openshift-default-584cbd6489-vbqmp\" (UID: \"232038b8-4749-4401-959f-d4456462eff3\") " pod="openshift-ingress/e2e-inf-b-fec108-openshift-default-584cbd6489-vbqmp" Jul 22 03:14:02.418081 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:14:02.418076 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/232038b8-4749-4401-959f-d4456462eff3-istiod-ca-cert\") pod \"e2e-inf-b-fec108-openshift-default-584cbd6489-vbqmp\" (UID: \"232038b8-4749-4401-959f-d4456462eff3\") " pod="openshift-ingress/e2e-inf-b-fec108-openshift-default-584cbd6489-vbqmp" Jul 22 03:14:02.418370 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:14:02.418109 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/232038b8-4749-4401-959f-d4456462eff3-istio-token\") pod \"e2e-inf-b-fec108-openshift-default-584cbd6489-vbqmp\" (UID: \"232038b8-4749-4401-959f-d4456462eff3\") " pod="openshift-ingress/e2e-inf-b-fec108-openshift-default-584cbd6489-vbqmp" Jul 22 03:14:02.418370 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:14:02.418131 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6b5j6\" (UniqueName: \"kubernetes.io/projected/232038b8-4749-4401-959f-d4456462eff3-kube-api-access-6b5j6\") pod \"e2e-inf-b-fec108-openshift-default-584cbd6489-vbqmp\" (UID: \"232038b8-4749-4401-959f-d4456462eff3\") " pod="openshift-ingress/e2e-inf-b-fec108-openshift-default-584cbd6489-vbqmp" Jul 22 03:14:02.418370 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:14:02.418307 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/232038b8-4749-4401-959f-d4456462eff3-workload-certs\") pod \"e2e-inf-b-fec108-openshift-default-584cbd6489-vbqmp\" (UID: \"232038b8-4749-4401-959f-d4456462eff3\") " pod="openshift-ingress/e2e-inf-b-fec108-openshift-default-584cbd6489-vbqmp" Jul 22 03:14:02.418370 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:14:02.418329 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/232038b8-4749-4401-959f-d4456462eff3-credential-socket\") pod \"e2e-inf-b-fec108-openshift-default-584cbd6489-vbqmp\" (UID: \"232038b8-4749-4401-959f-d4456462eff3\") " pod="openshift-ingress/e2e-inf-b-fec108-openshift-default-584cbd6489-vbqmp" Jul 22 03:14:02.418603 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:14:02.418488 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/232038b8-4749-4401-959f-d4456462eff3-istio-data\") pod \"e2e-inf-b-fec108-openshift-default-584cbd6489-vbqmp\" (UID: \"232038b8-4749-4401-959f-d4456462eff3\") " pod="openshift-ingress/e2e-inf-b-fec108-openshift-default-584cbd6489-vbqmp" Jul 22 03:14:02.418696 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:14:02.418675 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/232038b8-4749-4401-959f-d4456462eff3-workload-socket\") pod \"e2e-inf-b-fec108-openshift-default-584cbd6489-vbqmp\" (UID: \"232038b8-4749-4401-959f-d4456462eff3\") " pod="openshift-ingress/e2e-inf-b-fec108-openshift-default-584cbd6489-vbqmp" Jul 22 03:14:02.418884 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:14:02.418863 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/232038b8-4749-4401-959f-d4456462eff3-istiod-ca-cert\") pod \"e2e-inf-b-fec108-openshift-default-584cbd6489-vbqmp\" (UID: \"232038b8-4749-4401-959f-d4456462eff3\") " pod="openshift-ingress/e2e-inf-b-fec108-openshift-default-584cbd6489-vbqmp" Jul 22 03:14:02.420418 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:14:02.420397 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/232038b8-4749-4401-959f-d4456462eff3-istio-envoy\") pod \"e2e-inf-b-fec108-openshift-default-584cbd6489-vbqmp\" (UID: \"232038b8-4749-4401-959f-d4456462eff3\") " pod="openshift-ingress/e2e-inf-b-fec108-openshift-default-584cbd6489-vbqmp" Jul 22 03:14:02.420703 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:14:02.420685 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/232038b8-4749-4401-959f-d4456462eff3-istio-podinfo\") pod \"e2e-inf-b-fec108-openshift-default-584cbd6489-vbqmp\" (UID: \"232038b8-4749-4401-959f-d4456462eff3\") " pod="openshift-ingress/e2e-inf-b-fec108-openshift-default-584cbd6489-vbqmp" Jul 22 03:14:02.437244 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:14:02.437212 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6b5j6\" (UniqueName: \"kubernetes.io/projected/232038b8-4749-4401-959f-d4456462eff3-kube-api-access-6b5j6\") pod \"e2e-inf-b-fec108-openshift-default-584cbd6489-vbqmp\" (UID: \"232038b8-4749-4401-959f-d4456462eff3\") " pod="openshift-ingress/e2e-inf-b-fec108-openshift-default-584cbd6489-vbqmp" Jul 22 03:14:02.437439 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:14:02.437419 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/232038b8-4749-4401-959f-d4456462eff3-istio-token\") pod \"e2e-inf-b-fec108-openshift-default-584cbd6489-vbqmp\" (UID: \"232038b8-4749-4401-959f-d4456462eff3\") " pod="openshift-ingress/e2e-inf-b-fec108-openshift-default-584cbd6489-vbqmp" Jul 22 03:14:02.583596 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:14:02.583553 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-fec108-openshift-default-584cbd6489-vbqmp" Jul 22 03:14:02.732347 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:14:02.732303 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-fec108-openshift-default-584cbd6489-vbqmp"] Jul 22 03:14:02.734159 ip-10-0-133-31 kubenswrapper[2578]: W0722 03:14:02.734133 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod232038b8_4749_4401_959f_d4456462eff3.slice/crio-74ab63fc7220697d8406b53061ded846bb82d6f07dd03527e5ea683369c956b6 WatchSource:0}: Error finding container 74ab63fc7220697d8406b53061ded846bb82d6f07dd03527e5ea683369c956b6: Status 404 returned error can't find the container with id 74ab63fc7220697d8406b53061ded846bb82d6f07dd03527e5ea683369c956b6 Jul 22 03:14:02.737668 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:14:02.737637 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 22 03:14:02.737769 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:14:02.737705 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 22 03:14:02.737769 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:14:02.737732 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 22 03:14:03.039363 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:14:03.039279 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-fec108-openshift-default-584cbd6489-vbqmp" event={"ID":"232038b8-4749-4401-959f-d4456462eff3","Type":"ContainerStarted","Data":"902c3e8e8d43c7344902b7d30d927e5ab71b85a476ebf12255c4c028fb57857e"} Jul 22 03:14:03.039363 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:14:03.039340 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-fec108-openshift-default-584cbd6489-vbqmp" event={"ID":"232038b8-4749-4401-959f-d4456462eff3","Type":"ContainerStarted","Data":"74ab63fc7220697d8406b53061ded846bb82d6f07dd03527e5ea683369c956b6"} Jul 22 03:14:03.070497 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:14:03.070448 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-fec108-openshift-default-584cbd6489-vbqmp" podStartSLOduration=1.0704335249999999 podStartE2EDuration="1.070433525s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 03:14:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 03:14:03.069735512 +0000 UTC m=+2209.388910813" watchObservedRunningTime="2026-07-22 03:14:03.070433525 +0000 UTC m=+2209.389608827" Jul 22 03:14:03.584747 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:14:03.584711 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-fec108-openshift-default-584cbd6489-vbqmp" Jul 22 03:14:03.594081 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:14:03.594058 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-fec108-openshift-default-584cbd6489-vbqmp" Jul 22 03:14:04.044543 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:14:04.044451 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-fec108-openshift-default-584cbd6489-vbqmp" Jul 22 03:14:04.045426 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:14:04.045404 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-fec108-openshift-default-584cbd6489-vbqmp" Jul 22 03:14:10.079956 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:14:10.079921 2578 generic.go:356] "Generic (PLEG): container finished" podID="ca96aeae-09aa-46f7-88e2-8a15233737b6" containerID="21ab39aaa43de9950fcc75e94d63cb3f5139ab210d1901b75161496742e85598" exitCode=0 Jul 22 03:14:10.079956 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:14:10.079966 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-9ded6d-openshift-default-7f969bd769-l268z" event={"ID":"ca96aeae-09aa-46f7-88e2-8a15233737b6","Type":"ContainerDied","Data":"21ab39aaa43de9950fcc75e94d63cb3f5139ab210d1901b75161496742e85598"} Jul 22 03:14:10.080541 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:14:10.080324 2578 scope.go:117] "RemoveContainer" containerID="21ab39aaa43de9950fcc75e94d63cb3f5139ab210d1901b75161496742e85598" Jul 22 03:14:10.081398 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:14:10.081373 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 22 03:14:10.081472 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:14:10.081444 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 22 03:14:10.081515 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:14:10.081474 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 22 03:14:11.001061 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:14:11.001010 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-9ded6d-openshift-default-7f969bd769-l268z" Jul 22 03:14:11.097823 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:14:11.097765 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-9ded6d-openshift-default-7f969bd769-l268z" event={"ID":"ca96aeae-09aa-46f7-88e2-8a15233737b6","Type":"ContainerStarted","Data":"e7845aa99734067e03b639e0c9290d18aa97d765932c34524e0480215829f23a"} Jul 22 03:14:11.434546 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:14:11.434499 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-9ded6d-openshift-default-7f969bd769-l268z" Jul 22 03:14:11.434776 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:14:11.434660 2578 patch_prober.go:28] interesting pod/e2e-inf-a-9ded6d-openshift-default-7f969bd769-l268z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Jul 22 03:14:11.434776 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:14:11.434722 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-9ded6d-openshift-default-7f969bd769-l268z" podUID="ca96aeae-09aa-46f7-88e2-8a15233737b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Jul 22 03:14:12.433260 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:14:12.433222 2578 patch_prober.go:28] interesting pod/e2e-inf-a-9ded6d-openshift-default-7f969bd769-l268z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Jul 22 03:14:12.433891 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:14:12.433858 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-9ded6d-openshift-default-7f969bd769-l268z" podUID="ca96aeae-09aa-46f7-88e2-8a15233737b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Jul 22 03:14:13.433172 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:14:13.433130 2578 patch_prober.go:28] interesting pod/e2e-inf-a-9ded6d-openshift-default-7f969bd769-l268z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Jul 22 03:14:13.433905 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:14:13.433478 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-9ded6d-openshift-default-7f969bd769-l268z" podUID="ca96aeae-09aa-46f7-88e2-8a15233737b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Jul 22 03:14:14.432872 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:14:14.432825 2578 patch_prober.go:28] interesting pod/e2e-inf-a-9ded6d-openshift-default-7f969bd769-l268z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Jul 22 03:14:14.433065 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:14:14.432894 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-9ded6d-openshift-default-7f969bd769-l268z" podUID="ca96aeae-09aa-46f7-88e2-8a15233737b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Jul 22 03:14:15.436395 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:14:15.436352 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-9ded6d-openshift-default-7f969bd769-l268z" Jul 22 03:14:15.436909 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:14:15.436740 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-9ded6d-openshift-default-7f969bd769-l268z" Jul 22 03:14:15.437692 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:14:15.437662 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-9ded6d-openshift-default-7f969bd769-l268z" Jul 22 03:14:18.144967 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:14:18.144932 2578 generic.go:356] "Generic (PLEG): container finished" podID="232038b8-4749-4401-959f-d4456462eff3" containerID="902c3e8e8d43c7344902b7d30d927e5ab71b85a476ebf12255c4c028fb57857e" exitCode=0 Jul 22 03:14:18.145389 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:14:18.145005 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-fec108-openshift-default-584cbd6489-vbqmp" event={"ID":"232038b8-4749-4401-959f-d4456462eff3","Type":"ContainerDied","Data":"902c3e8e8d43c7344902b7d30d927e5ab71b85a476ebf12255c4c028fb57857e"} Jul 22 03:14:18.145431 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:14:18.145419 2578 scope.go:117] "RemoveContainer" containerID="902c3e8e8d43c7344902b7d30d927e5ab71b85a476ebf12255c4c028fb57857e" Jul 22 03:14:18.146433 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:14:18.146411 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 22 03:14:18.146485 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:14:18.146475 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 22 03:14:18.146519 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:14:18.146509 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 22 03:14:19.045110 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:14:19.045059 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-fec108-openshift-default-584cbd6489-vbqmp" Jul 22 03:14:19.151670 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:14:19.151632 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-fec108-openshift-default-584cbd6489-vbqmp" event={"ID":"232038b8-4749-4401-959f-d4456462eff3","Type":"ContainerStarted","Data":"f36dc69cb54f3a024ed4e0e35740fa58b70f04d9b4cf99735ac0aedecf3d6b8e"} Jul 22 03:14:19.584483 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:14:19.584453 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-fec108-openshift-default-584cbd6489-vbqmp" Jul 22 03:14:19.585712 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:14:19.585688 2578 patch_prober.go:28] interesting pod/e2e-inf-b-fec108-openshift-default-584cbd6489-vbqmp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body= Jul 22 03:14:19.585838 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:14:19.585746 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-fec108-openshift-default-584cbd6489-vbqmp" podUID="232038b8-4749-4401-959f-d4456462eff3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" Jul 22 03:14:20.584626 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:14:20.584593 2578 patch_prober.go:28] interesting pod/e2e-inf-b-fec108-openshift-default-584cbd6489-vbqmp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body= Jul 22 03:14:20.585000 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:14:20.584649 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-fec108-openshift-default-584cbd6489-vbqmp" podUID="232038b8-4749-4401-959f-d4456462eff3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" Jul 22 03:14:21.583932 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:14:21.583900 2578 patch_prober.go:28] interesting pod/e2e-inf-b-fec108-openshift-default-584cbd6489-vbqmp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body= Jul 22 03:14:21.584110 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:14:21.583955 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-fec108-openshift-default-584cbd6489-vbqmp" podUID="232038b8-4749-4401-959f-d4456462eff3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" Jul 22 03:14:22.584893 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:14:22.584853 2578 patch_prober.go:28] interesting pod/e2e-inf-b-fec108-openshift-default-584cbd6489-vbqmp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body= Jul 22 03:14:22.585271 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:14:22.584911 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-fec108-openshift-default-584cbd6489-vbqmp" podUID="232038b8-4749-4401-959f-d4456462eff3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" Jul 22 03:14:23.588289 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:14:23.588257 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-fec108-openshift-default-584cbd6489-vbqmp" Jul 22 03:14:23.588725 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:14:23.588615 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-fec108-openshift-default-584cbd6489-vbqmp" Jul 22 03:14:23.589543 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:14:23.589525 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-fec108-openshift-default-584cbd6489-vbqmp" Jul 22 03:16:12.531436 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:16:12.531400 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-9ded6d-openshift-default-7f969bd769-l268z"] Jul 22 03:16:12.531904 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:16:12.531661 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-9ded6d-openshift-default-7f969bd769-l268z" podUID="ca96aeae-09aa-46f7-88e2-8a15233737b6" containerName="istio-proxy" containerID="cri-o://e7845aa99734067e03b639e0c9290d18aa97d765932c34524e0480215829f23a" gracePeriod=30 Jul 22 03:16:15.438422 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:16:15.438385 2578 patch_prober.go:28] interesting pod/e2e-inf-a-9ded6d-openshift-default-7f969bd769-l268z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 22 03:16:15.438794 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:16:15.438447 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-9ded6d-openshift-default-7f969bd769-l268z" podUID="ca96aeae-09aa-46f7-88e2-8a15233737b6" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 22 03:16:17.784684 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:16:17.784660 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-9ded6d-openshift-default-7f969bd769-l268z" Jul 22 03:16:17.792620 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:16:17.792596 2578 generic.go:356] "Generic (PLEG): container finished" podID="ca96aeae-09aa-46f7-88e2-8a15233737b6" containerID="e7845aa99734067e03b639e0c9290d18aa97d765932c34524e0480215829f23a" exitCode=0 Jul 22 03:16:17.792727 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:16:17.792700 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-9ded6d-openshift-default-7f969bd769-l268z" Jul 22 03:16:17.792767 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:16:17.792734 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-9ded6d-openshift-default-7f969bd769-l268z" event={"ID":"ca96aeae-09aa-46f7-88e2-8a15233737b6","Type":"ContainerDied","Data":"e7845aa99734067e03b639e0c9290d18aa97d765932c34524e0480215829f23a"} Jul 22 03:16:17.792802 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:16:17.792788 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-9ded6d-openshift-default-7f969bd769-l268z" event={"ID":"ca96aeae-09aa-46f7-88e2-8a15233737b6","Type":"ContainerDied","Data":"38f7fd9be2c3232684cf5e0f403c2052929a1b2558c9a3c7ebc6bfac721d34d8"} Jul 22 03:16:17.792835 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:16:17.792811 2578 scope.go:117] "RemoveContainer" containerID="e7845aa99734067e03b639e0c9290d18aa97d765932c34524e0480215829f23a" Jul 22 03:16:17.801066 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:16:17.801050 2578 scope.go:117] "RemoveContainer" containerID="21ab39aaa43de9950fcc75e94d63cb3f5139ab210d1901b75161496742e85598" Jul 22 03:16:17.810004 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:16:17.809235 2578 scope.go:117] "RemoveContainer" containerID="e7845aa99734067e03b639e0c9290d18aa97d765932c34524e0480215829f23a" Jul 22 03:16:17.810004 ip-10-0-133-31 kubenswrapper[2578]: E0722 03:16:17.809593 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e7845aa99734067e03b639e0c9290d18aa97d765932c34524e0480215829f23a\": container with ID starting with e7845aa99734067e03b639e0c9290d18aa97d765932c34524e0480215829f23a not found: ID does not exist" containerID="e7845aa99734067e03b639e0c9290d18aa97d765932c34524e0480215829f23a" Jul 22 03:16:17.810004 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:16:17.809630 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e7845aa99734067e03b639e0c9290d18aa97d765932c34524e0480215829f23a"} err="failed to get container status \"e7845aa99734067e03b639e0c9290d18aa97d765932c34524e0480215829f23a\": rpc error: code = NotFound desc = could not find container \"e7845aa99734067e03b639e0c9290d18aa97d765932c34524e0480215829f23a\": container with ID starting with e7845aa99734067e03b639e0c9290d18aa97d765932c34524e0480215829f23a not found: ID does not exist" Jul 22 03:16:17.810004 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:16:17.809653 2578 scope.go:117] "RemoveContainer" containerID="21ab39aaa43de9950fcc75e94d63cb3f5139ab210d1901b75161496742e85598" Jul 22 03:16:17.810004 ip-10-0-133-31 kubenswrapper[2578]: E0722 03:16:17.809958 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"21ab39aaa43de9950fcc75e94d63cb3f5139ab210d1901b75161496742e85598\": container with ID starting with 21ab39aaa43de9950fcc75e94d63cb3f5139ab210d1901b75161496742e85598 not found: ID does not exist" containerID="21ab39aaa43de9950fcc75e94d63cb3f5139ab210d1901b75161496742e85598" Jul 22 03:16:17.810004 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:16:17.809987 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21ab39aaa43de9950fcc75e94d63cb3f5139ab210d1901b75161496742e85598"} err="failed to get container status \"21ab39aaa43de9950fcc75e94d63cb3f5139ab210d1901b75161496742e85598\": rpc error: code = NotFound desc = could not find container \"21ab39aaa43de9950fcc75e94d63cb3f5139ab210d1901b75161496742e85598\": container with ID starting with 21ab39aaa43de9950fcc75e94d63cb3f5139ab210d1901b75161496742e85598 not found: ID does not exist" Jul 22 03:16:17.905935 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:16:17.905912 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca96aeae-09aa-46f7-88e2-8a15233737b6-istio-data\") pod \"ca96aeae-09aa-46f7-88e2-8a15233737b6\" (UID: \"ca96aeae-09aa-46f7-88e2-8a15233737b6\") " Jul 22 03:16:17.906042 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:16:17.905956 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ca96aeae-09aa-46f7-88e2-8a15233737b6-istio-token\") pod \"ca96aeae-09aa-46f7-88e2-8a15233737b6\" (UID: \"ca96aeae-09aa-46f7-88e2-8a15233737b6\") " Jul 22 03:16:17.906042 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:16:17.906013 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca96aeae-09aa-46f7-88e2-8a15233737b6-credential-socket\") pod \"ca96aeae-09aa-46f7-88e2-8a15233737b6\" (UID: \"ca96aeae-09aa-46f7-88e2-8a15233737b6\") " Jul 22 03:16:17.906153 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:16:17.906065 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca96aeae-09aa-46f7-88e2-8a15233737b6-workload-socket\") pod \"ca96aeae-09aa-46f7-88e2-8a15233737b6\" (UID: \"ca96aeae-09aa-46f7-88e2-8a15233737b6\") " Jul 22 03:16:17.906153 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:16:17.906090 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca96aeae-09aa-46f7-88e2-8a15233737b6-istiod-ca-cert\") pod \"ca96aeae-09aa-46f7-88e2-8a15233737b6\" (UID: \"ca96aeae-09aa-46f7-88e2-8a15233737b6\") " Jul 22 03:16:17.906305 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:16:17.906151 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca96aeae-09aa-46f7-88e2-8a15233737b6-workload-certs\") pod \"ca96aeae-09aa-46f7-88e2-8a15233737b6\" (UID: \"ca96aeae-09aa-46f7-88e2-8a15233737b6\") " Jul 22 03:16:17.906305 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:16:17.906276 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca96aeae-09aa-46f7-88e2-8a15233737b6-istio-podinfo\") pod \"ca96aeae-09aa-46f7-88e2-8a15233737b6\" (UID: \"ca96aeae-09aa-46f7-88e2-8a15233737b6\") " Jul 22 03:16:17.906413 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:16:17.906339 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lx8x6\" (UniqueName: \"kubernetes.io/projected/ca96aeae-09aa-46f7-88e2-8a15233737b6-kube-api-access-lx8x6\") pod \"ca96aeae-09aa-46f7-88e2-8a15233737b6\" (UID: \"ca96aeae-09aa-46f7-88e2-8a15233737b6\") " Jul 22 03:16:17.906413 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:16:17.906364 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca96aeae-09aa-46f7-88e2-8a15233737b6-istio-envoy\") pod \"ca96aeae-09aa-46f7-88e2-8a15233737b6\" (UID: \"ca96aeae-09aa-46f7-88e2-8a15233737b6\") " Jul 22 03:16:17.906413 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:16:17.906358 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca96aeae-09aa-46f7-88e2-8a15233737b6-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "ca96aeae-09aa-46f7-88e2-8a15233737b6" (UID: "ca96aeae-09aa-46f7-88e2-8a15233737b6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 03:16:17.906413 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:16:17.906365 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca96aeae-09aa-46f7-88e2-8a15233737b6-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "ca96aeae-09aa-46f7-88e2-8a15233737b6" (UID: "ca96aeae-09aa-46f7-88e2-8a15233737b6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 03:16:17.906590 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:16:17.906429 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca96aeae-09aa-46f7-88e2-8a15233737b6-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "ca96aeae-09aa-46f7-88e2-8a15233737b6" (UID: "ca96aeae-09aa-46f7-88e2-8a15233737b6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 03:16:17.906590 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:16:17.906484 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca96aeae-09aa-46f7-88e2-8a15233737b6-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "ca96aeae-09aa-46f7-88e2-8a15233737b6" (UID: "ca96aeae-09aa-46f7-88e2-8a15233737b6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 03:16:17.906718 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:16:17.906696 2578 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca96aeae-09aa-46f7-88e2-8a15233737b6-credential-socket\") on node \"ip-10-0-133-31.ec2.internal\" DevicePath \"\"" Jul 22 03:16:17.906718 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:16:17.906719 2578 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca96aeae-09aa-46f7-88e2-8a15233737b6-workload-socket\") on node \"ip-10-0-133-31.ec2.internal\" DevicePath \"\"" Jul 22 03:16:17.907009 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:16:17.906732 2578 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca96aeae-09aa-46f7-88e2-8a15233737b6-istiod-ca-cert\") on node \"ip-10-0-133-31.ec2.internal\" DevicePath \"\"" Jul 22 03:16:17.907009 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:16:17.906746 2578 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca96aeae-09aa-46f7-88e2-8a15233737b6-workload-certs\") on node \"ip-10-0-133-31.ec2.internal\" DevicePath \"\"" Jul 22 03:16:17.907009 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:16:17.906746 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca96aeae-09aa-46f7-88e2-8a15233737b6-istio-data" (OuterVolumeSpecName: "istio-data") pod "ca96aeae-09aa-46f7-88e2-8a15233737b6" (UID: "ca96aeae-09aa-46f7-88e2-8a15233737b6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 03:16:17.908387 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:16:17.908348 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ca96aeae-09aa-46f7-88e2-8a15233737b6-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "ca96aeae-09aa-46f7-88e2-8a15233737b6" (UID: "ca96aeae-09aa-46f7-88e2-8a15233737b6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 22 03:16:17.908482 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:16:17.908443 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca96aeae-09aa-46f7-88e2-8a15233737b6-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "ca96aeae-09aa-46f7-88e2-8a15233737b6" (UID: "ca96aeae-09aa-46f7-88e2-8a15233737b6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 03:16:17.908880 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:16:17.908853 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca96aeae-09aa-46f7-88e2-8a15233737b6-istio-token" (OuterVolumeSpecName: "istio-token") pod "ca96aeae-09aa-46f7-88e2-8a15233737b6" (UID: "ca96aeae-09aa-46f7-88e2-8a15233737b6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 03:16:17.908961 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:16:17.908925 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca96aeae-09aa-46f7-88e2-8a15233737b6-kube-api-access-lx8x6" (OuterVolumeSpecName: "kube-api-access-lx8x6") pod "ca96aeae-09aa-46f7-88e2-8a15233737b6" (UID: "ca96aeae-09aa-46f7-88e2-8a15233737b6"). InnerVolumeSpecName "kube-api-access-lx8x6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 03:16:18.008092 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:16:18.008071 2578 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca96aeae-09aa-46f7-88e2-8a15233737b6-istio-podinfo\") on node \"ip-10-0-133-31.ec2.internal\" DevicePath \"\"" Jul 22 03:16:18.008092 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:16:18.008090 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lx8x6\" (UniqueName: \"kubernetes.io/projected/ca96aeae-09aa-46f7-88e2-8a15233737b6-kube-api-access-lx8x6\") on node \"ip-10-0-133-31.ec2.internal\" DevicePath \"\"" Jul 22 03:16:18.008210 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:16:18.008099 2578 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca96aeae-09aa-46f7-88e2-8a15233737b6-istio-envoy\") on node \"ip-10-0-133-31.ec2.internal\" DevicePath \"\"" Jul 22 03:16:18.008210 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:16:18.008107 2578 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca96aeae-09aa-46f7-88e2-8a15233737b6-istio-data\") on node \"ip-10-0-133-31.ec2.internal\" DevicePath \"\"" Jul 22 03:16:18.008210 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:16:18.008114 2578 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ca96aeae-09aa-46f7-88e2-8a15233737b6-istio-token\") on node \"ip-10-0-133-31.ec2.internal\" DevicePath \"\"" Jul 22 03:16:18.115277 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:16:18.115253 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-9ded6d-openshift-default-7f969bd769-l268z"] Jul 22 03:16:18.118896 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:16:18.118877 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-9ded6d-openshift-default-7f969bd769-l268z"] Jul 22 03:16:18.142204 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:16:18.142177 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ca96aeae-09aa-46f7-88e2-8a15233737b6" path="/var/lib/kubelet/pods/ca96aeae-09aa-46f7-88e2-8a15233737b6/volumes" Jul 22 03:16:34.707009 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:16:34.706977 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-fec108-openshift-default-584cbd6489-vbqmp"] Jul 22 03:16:34.707488 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:16:34.707298 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-fec108-openshift-default-584cbd6489-vbqmp" podUID="232038b8-4749-4401-959f-d4456462eff3" containerName="istio-proxy" containerID="cri-o://f36dc69cb54f3a024ed4e0e35740fa58b70f04d9b4cf99735ac0aedecf3d6b8e" gracePeriod=30 Jul 22 03:16:35.604697 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:16:35.604662 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-34fcb8-openshift-default-7fbff9ccc9-c6tqv"] Jul 22 03:16:35.605032 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:16:35.605020 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ca96aeae-09aa-46f7-88e2-8a15233737b6" containerName="istio-proxy" Jul 22 03:16:35.605082 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:16:35.605034 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="ca96aeae-09aa-46f7-88e2-8a15233737b6" containerName="istio-proxy" Jul 22 03:16:35.605082 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:16:35.605046 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ca96aeae-09aa-46f7-88e2-8a15233737b6" containerName="istio-proxy" Jul 22 03:16:35.605082 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:16:35.605052 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="ca96aeae-09aa-46f7-88e2-8a15233737b6" containerName="istio-proxy" Jul 22 03:16:35.605174 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:16:35.605109 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="ca96aeae-09aa-46f7-88e2-8a15233737b6" containerName="istio-proxy" Jul 22 03:16:35.605174 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:16:35.605117 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="ca96aeae-09aa-46f7-88e2-8a15233737b6" containerName="istio-proxy" Jul 22 03:16:35.608039 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:16:35.608015 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-34fcb8-openshift-default-7fbff9ccc9-c6tqv" Jul 22 03:16:35.610806 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:16:35.610782 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-34fcb8-openshift-default-dockercfg-hls8g\"" Jul 22 03:16:35.612585 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:16:35.612543 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-34fcb8-openshift-default-7fbff9ccc9-c6tqv"] Jul 22 03:16:35.743852 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:16:35.743813 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3f4818a-8157-4536-91b6-306280ddb7ef-credential-socket\") pod \"e2e-api-a-34fcb8-openshift-default-7fbff9ccc9-c6tqv\" (UID: \"e3f4818a-8157-4536-91b6-306280ddb7ef\") " pod="openshift-ingress/e2e-api-a-34fcb8-openshift-default-7fbff9ccc9-c6tqv" Jul 22 03:16:35.743852 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:16:35.743854 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3f4818a-8157-4536-91b6-306280ddb7ef-workload-socket\") pod \"e2e-api-a-34fcb8-openshift-default-7fbff9ccc9-c6tqv\" (UID: \"e3f4818a-8157-4536-91b6-306280ddb7ef\") " pod="openshift-ingress/e2e-api-a-34fcb8-openshift-default-7fbff9ccc9-c6tqv" Jul 22 03:16:35.744269 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:16:35.743878 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3f4818a-8157-4536-91b6-306280ddb7ef-istio-data\") pod \"e2e-api-a-34fcb8-openshift-default-7fbff9ccc9-c6tqv\" (UID: \"e3f4818a-8157-4536-91b6-306280ddb7ef\") " pod="openshift-ingress/e2e-api-a-34fcb8-openshift-default-7fbff9ccc9-c6tqv" Jul 22 03:16:35.744269 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:16:35.743959 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3f4818a-8157-4536-91b6-306280ddb7ef-istio-token\") pod \"e2e-api-a-34fcb8-openshift-default-7fbff9ccc9-c6tqv\" (UID: \"e3f4818a-8157-4536-91b6-306280ddb7ef\") " pod="openshift-ingress/e2e-api-a-34fcb8-openshift-default-7fbff9ccc9-c6tqv" Jul 22 03:16:35.744269 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:16:35.743984 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3f4818a-8157-4536-91b6-306280ddb7ef-istio-podinfo\") pod \"e2e-api-a-34fcb8-openshift-default-7fbff9ccc9-c6tqv\" (UID: \"e3f4818a-8157-4536-91b6-306280ddb7ef\") " pod="openshift-ingress/e2e-api-a-34fcb8-openshift-default-7fbff9ccc9-c6tqv" Jul 22 03:16:35.744269 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:16:35.744079 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3f4818a-8157-4536-91b6-306280ddb7ef-istio-envoy\") pod \"e2e-api-a-34fcb8-openshift-default-7fbff9ccc9-c6tqv\" (UID: \"e3f4818a-8157-4536-91b6-306280ddb7ef\") " pod="openshift-ingress/e2e-api-a-34fcb8-openshift-default-7fbff9ccc9-c6tqv" Jul 22 03:16:35.744269 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:16:35.744119 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vx7b6\" (UniqueName: \"kubernetes.io/projected/e3f4818a-8157-4536-91b6-306280ddb7ef-kube-api-access-vx7b6\") pod \"e2e-api-a-34fcb8-openshift-default-7fbff9ccc9-c6tqv\" (UID: \"e3f4818a-8157-4536-91b6-306280ddb7ef\") " pod="openshift-ingress/e2e-api-a-34fcb8-openshift-default-7fbff9ccc9-c6tqv" Jul 22 03:16:35.744269 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:16:35.744165 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3f4818a-8157-4536-91b6-306280ddb7ef-istiod-ca-cert\") pod \"e2e-api-a-34fcb8-openshift-default-7fbff9ccc9-c6tqv\" (UID: \"e3f4818a-8157-4536-91b6-306280ddb7ef\") " pod="openshift-ingress/e2e-api-a-34fcb8-openshift-default-7fbff9ccc9-c6tqv" Jul 22 03:16:35.744269 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:16:35.744192 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3f4818a-8157-4536-91b6-306280ddb7ef-workload-certs\") pod \"e2e-api-a-34fcb8-openshift-default-7fbff9ccc9-c6tqv\" (UID: \"e3f4818a-8157-4536-91b6-306280ddb7ef\") " pod="openshift-ingress/e2e-api-a-34fcb8-openshift-default-7fbff9ccc9-c6tqv" Jul 22 03:16:35.845581 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:16:35.845541 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3f4818a-8157-4536-91b6-306280ddb7ef-istiod-ca-cert\") pod \"e2e-api-a-34fcb8-openshift-default-7fbff9ccc9-c6tqv\" (UID: \"e3f4818a-8157-4536-91b6-306280ddb7ef\") " pod="openshift-ingress/e2e-api-a-34fcb8-openshift-default-7fbff9ccc9-c6tqv" Jul 22 03:16:35.845730 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:16:35.845596 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3f4818a-8157-4536-91b6-306280ddb7ef-workload-certs\") pod \"e2e-api-a-34fcb8-openshift-default-7fbff9ccc9-c6tqv\" (UID: \"e3f4818a-8157-4536-91b6-306280ddb7ef\") " pod="openshift-ingress/e2e-api-a-34fcb8-openshift-default-7fbff9ccc9-c6tqv" Jul 22 03:16:35.845730 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:16:35.845634 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3f4818a-8157-4536-91b6-306280ddb7ef-credential-socket\") pod \"e2e-api-a-34fcb8-openshift-default-7fbff9ccc9-c6tqv\" (UID: \"e3f4818a-8157-4536-91b6-306280ddb7ef\") " pod="openshift-ingress/e2e-api-a-34fcb8-openshift-default-7fbff9ccc9-c6tqv" Jul 22 03:16:35.845730 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:16:35.845652 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3f4818a-8157-4536-91b6-306280ddb7ef-workload-socket\") pod \"e2e-api-a-34fcb8-openshift-default-7fbff9ccc9-c6tqv\" (UID: \"e3f4818a-8157-4536-91b6-306280ddb7ef\") " pod="openshift-ingress/e2e-api-a-34fcb8-openshift-default-7fbff9ccc9-c6tqv" Jul 22 03:16:35.845730 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:16:35.845679 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3f4818a-8157-4536-91b6-306280ddb7ef-istio-data\") pod \"e2e-api-a-34fcb8-openshift-default-7fbff9ccc9-c6tqv\" (UID: \"e3f4818a-8157-4536-91b6-306280ddb7ef\") " pod="openshift-ingress/e2e-api-a-34fcb8-openshift-default-7fbff9ccc9-c6tqv" Jul 22 03:16:35.845730 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:16:35.845718 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3f4818a-8157-4536-91b6-306280ddb7ef-istio-token\") pod \"e2e-api-a-34fcb8-openshift-default-7fbff9ccc9-c6tqv\" (UID: \"e3f4818a-8157-4536-91b6-306280ddb7ef\") " pod="openshift-ingress/e2e-api-a-34fcb8-openshift-default-7fbff9ccc9-c6tqv" Jul 22 03:16:35.845957 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:16:35.845738 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3f4818a-8157-4536-91b6-306280ddb7ef-istio-podinfo\") pod \"e2e-api-a-34fcb8-openshift-default-7fbff9ccc9-c6tqv\" (UID: \"e3f4818a-8157-4536-91b6-306280ddb7ef\") " pod="openshift-ingress/e2e-api-a-34fcb8-openshift-default-7fbff9ccc9-c6tqv" Jul 22 03:16:35.845957 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:16:35.845801 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3f4818a-8157-4536-91b6-306280ddb7ef-istio-envoy\") pod \"e2e-api-a-34fcb8-openshift-default-7fbff9ccc9-c6tqv\" (UID: \"e3f4818a-8157-4536-91b6-306280ddb7ef\") " pod="openshift-ingress/e2e-api-a-34fcb8-openshift-default-7fbff9ccc9-c6tqv" Jul 22 03:16:35.845957 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:16:35.845823 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vx7b6\" (UniqueName: \"kubernetes.io/projected/e3f4818a-8157-4536-91b6-306280ddb7ef-kube-api-access-vx7b6\") pod \"e2e-api-a-34fcb8-openshift-default-7fbff9ccc9-c6tqv\" (UID: \"e3f4818a-8157-4536-91b6-306280ddb7ef\") " pod="openshift-ingress/e2e-api-a-34fcb8-openshift-default-7fbff9ccc9-c6tqv" Jul 22 03:16:35.846106 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:16:35.845985 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3f4818a-8157-4536-91b6-306280ddb7ef-workload-socket\") pod \"e2e-api-a-34fcb8-openshift-default-7fbff9ccc9-c6tqv\" (UID: \"e3f4818a-8157-4536-91b6-306280ddb7ef\") " pod="openshift-ingress/e2e-api-a-34fcb8-openshift-default-7fbff9ccc9-c6tqv" Jul 22 03:16:35.846167 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:16:35.846151 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3f4818a-8157-4536-91b6-306280ddb7ef-workload-certs\") pod \"e2e-api-a-34fcb8-openshift-default-7fbff9ccc9-c6tqv\" (UID: \"e3f4818a-8157-4536-91b6-306280ddb7ef\") " pod="openshift-ingress/e2e-api-a-34fcb8-openshift-default-7fbff9ccc9-c6tqv" Jul 22 03:16:35.846339 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:16:35.846321 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3f4818a-8157-4536-91b6-306280ddb7ef-credential-socket\") pod \"e2e-api-a-34fcb8-openshift-default-7fbff9ccc9-c6tqv\" (UID: \"e3f4818a-8157-4536-91b6-306280ddb7ef\") " pod="openshift-ingress/e2e-api-a-34fcb8-openshift-default-7fbff9ccc9-c6tqv" Jul 22 03:16:35.846413 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:16:35.846368 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3f4818a-8157-4536-91b6-306280ddb7ef-istiod-ca-cert\") pod \"e2e-api-a-34fcb8-openshift-default-7fbff9ccc9-c6tqv\" (UID: \"e3f4818a-8157-4536-91b6-306280ddb7ef\") " pod="openshift-ingress/e2e-api-a-34fcb8-openshift-default-7fbff9ccc9-c6tqv" Jul 22 03:16:35.846413 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:16:35.846379 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3f4818a-8157-4536-91b6-306280ddb7ef-istio-data\") pod \"e2e-api-a-34fcb8-openshift-default-7fbff9ccc9-c6tqv\" (UID: \"e3f4818a-8157-4536-91b6-306280ddb7ef\") " pod="openshift-ingress/e2e-api-a-34fcb8-openshift-default-7fbff9ccc9-c6tqv" Jul 22 03:16:35.848203 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:16:35.848182 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3f4818a-8157-4536-91b6-306280ddb7ef-istio-envoy\") pod \"e2e-api-a-34fcb8-openshift-default-7fbff9ccc9-c6tqv\" (UID: \"e3f4818a-8157-4536-91b6-306280ddb7ef\") " pod="openshift-ingress/e2e-api-a-34fcb8-openshift-default-7fbff9ccc9-c6tqv" Jul 22 03:16:35.848486 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:16:35.848465 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3f4818a-8157-4536-91b6-306280ddb7ef-istio-podinfo\") pod \"e2e-api-a-34fcb8-openshift-default-7fbff9ccc9-c6tqv\" (UID: \"e3f4818a-8157-4536-91b6-306280ddb7ef\") " pod="openshift-ingress/e2e-api-a-34fcb8-openshift-default-7fbff9ccc9-c6tqv" Jul 22 03:16:35.854102 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:16:35.854079 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3f4818a-8157-4536-91b6-306280ddb7ef-istio-token\") pod \"e2e-api-a-34fcb8-openshift-default-7fbff9ccc9-c6tqv\" (UID: \"e3f4818a-8157-4536-91b6-306280ddb7ef\") " pod="openshift-ingress/e2e-api-a-34fcb8-openshift-default-7fbff9ccc9-c6tqv" Jul 22 03:16:35.854300 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:16:35.854280 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vx7b6\" (UniqueName: \"kubernetes.io/projected/e3f4818a-8157-4536-91b6-306280ddb7ef-kube-api-access-vx7b6\") pod \"e2e-api-a-34fcb8-openshift-default-7fbff9ccc9-c6tqv\" (UID: \"e3f4818a-8157-4536-91b6-306280ddb7ef\") " pod="openshift-ingress/e2e-api-a-34fcb8-openshift-default-7fbff9ccc9-c6tqv" Jul 22 03:16:35.920799 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:16:35.920743 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-34fcb8-openshift-default-7fbff9ccc9-c6tqv" Jul 22 03:16:36.060013 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:16:36.059973 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-34fcb8-openshift-default-7fbff9ccc9-c6tqv"] Jul 22 03:16:36.062070 ip-10-0-133-31 kubenswrapper[2578]: W0722 03:16:36.062031 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3f4818a_8157_4536_91b6_306280ddb7ef.slice/crio-4bada0eddefb122da6e26bede80a21f1625c4ec6c669889ece633d45a0b568d6 WatchSource:0}: Error finding container 4bada0eddefb122da6e26bede80a21f1625c4ec6c669889ece633d45a0b568d6: Status 404 returned error can't find the container with id 4bada0eddefb122da6e26bede80a21f1625c4ec6c669889ece633d45a0b568d6 Jul 22 03:16:36.064366 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:16:36.064325 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 22 03:16:36.064448 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:16:36.064428 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 22 03:16:36.064487 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:16:36.064477 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 22 03:16:36.891997 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:16:36.891966 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-34fcb8-openshift-default-7fbff9ccc9-c6tqv" event={"ID":"e3f4818a-8157-4536-91b6-306280ddb7ef","Type":"ContainerStarted","Data":"b1215eb5895cedfa365822723227861286ddeee6023715e4f3d73f227ccae350"} Jul 22 03:16:36.892453 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:16:36.892022 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-34fcb8-openshift-default-7fbff9ccc9-c6tqv" event={"ID":"e3f4818a-8157-4536-91b6-306280ddb7ef","Type":"ContainerStarted","Data":"4bada0eddefb122da6e26bede80a21f1625c4ec6c669889ece633d45a0b568d6"} Jul 22 03:16:36.913904 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:16:36.913858 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-34fcb8-openshift-default-7fbff9ccc9-c6tqv" podStartSLOduration=1.913842924 podStartE2EDuration="1.913842924s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 03:16:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 03:16:36.913220523 +0000 UTC m=+2363.232395824" watchObservedRunningTime="2026-07-22 03:16:36.913842924 +0000 UTC m=+2363.233018223" Jul 22 03:16:36.921275 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:16:36.921255 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-34fcb8-openshift-default-7fbff9ccc9-c6tqv" Jul 22 03:16:37.925854 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:16:37.925821 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-34fcb8-openshift-default-7fbff9ccc9-c6tqv" Jul 22 03:16:38.591043 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:16:38.591010 2578 patch_prober.go:28] interesting pod/e2e-inf-b-fec108-openshift-default-584cbd6489-vbqmp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 22 03:16:38.591235 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:16:38.591066 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-fec108-openshift-default-584cbd6489-vbqmp" podUID="232038b8-4749-4401-959f-d4456462eff3" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 22 03:16:38.901832 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:16:38.901757 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-34fcb8-openshift-default-7fbff9ccc9-c6tqv" Jul 22 03:16:38.902649 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:16:38.902629 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-34fcb8-openshift-default-7fbff9ccc9-c6tqv" Jul 22 03:16:39.911197 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:16:39.911098 2578 generic.go:356] "Generic (PLEG): container finished" podID="232038b8-4749-4401-959f-d4456462eff3" containerID="f36dc69cb54f3a024ed4e0e35740fa58b70f04d9b4cf99735ac0aedecf3d6b8e" exitCode=0 Jul 22 03:16:39.911197 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:16:39.911149 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-fec108-openshift-default-584cbd6489-vbqmp" event={"ID":"232038b8-4749-4401-959f-d4456462eff3","Type":"ContainerDied","Data":"f36dc69cb54f3a024ed4e0e35740fa58b70f04d9b4cf99735ac0aedecf3d6b8e"} Jul 22 03:16:39.911710 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:16:39.911217 2578 scope.go:117] "RemoveContainer" containerID="902c3e8e8d43c7344902b7d30d927e5ab71b85a476ebf12255c4c028fb57857e" Jul 22 03:16:39.972370 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:16:39.972320 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-fec108-openshift-default-584cbd6489-vbqmp" Jul 22 03:16:39.979891 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:16:39.979870 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/232038b8-4749-4401-959f-d4456462eff3-istio-token\") pod \"232038b8-4749-4401-959f-d4456462eff3\" (UID: \"232038b8-4749-4401-959f-d4456462eff3\") " Jul 22 03:16:39.980001 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:16:39.979904 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/232038b8-4749-4401-959f-d4456462eff3-credential-socket\") pod \"232038b8-4749-4401-959f-d4456462eff3\" (UID: \"232038b8-4749-4401-959f-d4456462eff3\") " Jul 22 03:16:39.980001 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:16:39.979958 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/232038b8-4749-4401-959f-d4456462eff3-workload-socket\") pod \"232038b8-4749-4401-959f-d4456462eff3\" (UID: \"232038b8-4749-4401-959f-d4456462eff3\") " Jul 22 03:16:39.980001 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:16:39.979995 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/232038b8-4749-4401-959f-d4456462eff3-istio-data\") pod \"232038b8-4749-4401-959f-d4456462eff3\" (UID: \"232038b8-4749-4401-959f-d4456462eff3\") " Jul 22 03:16:39.980168 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:16:39.980024 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/232038b8-4749-4401-959f-d4456462eff3-istiod-ca-cert\") pod \"232038b8-4749-4401-959f-d4456462eff3\" (UID: \"232038b8-4749-4401-959f-d4456462eff3\") " Jul 22 03:16:39.980168 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:16:39.980060 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6b5j6\" (UniqueName: \"kubernetes.io/projected/232038b8-4749-4401-959f-d4456462eff3-kube-api-access-6b5j6\") pod \"232038b8-4749-4401-959f-d4456462eff3\" (UID: \"232038b8-4749-4401-959f-d4456462eff3\") " Jul 22 03:16:39.980168 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:16:39.980089 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/232038b8-4749-4401-959f-d4456462eff3-istio-podinfo\") pod \"232038b8-4749-4401-959f-d4456462eff3\" (UID: \"232038b8-4749-4401-959f-d4456462eff3\") " Jul 22 03:16:39.980168 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:16:39.980112 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/232038b8-4749-4401-959f-d4456462eff3-workload-certs\") pod \"232038b8-4749-4401-959f-d4456462eff3\" (UID: \"232038b8-4749-4401-959f-d4456462eff3\") " Jul 22 03:16:39.980168 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:16:39.980151 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/232038b8-4749-4401-959f-d4456462eff3-istio-envoy\") pod \"232038b8-4749-4401-959f-d4456462eff3\" (UID: \"232038b8-4749-4401-959f-d4456462eff3\") " Jul 22 03:16:39.980429 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:16:39.980280 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/232038b8-4749-4401-959f-d4456462eff3-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "232038b8-4749-4401-959f-d4456462eff3" (UID: "232038b8-4749-4401-959f-d4456462eff3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 03:16:39.980429 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:16:39.980293 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/232038b8-4749-4401-959f-d4456462eff3-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "232038b8-4749-4401-959f-d4456462eff3" (UID: "232038b8-4749-4401-959f-d4456462eff3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 03:16:39.980535 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:16:39.980431 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/232038b8-4749-4401-959f-d4456462eff3-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "232038b8-4749-4401-959f-d4456462eff3" (UID: "232038b8-4749-4401-959f-d4456462eff3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 03:16:39.980621 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:16:39.980553 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/232038b8-4749-4401-959f-d4456462eff3-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "232038b8-4749-4401-959f-d4456462eff3" (UID: "232038b8-4749-4401-959f-d4456462eff3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 03:16:39.981003 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:16:39.980817 2578 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/232038b8-4749-4401-959f-d4456462eff3-credential-socket\") on node \"ip-10-0-133-31.ec2.internal\" DevicePath \"\"" Jul 22 03:16:39.981003 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:16:39.980990 2578 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/232038b8-4749-4401-959f-d4456462eff3-workload-socket\") on node \"ip-10-0-133-31.ec2.internal\" DevicePath \"\"" Jul 22 03:16:39.981132 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:16:39.981009 2578 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/232038b8-4749-4401-959f-d4456462eff3-istiod-ca-cert\") on node \"ip-10-0-133-31.ec2.internal\" DevicePath \"\"" Jul 22 03:16:39.981132 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:16:39.981022 2578 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/232038b8-4749-4401-959f-d4456462eff3-workload-certs\") on node \"ip-10-0-133-31.ec2.internal\" DevicePath \"\"" Jul 22 03:16:39.981132 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:16:39.981030 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/232038b8-4749-4401-959f-d4456462eff3-istio-data" (OuterVolumeSpecName: "istio-data") pod "232038b8-4749-4401-959f-d4456462eff3" (UID: "232038b8-4749-4401-959f-d4456462eff3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 03:16:39.982216 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:16:39.982196 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/232038b8-4749-4401-959f-d4456462eff3-kube-api-access-6b5j6" (OuterVolumeSpecName: "kube-api-access-6b5j6") pod "232038b8-4749-4401-959f-d4456462eff3" (UID: "232038b8-4749-4401-959f-d4456462eff3"). InnerVolumeSpecName "kube-api-access-6b5j6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 03:16:39.982499 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:16:39.982480 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/232038b8-4749-4401-959f-d4456462eff3-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "232038b8-4749-4401-959f-d4456462eff3" (UID: "232038b8-4749-4401-959f-d4456462eff3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 03:16:39.982778 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:16:39.982764 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/232038b8-4749-4401-959f-d4456462eff3-istio-token" (OuterVolumeSpecName: "istio-token") pod "232038b8-4749-4401-959f-d4456462eff3" (UID: "232038b8-4749-4401-959f-d4456462eff3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 03:16:39.982874 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:16:39.982855 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/232038b8-4749-4401-959f-d4456462eff3-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "232038b8-4749-4401-959f-d4456462eff3" (UID: "232038b8-4749-4401-959f-d4456462eff3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 22 03:16:40.082238 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:16:40.082207 2578 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/232038b8-4749-4401-959f-d4456462eff3-istio-token\") on node \"ip-10-0-133-31.ec2.internal\" DevicePath \"\"" Jul 22 03:16:40.082238 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:16:40.082234 2578 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/232038b8-4749-4401-959f-d4456462eff3-istio-data\") on node \"ip-10-0-133-31.ec2.internal\" DevicePath \"\"" Jul 22 03:16:40.082238 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:16:40.082243 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6b5j6\" (UniqueName: \"kubernetes.io/projected/232038b8-4749-4401-959f-d4456462eff3-kube-api-access-6b5j6\") on node \"ip-10-0-133-31.ec2.internal\" DevicePath \"\"" Jul 22 03:16:40.082460 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:16:40.082251 2578 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/232038b8-4749-4401-959f-d4456462eff3-istio-podinfo\") on node \"ip-10-0-133-31.ec2.internal\" DevicePath \"\"" Jul 22 03:16:40.082460 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:16:40.082260 2578 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/232038b8-4749-4401-959f-d4456462eff3-istio-envoy\") on node \"ip-10-0-133-31.ec2.internal\" DevicePath \"\"" Jul 22 03:16:40.917949 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:16:40.917910 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-fec108-openshift-default-584cbd6489-vbqmp" Jul 22 03:16:40.917949 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:16:40.917940 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-fec108-openshift-default-584cbd6489-vbqmp" event={"ID":"232038b8-4749-4401-959f-d4456462eff3","Type":"ContainerDied","Data":"74ab63fc7220697d8406b53061ded846bb82d6f07dd03527e5ea683369c956b6"} Jul 22 03:16:40.918460 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:16:40.918041 2578 scope.go:117] "RemoveContainer" containerID="f36dc69cb54f3a024ed4e0e35740fa58b70f04d9b4cf99735ac0aedecf3d6b8e" Jul 22 03:16:40.936927 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:16:40.936905 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-fec108-openshift-default-584cbd6489-vbqmp"] Jul 22 03:16:40.942993 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:16:40.942973 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-fec108-openshift-default-584cbd6489-vbqmp"] Jul 22 03:16:42.142367 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:16:42.142330 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="232038b8-4749-4401-959f-d4456462eff3" path="/var/lib/kubelet/pods/232038b8-4749-4401-959f-d4456462eff3/volumes" Jul 22 03:16:44.080173 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:16:44.080142 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-8dba71-openshift-default-55794d6cbc-5b4wq"] Jul 22 03:16:44.080697 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:16:44.080679 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="232038b8-4749-4401-959f-d4456462eff3" containerName="istio-proxy" Jul 22 03:16:44.080795 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:16:44.080700 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="232038b8-4749-4401-959f-d4456462eff3" containerName="istio-proxy" Jul 22 03:16:44.080851 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:16:44.080824 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="232038b8-4749-4401-959f-d4456462eff3" containerName="istio-proxy" Jul 22 03:16:44.080851 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:16:44.080840 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="232038b8-4749-4401-959f-d4456462eff3" containerName="istio-proxy" Jul 22 03:16:44.085284 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:16:44.085260 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-8dba71-openshift-default-55794d6cbc-5b4wq" Jul 22 03:16:44.087880 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:16:44.087859 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-8dba71-openshift-default-dockercfg-77nkz\"" Jul 22 03:16:44.099648 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:16:44.098501 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-8dba71-openshift-default-55794d6cbc-5b4wq"] Jul 22 03:16:44.116313 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:16:44.116288 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5ccbe86-eef3-4024-b02c-90951588a6ac-workload-certs\") pod \"e2e-api-b-8dba71-openshift-default-55794d6cbc-5b4wq\" (UID: \"e5ccbe86-eef3-4024-b02c-90951588a6ac\") " pod="openshift-ingress/e2e-api-b-8dba71-openshift-default-55794d6cbc-5b4wq" Jul 22 03:16:44.116442 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:16:44.116322 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dfmjk\" (UniqueName: \"kubernetes.io/projected/e5ccbe86-eef3-4024-b02c-90951588a6ac-kube-api-access-dfmjk\") pod \"e2e-api-b-8dba71-openshift-default-55794d6cbc-5b4wq\" (UID: \"e5ccbe86-eef3-4024-b02c-90951588a6ac\") " pod="openshift-ingress/e2e-api-b-8dba71-openshift-default-55794d6cbc-5b4wq" Jul 22 03:16:44.116442 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:16:44.116344 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5ccbe86-eef3-4024-b02c-90951588a6ac-workload-socket\") pod \"e2e-api-b-8dba71-openshift-default-55794d6cbc-5b4wq\" (UID: \"e5ccbe86-eef3-4024-b02c-90951588a6ac\") " pod="openshift-ingress/e2e-api-b-8dba71-openshift-default-55794d6cbc-5b4wq" Jul 22 03:16:44.116557 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:16:44.116471 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5ccbe86-eef3-4024-b02c-90951588a6ac-istiod-ca-cert\") pod \"e2e-api-b-8dba71-openshift-default-55794d6cbc-5b4wq\" (UID: \"e5ccbe86-eef3-4024-b02c-90951588a6ac\") " pod="openshift-ingress/e2e-api-b-8dba71-openshift-default-55794d6cbc-5b4wq" Jul 22 03:16:44.116557 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:16:44.116525 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5ccbe86-eef3-4024-b02c-90951588a6ac-istio-podinfo\") pod \"e2e-api-b-8dba71-openshift-default-55794d6cbc-5b4wq\" (UID: \"e5ccbe86-eef3-4024-b02c-90951588a6ac\") " pod="openshift-ingress/e2e-api-b-8dba71-openshift-default-55794d6cbc-5b4wq" Jul 22 03:16:44.116685 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:16:44.116556 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5ccbe86-eef3-4024-b02c-90951588a6ac-istio-data\") pod \"e2e-api-b-8dba71-openshift-default-55794d6cbc-5b4wq\" (UID: \"e5ccbe86-eef3-4024-b02c-90951588a6ac\") " pod="openshift-ingress/e2e-api-b-8dba71-openshift-default-55794d6cbc-5b4wq" Jul 22 03:16:44.116685 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:16:44.116630 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5ccbe86-eef3-4024-b02c-90951588a6ac-istio-envoy\") pod \"e2e-api-b-8dba71-openshift-default-55794d6cbc-5b4wq\" (UID: \"e5ccbe86-eef3-4024-b02c-90951588a6ac\") " pod="openshift-ingress/e2e-api-b-8dba71-openshift-default-55794d6cbc-5b4wq" Jul 22 03:16:44.116685 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:16:44.116675 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5ccbe86-eef3-4024-b02c-90951588a6ac-credential-socket\") pod \"e2e-api-b-8dba71-openshift-default-55794d6cbc-5b4wq\" (UID: \"e5ccbe86-eef3-4024-b02c-90951588a6ac\") " pod="openshift-ingress/e2e-api-b-8dba71-openshift-default-55794d6cbc-5b4wq" Jul 22 03:16:44.116832 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:16:44.116705 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5ccbe86-eef3-4024-b02c-90951588a6ac-istio-token\") pod \"e2e-api-b-8dba71-openshift-default-55794d6cbc-5b4wq\" (UID: \"e5ccbe86-eef3-4024-b02c-90951588a6ac\") " pod="openshift-ingress/e2e-api-b-8dba71-openshift-default-55794d6cbc-5b4wq" Jul 22 03:16:44.217832 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:16:44.217789 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5ccbe86-eef3-4024-b02c-90951588a6ac-istiod-ca-cert\") pod \"e2e-api-b-8dba71-openshift-default-55794d6cbc-5b4wq\" (UID: \"e5ccbe86-eef3-4024-b02c-90951588a6ac\") " pod="openshift-ingress/e2e-api-b-8dba71-openshift-default-55794d6cbc-5b4wq" Jul 22 03:16:44.218016 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:16:44.217852 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5ccbe86-eef3-4024-b02c-90951588a6ac-istio-podinfo\") pod \"e2e-api-b-8dba71-openshift-default-55794d6cbc-5b4wq\" (UID: \"e5ccbe86-eef3-4024-b02c-90951588a6ac\") " pod="openshift-ingress/e2e-api-b-8dba71-openshift-default-55794d6cbc-5b4wq" Jul 22 03:16:44.218016 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:16:44.217880 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5ccbe86-eef3-4024-b02c-90951588a6ac-istio-data\") pod \"e2e-api-b-8dba71-openshift-default-55794d6cbc-5b4wq\" (UID: \"e5ccbe86-eef3-4024-b02c-90951588a6ac\") " pod="openshift-ingress/e2e-api-b-8dba71-openshift-default-55794d6cbc-5b4wq" Jul 22 03:16:44.218016 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:16:44.217926 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5ccbe86-eef3-4024-b02c-90951588a6ac-istio-envoy\") pod \"e2e-api-b-8dba71-openshift-default-55794d6cbc-5b4wq\" (UID: \"e5ccbe86-eef3-4024-b02c-90951588a6ac\") " pod="openshift-ingress/e2e-api-b-8dba71-openshift-default-55794d6cbc-5b4wq" Jul 22 03:16:44.218187 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:16:44.218111 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5ccbe86-eef3-4024-b02c-90951588a6ac-credential-socket\") pod \"e2e-api-b-8dba71-openshift-default-55794d6cbc-5b4wq\" (UID: \"e5ccbe86-eef3-4024-b02c-90951588a6ac\") " pod="openshift-ingress/e2e-api-b-8dba71-openshift-default-55794d6cbc-5b4wq" Jul 22 03:16:44.218187 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:16:44.218148 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5ccbe86-eef3-4024-b02c-90951588a6ac-istio-token\") pod \"e2e-api-b-8dba71-openshift-default-55794d6cbc-5b4wq\" (UID: \"e5ccbe86-eef3-4024-b02c-90951588a6ac\") " pod="openshift-ingress/e2e-api-b-8dba71-openshift-default-55794d6cbc-5b4wq" Jul 22 03:16:44.218302 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:16:44.218199 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5ccbe86-eef3-4024-b02c-90951588a6ac-workload-certs\") pod \"e2e-api-b-8dba71-openshift-default-55794d6cbc-5b4wq\" (UID: \"e5ccbe86-eef3-4024-b02c-90951588a6ac\") " pod="openshift-ingress/e2e-api-b-8dba71-openshift-default-55794d6cbc-5b4wq" Jul 22 03:16:44.218302 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:16:44.218232 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dfmjk\" (UniqueName: \"kubernetes.io/projected/e5ccbe86-eef3-4024-b02c-90951588a6ac-kube-api-access-dfmjk\") pod \"e2e-api-b-8dba71-openshift-default-55794d6cbc-5b4wq\" (UID: \"e5ccbe86-eef3-4024-b02c-90951588a6ac\") " pod="openshift-ingress/e2e-api-b-8dba71-openshift-default-55794d6cbc-5b4wq" Jul 22 03:16:44.218302 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:16:44.218268 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5ccbe86-eef3-4024-b02c-90951588a6ac-workload-socket\") pod \"e2e-api-b-8dba71-openshift-default-55794d6cbc-5b4wq\" (UID: \"e5ccbe86-eef3-4024-b02c-90951588a6ac\") " pod="openshift-ingress/e2e-api-b-8dba71-openshift-default-55794d6cbc-5b4wq" Jul 22 03:16:44.218438 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:16:44.218362 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5ccbe86-eef3-4024-b02c-90951588a6ac-istio-data\") pod \"e2e-api-b-8dba71-openshift-default-55794d6cbc-5b4wq\" (UID: \"e5ccbe86-eef3-4024-b02c-90951588a6ac\") " pod="openshift-ingress/e2e-api-b-8dba71-openshift-default-55794d6cbc-5b4wq" Jul 22 03:16:44.218490 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:16:44.218443 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5ccbe86-eef3-4024-b02c-90951588a6ac-credential-socket\") pod \"e2e-api-b-8dba71-openshift-default-55794d6cbc-5b4wq\" (UID: \"e5ccbe86-eef3-4024-b02c-90951588a6ac\") " pod="openshift-ingress/e2e-api-b-8dba71-openshift-default-55794d6cbc-5b4wq" Jul 22 03:16:44.218686 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:16:44.218629 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5ccbe86-eef3-4024-b02c-90951588a6ac-workload-socket\") pod \"e2e-api-b-8dba71-openshift-default-55794d6cbc-5b4wq\" (UID: \"e5ccbe86-eef3-4024-b02c-90951588a6ac\") " pod="openshift-ingress/e2e-api-b-8dba71-openshift-default-55794d6cbc-5b4wq" Jul 22 03:16:44.218839 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:16:44.218745 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5ccbe86-eef3-4024-b02c-90951588a6ac-workload-certs\") pod \"e2e-api-b-8dba71-openshift-default-55794d6cbc-5b4wq\" (UID: \"e5ccbe86-eef3-4024-b02c-90951588a6ac\") " pod="openshift-ingress/e2e-api-b-8dba71-openshift-default-55794d6cbc-5b4wq" Jul 22 03:16:44.218946 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:16:44.218928 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5ccbe86-eef3-4024-b02c-90951588a6ac-istiod-ca-cert\") pod \"e2e-api-b-8dba71-openshift-default-55794d6cbc-5b4wq\" (UID: \"e5ccbe86-eef3-4024-b02c-90951588a6ac\") " pod="openshift-ingress/e2e-api-b-8dba71-openshift-default-55794d6cbc-5b4wq" Jul 22 03:16:44.220447 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:16:44.220421 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5ccbe86-eef3-4024-b02c-90951588a6ac-istio-podinfo\") pod \"e2e-api-b-8dba71-openshift-default-55794d6cbc-5b4wq\" (UID: \"e5ccbe86-eef3-4024-b02c-90951588a6ac\") " pod="openshift-ingress/e2e-api-b-8dba71-openshift-default-55794d6cbc-5b4wq" Jul 22 03:16:44.220596 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:16:44.220555 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5ccbe86-eef3-4024-b02c-90951588a6ac-istio-envoy\") pod \"e2e-api-b-8dba71-openshift-default-55794d6cbc-5b4wq\" (UID: \"e5ccbe86-eef3-4024-b02c-90951588a6ac\") " pod="openshift-ingress/e2e-api-b-8dba71-openshift-default-55794d6cbc-5b4wq" Jul 22 03:16:44.226340 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:16:44.226322 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5ccbe86-eef3-4024-b02c-90951588a6ac-istio-token\") pod \"e2e-api-b-8dba71-openshift-default-55794d6cbc-5b4wq\" (UID: \"e5ccbe86-eef3-4024-b02c-90951588a6ac\") " pod="openshift-ingress/e2e-api-b-8dba71-openshift-default-55794d6cbc-5b4wq" Jul 22 03:16:44.226512 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:16:44.226492 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dfmjk\" (UniqueName: \"kubernetes.io/projected/e5ccbe86-eef3-4024-b02c-90951588a6ac-kube-api-access-dfmjk\") pod \"e2e-api-b-8dba71-openshift-default-55794d6cbc-5b4wq\" (UID: \"e5ccbe86-eef3-4024-b02c-90951588a6ac\") " pod="openshift-ingress/e2e-api-b-8dba71-openshift-default-55794d6cbc-5b4wq" Jul 22 03:16:44.402285 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:16:44.402202 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-8dba71-openshift-default-55794d6cbc-5b4wq" Jul 22 03:16:44.546733 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:16:44.546706 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-8dba71-openshift-default-55794d6cbc-5b4wq"] Jul 22 03:16:44.548357 ip-10-0-133-31 kubenswrapper[2578]: W0722 03:16:44.548325 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5ccbe86_eef3_4024_b02c_90951588a6ac.slice/crio-10f175e033c81e4db8fd9a3538b907ccfe9054374293108beb3ca2e48f854a9b WatchSource:0}: Error finding container 10f175e033c81e4db8fd9a3538b907ccfe9054374293108beb3ca2e48f854a9b: Status 404 returned error can't find the container with id 10f175e033c81e4db8fd9a3538b907ccfe9054374293108beb3ca2e48f854a9b Jul 22 03:16:44.550514 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:16:44.550484 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 22 03:16:44.550634 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:16:44.550593 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 22 03:16:44.550698 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:16:44.550642 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 22 03:16:44.940495 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:16:44.940446 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-8dba71-openshift-default-55794d6cbc-5b4wq" event={"ID":"e5ccbe86-eef3-4024-b02c-90951588a6ac","Type":"ContainerStarted","Data":"91757b8e01745a4cf186fe2431d488a1566a0df30ec1c0d10c107d521983dc96"} Jul 22 03:16:44.940671 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:16:44.940511 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-8dba71-openshift-default-55794d6cbc-5b4wq" event={"ID":"e5ccbe86-eef3-4024-b02c-90951588a6ac","Type":"ContainerStarted","Data":"10f175e033c81e4db8fd9a3538b907ccfe9054374293108beb3ca2e48f854a9b"} Jul 22 03:16:44.963533 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:16:44.963489 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-8dba71-openshift-default-55794d6cbc-5b4wq" podStartSLOduration=0.963475169 podStartE2EDuration="963.475169ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 03:16:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 03:16:44.961355012 +0000 UTC m=+2371.280530310" watchObservedRunningTime="2026-07-22 03:16:44.963475169 +0000 UTC m=+2371.282650462" Jul 22 03:16:45.402568 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:16:45.402540 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-8dba71-openshift-default-55794d6cbc-5b4wq" Jul 22 03:16:45.406534 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:16:45.406515 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-8dba71-openshift-default-55794d6cbc-5b4wq" Jul 22 03:16:45.946429 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:16:45.946394 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-8dba71-openshift-default-55794d6cbc-5b4wq" Jul 22 03:16:45.947362 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:16:45.947337 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-8dba71-openshift-default-55794d6cbc-5b4wq" Jul 22 03:16:52.985491 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:16:52.985452 2578 generic.go:356] "Generic (PLEG): container finished" podID="e3f4818a-8157-4536-91b6-306280ddb7ef" containerID="b1215eb5895cedfa365822723227861286ddeee6023715e4f3d73f227ccae350" exitCode=0 Jul 22 03:16:52.985907 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:16:52.985512 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-34fcb8-openshift-default-7fbff9ccc9-c6tqv" event={"ID":"e3f4818a-8157-4536-91b6-306280ddb7ef","Type":"ContainerDied","Data":"b1215eb5895cedfa365822723227861286ddeee6023715e4f3d73f227ccae350"} Jul 22 03:16:52.985958 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:16:52.985921 2578 scope.go:117] "RemoveContainer" containerID="b1215eb5895cedfa365822723227861286ddeee6023715e4f3d73f227ccae350" Jul 22 03:16:52.986897 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:16:52.986869 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 22 03:16:52.986973 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:16:52.986937 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 22 03:16:52.987010 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:16:52.986974 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 22 03:16:53.901793 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:16:53.901730 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-34fcb8-openshift-default-7fbff9ccc9-c6tqv" Jul 22 03:16:53.996786 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:16:53.996744 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-34fcb8-openshift-default-7fbff9ccc9-c6tqv" event={"ID":"e3f4818a-8157-4536-91b6-306280ddb7ef","Type":"ContainerStarted","Data":"5d68099623396dc526e390eb06af7f3f052ba35825d55e5ebfc365efaef3e34e"} Jul 22 03:16:54.921775 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:16:54.921737 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-34fcb8-openshift-default-7fbff9ccc9-c6tqv" Jul 22 03:16:54.922926 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:16:54.922894 2578 patch_prober.go:28] interesting pod/e2e-api-a-34fcb8-openshift-default-7fbff9ccc9-c6tqv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Jul 22 03:16:54.923065 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:16:54.922965 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-34fcb8-openshift-default-7fbff9ccc9-c6tqv" podUID="e3f4818a-8157-4536-91b6-306280ddb7ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Jul 22 03:16:55.921962 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:16:55.921926 2578 patch_prober.go:28] interesting pod/e2e-api-a-34fcb8-openshift-default-7fbff9ccc9-c6tqv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Jul 22 03:16:55.922579 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:16:55.922528 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-34fcb8-openshift-default-7fbff9ccc9-c6tqv" podUID="e3f4818a-8157-4536-91b6-306280ddb7ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Jul 22 03:16:56.922143 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:16:56.922110 2578 patch_prober.go:28] interesting pod/e2e-api-a-34fcb8-openshift-default-7fbff9ccc9-c6tqv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Jul 22 03:16:56.922493 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:16:56.922175 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-34fcb8-openshift-default-7fbff9ccc9-c6tqv" podUID="e3f4818a-8157-4536-91b6-306280ddb7ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Jul 22 03:16:57.924991 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:16:57.924960 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-34fcb8-openshift-default-7fbff9ccc9-c6tqv" Jul 22 03:16:57.925381 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:16:57.925250 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-34fcb8-openshift-default-7fbff9ccc9-c6tqv" Jul 22 03:16:57.925922 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:16:57.925899 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-34fcb8-openshift-default-7fbff9ccc9-c6tqv" Jul 22 03:17:00.039717 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:17:00.039686 2578 generic.go:356] "Generic (PLEG): container finished" podID="e5ccbe86-eef3-4024-b02c-90951588a6ac" containerID="91757b8e01745a4cf186fe2431d488a1566a0df30ec1c0d10c107d521983dc96" exitCode=0 Jul 22 03:17:00.040117 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:17:00.039723 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-8dba71-openshift-default-55794d6cbc-5b4wq" event={"ID":"e5ccbe86-eef3-4024-b02c-90951588a6ac","Type":"ContainerDied","Data":"91757b8e01745a4cf186fe2431d488a1566a0df30ec1c0d10c107d521983dc96"} Jul 22 03:17:00.040159 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:17:00.040131 2578 scope.go:117] "RemoveContainer" containerID="91757b8e01745a4cf186fe2431d488a1566a0df30ec1c0d10c107d521983dc96" Jul 22 03:17:00.041174 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:17:00.041148 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 22 03:17:00.041232 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:17:00.041219 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 22 03:17:00.041266 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:17:00.041253 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 22 03:17:00.946644 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:17:00.946610 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-8dba71-openshift-default-55794d6cbc-5b4wq" Jul 22 03:17:01.047323 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:17:01.047292 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-8dba71-openshift-default-55794d6cbc-5b4wq" event={"ID":"e5ccbe86-eef3-4024-b02c-90951588a6ac","Type":"ContainerStarted","Data":"56b67360792e1fa2f9d10b5f4e9e56534027c93bb17094ed6941d9c572d7f71f"} Jul 22 03:17:01.403206 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:17:01.403162 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-8dba71-openshift-default-55794d6cbc-5b4wq" Jul 22 03:17:01.404760 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:17:01.404731 2578 patch_prober.go:28] interesting pod/e2e-api-b-8dba71-openshift-default-55794d6cbc-5b4wq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body= Jul 22 03:17:01.404997 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:17:01.404967 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-8dba71-openshift-default-55794d6cbc-5b4wq" podUID="e5ccbe86-eef3-4024-b02c-90951588a6ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" Jul 22 03:17:02.403418 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:17:02.403371 2578 patch_prober.go:28] interesting pod/e2e-api-b-8dba71-openshift-default-55794d6cbc-5b4wq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body= Jul 22 03:17:02.403908 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:17:02.403448 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-8dba71-openshift-default-55794d6cbc-5b4wq" podUID="e5ccbe86-eef3-4024-b02c-90951588a6ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" Jul 22 03:17:03.403004 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:17:03.402964 2578 patch_prober.go:28] interesting pod/e2e-api-b-8dba71-openshift-default-55794d6cbc-5b4wq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body= Jul 22 03:17:03.403259 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:17:03.403040 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-8dba71-openshift-default-55794d6cbc-5b4wq" podUID="e5ccbe86-eef3-4024-b02c-90951588a6ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" Jul 22 03:17:04.405999 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:17:04.405907 2578 patch_prober.go:28] interesting pod/e2e-api-b-8dba71-openshift-default-55794d6cbc-5b4wq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body= Jul 22 03:17:04.406625 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:17:04.406592 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-8dba71-openshift-default-55794d6cbc-5b4wq" podUID="e5ccbe86-eef3-4024-b02c-90951588a6ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" Jul 22 03:17:05.406350 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:17:05.406314 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-8dba71-openshift-default-55794d6cbc-5b4wq" Jul 22 03:17:05.406877 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:17:05.406653 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-8dba71-openshift-default-55794d6cbc-5b4wq" Jul 22 03:17:05.407727 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:17:05.407708 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-8dba71-openshift-default-55794d6cbc-5b4wq" Jul 22 03:17:14.262881 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:17:14.262844 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-prfql_fd7bf923-8d16-476c-a8d0-a3e6fd645061/ovn-acl-logging/0.log" Jul 22 03:17:14.270630 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:17:14.270604 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-prfql_fd7bf923-8d16-476c-a8d0-a3e6fd645061/ovn-acl-logging/0.log" Jul 22 03:17:26.199006 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:17:26.198972 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-34fcb8-openshift-default-7fbff9ccc9-c6tqv"] Jul 22 03:17:26.199424 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:17:26.199178 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-34fcb8-openshift-default-7fbff9ccc9-c6tqv" podUID="e3f4818a-8157-4536-91b6-306280ddb7ef" containerName="istio-proxy" containerID="cri-o://5d68099623396dc526e390eb06af7f3f052ba35825d55e5ebfc365efaef3e34e" gracePeriod=30 Jul 22 03:17:27.927903 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:17:27.927869 2578 patch_prober.go:28] interesting pod/e2e-api-a-34fcb8-openshift-default-7fbff9ccc9-c6tqv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=upstream connect error or disconnect/reset before headers. reset reason: remote connection failure, transport failure reason: delayed connect error: Connection refused Jul 22 03:17:27.928256 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:17:27.927930 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-34fcb8-openshift-default-7fbff9ccc9-c6tqv" podUID="e3f4818a-8157-4536-91b6-306280ddb7ef" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 22 03:17:31.458335 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:17:31.458313 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-34fcb8-openshift-default-7fbff9ccc9-c6tqv" Jul 22 03:17:31.551554 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:17:31.551497 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vx7b6\" (UniqueName: \"kubernetes.io/projected/e3f4818a-8157-4536-91b6-306280ddb7ef-kube-api-access-vx7b6\") pod \"e3f4818a-8157-4536-91b6-306280ddb7ef\" (UID: \"e3f4818a-8157-4536-91b6-306280ddb7ef\") " Jul 22 03:17:31.551554 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:17:31.551527 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3f4818a-8157-4536-91b6-306280ddb7ef-workload-certs\") pod \"e3f4818a-8157-4536-91b6-306280ddb7ef\" (UID: \"e3f4818a-8157-4536-91b6-306280ddb7ef\") " Jul 22 03:17:31.551554 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:17:31.551550 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3f4818a-8157-4536-91b6-306280ddb7ef-istiod-ca-cert\") pod \"e3f4818a-8157-4536-91b6-306280ddb7ef\" (UID: \"e3f4818a-8157-4536-91b6-306280ddb7ef\") " Jul 22 03:17:31.551823 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:17:31.551611 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3f4818a-8157-4536-91b6-306280ddb7ef-credential-socket\") pod \"e3f4818a-8157-4536-91b6-306280ddb7ef\" (UID: \"e3f4818a-8157-4536-91b6-306280ddb7ef\") " Jul 22 03:17:31.551823 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:17:31.551635 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3f4818a-8157-4536-91b6-306280ddb7ef-istio-token\") pod \"e3f4818a-8157-4536-91b6-306280ddb7ef\" (UID: \"e3f4818a-8157-4536-91b6-306280ddb7ef\") " Jul 22 03:17:31.551823 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:17:31.551671 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3f4818a-8157-4536-91b6-306280ddb7ef-istio-podinfo\") pod \"e3f4818a-8157-4536-91b6-306280ddb7ef\" (UID: \"e3f4818a-8157-4536-91b6-306280ddb7ef\") " Jul 22 03:17:31.551823 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:17:31.551710 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3f4818a-8157-4536-91b6-306280ddb7ef-istio-data\") pod \"e3f4818a-8157-4536-91b6-306280ddb7ef\" (UID: \"e3f4818a-8157-4536-91b6-306280ddb7ef\") " Jul 22 03:17:31.551823 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:17:31.551749 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3f4818a-8157-4536-91b6-306280ddb7ef-workload-socket\") pod \"e3f4818a-8157-4536-91b6-306280ddb7ef\" (UID: \"e3f4818a-8157-4536-91b6-306280ddb7ef\") " Jul 22 03:17:31.551823 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:17:31.551775 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3f4818a-8157-4536-91b6-306280ddb7ef-istio-envoy\") pod \"e3f4818a-8157-4536-91b6-306280ddb7ef\" (UID: \"e3f4818a-8157-4536-91b6-306280ddb7ef\") " Jul 22 03:17:31.552127 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:17:31.551853 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3f4818a-8157-4536-91b6-306280ddb7ef-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "e3f4818a-8157-4536-91b6-306280ddb7ef" (UID: "e3f4818a-8157-4536-91b6-306280ddb7ef"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 03:17:31.552127 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:17:31.551954 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3f4818a-8157-4536-91b6-306280ddb7ef-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "e3f4818a-8157-4536-91b6-306280ddb7ef" (UID: "e3f4818a-8157-4536-91b6-306280ddb7ef"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 03:17:31.552127 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:17:31.552062 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3f4818a-8157-4536-91b6-306280ddb7ef-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "e3f4818a-8157-4536-91b6-306280ddb7ef" (UID: "e3f4818a-8157-4536-91b6-306280ddb7ef"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 03:17:31.552127 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:17:31.552111 2578 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3f4818a-8157-4536-91b6-306280ddb7ef-istiod-ca-cert\") on node \"ip-10-0-133-31.ec2.internal\" DevicePath \"\"" Jul 22 03:17:31.552317 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:17:31.552130 2578 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3f4818a-8157-4536-91b6-306280ddb7ef-workload-certs\") on node \"ip-10-0-133-31.ec2.internal\" DevicePath \"\"" Jul 22 03:17:31.552317 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:17:31.552219 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3f4818a-8157-4536-91b6-306280ddb7ef-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "e3f4818a-8157-4536-91b6-306280ddb7ef" (UID: "e3f4818a-8157-4536-91b6-306280ddb7ef"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 03:17:31.553142 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:17:31.553118 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3f4818a-8157-4536-91b6-306280ddb7ef-istio-data" (OuterVolumeSpecName: "istio-data") pod "e3f4818a-8157-4536-91b6-306280ddb7ef" (UID: "e3f4818a-8157-4536-91b6-306280ddb7ef"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 03:17:31.553897 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:17:31.553865 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3f4818a-8157-4536-91b6-306280ddb7ef-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "e3f4818a-8157-4536-91b6-306280ddb7ef" (UID: "e3f4818a-8157-4536-91b6-306280ddb7ef"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 03:17:31.554016 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:17:31.553942 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3f4818a-8157-4536-91b6-306280ddb7ef-istio-token" (OuterVolumeSpecName: "istio-token") pod "e3f4818a-8157-4536-91b6-306280ddb7ef" (UID: "e3f4818a-8157-4536-91b6-306280ddb7ef"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 03:17:31.554285 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:17:31.554258 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e3f4818a-8157-4536-91b6-306280ddb7ef-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "e3f4818a-8157-4536-91b6-306280ddb7ef" (UID: "e3f4818a-8157-4536-91b6-306280ddb7ef"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 22 03:17:31.554285 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:17:31.554262 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3f4818a-8157-4536-91b6-306280ddb7ef-kube-api-access-vx7b6" (OuterVolumeSpecName: "kube-api-access-vx7b6") pod "e3f4818a-8157-4536-91b6-306280ddb7ef" (UID: "e3f4818a-8157-4536-91b6-306280ddb7ef"). InnerVolumeSpecName "kube-api-access-vx7b6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 03:17:31.652807 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:17:31.652785 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vx7b6\" (UniqueName: \"kubernetes.io/projected/e3f4818a-8157-4536-91b6-306280ddb7ef-kube-api-access-vx7b6\") on node \"ip-10-0-133-31.ec2.internal\" DevicePath \"\"" Jul 22 03:17:31.652807 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:17:31.652804 2578 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3f4818a-8157-4536-91b6-306280ddb7ef-credential-socket\") on node \"ip-10-0-133-31.ec2.internal\" DevicePath \"\"" Jul 22 03:17:31.652933 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:17:31.652813 2578 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3f4818a-8157-4536-91b6-306280ddb7ef-istio-token\") on node \"ip-10-0-133-31.ec2.internal\" DevicePath \"\"" Jul 22 03:17:31.652933 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:17:31.652821 2578 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3f4818a-8157-4536-91b6-306280ddb7ef-istio-podinfo\") on node \"ip-10-0-133-31.ec2.internal\" DevicePath \"\"" Jul 22 03:17:31.652933 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:17:31.652829 2578 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3f4818a-8157-4536-91b6-306280ddb7ef-istio-data\") on node \"ip-10-0-133-31.ec2.internal\" DevicePath \"\"" Jul 22 03:17:31.652933 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:17:31.652837 2578 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3f4818a-8157-4536-91b6-306280ddb7ef-workload-socket\") on node \"ip-10-0-133-31.ec2.internal\" DevicePath \"\"" Jul 22 03:17:31.652933 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:17:31.652844 2578 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3f4818a-8157-4536-91b6-306280ddb7ef-istio-envoy\") on node \"ip-10-0-133-31.ec2.internal\" DevicePath \"\"" Jul 22 03:17:32.225431 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:17:32.225397 2578 generic.go:356] "Generic (PLEG): container finished" podID="e3f4818a-8157-4536-91b6-306280ddb7ef" containerID="5d68099623396dc526e390eb06af7f3f052ba35825d55e5ebfc365efaef3e34e" exitCode=0 Jul 22 03:17:32.225552 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:17:32.225448 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-34fcb8-openshift-default-7fbff9ccc9-c6tqv" Jul 22 03:17:32.225552 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:17:32.225471 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-34fcb8-openshift-default-7fbff9ccc9-c6tqv" event={"ID":"e3f4818a-8157-4536-91b6-306280ddb7ef","Type":"ContainerDied","Data":"5d68099623396dc526e390eb06af7f3f052ba35825d55e5ebfc365efaef3e34e"} Jul 22 03:17:32.225552 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:17:32.225522 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-34fcb8-openshift-default-7fbff9ccc9-c6tqv" event={"ID":"e3f4818a-8157-4536-91b6-306280ddb7ef","Type":"ContainerDied","Data":"4bada0eddefb122da6e26bede80a21f1625c4ec6c669889ece633d45a0b568d6"} Jul 22 03:17:32.225704 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:17:32.225552 2578 scope.go:117] "RemoveContainer" containerID="5d68099623396dc526e390eb06af7f3f052ba35825d55e5ebfc365efaef3e34e" Jul 22 03:17:32.234277 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:17:32.234261 2578 scope.go:117] "RemoveContainer" containerID="b1215eb5895cedfa365822723227861286ddeee6023715e4f3d73f227ccae350" Jul 22 03:17:32.241946 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:17:32.241917 2578 scope.go:117] "RemoveContainer" containerID="5d68099623396dc526e390eb06af7f3f052ba35825d55e5ebfc365efaef3e34e" Jul 22 03:17:32.242219 ip-10-0-133-31 kubenswrapper[2578]: E0722 03:17:32.242195 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d68099623396dc526e390eb06af7f3f052ba35825d55e5ebfc365efaef3e34e\": container with ID starting with 5d68099623396dc526e390eb06af7f3f052ba35825d55e5ebfc365efaef3e34e not found: ID does not exist" containerID="5d68099623396dc526e390eb06af7f3f052ba35825d55e5ebfc365efaef3e34e" Jul 22 03:17:32.242297 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:17:32.242229 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d68099623396dc526e390eb06af7f3f052ba35825d55e5ebfc365efaef3e34e"} err="failed to get container status \"5d68099623396dc526e390eb06af7f3f052ba35825d55e5ebfc365efaef3e34e\": rpc error: code = NotFound desc = could not find container \"5d68099623396dc526e390eb06af7f3f052ba35825d55e5ebfc365efaef3e34e\": container with ID starting with 5d68099623396dc526e390eb06af7f3f052ba35825d55e5ebfc365efaef3e34e not found: ID does not exist" Jul 22 03:17:32.242297 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:17:32.242251 2578 scope.go:117] "RemoveContainer" containerID="b1215eb5895cedfa365822723227861286ddeee6023715e4f3d73f227ccae350" Jul 22 03:17:32.242549 ip-10-0-133-31 kubenswrapper[2578]: E0722 03:17:32.242508 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b1215eb5895cedfa365822723227861286ddeee6023715e4f3d73f227ccae350\": container with ID starting with b1215eb5895cedfa365822723227861286ddeee6023715e4f3d73f227ccae350 not found: ID does not exist" containerID="b1215eb5895cedfa365822723227861286ddeee6023715e4f3d73f227ccae350" Jul 22 03:17:32.242649 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:17:32.242542 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b1215eb5895cedfa365822723227861286ddeee6023715e4f3d73f227ccae350"} err="failed to get container status \"b1215eb5895cedfa365822723227861286ddeee6023715e4f3d73f227ccae350\": rpc error: code = NotFound desc = could not find container \"b1215eb5895cedfa365822723227861286ddeee6023715e4f3d73f227ccae350\": container with ID starting with b1215eb5895cedfa365822723227861286ddeee6023715e4f3d73f227ccae350 not found: ID does not exist" Jul 22 03:17:32.244982 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:17:32.244963 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-34fcb8-openshift-default-7fbff9ccc9-c6tqv"] Jul 22 03:17:32.248691 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:17:32.248671 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-34fcb8-openshift-default-7fbff9ccc9-c6tqv"] Jul 22 03:17:34.142614 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:17:34.142585 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e3f4818a-8157-4536-91b6-306280ddb7ef" path="/var/lib/kubelet/pods/e3f4818a-8157-4536-91b6-306280ddb7ef/volumes" Jul 22 03:17:53.778520 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:17:53.778434 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-8dba71-openshift-default-55794d6cbc-5b4wq"] Jul 22 03:17:53.779074 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:17:53.778723 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-8dba71-openshift-default-55794d6cbc-5b4wq" podUID="e5ccbe86-eef3-4024-b02c-90951588a6ac" containerName="istio-proxy" containerID="cri-o://56b67360792e1fa2f9d10b5f4e9e56534027c93bb17094ed6941d9c572d7f71f" gracePeriod=30 Jul 22 03:17:54.697987 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:17:54.697951 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-72501c-openshift-default-7f7d685fff-dvkn2"] Jul 22 03:17:54.698453 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:17:54.698432 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e3f4818a-8157-4536-91b6-306280ddb7ef" containerName="istio-proxy" Jul 22 03:17:54.698453 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:17:54.698451 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="e3f4818a-8157-4536-91b6-306280ddb7ef" containerName="istio-proxy" Jul 22 03:17:54.698691 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:17:54.698474 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="232038b8-4749-4401-959f-d4456462eff3" containerName="istio-proxy" Jul 22 03:17:54.698691 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:17:54.698482 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="232038b8-4749-4401-959f-d4456462eff3" containerName="istio-proxy" Jul 22 03:17:54.698691 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:17:54.698586 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="e3f4818a-8157-4536-91b6-306280ddb7ef" containerName="istio-proxy" Jul 22 03:17:54.698691 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:17:54.698598 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="e3f4818a-8157-4536-91b6-306280ddb7ef" containerName="istio-proxy" Jul 22 03:17:54.701612 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:17:54.701593 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-72501c-openshift-default-7f7d685fff-dvkn2" Jul 22 03:17:54.704510 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:17:54.704489 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-72501c-openshift-default-dockercfg-pr5fg\"" Jul 22 03:17:54.714605 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:17:54.714576 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-72501c-openshift-default-7f7d685fff-dvkn2"] Jul 22 03:17:54.754890 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:17:54.754863 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c52d4866-0bd6-4fdf-ab45-166f1778ba19-istio-data\") pod \"e2e-shared-a-72501c-openshift-default-7f7d685fff-dvkn2\" (UID: \"c52d4866-0bd6-4fdf-ab45-166f1778ba19\") " pod="openshift-ingress/e2e-shared-a-72501c-openshift-default-7f7d685fff-dvkn2" Jul 22 03:17:54.755018 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:17:54.754907 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c52d4866-0bd6-4fdf-ab45-166f1778ba19-istio-envoy\") pod \"e2e-shared-a-72501c-openshift-default-7f7d685fff-dvkn2\" (UID: \"c52d4866-0bd6-4fdf-ab45-166f1778ba19\") " pod="openshift-ingress/e2e-shared-a-72501c-openshift-default-7f7d685fff-dvkn2" Jul 22 03:17:54.755018 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:17:54.754936 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c52d4866-0bd6-4fdf-ab45-166f1778ba19-istio-token\") pod \"e2e-shared-a-72501c-openshift-default-7f7d685fff-dvkn2\" (UID: \"c52d4866-0bd6-4fdf-ab45-166f1778ba19\") " pod="openshift-ingress/e2e-shared-a-72501c-openshift-default-7f7d685fff-dvkn2" Jul 22 03:17:54.755018 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:17:54.754968 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c52d4866-0bd6-4fdf-ab45-166f1778ba19-istio-podinfo\") pod \"e2e-shared-a-72501c-openshift-default-7f7d685fff-dvkn2\" (UID: \"c52d4866-0bd6-4fdf-ab45-166f1778ba19\") " pod="openshift-ingress/e2e-shared-a-72501c-openshift-default-7f7d685fff-dvkn2" Jul 22 03:17:54.755018 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:17:54.754996 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c52d4866-0bd6-4fdf-ab45-166f1778ba19-workload-certs\") pod \"e2e-shared-a-72501c-openshift-default-7f7d685fff-dvkn2\" (UID: \"c52d4866-0bd6-4fdf-ab45-166f1778ba19\") " pod="openshift-ingress/e2e-shared-a-72501c-openshift-default-7f7d685fff-dvkn2" Jul 22 03:17:54.755219 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:17:54.755025 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c52d4866-0bd6-4fdf-ab45-166f1778ba19-workload-socket\") pod \"e2e-shared-a-72501c-openshift-default-7f7d685fff-dvkn2\" (UID: \"c52d4866-0bd6-4fdf-ab45-166f1778ba19\") " pod="openshift-ingress/e2e-shared-a-72501c-openshift-default-7f7d685fff-dvkn2" Jul 22 03:17:54.755219 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:17:54.755044 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c52d4866-0bd6-4fdf-ab45-166f1778ba19-credential-socket\") pod \"e2e-shared-a-72501c-openshift-default-7f7d685fff-dvkn2\" (UID: \"c52d4866-0bd6-4fdf-ab45-166f1778ba19\") " pod="openshift-ingress/e2e-shared-a-72501c-openshift-default-7f7d685fff-dvkn2" Jul 22 03:17:54.755219 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:17:54.755059 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c52d4866-0bd6-4fdf-ab45-166f1778ba19-istiod-ca-cert\") pod \"e2e-shared-a-72501c-openshift-default-7f7d685fff-dvkn2\" (UID: \"c52d4866-0bd6-4fdf-ab45-166f1778ba19\") " pod="openshift-ingress/e2e-shared-a-72501c-openshift-default-7f7d685fff-dvkn2" Jul 22 03:17:54.755219 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:17:54.755104 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ssmm2\" (UniqueName: \"kubernetes.io/projected/c52d4866-0bd6-4fdf-ab45-166f1778ba19-kube-api-access-ssmm2\") pod \"e2e-shared-a-72501c-openshift-default-7f7d685fff-dvkn2\" (UID: \"c52d4866-0bd6-4fdf-ab45-166f1778ba19\") " pod="openshift-ingress/e2e-shared-a-72501c-openshift-default-7f7d685fff-dvkn2" Jul 22 03:17:54.855901 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:17:54.855867 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c52d4866-0bd6-4fdf-ab45-166f1778ba19-workload-socket\") pod \"e2e-shared-a-72501c-openshift-default-7f7d685fff-dvkn2\" (UID: \"c52d4866-0bd6-4fdf-ab45-166f1778ba19\") " pod="openshift-ingress/e2e-shared-a-72501c-openshift-default-7f7d685fff-dvkn2" Jul 22 03:17:54.856344 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:17:54.855909 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c52d4866-0bd6-4fdf-ab45-166f1778ba19-credential-socket\") pod \"e2e-shared-a-72501c-openshift-default-7f7d685fff-dvkn2\" (UID: \"c52d4866-0bd6-4fdf-ab45-166f1778ba19\") " pod="openshift-ingress/e2e-shared-a-72501c-openshift-default-7f7d685fff-dvkn2" Jul 22 03:17:54.856344 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:17:54.855927 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c52d4866-0bd6-4fdf-ab45-166f1778ba19-istiod-ca-cert\") pod \"e2e-shared-a-72501c-openshift-default-7f7d685fff-dvkn2\" (UID: \"c52d4866-0bd6-4fdf-ab45-166f1778ba19\") " pod="openshift-ingress/e2e-shared-a-72501c-openshift-default-7f7d685fff-dvkn2" Jul 22 03:17:54.856344 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:17:54.856044 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ssmm2\" (UniqueName: \"kubernetes.io/projected/c52d4866-0bd6-4fdf-ab45-166f1778ba19-kube-api-access-ssmm2\") pod \"e2e-shared-a-72501c-openshift-default-7f7d685fff-dvkn2\" (UID: \"c52d4866-0bd6-4fdf-ab45-166f1778ba19\") " pod="openshift-ingress/e2e-shared-a-72501c-openshift-default-7f7d685fff-dvkn2" Jul 22 03:17:54.856344 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:17:54.856140 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c52d4866-0bd6-4fdf-ab45-166f1778ba19-istio-data\") pod \"e2e-shared-a-72501c-openshift-default-7f7d685fff-dvkn2\" (UID: \"c52d4866-0bd6-4fdf-ab45-166f1778ba19\") " pod="openshift-ingress/e2e-shared-a-72501c-openshift-default-7f7d685fff-dvkn2" Jul 22 03:17:54.856344 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:17:54.856198 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c52d4866-0bd6-4fdf-ab45-166f1778ba19-istio-envoy\") pod \"e2e-shared-a-72501c-openshift-default-7f7d685fff-dvkn2\" (UID: \"c52d4866-0bd6-4fdf-ab45-166f1778ba19\") " pod="openshift-ingress/e2e-shared-a-72501c-openshift-default-7f7d685fff-dvkn2" Jul 22 03:17:54.856344 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:17:54.856310 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c52d4866-0bd6-4fdf-ab45-166f1778ba19-workload-socket\") pod \"e2e-shared-a-72501c-openshift-default-7f7d685fff-dvkn2\" (UID: \"c52d4866-0bd6-4fdf-ab45-166f1778ba19\") " pod="openshift-ingress/e2e-shared-a-72501c-openshift-default-7f7d685fff-dvkn2" Jul 22 03:17:54.856725 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:17:54.856346 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c52d4866-0bd6-4fdf-ab45-166f1778ba19-credential-socket\") pod \"e2e-shared-a-72501c-openshift-default-7f7d685fff-dvkn2\" (UID: \"c52d4866-0bd6-4fdf-ab45-166f1778ba19\") " pod="openshift-ingress/e2e-shared-a-72501c-openshift-default-7f7d685fff-dvkn2" Jul 22 03:17:54.856725 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:17:54.856420 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c52d4866-0bd6-4fdf-ab45-166f1778ba19-istio-token\") pod \"e2e-shared-a-72501c-openshift-default-7f7d685fff-dvkn2\" (UID: \"c52d4866-0bd6-4fdf-ab45-166f1778ba19\") " pod="openshift-ingress/e2e-shared-a-72501c-openshift-default-7f7d685fff-dvkn2" Jul 22 03:17:54.856725 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:17:54.856449 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c52d4866-0bd6-4fdf-ab45-166f1778ba19-istio-podinfo\") pod \"e2e-shared-a-72501c-openshift-default-7f7d685fff-dvkn2\" (UID: \"c52d4866-0bd6-4fdf-ab45-166f1778ba19\") " pod="openshift-ingress/e2e-shared-a-72501c-openshift-default-7f7d685fff-dvkn2" Jul 22 03:17:54.856725 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:17:54.856484 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c52d4866-0bd6-4fdf-ab45-166f1778ba19-workload-certs\") pod \"e2e-shared-a-72501c-openshift-default-7f7d685fff-dvkn2\" (UID: \"c52d4866-0bd6-4fdf-ab45-166f1778ba19\") " pod="openshift-ingress/e2e-shared-a-72501c-openshift-default-7f7d685fff-dvkn2" Jul 22 03:17:54.856725 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:17:54.856521 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c52d4866-0bd6-4fdf-ab45-166f1778ba19-istio-data\") pod \"e2e-shared-a-72501c-openshift-default-7f7d685fff-dvkn2\" (UID: \"c52d4866-0bd6-4fdf-ab45-166f1778ba19\") " pod="openshift-ingress/e2e-shared-a-72501c-openshift-default-7f7d685fff-dvkn2" Jul 22 03:17:54.856936 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:17:54.856748 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c52d4866-0bd6-4fdf-ab45-166f1778ba19-workload-certs\") pod \"e2e-shared-a-72501c-openshift-default-7f7d685fff-dvkn2\" (UID: \"c52d4866-0bd6-4fdf-ab45-166f1778ba19\") " pod="openshift-ingress/e2e-shared-a-72501c-openshift-default-7f7d685fff-dvkn2" Jul 22 03:17:54.856936 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:17:54.856773 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c52d4866-0bd6-4fdf-ab45-166f1778ba19-istiod-ca-cert\") pod \"e2e-shared-a-72501c-openshift-default-7f7d685fff-dvkn2\" (UID: \"c52d4866-0bd6-4fdf-ab45-166f1778ba19\") " pod="openshift-ingress/e2e-shared-a-72501c-openshift-default-7f7d685fff-dvkn2" Jul 22 03:17:54.858475 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:17:54.858448 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c52d4866-0bd6-4fdf-ab45-166f1778ba19-istio-envoy\") pod \"e2e-shared-a-72501c-openshift-default-7f7d685fff-dvkn2\" (UID: \"c52d4866-0bd6-4fdf-ab45-166f1778ba19\") " pod="openshift-ingress/e2e-shared-a-72501c-openshift-default-7f7d685fff-dvkn2" Jul 22 03:17:54.858943 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:17:54.858925 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c52d4866-0bd6-4fdf-ab45-166f1778ba19-istio-podinfo\") pod \"e2e-shared-a-72501c-openshift-default-7f7d685fff-dvkn2\" (UID: \"c52d4866-0bd6-4fdf-ab45-166f1778ba19\") " pod="openshift-ingress/e2e-shared-a-72501c-openshift-default-7f7d685fff-dvkn2" Jul 22 03:17:54.866619 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:17:54.866596 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c52d4866-0bd6-4fdf-ab45-166f1778ba19-istio-token\") pod \"e2e-shared-a-72501c-openshift-default-7f7d685fff-dvkn2\" (UID: \"c52d4866-0bd6-4fdf-ab45-166f1778ba19\") " pod="openshift-ingress/e2e-shared-a-72501c-openshift-default-7f7d685fff-dvkn2" Jul 22 03:17:54.867042 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:17:54.867022 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ssmm2\" (UniqueName: \"kubernetes.io/projected/c52d4866-0bd6-4fdf-ab45-166f1778ba19-kube-api-access-ssmm2\") pod \"e2e-shared-a-72501c-openshift-default-7f7d685fff-dvkn2\" (UID: \"c52d4866-0bd6-4fdf-ab45-166f1778ba19\") " pod="openshift-ingress/e2e-shared-a-72501c-openshift-default-7f7d685fff-dvkn2" Jul 22 03:17:55.012870 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:17:55.012786 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-72501c-openshift-default-7f7d685fff-dvkn2" Jul 22 03:17:55.155250 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:17:55.155225 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-72501c-openshift-default-7f7d685fff-dvkn2"] Jul 22 03:17:55.156197 ip-10-0-133-31 kubenswrapper[2578]: W0722 03:17:55.156172 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc52d4866_0bd6_4fdf_ab45_166f1778ba19.slice/crio-b6503724ca321b83be53e16f588c022a534494af83c832a76e070409b42d86ff WatchSource:0}: Error finding container b6503724ca321b83be53e16f588c022a534494af83c832a76e070409b42d86ff: Status 404 returned error can't find the container with id b6503724ca321b83be53e16f588c022a534494af83c832a76e070409b42d86ff Jul 22 03:17:55.158464 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:17:55.158430 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 22 03:17:55.158529 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:17:55.158501 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 22 03:17:55.158581 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:17:55.158528 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 22 03:17:55.349339 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:17:55.349302 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-72501c-openshift-default-7f7d685fff-dvkn2" event={"ID":"c52d4866-0bd6-4fdf-ab45-166f1778ba19","Type":"ContainerStarted","Data":"ff0dfc1c872ff0dc11924df062fed85c0796e5823060a10d0ee0c86ce046e1f7"} Jul 22 03:17:55.349499 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:17:55.349374 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-72501c-openshift-default-7f7d685fff-dvkn2" event={"ID":"c52d4866-0bd6-4fdf-ab45-166f1778ba19","Type":"ContainerStarted","Data":"b6503724ca321b83be53e16f588c022a534494af83c832a76e070409b42d86ff"} Jul 22 03:17:55.372443 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:17:55.372390 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-72501c-openshift-default-7f7d685fff-dvkn2" podStartSLOduration=1.372373963 podStartE2EDuration="1.372373963s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 03:17:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 03:17:55.371786012 +0000 UTC m=+2441.690961313" watchObservedRunningTime="2026-07-22 03:17:55.372373963 +0000 UTC m=+2441.691549262" Jul 22 03:17:56.013867 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:17:56.013831 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-72501c-openshift-default-7f7d685fff-dvkn2" Jul 22 03:17:56.018760 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:17:56.018737 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-72501c-openshift-default-7f7d685fff-dvkn2" Jul 22 03:17:56.354346 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:17:56.354317 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-72501c-openshift-default-7f7d685fff-dvkn2" Jul 22 03:17:56.355272 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:17:56.355254 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-72501c-openshift-default-7f7d685fff-dvkn2" Jul 22 03:17:59.036960 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:17:59.036939 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-8dba71-openshift-default-55794d6cbc-5b4wq" Jul 22 03:17:59.097901 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:17:59.097877 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5ccbe86-eef3-4024-b02c-90951588a6ac-workload-certs\") pod \"e5ccbe86-eef3-4024-b02c-90951588a6ac\" (UID: \"e5ccbe86-eef3-4024-b02c-90951588a6ac\") " Jul 22 03:17:59.098064 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:17:59.097912 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5ccbe86-eef3-4024-b02c-90951588a6ac-istio-envoy\") pod \"e5ccbe86-eef3-4024-b02c-90951588a6ac\" (UID: \"e5ccbe86-eef3-4024-b02c-90951588a6ac\") " Jul 22 03:17:59.098064 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:17:59.097930 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dfmjk\" (UniqueName: \"kubernetes.io/projected/e5ccbe86-eef3-4024-b02c-90951588a6ac-kube-api-access-dfmjk\") pod \"e5ccbe86-eef3-4024-b02c-90951588a6ac\" (UID: \"e5ccbe86-eef3-4024-b02c-90951588a6ac\") " Jul 22 03:17:59.098064 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:17:59.097949 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5ccbe86-eef3-4024-b02c-90951588a6ac-istiod-ca-cert\") pod \"e5ccbe86-eef3-4024-b02c-90951588a6ac\" (UID: \"e5ccbe86-eef3-4024-b02c-90951588a6ac\") " Jul 22 03:17:59.098064 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:17:59.097977 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5ccbe86-eef3-4024-b02c-90951588a6ac-istio-data\") pod \"e5ccbe86-eef3-4024-b02c-90951588a6ac\" (UID: \"e5ccbe86-eef3-4024-b02c-90951588a6ac\") " Jul 22 03:17:59.098064 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:17:59.098031 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5ccbe86-eef3-4024-b02c-90951588a6ac-credential-socket\") pod \"e5ccbe86-eef3-4024-b02c-90951588a6ac\" (UID: \"e5ccbe86-eef3-4024-b02c-90951588a6ac\") " Jul 22 03:17:59.098064 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:17:59.098061 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5ccbe86-eef3-4024-b02c-90951588a6ac-workload-socket\") pod \"e5ccbe86-eef3-4024-b02c-90951588a6ac\" (UID: \"e5ccbe86-eef3-4024-b02c-90951588a6ac\") " Jul 22 03:17:59.098393 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:17:59.098091 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5ccbe86-eef3-4024-b02c-90951588a6ac-istio-token\") pod \"e5ccbe86-eef3-4024-b02c-90951588a6ac\" (UID: \"e5ccbe86-eef3-4024-b02c-90951588a6ac\") " Jul 22 03:17:59.098393 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:17:59.098130 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5ccbe86-eef3-4024-b02c-90951588a6ac-istio-podinfo\") pod \"e5ccbe86-eef3-4024-b02c-90951588a6ac\" (UID: \"e5ccbe86-eef3-4024-b02c-90951588a6ac\") " Jul 22 03:17:59.098393 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:17:59.098173 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5ccbe86-eef3-4024-b02c-90951588a6ac-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "e5ccbe86-eef3-4024-b02c-90951588a6ac" (UID: "e5ccbe86-eef3-4024-b02c-90951588a6ac"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 03:17:59.098393 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:17:59.098334 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5ccbe86-eef3-4024-b02c-90951588a6ac-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "e5ccbe86-eef3-4024-b02c-90951588a6ac" (UID: "e5ccbe86-eef3-4024-b02c-90951588a6ac"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 03:17:59.098629 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:17:59.098440 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5ccbe86-eef3-4024-b02c-90951588a6ac-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "e5ccbe86-eef3-4024-b02c-90951588a6ac" (UID: "e5ccbe86-eef3-4024-b02c-90951588a6ac"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 03:17:59.098629 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:17:59.098589 2578 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5ccbe86-eef3-4024-b02c-90951588a6ac-workload-certs\") on node \"ip-10-0-133-31.ec2.internal\" DevicePath \"\"" Jul 22 03:17:59.098629 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:17:59.098608 2578 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5ccbe86-eef3-4024-b02c-90951588a6ac-istiod-ca-cert\") on node \"ip-10-0-133-31.ec2.internal\" DevicePath \"\"" Jul 22 03:17:59.098629 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:17:59.098619 2578 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5ccbe86-eef3-4024-b02c-90951588a6ac-workload-socket\") on node \"ip-10-0-133-31.ec2.internal\" DevicePath \"\"" Jul 22 03:17:59.098991 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:17:59.098951 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5ccbe86-eef3-4024-b02c-90951588a6ac-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "e5ccbe86-eef3-4024-b02c-90951588a6ac" (UID: "e5ccbe86-eef3-4024-b02c-90951588a6ac"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 03:17:59.099747 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:17:59.099718 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5ccbe86-eef3-4024-b02c-90951588a6ac-istio-data" (OuterVolumeSpecName: "istio-data") pod "e5ccbe86-eef3-4024-b02c-90951588a6ac" (UID: "e5ccbe86-eef3-4024-b02c-90951588a6ac"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 03:17:59.100245 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:17:59.100216 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5ccbe86-eef3-4024-b02c-90951588a6ac-kube-api-access-dfmjk" (OuterVolumeSpecName: "kube-api-access-dfmjk") pod "e5ccbe86-eef3-4024-b02c-90951588a6ac" (UID: "e5ccbe86-eef3-4024-b02c-90951588a6ac"). InnerVolumeSpecName "kube-api-access-dfmjk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 03:17:59.100341 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:17:59.100321 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5ccbe86-eef3-4024-b02c-90951588a6ac-istio-token" (OuterVolumeSpecName: "istio-token") pod "e5ccbe86-eef3-4024-b02c-90951588a6ac" (UID: "e5ccbe86-eef3-4024-b02c-90951588a6ac"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 03:17:59.100387 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:17:59.100361 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5ccbe86-eef3-4024-b02c-90951588a6ac-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "e5ccbe86-eef3-4024-b02c-90951588a6ac" (UID: "e5ccbe86-eef3-4024-b02c-90951588a6ac"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 03:17:59.100829 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:17:59.100814 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e5ccbe86-eef3-4024-b02c-90951588a6ac-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "e5ccbe86-eef3-4024-b02c-90951588a6ac" (UID: "e5ccbe86-eef3-4024-b02c-90951588a6ac"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 22 03:17:59.199445 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:17:59.199417 2578 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5ccbe86-eef3-4024-b02c-90951588a6ac-istio-envoy\") on node \"ip-10-0-133-31.ec2.internal\" DevicePath \"\"" Jul 22 03:17:59.199445 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:17:59.199442 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dfmjk\" (UniqueName: \"kubernetes.io/projected/e5ccbe86-eef3-4024-b02c-90951588a6ac-kube-api-access-dfmjk\") on node \"ip-10-0-133-31.ec2.internal\" DevicePath \"\"" Jul 22 03:17:59.199623 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:17:59.199451 2578 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5ccbe86-eef3-4024-b02c-90951588a6ac-istio-data\") on node \"ip-10-0-133-31.ec2.internal\" DevicePath \"\"" Jul 22 03:17:59.199623 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:17:59.199459 2578 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5ccbe86-eef3-4024-b02c-90951588a6ac-credential-socket\") on node \"ip-10-0-133-31.ec2.internal\" DevicePath \"\"" Jul 22 03:17:59.199623 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:17:59.199466 2578 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5ccbe86-eef3-4024-b02c-90951588a6ac-istio-token\") on node \"ip-10-0-133-31.ec2.internal\" DevicePath \"\"" Jul 22 03:17:59.199623 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:17:59.199474 2578 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5ccbe86-eef3-4024-b02c-90951588a6ac-istio-podinfo\") on node \"ip-10-0-133-31.ec2.internal\" DevicePath \"\"" Jul 22 03:17:59.372454 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:17:59.372392 2578 generic.go:356] "Generic (PLEG): container finished" podID="e5ccbe86-eef3-4024-b02c-90951588a6ac" containerID="56b67360792e1fa2f9d10b5f4e9e56534027c93bb17094ed6941d9c572d7f71f" exitCode=0 Jul 22 03:17:59.372454 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:17:59.372444 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-8dba71-openshift-default-55794d6cbc-5b4wq" Jul 22 03:17:59.372652 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:17:59.372484 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-8dba71-openshift-default-55794d6cbc-5b4wq" event={"ID":"e5ccbe86-eef3-4024-b02c-90951588a6ac","Type":"ContainerDied","Data":"56b67360792e1fa2f9d10b5f4e9e56534027c93bb17094ed6941d9c572d7f71f"} Jul 22 03:17:59.372652 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:17:59.372548 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-8dba71-openshift-default-55794d6cbc-5b4wq" event={"ID":"e5ccbe86-eef3-4024-b02c-90951588a6ac","Type":"ContainerDied","Data":"10f175e033c81e4db8fd9a3538b907ccfe9054374293108beb3ca2e48f854a9b"} Jul 22 03:17:59.372652 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:17:59.372593 2578 scope.go:117] "RemoveContainer" containerID="56b67360792e1fa2f9d10b5f4e9e56534027c93bb17094ed6941d9c572d7f71f" Jul 22 03:17:59.381590 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:17:59.381553 2578 scope.go:117] "RemoveContainer" containerID="91757b8e01745a4cf186fe2431d488a1566a0df30ec1c0d10c107d521983dc96" Jul 22 03:17:59.389214 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:17:59.389196 2578 scope.go:117] "RemoveContainer" containerID="56b67360792e1fa2f9d10b5f4e9e56534027c93bb17094ed6941d9c572d7f71f" Jul 22 03:17:59.389473 ip-10-0-133-31 kubenswrapper[2578]: E0722 03:17:59.389452 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"56b67360792e1fa2f9d10b5f4e9e56534027c93bb17094ed6941d9c572d7f71f\": container with ID starting with 56b67360792e1fa2f9d10b5f4e9e56534027c93bb17094ed6941d9c572d7f71f not found: ID does not exist" containerID="56b67360792e1fa2f9d10b5f4e9e56534027c93bb17094ed6941d9c572d7f71f" Jul 22 03:17:59.389551 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:17:59.389480 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56b67360792e1fa2f9d10b5f4e9e56534027c93bb17094ed6941d9c572d7f71f"} err="failed to get container status \"56b67360792e1fa2f9d10b5f4e9e56534027c93bb17094ed6941d9c572d7f71f\": rpc error: code = NotFound desc = could not find container \"56b67360792e1fa2f9d10b5f4e9e56534027c93bb17094ed6941d9c572d7f71f\": container with ID starting with 56b67360792e1fa2f9d10b5f4e9e56534027c93bb17094ed6941d9c572d7f71f not found: ID does not exist" Jul 22 03:17:59.389551 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:17:59.389498 2578 scope.go:117] "RemoveContainer" containerID="91757b8e01745a4cf186fe2431d488a1566a0df30ec1c0d10c107d521983dc96" Jul 22 03:17:59.389799 ip-10-0-133-31 kubenswrapper[2578]: E0722 03:17:59.389785 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"91757b8e01745a4cf186fe2431d488a1566a0df30ec1c0d10c107d521983dc96\": container with ID starting with 91757b8e01745a4cf186fe2431d488a1566a0df30ec1c0d10c107d521983dc96 not found: ID does not exist" containerID="91757b8e01745a4cf186fe2431d488a1566a0df30ec1c0d10c107d521983dc96" Jul 22 03:17:59.389842 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:17:59.389803 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91757b8e01745a4cf186fe2431d488a1566a0df30ec1c0d10c107d521983dc96"} err="failed to get container status \"91757b8e01745a4cf186fe2431d488a1566a0df30ec1c0d10c107d521983dc96\": rpc error: code = NotFound desc = could not find container \"91757b8e01745a4cf186fe2431d488a1566a0df30ec1c0d10c107d521983dc96\": container with ID starting with 91757b8e01745a4cf186fe2431d488a1566a0df30ec1c0d10c107d521983dc96 not found: ID does not exist" Jul 22 03:17:59.395701 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:17:59.395681 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-8dba71-openshift-default-55794d6cbc-5b4wq"] Jul 22 03:17:59.399252 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:17:59.399233 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-8dba71-openshift-default-55794d6cbc-5b4wq"] Jul 22 03:18:00.142662 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:18:00.142622 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e5ccbe86-eef3-4024-b02c-90951588a6ac" path="/var/lib/kubelet/pods/e5ccbe86-eef3-4024-b02c-90951588a6ac/volumes" Jul 22 03:18:02.466332 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:18:02.466297 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-e9ef33-openshift-default-76c5696ddc-5g9gv"] Jul 22 03:18:02.466721 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:18:02.466694 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e3f4818a-8157-4536-91b6-306280ddb7ef" containerName="istio-proxy" Jul 22 03:18:02.466721 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:18:02.466707 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="e3f4818a-8157-4536-91b6-306280ddb7ef" containerName="istio-proxy" Jul 22 03:18:02.466721 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:18:02.466716 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e5ccbe86-eef3-4024-b02c-90951588a6ac" containerName="istio-proxy" Jul 22 03:18:02.466721 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:18:02.466721 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="e5ccbe86-eef3-4024-b02c-90951588a6ac" containerName="istio-proxy" Jul 22 03:18:02.466868 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:18:02.466740 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e5ccbe86-eef3-4024-b02c-90951588a6ac" containerName="istio-proxy" Jul 22 03:18:02.466868 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:18:02.466747 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="e5ccbe86-eef3-4024-b02c-90951588a6ac" containerName="istio-proxy" Jul 22 03:18:02.466868 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:18:02.466806 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="e5ccbe86-eef3-4024-b02c-90951588a6ac" containerName="istio-proxy" Jul 22 03:18:02.466868 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:18:02.466814 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="e5ccbe86-eef3-4024-b02c-90951588a6ac" containerName="istio-proxy" Jul 22 03:18:02.471189 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:18:02.471168 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-e9ef33-openshift-default-76c5696ddc-5g9gv" Jul 22 03:18:02.474443 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:18:02.474426 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-e9ef33-openshift-default-dockercfg-zjv4b\"" Jul 22 03:18:02.486962 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:18:02.486943 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-e9ef33-openshift-default-76c5696ddc-5g9gv"] Jul 22 03:18:02.629202 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:18:02.629169 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-chlpc\" (UniqueName: \"kubernetes.io/projected/b8649024-1788-46ee-959c-d521b00975f4-kube-api-access-chlpc\") pod \"e2e-shared-b-e9ef33-openshift-default-76c5696ddc-5g9gv\" (UID: \"b8649024-1788-46ee-959c-d521b00975f4\") " pod="openshift-ingress/e2e-shared-b-e9ef33-openshift-default-76c5696ddc-5g9gv" Jul 22 03:18:02.629394 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:18:02.629212 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8649024-1788-46ee-959c-d521b00975f4-istiod-ca-cert\") pod \"e2e-shared-b-e9ef33-openshift-default-76c5696ddc-5g9gv\" (UID: \"b8649024-1788-46ee-959c-d521b00975f4\") " pod="openshift-ingress/e2e-shared-b-e9ef33-openshift-default-76c5696ddc-5g9gv" Jul 22 03:18:02.629394 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:18:02.629256 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8649024-1788-46ee-959c-d521b00975f4-istio-data\") pod \"e2e-shared-b-e9ef33-openshift-default-76c5696ddc-5g9gv\" (UID: \"b8649024-1788-46ee-959c-d521b00975f4\") " pod="openshift-ingress/e2e-shared-b-e9ef33-openshift-default-76c5696ddc-5g9gv" Jul 22 03:18:02.629394 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:18:02.629278 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b8649024-1788-46ee-959c-d521b00975f4-istio-token\") pod \"e2e-shared-b-e9ef33-openshift-default-76c5696ddc-5g9gv\" (UID: \"b8649024-1788-46ee-959c-d521b00975f4\") " pod="openshift-ingress/e2e-shared-b-e9ef33-openshift-default-76c5696ddc-5g9gv" Jul 22 03:18:02.629394 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:18:02.629300 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8649024-1788-46ee-959c-d521b00975f4-istio-podinfo\") pod \"e2e-shared-b-e9ef33-openshift-default-76c5696ddc-5g9gv\" (UID: \"b8649024-1788-46ee-959c-d521b00975f4\") " pod="openshift-ingress/e2e-shared-b-e9ef33-openshift-default-76c5696ddc-5g9gv" Jul 22 03:18:02.629394 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:18:02.629386 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8649024-1788-46ee-959c-d521b00975f4-workload-certs\") pod \"e2e-shared-b-e9ef33-openshift-default-76c5696ddc-5g9gv\" (UID: \"b8649024-1788-46ee-959c-d521b00975f4\") " pod="openshift-ingress/e2e-shared-b-e9ef33-openshift-default-76c5696ddc-5g9gv" Jul 22 03:18:02.629625 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:18:02.629419 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8649024-1788-46ee-959c-d521b00975f4-credential-socket\") pod \"e2e-shared-b-e9ef33-openshift-default-76c5696ddc-5g9gv\" (UID: \"b8649024-1788-46ee-959c-d521b00975f4\") " pod="openshift-ingress/e2e-shared-b-e9ef33-openshift-default-76c5696ddc-5g9gv" Jul 22 03:18:02.629625 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:18:02.629457 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8649024-1788-46ee-959c-d521b00975f4-workload-socket\") pod \"e2e-shared-b-e9ef33-openshift-default-76c5696ddc-5g9gv\" (UID: \"b8649024-1788-46ee-959c-d521b00975f4\") " pod="openshift-ingress/e2e-shared-b-e9ef33-openshift-default-76c5696ddc-5g9gv" Jul 22 03:18:02.629625 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:18:02.629484 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8649024-1788-46ee-959c-d521b00975f4-istio-envoy\") pod \"e2e-shared-b-e9ef33-openshift-default-76c5696ddc-5g9gv\" (UID: \"b8649024-1788-46ee-959c-d521b00975f4\") " pod="openshift-ingress/e2e-shared-b-e9ef33-openshift-default-76c5696ddc-5g9gv" Jul 22 03:18:02.730252 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:18:02.730173 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-chlpc\" (UniqueName: \"kubernetes.io/projected/b8649024-1788-46ee-959c-d521b00975f4-kube-api-access-chlpc\") pod \"e2e-shared-b-e9ef33-openshift-default-76c5696ddc-5g9gv\" (UID: \"b8649024-1788-46ee-959c-d521b00975f4\") " pod="openshift-ingress/e2e-shared-b-e9ef33-openshift-default-76c5696ddc-5g9gv" Jul 22 03:18:02.730252 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:18:02.730209 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8649024-1788-46ee-959c-d521b00975f4-istiod-ca-cert\") pod \"e2e-shared-b-e9ef33-openshift-default-76c5696ddc-5g9gv\" (UID: \"b8649024-1788-46ee-959c-d521b00975f4\") " pod="openshift-ingress/e2e-shared-b-e9ef33-openshift-default-76c5696ddc-5g9gv" Jul 22 03:18:02.730252 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:18:02.730238 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8649024-1788-46ee-959c-d521b00975f4-istio-data\") pod \"e2e-shared-b-e9ef33-openshift-default-76c5696ddc-5g9gv\" (UID: \"b8649024-1788-46ee-959c-d521b00975f4\") " pod="openshift-ingress/e2e-shared-b-e9ef33-openshift-default-76c5696ddc-5g9gv" Jul 22 03:18:02.730529 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:18:02.730420 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b8649024-1788-46ee-959c-d521b00975f4-istio-token\") pod \"e2e-shared-b-e9ef33-openshift-default-76c5696ddc-5g9gv\" (UID: \"b8649024-1788-46ee-959c-d521b00975f4\") " pod="openshift-ingress/e2e-shared-b-e9ef33-openshift-default-76c5696ddc-5g9gv" Jul 22 03:18:02.730529 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:18:02.730459 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8649024-1788-46ee-959c-d521b00975f4-istio-podinfo\") pod \"e2e-shared-b-e9ef33-openshift-default-76c5696ddc-5g9gv\" (UID: \"b8649024-1788-46ee-959c-d521b00975f4\") " pod="openshift-ingress/e2e-shared-b-e9ef33-openshift-default-76c5696ddc-5g9gv" Jul 22 03:18:02.730655 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:18:02.730606 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8649024-1788-46ee-959c-d521b00975f4-workload-certs\") pod \"e2e-shared-b-e9ef33-openshift-default-76c5696ddc-5g9gv\" (UID: \"b8649024-1788-46ee-959c-d521b00975f4\") " pod="openshift-ingress/e2e-shared-b-e9ef33-openshift-default-76c5696ddc-5g9gv" Jul 22 03:18:02.730709 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:18:02.730656 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8649024-1788-46ee-959c-d521b00975f4-credential-socket\") pod \"e2e-shared-b-e9ef33-openshift-default-76c5696ddc-5g9gv\" (UID: \"b8649024-1788-46ee-959c-d521b00975f4\") " pod="openshift-ingress/e2e-shared-b-e9ef33-openshift-default-76c5696ddc-5g9gv" Jul 22 03:18:02.730709 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:18:02.730673 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8649024-1788-46ee-959c-d521b00975f4-istio-data\") pod \"e2e-shared-b-e9ef33-openshift-default-76c5696ddc-5g9gv\" (UID: \"b8649024-1788-46ee-959c-d521b00975f4\") " pod="openshift-ingress/e2e-shared-b-e9ef33-openshift-default-76c5696ddc-5g9gv" Jul 22 03:18:02.730813 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:18:02.730724 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8649024-1788-46ee-959c-d521b00975f4-workload-socket\") pod \"e2e-shared-b-e9ef33-openshift-default-76c5696ddc-5g9gv\" (UID: \"b8649024-1788-46ee-959c-d521b00975f4\") " pod="openshift-ingress/e2e-shared-b-e9ef33-openshift-default-76c5696ddc-5g9gv" Jul 22 03:18:02.730813 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:18:02.730766 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8649024-1788-46ee-959c-d521b00975f4-istio-envoy\") pod \"e2e-shared-b-e9ef33-openshift-default-76c5696ddc-5g9gv\" (UID: \"b8649024-1788-46ee-959c-d521b00975f4\") " pod="openshift-ingress/e2e-shared-b-e9ef33-openshift-default-76c5696ddc-5g9gv" Jul 22 03:18:02.730970 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:18:02.730951 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8649024-1788-46ee-959c-d521b00975f4-workload-certs\") pod \"e2e-shared-b-e9ef33-openshift-default-76c5696ddc-5g9gv\" (UID: \"b8649024-1788-46ee-959c-d521b00975f4\") " pod="openshift-ingress/e2e-shared-b-e9ef33-openshift-default-76c5696ddc-5g9gv" Jul 22 03:18:02.731020 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:18:02.730979 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8649024-1788-46ee-959c-d521b00975f4-istiod-ca-cert\") pod \"e2e-shared-b-e9ef33-openshift-default-76c5696ddc-5g9gv\" (UID: \"b8649024-1788-46ee-959c-d521b00975f4\") " pod="openshift-ingress/e2e-shared-b-e9ef33-openshift-default-76c5696ddc-5g9gv" Jul 22 03:18:02.731020 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:18:02.730996 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8649024-1788-46ee-959c-d521b00975f4-credential-socket\") pod \"e2e-shared-b-e9ef33-openshift-default-76c5696ddc-5g9gv\" (UID: \"b8649024-1788-46ee-959c-d521b00975f4\") " pod="openshift-ingress/e2e-shared-b-e9ef33-openshift-default-76c5696ddc-5g9gv" Jul 22 03:18:02.731103 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:18:02.731082 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8649024-1788-46ee-959c-d521b00975f4-workload-socket\") pod \"e2e-shared-b-e9ef33-openshift-default-76c5696ddc-5g9gv\" (UID: \"b8649024-1788-46ee-959c-d521b00975f4\") " pod="openshift-ingress/e2e-shared-b-e9ef33-openshift-default-76c5696ddc-5g9gv" Jul 22 03:18:02.732960 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:18:02.732925 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8649024-1788-46ee-959c-d521b00975f4-istio-envoy\") pod \"e2e-shared-b-e9ef33-openshift-default-76c5696ddc-5g9gv\" (UID: \"b8649024-1788-46ee-959c-d521b00975f4\") " pod="openshift-ingress/e2e-shared-b-e9ef33-openshift-default-76c5696ddc-5g9gv" Jul 22 03:18:02.733059 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:18:02.733020 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8649024-1788-46ee-959c-d521b00975f4-istio-podinfo\") pod \"e2e-shared-b-e9ef33-openshift-default-76c5696ddc-5g9gv\" (UID: \"b8649024-1788-46ee-959c-d521b00975f4\") " pod="openshift-ingress/e2e-shared-b-e9ef33-openshift-default-76c5696ddc-5g9gv" Jul 22 03:18:02.738745 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:18:02.738725 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-chlpc\" (UniqueName: \"kubernetes.io/projected/b8649024-1788-46ee-959c-d521b00975f4-kube-api-access-chlpc\") pod \"e2e-shared-b-e9ef33-openshift-default-76c5696ddc-5g9gv\" (UID: \"b8649024-1788-46ee-959c-d521b00975f4\") " pod="openshift-ingress/e2e-shared-b-e9ef33-openshift-default-76c5696ddc-5g9gv" Jul 22 03:18:02.738854 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:18:02.738800 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b8649024-1788-46ee-959c-d521b00975f4-istio-token\") pod \"e2e-shared-b-e9ef33-openshift-default-76c5696ddc-5g9gv\" (UID: \"b8649024-1788-46ee-959c-d521b00975f4\") " pod="openshift-ingress/e2e-shared-b-e9ef33-openshift-default-76c5696ddc-5g9gv" Jul 22 03:18:02.781548 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:18:02.781529 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-e9ef33-openshift-default-76c5696ddc-5g9gv" Jul 22 03:18:02.922600 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:18:02.922555 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-e9ef33-openshift-default-76c5696ddc-5g9gv"] Jul 22 03:18:02.924387 ip-10-0-133-31 kubenswrapper[2578]: W0722 03:18:02.924364 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8649024_1788_46ee_959c_d521b00975f4.slice/crio-b3f44d6ef1c1f66f633686b72d6512394ddd448509f43a1d9ba5d0ca7453d1b2 WatchSource:0}: Error finding container b3f44d6ef1c1f66f633686b72d6512394ddd448509f43a1d9ba5d0ca7453d1b2: Status 404 returned error can't find the container with id b3f44d6ef1c1f66f633686b72d6512394ddd448509f43a1d9ba5d0ca7453d1b2 Jul 22 03:18:02.926429 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:18:02.926401 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 22 03:18:02.926502 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:18:02.926484 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 22 03:18:02.926541 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:18:02.926513 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 22 03:18:03.398089 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:18:03.398048 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-e9ef33-openshift-default-76c5696ddc-5g9gv" event={"ID":"b8649024-1788-46ee-959c-d521b00975f4","Type":"ContainerStarted","Data":"694b178fd7e3f0d6187134a6ee626bd2c40a132cda4646f785ea20b6775db2a0"} Jul 22 03:18:03.398273 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:18:03.398116 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-e9ef33-openshift-default-76c5696ddc-5g9gv" event={"ID":"b8649024-1788-46ee-959c-d521b00975f4","Type":"ContainerStarted","Data":"b3f44d6ef1c1f66f633686b72d6512394ddd448509f43a1d9ba5d0ca7453d1b2"} Jul 22 03:18:03.422882 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:18:03.422834 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-e9ef33-openshift-default-76c5696ddc-5g9gv" podStartSLOduration=1.4228199959999999 podStartE2EDuration="1.422819996s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 03:18:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 03:18:03.420737658 +0000 UTC m=+2449.739912957" watchObservedRunningTime="2026-07-22 03:18:03.422819996 +0000 UTC m=+2449.741995296" Jul 22 03:18:03.782619 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:18:03.782539 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-e9ef33-openshift-default-76c5696ddc-5g9gv" Jul 22 03:18:03.787240 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:18:03.787218 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-e9ef33-openshift-default-76c5696ddc-5g9gv" Jul 22 03:18:04.403730 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:18:04.403650 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-e9ef33-openshift-default-76c5696ddc-5g9gv" Jul 22 03:18:04.404747 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:18:04.404642 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-e9ef33-openshift-default-76c5696ddc-5g9gv" Jul 22 03:18:12.137377 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:18:12.136539 2578 patch_prober.go:28] interesting pod/e2e-shared-a-72501c-openshift-default-7f7d685fff-dvkn2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": read tcp 10.133.0.2:54074->10.133.0.62:15021: read: connection reset by peer" start-of-body= Jul 22 03:18:12.137377 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:18:12.136659 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-72501c-openshift-default-7f7d685fff-dvkn2" podUID="c52d4866-0bd6-4fdf-ab45-166f1778ba19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": read tcp 10.133.0.2:54074->10.133.0.62:15021: read: connection reset by peer" Jul 22 03:18:12.454589 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:18:12.454483 2578 generic.go:356] "Generic (PLEG): container finished" podID="c52d4866-0bd6-4fdf-ab45-166f1778ba19" containerID="ff0dfc1c872ff0dc11924df062fed85c0796e5823060a10d0ee0c86ce046e1f7" exitCode=0 Jul 22 03:18:12.454589 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:18:12.454553 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-72501c-openshift-default-7f7d685fff-dvkn2" event={"ID":"c52d4866-0bd6-4fdf-ab45-166f1778ba19","Type":"ContainerDied","Data":"ff0dfc1c872ff0dc11924df062fed85c0796e5823060a10d0ee0c86ce046e1f7"} Jul 22 03:18:12.454982 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:18:12.454969 2578 scope.go:117] "RemoveContainer" containerID="ff0dfc1c872ff0dc11924df062fed85c0796e5823060a10d0ee0c86ce046e1f7" Jul 22 03:18:12.456025 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:18:12.455988 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 22 03:18:12.456113 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:18:12.456074 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 22 03:18:12.456113 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:18:12.456106 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 22 03:18:13.462443 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:18:13.462407 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-72501c-openshift-default-7f7d685fff-dvkn2" event={"ID":"c52d4866-0bd6-4fdf-ab45-166f1778ba19","Type":"ContainerStarted","Data":"0ef2bbfe527acbe8a98853f85b223abb50d78ce954557f20aaee8dc8d2fdcdcb"} Jul 22 03:18:14.013747 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:18:14.013706 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-72501c-openshift-default-7f7d685fff-dvkn2" Jul 22 03:18:14.013949 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:18:14.013839 2578 patch_prober.go:28] interesting pod/e2e-shared-a-72501c-openshift-default-7f7d685fff-dvkn2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Jul 22 03:18:14.013949 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:18:14.013883 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-72501c-openshift-default-7f7d685fff-dvkn2" podUID="c52d4866-0bd6-4fdf-ab45-166f1778ba19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Jul 22 03:18:15.013824 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:18:15.013768 2578 patch_prober.go:28] interesting pod/e2e-shared-a-72501c-openshift-default-7f7d685fff-dvkn2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Jul 22 03:18:15.014315 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:18:15.013864 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-72501c-openshift-default-7f7d685fff-dvkn2" podUID="c52d4866-0bd6-4fdf-ab45-166f1778ba19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Jul 22 03:18:16.014061 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:18:16.014022 2578 patch_prober.go:28] interesting pod/e2e-shared-a-72501c-openshift-default-7f7d685fff-dvkn2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Jul 22 03:18:16.014652 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:18:16.014616 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-72501c-openshift-default-7f7d685fff-dvkn2" podUID="c52d4866-0bd6-4fdf-ab45-166f1778ba19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Jul 22 03:18:17.013510 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:18:17.013469 2578 patch_prober.go:28] interesting pod/e2e-shared-a-72501c-openshift-default-7f7d685fff-dvkn2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Jul 22 03:18:17.013721 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:18:17.013554 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-72501c-openshift-default-7f7d685fff-dvkn2" podUID="c52d4866-0bd6-4fdf-ab45-166f1778ba19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Jul 22 03:18:18.017397 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:18:18.017362 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-72501c-openshift-default-7f7d685fff-dvkn2" Jul 22 03:18:18.017828 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:18:18.017763 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-72501c-openshift-default-7f7d685fff-dvkn2" Jul 22 03:18:18.018537 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:18:18.018520 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-72501c-openshift-default-7f7d685fff-dvkn2" Jul 22 03:18:20.176072 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:18:20.176031 2578 patch_prober.go:28] interesting pod/e2e-shared-b-e9ef33-openshift-default-76c5696ddc-5g9gv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": read tcp 10.133.0.2:59588->10.133.0.63:15021: read: connection reset by peer" start-of-body= Jul 22 03:18:20.176481 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:18:20.176119 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-e9ef33-openshift-default-76c5696ddc-5g9gv" podUID="b8649024-1788-46ee-959c-d521b00975f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": read tcp 10.133.0.2:59588->10.133.0.63:15021: read: connection reset by peer" Jul 22 03:18:20.500106 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:18:20.500006 2578 generic.go:356] "Generic (PLEG): container finished" podID="b8649024-1788-46ee-959c-d521b00975f4" containerID="694b178fd7e3f0d6187134a6ee626bd2c40a132cda4646f785ea20b6775db2a0" exitCode=0 Jul 22 03:18:20.500106 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:18:20.500081 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-e9ef33-openshift-default-76c5696ddc-5g9gv" event={"ID":"b8649024-1788-46ee-959c-d521b00975f4","Type":"ContainerDied","Data":"694b178fd7e3f0d6187134a6ee626bd2c40a132cda4646f785ea20b6775db2a0"} Jul 22 03:18:20.500475 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:18:20.500460 2578 scope.go:117] "RemoveContainer" containerID="694b178fd7e3f0d6187134a6ee626bd2c40a132cda4646f785ea20b6775db2a0" Jul 22 03:18:20.501547 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:18:20.501522 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 22 03:18:20.501647 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:18:20.501610 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 22 03:18:20.501687 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:18:20.501645 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 22 03:18:21.523380 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:18:21.523349 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-e9ef33-openshift-default-76c5696ddc-5g9gv" event={"ID":"b8649024-1788-46ee-959c-d521b00975f4","Type":"ContainerStarted","Data":"8b5fbc9196b074164d88522c300a6994df416fe1cc27f511b15aae80115862c6"} Jul 22 03:18:21.782151 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:18:21.782051 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-e9ef33-openshift-default-76c5696ddc-5g9gv" Jul 22 03:18:21.782351 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:18:21.782320 2578 patch_prober.go:28] interesting pod/e2e-shared-b-e9ef33-openshift-default-76c5696ddc-5g9gv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Jul 22 03:18:21.782476 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:18:21.782440 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-e9ef33-openshift-default-76c5696ddc-5g9gv" podUID="b8649024-1788-46ee-959c-d521b00975f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Jul 22 03:18:22.781985 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:18:22.781948 2578 patch_prober.go:28] interesting pod/e2e-shared-b-e9ef33-openshift-default-76c5696ddc-5g9gv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Jul 22 03:18:22.782433 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:18:22.782021 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-e9ef33-openshift-default-76c5696ddc-5g9gv" podUID="b8649024-1788-46ee-959c-d521b00975f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Jul 22 03:18:23.782106 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:18:23.782076 2578 patch_prober.go:28] interesting pod/e2e-shared-b-e9ef33-openshift-default-76c5696ddc-5g9gv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Jul 22 03:18:23.782528 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:18:23.782138 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-e9ef33-openshift-default-76c5696ddc-5g9gv" podUID="b8649024-1788-46ee-959c-d521b00975f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Jul 22 03:18:24.782649 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:18:24.782618 2578 patch_prober.go:28] interesting pod/e2e-shared-b-e9ef33-openshift-default-76c5696ddc-5g9gv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Jul 22 03:18:24.783093 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:18:24.782671 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-e9ef33-openshift-default-76c5696ddc-5g9gv" podUID="b8649024-1788-46ee-959c-d521b00975f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Jul 22 03:18:25.785236 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:18:25.785205 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-e9ef33-openshift-default-76c5696ddc-5g9gv" Jul 22 03:18:25.785609 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:18:25.785459 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-e9ef33-openshift-default-76c5696ddc-5g9gv" Jul 22 03:18:25.786149 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:18:25.786136 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-e9ef33-openshift-default-76c5696ddc-5g9gv" Jul 22 03:20:47.579754 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:20:47.579674 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-72501c/sub-test-model-72501c-kserve-ffd4675ff-t2k68"] Jul 22 03:20:47.583010 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:20:47.582993 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-72501c/sub-test-model-72501c-kserve-ffd4675ff-t2k68" Jul 22 03:20:47.587188 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:20:47.587168 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-72501c\"/\"kube-root-ca.crt\"" Jul 22 03:20:47.587325 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:20:47.587172 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-72501c\"/\"default-dockercfg-k5gzz\"" Jul 22 03:20:47.587325 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:20:47.587174 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-72501c\"/\"sub-test-model-72501c-kserve-self-signed-certs\"" Jul 22 03:20:47.587325 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:20:47.587286 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-72501c\"/\"openshift-service-ca.crt\"" Jul 22 03:20:47.594094 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:20:47.594075 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-72501c/sub-test-model-72501c-kserve-ffd4675ff-t2k68"] Jul 22 03:20:47.721175 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:20:47.721137 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e4b99269-7b0c-48e4-8617-82aae55cbdd0-tmp-dir\") pod \"sub-test-model-72501c-kserve-ffd4675ff-t2k68\" (UID: \"e4b99269-7b0c-48e4-8617-82aae55cbdd0\") " pod="ai-tenant-e2e-shared-a-72501c/sub-test-model-72501c-kserve-ffd4675ff-t2k68" Jul 22 03:20:47.721175 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:20:47.721180 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e4b99269-7b0c-48e4-8617-82aae55cbdd0-tls-certs\") pod \"sub-test-model-72501c-kserve-ffd4675ff-t2k68\" (UID: \"e4b99269-7b0c-48e4-8617-82aae55cbdd0\") " pod="ai-tenant-e2e-shared-a-72501c/sub-test-model-72501c-kserve-ffd4675ff-t2k68" Jul 22 03:20:47.721404 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:20:47.721260 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e4b99269-7b0c-48e4-8617-82aae55cbdd0-dshm\") pod \"sub-test-model-72501c-kserve-ffd4675ff-t2k68\" (UID: \"e4b99269-7b0c-48e4-8617-82aae55cbdd0\") " pod="ai-tenant-e2e-shared-a-72501c/sub-test-model-72501c-kserve-ffd4675ff-t2k68" Jul 22 03:20:47.721404 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:20:47.721337 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e4b99269-7b0c-48e4-8617-82aae55cbdd0-model-cache\") pod \"sub-test-model-72501c-kserve-ffd4675ff-t2k68\" (UID: \"e4b99269-7b0c-48e4-8617-82aae55cbdd0\") " pod="ai-tenant-e2e-shared-a-72501c/sub-test-model-72501c-kserve-ffd4675ff-t2k68" Jul 22 03:20:47.721404 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:20:47.721386 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e4b99269-7b0c-48e4-8617-82aae55cbdd0-home\") pod \"sub-test-model-72501c-kserve-ffd4675ff-t2k68\" (UID: \"e4b99269-7b0c-48e4-8617-82aae55cbdd0\") " pod="ai-tenant-e2e-shared-a-72501c/sub-test-model-72501c-kserve-ffd4675ff-t2k68" Jul 22 03:20:47.721550 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:20:47.721445 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tk72l\" (UniqueName: \"kubernetes.io/projected/e4b99269-7b0c-48e4-8617-82aae55cbdd0-kube-api-access-tk72l\") pod \"sub-test-model-72501c-kserve-ffd4675ff-t2k68\" (UID: \"e4b99269-7b0c-48e4-8617-82aae55cbdd0\") " pod="ai-tenant-e2e-shared-a-72501c/sub-test-model-72501c-kserve-ffd4675ff-t2k68" Jul 22 03:20:47.822269 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:20:47.822238 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e4b99269-7b0c-48e4-8617-82aae55cbdd0-home\") pod \"sub-test-model-72501c-kserve-ffd4675ff-t2k68\" (UID: \"e4b99269-7b0c-48e4-8617-82aae55cbdd0\") " pod="ai-tenant-e2e-shared-a-72501c/sub-test-model-72501c-kserve-ffd4675ff-t2k68" Jul 22 03:20:47.822433 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:20:47.822287 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tk72l\" (UniqueName: \"kubernetes.io/projected/e4b99269-7b0c-48e4-8617-82aae55cbdd0-kube-api-access-tk72l\") pod \"sub-test-model-72501c-kserve-ffd4675ff-t2k68\" (UID: \"e4b99269-7b0c-48e4-8617-82aae55cbdd0\") " pod="ai-tenant-e2e-shared-a-72501c/sub-test-model-72501c-kserve-ffd4675ff-t2k68" Jul 22 03:20:47.822433 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:20:47.822341 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e4b99269-7b0c-48e4-8617-82aae55cbdd0-tmp-dir\") pod \"sub-test-model-72501c-kserve-ffd4675ff-t2k68\" (UID: \"e4b99269-7b0c-48e4-8617-82aae55cbdd0\") " pod="ai-tenant-e2e-shared-a-72501c/sub-test-model-72501c-kserve-ffd4675ff-t2k68" Jul 22 03:20:47.822433 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:20:47.822370 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e4b99269-7b0c-48e4-8617-82aae55cbdd0-tls-certs\") pod \"sub-test-model-72501c-kserve-ffd4675ff-t2k68\" (UID: \"e4b99269-7b0c-48e4-8617-82aae55cbdd0\") " pod="ai-tenant-e2e-shared-a-72501c/sub-test-model-72501c-kserve-ffd4675ff-t2k68" Jul 22 03:20:47.822433 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:20:47.822427 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e4b99269-7b0c-48e4-8617-82aae55cbdd0-dshm\") pod \"sub-test-model-72501c-kserve-ffd4675ff-t2k68\" (UID: \"e4b99269-7b0c-48e4-8617-82aae55cbdd0\") " pod="ai-tenant-e2e-shared-a-72501c/sub-test-model-72501c-kserve-ffd4675ff-t2k68" Jul 22 03:20:47.822689 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:20:47.822483 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e4b99269-7b0c-48e4-8617-82aae55cbdd0-model-cache\") pod \"sub-test-model-72501c-kserve-ffd4675ff-t2k68\" (UID: \"e4b99269-7b0c-48e4-8617-82aae55cbdd0\") " pod="ai-tenant-e2e-shared-a-72501c/sub-test-model-72501c-kserve-ffd4675ff-t2k68" Jul 22 03:20:47.822749 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:20:47.822683 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e4b99269-7b0c-48e4-8617-82aae55cbdd0-home\") pod \"sub-test-model-72501c-kserve-ffd4675ff-t2k68\" (UID: \"e4b99269-7b0c-48e4-8617-82aae55cbdd0\") " pod="ai-tenant-e2e-shared-a-72501c/sub-test-model-72501c-kserve-ffd4675ff-t2k68" Jul 22 03:20:47.822802 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:20:47.822764 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e4b99269-7b0c-48e4-8617-82aae55cbdd0-model-cache\") pod \"sub-test-model-72501c-kserve-ffd4675ff-t2k68\" (UID: \"e4b99269-7b0c-48e4-8617-82aae55cbdd0\") " pod="ai-tenant-e2e-shared-a-72501c/sub-test-model-72501c-kserve-ffd4675ff-t2k68" Jul 22 03:20:47.822947 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:20:47.822922 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e4b99269-7b0c-48e4-8617-82aae55cbdd0-tmp-dir\") pod \"sub-test-model-72501c-kserve-ffd4675ff-t2k68\" (UID: \"e4b99269-7b0c-48e4-8617-82aae55cbdd0\") " pod="ai-tenant-e2e-shared-a-72501c/sub-test-model-72501c-kserve-ffd4675ff-t2k68" Jul 22 03:20:47.825629 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:20:47.825605 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e4b99269-7b0c-48e4-8617-82aae55cbdd0-dshm\") pod \"sub-test-model-72501c-kserve-ffd4675ff-t2k68\" (UID: \"e4b99269-7b0c-48e4-8617-82aae55cbdd0\") " pod="ai-tenant-e2e-shared-a-72501c/sub-test-model-72501c-kserve-ffd4675ff-t2k68" Jul 22 03:20:47.826045 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:20:47.826017 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e4b99269-7b0c-48e4-8617-82aae55cbdd0-tls-certs\") pod \"sub-test-model-72501c-kserve-ffd4675ff-t2k68\" (UID: \"e4b99269-7b0c-48e4-8617-82aae55cbdd0\") " pod="ai-tenant-e2e-shared-a-72501c/sub-test-model-72501c-kserve-ffd4675ff-t2k68" Jul 22 03:20:47.832487 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:20:47.832410 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tk72l\" (UniqueName: \"kubernetes.io/projected/e4b99269-7b0c-48e4-8617-82aae55cbdd0-kube-api-access-tk72l\") pod \"sub-test-model-72501c-kserve-ffd4675ff-t2k68\" (UID: \"e4b99269-7b0c-48e4-8617-82aae55cbdd0\") " pod="ai-tenant-e2e-shared-a-72501c/sub-test-model-72501c-kserve-ffd4675ff-t2k68" Jul 22 03:20:47.893462 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:20:47.893426 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-72501c/sub-test-model-72501c-kserve-ffd4675ff-t2k68" Jul 22 03:20:48.027197 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:20:48.027171 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-72501c/sub-test-model-72501c-kserve-ffd4675ff-t2k68"] Jul 22 03:20:48.029000 ip-10-0-133-31 kubenswrapper[2578]: W0722 03:20:48.028973 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode4b99269_7b0c_48e4_8617_82aae55cbdd0.slice/crio-89aa03530a311d842fdfb9134427492af26130e93c1011f49a525c3f7eea9594 WatchSource:0}: Error finding container 89aa03530a311d842fdfb9134427492af26130e93c1011f49a525c3f7eea9594: Status 404 returned error can't find the container with id 89aa03530a311d842fdfb9134427492af26130e93c1011f49a525c3f7eea9594 Jul 22 03:20:48.294211 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:20:48.294126 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-72501c/sub-test-model-72501c-kserve-ffd4675ff-t2k68" event={"ID":"e4b99269-7b0c-48e4-8617-82aae55cbdd0","Type":"ContainerStarted","Data":"71442fd01d98cdc716bb2ddccfd50abf2ca858ff39b7bdd8adc973e740a1962e"} Jul 22 03:20:48.294211 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:20:48.294178 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-72501c/sub-test-model-72501c-kserve-ffd4675ff-t2k68" event={"ID":"e4b99269-7b0c-48e4-8617-82aae55cbdd0","Type":"ContainerStarted","Data":"89aa03530a311d842fdfb9134427492af26130e93c1011f49a525c3f7eea9594"} Jul 22 03:20:48.320253 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:20:48.320202 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-72501c/sub-test-model-72501c-kserve-ffd4675ff-t2k68" podStartSLOduration=1.320187487 podStartE2EDuration="1.320187487s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 03:20:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 03:20:48.317781319 +0000 UTC m=+2614.636956630" watchObservedRunningTime="2026-07-22 03:20:48.320187487 +0000 UTC m=+2614.639362833" Jul 22 03:20:57.894278 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:20:57.894244 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-72501c/sub-test-model-72501c-kserve-ffd4675ff-t2k68" Jul 22 03:20:57.894680 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:20:57.894293 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-72501c/sub-test-model-72501c-kserve-ffd4675ff-t2k68" Jul 22 03:20:57.906696 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:20:57.906670 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-72501c/sub-test-model-72501c-kserve-ffd4675ff-t2k68" Jul 22 03:20:58.360099 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:20:58.360070 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-72501c/sub-test-model-72501c-kserve-ffd4675ff-t2k68" Jul 22 03:22:14.300081 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:22:14.300057 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-prfql_fd7bf923-8d16-476c-a8d0-a3e6fd645061/ovn-acl-logging/0.log" Jul 22 03:22:14.308416 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:22:14.308388 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-prfql_fd7bf923-8d16-476c-a8d0-a3e6fd645061/ovn-acl-logging/0.log" Jul 22 03:22:34.894853 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:22:34.894822 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-72501c/sub-test-model-72501c-kserve-ffd4675ff-t2k68"] Jul 22 03:22:39.268724 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:22:39.268689 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-72501c/rate-test-model-72501c-kserve-5f8d857c9-pqx6q"] Jul 22 03:22:39.272143 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:22:39.272123 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-72501c/rate-test-model-72501c-kserve-5f8d857c9-pqx6q" Jul 22 03:22:39.275972 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:22:39.275956 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-72501c\"/\"rate-test-model-72501c-kserve-self-signed-certs\"" Jul 22 03:22:39.286017 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:22:39.285994 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-72501c/rate-test-model-72501c-kserve-5f8d857c9-pqx6q"] Jul 22 03:22:39.340189 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:22:39.340166 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6d898f3d-5f7c-4df7-8348-164cee485f36-tls-certs\") pod \"rate-test-model-72501c-kserve-5f8d857c9-pqx6q\" (UID: \"6d898f3d-5f7c-4df7-8348-164cee485f36\") " pod="ai-tenant-e2e-shared-a-72501c/rate-test-model-72501c-kserve-5f8d857c9-pqx6q" Jul 22 03:22:39.340293 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:22:39.340200 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6d898f3d-5f7c-4df7-8348-164cee485f36-dshm\") pod \"rate-test-model-72501c-kserve-5f8d857c9-pqx6q\" (UID: \"6d898f3d-5f7c-4df7-8348-164cee485f36\") " pod="ai-tenant-e2e-shared-a-72501c/rate-test-model-72501c-kserve-5f8d857c9-pqx6q" Jul 22 03:22:39.340293 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:22:39.340245 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6d898f3d-5f7c-4df7-8348-164cee485f36-tmp-dir\") pod \"rate-test-model-72501c-kserve-5f8d857c9-pqx6q\" (UID: \"6d898f3d-5f7c-4df7-8348-164cee485f36\") " pod="ai-tenant-e2e-shared-a-72501c/rate-test-model-72501c-kserve-5f8d857c9-pqx6q" Jul 22 03:22:39.340367 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:22:39.340306 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6d898f3d-5f7c-4df7-8348-164cee485f36-model-cache\") pod \"rate-test-model-72501c-kserve-5f8d857c9-pqx6q\" (UID: \"6d898f3d-5f7c-4df7-8348-164cee485f36\") " pod="ai-tenant-e2e-shared-a-72501c/rate-test-model-72501c-kserve-5f8d857c9-pqx6q" Jul 22 03:22:39.340367 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:22:39.340336 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6d898f3d-5f7c-4df7-8348-164cee485f36-home\") pod \"rate-test-model-72501c-kserve-5f8d857c9-pqx6q\" (UID: \"6d898f3d-5f7c-4df7-8348-164cee485f36\") " pod="ai-tenant-e2e-shared-a-72501c/rate-test-model-72501c-kserve-5f8d857c9-pqx6q" Jul 22 03:22:39.340440 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:22:39.340373 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f5rcg\" (UniqueName: \"kubernetes.io/projected/6d898f3d-5f7c-4df7-8348-164cee485f36-kube-api-access-f5rcg\") pod \"rate-test-model-72501c-kserve-5f8d857c9-pqx6q\" (UID: \"6d898f3d-5f7c-4df7-8348-164cee485f36\") " pod="ai-tenant-e2e-shared-a-72501c/rate-test-model-72501c-kserve-5f8d857c9-pqx6q" Jul 22 03:22:39.441103 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:22:39.441070 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6d898f3d-5f7c-4df7-8348-164cee485f36-tls-certs\") pod \"rate-test-model-72501c-kserve-5f8d857c9-pqx6q\" (UID: \"6d898f3d-5f7c-4df7-8348-164cee485f36\") " pod="ai-tenant-e2e-shared-a-72501c/rate-test-model-72501c-kserve-5f8d857c9-pqx6q" Jul 22 03:22:39.441293 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:22:39.441113 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6d898f3d-5f7c-4df7-8348-164cee485f36-dshm\") pod \"rate-test-model-72501c-kserve-5f8d857c9-pqx6q\" (UID: \"6d898f3d-5f7c-4df7-8348-164cee485f36\") " pod="ai-tenant-e2e-shared-a-72501c/rate-test-model-72501c-kserve-5f8d857c9-pqx6q" Jul 22 03:22:39.441293 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:22:39.441155 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6d898f3d-5f7c-4df7-8348-164cee485f36-tmp-dir\") pod \"rate-test-model-72501c-kserve-5f8d857c9-pqx6q\" (UID: \"6d898f3d-5f7c-4df7-8348-164cee485f36\") " pod="ai-tenant-e2e-shared-a-72501c/rate-test-model-72501c-kserve-5f8d857c9-pqx6q" Jul 22 03:22:39.441293 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:22:39.441187 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6d898f3d-5f7c-4df7-8348-164cee485f36-model-cache\") pod \"rate-test-model-72501c-kserve-5f8d857c9-pqx6q\" (UID: \"6d898f3d-5f7c-4df7-8348-164cee485f36\") " pod="ai-tenant-e2e-shared-a-72501c/rate-test-model-72501c-kserve-5f8d857c9-pqx6q" Jul 22 03:22:39.441293 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:22:39.441218 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6d898f3d-5f7c-4df7-8348-164cee485f36-home\") pod \"rate-test-model-72501c-kserve-5f8d857c9-pqx6q\" (UID: \"6d898f3d-5f7c-4df7-8348-164cee485f36\") " pod="ai-tenant-e2e-shared-a-72501c/rate-test-model-72501c-kserve-5f8d857c9-pqx6q" Jul 22 03:22:39.441293 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:22:39.441268 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f5rcg\" (UniqueName: \"kubernetes.io/projected/6d898f3d-5f7c-4df7-8348-164cee485f36-kube-api-access-f5rcg\") pod \"rate-test-model-72501c-kserve-5f8d857c9-pqx6q\" (UID: \"6d898f3d-5f7c-4df7-8348-164cee485f36\") " pod="ai-tenant-e2e-shared-a-72501c/rate-test-model-72501c-kserve-5f8d857c9-pqx6q" Jul 22 03:22:39.441689 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:22:39.441663 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6d898f3d-5f7c-4df7-8348-164cee485f36-model-cache\") pod \"rate-test-model-72501c-kserve-5f8d857c9-pqx6q\" (UID: \"6d898f3d-5f7c-4df7-8348-164cee485f36\") " pod="ai-tenant-e2e-shared-a-72501c/rate-test-model-72501c-kserve-5f8d857c9-pqx6q" Jul 22 03:22:39.441769 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:22:39.441702 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6d898f3d-5f7c-4df7-8348-164cee485f36-tmp-dir\") pod \"rate-test-model-72501c-kserve-5f8d857c9-pqx6q\" (UID: \"6d898f3d-5f7c-4df7-8348-164cee485f36\") " pod="ai-tenant-e2e-shared-a-72501c/rate-test-model-72501c-kserve-5f8d857c9-pqx6q" Jul 22 03:22:39.441769 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:22:39.441720 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6d898f3d-5f7c-4df7-8348-164cee485f36-home\") pod \"rate-test-model-72501c-kserve-5f8d857c9-pqx6q\" (UID: \"6d898f3d-5f7c-4df7-8348-164cee485f36\") " pod="ai-tenant-e2e-shared-a-72501c/rate-test-model-72501c-kserve-5f8d857c9-pqx6q" Jul 22 03:22:39.443589 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:22:39.443554 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6d898f3d-5f7c-4df7-8348-164cee485f36-dshm\") pod \"rate-test-model-72501c-kserve-5f8d857c9-pqx6q\" (UID: \"6d898f3d-5f7c-4df7-8348-164cee485f36\") " pod="ai-tenant-e2e-shared-a-72501c/rate-test-model-72501c-kserve-5f8d857c9-pqx6q" Jul 22 03:22:39.443735 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:22:39.443719 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6d898f3d-5f7c-4df7-8348-164cee485f36-tls-certs\") pod \"rate-test-model-72501c-kserve-5f8d857c9-pqx6q\" (UID: \"6d898f3d-5f7c-4df7-8348-164cee485f36\") " pod="ai-tenant-e2e-shared-a-72501c/rate-test-model-72501c-kserve-5f8d857c9-pqx6q" Jul 22 03:22:39.453328 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:22:39.453300 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f5rcg\" (UniqueName: \"kubernetes.io/projected/6d898f3d-5f7c-4df7-8348-164cee485f36-kube-api-access-f5rcg\") pod \"rate-test-model-72501c-kserve-5f8d857c9-pqx6q\" (UID: \"6d898f3d-5f7c-4df7-8348-164cee485f36\") " pod="ai-tenant-e2e-shared-a-72501c/rate-test-model-72501c-kserve-5f8d857c9-pqx6q" Jul 22 03:22:39.581449 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:22:39.581421 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-72501c/rate-test-model-72501c-kserve-5f8d857c9-pqx6q" Jul 22 03:22:39.718400 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:22:39.718374 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-72501c/rate-test-model-72501c-kserve-5f8d857c9-pqx6q"] Jul 22 03:22:39.719443 ip-10-0-133-31 kubenswrapper[2578]: W0722 03:22:39.719410 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6d898f3d_5f7c_4df7_8348_164cee485f36.slice/crio-ce0e64be7cf16e4b3b64ca51a073e5cc3aed32dc760042047723f0d3d86d8374 WatchSource:0}: Error finding container ce0e64be7cf16e4b3b64ca51a073e5cc3aed32dc760042047723f0d3d86d8374: Status 404 returned error can't find the container with id ce0e64be7cf16e4b3b64ca51a073e5cc3aed32dc760042047723f0d3d86d8374 Jul 22 03:22:39.887044 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:22:39.886960 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-72501c/rate-test-model-72501c-kserve-5f8d857c9-pqx6q" event={"ID":"6d898f3d-5f7c-4df7-8348-164cee485f36","Type":"ContainerStarted","Data":"d99f26001dc63961a33eee0b6d11a6159dc1088ecb5849deff4c0bf10fd83178"} Jul 22 03:22:39.887044 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:22:39.886998 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-72501c/rate-test-model-72501c-kserve-5f8d857c9-pqx6q" event={"ID":"6d898f3d-5f7c-4df7-8348-164cee485f36","Type":"ContainerStarted","Data":"ce0e64be7cf16e4b3b64ca51a073e5cc3aed32dc760042047723f0d3d86d8374"} Jul 22 03:22:39.914019 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:22:39.913966 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-72501c/rate-test-model-72501c-kserve-5f8d857c9-pqx6q" podStartSLOduration=0.913951461 podStartE2EDuration="913.951461ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 03:22:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 03:22:39.913492527 +0000 UTC m=+2726.232667828" watchObservedRunningTime="2026-07-22 03:22:39.913951461 +0000 UTC m=+2726.233126775" Jul 22 03:22:49.582486 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:22:49.582452 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-72501c/rate-test-model-72501c-kserve-5f8d857c9-pqx6q" Jul 22 03:22:49.582866 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:22:49.582501 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-72501c/rate-test-model-72501c-kserve-5f8d857c9-pqx6q" Jul 22 03:22:49.595458 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:22:49.595433 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-72501c/rate-test-model-72501c-kserve-5f8d857c9-pqx6q" Jul 22 03:22:49.914697 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:22:49.914607 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-72501c/sub-test-model-72501c-kserve-ffd4675ff-t2k68" podUID="e4b99269-7b0c-48e4-8617-82aae55cbdd0" containerName="main" containerID="cri-o://71442fd01d98cdc716bb2ddccfd50abf2ca858ff39b7bdd8adc973e740a1962e" gracePeriod=45 Jul 22 03:22:49.954593 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:22:49.954549 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-72501c/rate-test-model-72501c-kserve-5f8d857c9-pqx6q" Jul 22 03:22:50.163218 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:22:50.163193 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-72501c/sub-test-model-72501c-kserve-ffd4675ff-t2k68" Jul 22 03:22:50.339836 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:22:50.339813 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e4b99269-7b0c-48e4-8617-82aae55cbdd0-tls-certs\") pod \"e4b99269-7b0c-48e4-8617-82aae55cbdd0\" (UID: \"e4b99269-7b0c-48e4-8617-82aae55cbdd0\") " Jul 22 03:22:50.339960 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:22:50.339859 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e4b99269-7b0c-48e4-8617-82aae55cbdd0-tmp-dir\") pod \"e4b99269-7b0c-48e4-8617-82aae55cbdd0\" (UID: \"e4b99269-7b0c-48e4-8617-82aae55cbdd0\") " Jul 22 03:22:50.339960 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:22:50.339885 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tk72l\" (UniqueName: \"kubernetes.io/projected/e4b99269-7b0c-48e4-8617-82aae55cbdd0-kube-api-access-tk72l\") pod \"e4b99269-7b0c-48e4-8617-82aae55cbdd0\" (UID: \"e4b99269-7b0c-48e4-8617-82aae55cbdd0\") " Jul 22 03:22:50.339960 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:22:50.339907 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e4b99269-7b0c-48e4-8617-82aae55cbdd0-dshm\") pod \"e4b99269-7b0c-48e4-8617-82aae55cbdd0\" (UID: \"e4b99269-7b0c-48e4-8617-82aae55cbdd0\") " Jul 22 03:22:50.339960 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:22:50.339928 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e4b99269-7b0c-48e4-8617-82aae55cbdd0-home\") pod \"e4b99269-7b0c-48e4-8617-82aae55cbdd0\" (UID: \"e4b99269-7b0c-48e4-8617-82aae55cbdd0\") " Jul 22 03:22:50.339960 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:22:50.339952 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e4b99269-7b0c-48e4-8617-82aae55cbdd0-model-cache\") pod \"e4b99269-7b0c-48e4-8617-82aae55cbdd0\" (UID: \"e4b99269-7b0c-48e4-8617-82aae55cbdd0\") " Jul 22 03:22:50.340169 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:22:50.340098 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4b99269-7b0c-48e4-8617-82aae55cbdd0-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e4b99269-7b0c-48e4-8617-82aae55cbdd0" (UID: "e4b99269-7b0c-48e4-8617-82aae55cbdd0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 03:22:50.340256 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:22:50.340230 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4b99269-7b0c-48e4-8617-82aae55cbdd0-home" (OuterVolumeSpecName: "home") pod "e4b99269-7b0c-48e4-8617-82aae55cbdd0" (UID: "e4b99269-7b0c-48e4-8617-82aae55cbdd0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 03:22:50.340321 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:22:50.340264 2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e4b99269-7b0c-48e4-8617-82aae55cbdd0-tmp-dir\") on node \"ip-10-0-133-31.ec2.internal\" DevicePath \"\"" Jul 22 03:22:50.340364 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:22:50.340307 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4b99269-7b0c-48e4-8617-82aae55cbdd0-model-cache" (OuterVolumeSpecName: "model-cache") pod "e4b99269-7b0c-48e4-8617-82aae55cbdd0" (UID: "e4b99269-7b0c-48e4-8617-82aae55cbdd0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 03:22:50.342179 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:22:50.342152 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e4b99269-7b0c-48e4-8617-82aae55cbdd0-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e4b99269-7b0c-48e4-8617-82aae55cbdd0" (UID: "e4b99269-7b0c-48e4-8617-82aae55cbdd0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 03:22:50.342179 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:22:50.342163 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4b99269-7b0c-48e4-8617-82aae55cbdd0-dshm" (OuterVolumeSpecName: "dshm") pod "e4b99269-7b0c-48e4-8617-82aae55cbdd0" (UID: "e4b99269-7b0c-48e4-8617-82aae55cbdd0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 03:22:50.342314 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:22:50.342180 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4b99269-7b0c-48e4-8617-82aae55cbdd0-kube-api-access-tk72l" (OuterVolumeSpecName: "kube-api-access-tk72l") pod "e4b99269-7b0c-48e4-8617-82aae55cbdd0" (UID: "e4b99269-7b0c-48e4-8617-82aae55cbdd0"). InnerVolumeSpecName "kube-api-access-tk72l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 03:22:50.441413 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:22:50.441390 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tk72l\" (UniqueName: \"kubernetes.io/projected/e4b99269-7b0c-48e4-8617-82aae55cbdd0-kube-api-access-tk72l\") on node \"ip-10-0-133-31.ec2.internal\" DevicePath \"\"" Jul 22 03:22:50.441413 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:22:50.441411 2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e4b99269-7b0c-48e4-8617-82aae55cbdd0-dshm\") on node \"ip-10-0-133-31.ec2.internal\" DevicePath \"\"" Jul 22 03:22:50.441570 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:22:50.441420 2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e4b99269-7b0c-48e4-8617-82aae55cbdd0-home\") on node \"ip-10-0-133-31.ec2.internal\" DevicePath \"\"" Jul 22 03:22:50.441570 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:22:50.441427 2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e4b99269-7b0c-48e4-8617-82aae55cbdd0-model-cache\") on node \"ip-10-0-133-31.ec2.internal\" DevicePath \"\"" Jul 22 03:22:50.441570 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:22:50.441435 2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e4b99269-7b0c-48e4-8617-82aae55cbdd0-tls-certs\") on node \"ip-10-0-133-31.ec2.internal\" DevicePath \"\"" Jul 22 03:22:50.951270 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:22:50.951236 2578 generic.go:356] "Generic (PLEG): container finished" podID="e4b99269-7b0c-48e4-8617-82aae55cbdd0" containerID="71442fd01d98cdc716bb2ddccfd50abf2ca858ff39b7bdd8adc973e740a1962e" exitCode=0 Jul 22 03:22:50.951666 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:22:50.951280 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-72501c/sub-test-model-72501c-kserve-ffd4675ff-t2k68" Jul 22 03:22:50.951666 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:22:50.951323 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-72501c/sub-test-model-72501c-kserve-ffd4675ff-t2k68" event={"ID":"e4b99269-7b0c-48e4-8617-82aae55cbdd0","Type":"ContainerDied","Data":"71442fd01d98cdc716bb2ddccfd50abf2ca858ff39b7bdd8adc973e740a1962e"} Jul 22 03:22:50.951666 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:22:50.951360 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-72501c/sub-test-model-72501c-kserve-ffd4675ff-t2k68" event={"ID":"e4b99269-7b0c-48e4-8617-82aae55cbdd0","Type":"ContainerDied","Data":"89aa03530a311d842fdfb9134427492af26130e93c1011f49a525c3f7eea9594"} Jul 22 03:22:50.951666 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:22:50.951382 2578 scope.go:117] "RemoveContainer" containerID="71442fd01d98cdc716bb2ddccfd50abf2ca858ff39b7bdd8adc973e740a1962e" Jul 22 03:22:50.961876 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:22:50.961854 2578 scope.go:117] "RemoveContainer" containerID="71442fd01d98cdc716bb2ddccfd50abf2ca858ff39b7bdd8adc973e740a1962e" Jul 22 03:22:50.962135 ip-10-0-133-31 kubenswrapper[2578]: E0722 03:22:50.962118 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"71442fd01d98cdc716bb2ddccfd50abf2ca858ff39b7bdd8adc973e740a1962e\": container with ID starting with 71442fd01d98cdc716bb2ddccfd50abf2ca858ff39b7bdd8adc973e740a1962e not found: ID does not exist" containerID="71442fd01d98cdc716bb2ddccfd50abf2ca858ff39b7bdd8adc973e740a1962e" Jul 22 03:22:50.962197 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:22:50.962145 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71442fd01d98cdc716bb2ddccfd50abf2ca858ff39b7bdd8adc973e740a1962e"} err="failed to get container status \"71442fd01d98cdc716bb2ddccfd50abf2ca858ff39b7bdd8adc973e740a1962e\": rpc error: code = NotFound desc = could not find container \"71442fd01d98cdc716bb2ddccfd50abf2ca858ff39b7bdd8adc973e740a1962e\": container with ID starting with 71442fd01d98cdc716bb2ddccfd50abf2ca858ff39b7bdd8adc973e740a1962e not found: ID does not exist" Jul 22 03:22:50.979272 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:22:50.979253 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-72501c/sub-test-model-72501c-kserve-ffd4675ff-t2k68"] Jul 22 03:22:51.005125 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:22:51.005103 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-72501c/sub-test-model-72501c-kserve-ffd4675ff-t2k68"] Jul 22 03:22:52.142390 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:22:52.142360 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e4b99269-7b0c-48e4-8617-82aae55cbdd0" path="/var/lib/kubelet/pods/e4b99269-7b0c-48e4-8617-82aae55cbdd0/volumes" Jul 22 03:23:47.434615 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:23:47.434522 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-72501c/rate-test-model-72501c-kserve-5f8d857c9-pqx6q"] Jul 22 03:24:02.454000 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:24:02.453955 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-72501c/rate-test-model-72501c-kserve-5f8d857c9-pqx6q" podUID="6d898f3d-5f7c-4df7-8348-164cee485f36" containerName="main" containerID="cri-o://d99f26001dc63961a33eee0b6d11a6159dc1088ecb5849deff4c0bf10fd83178" gracePeriod=45 Jul 22 03:24:02.697381 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:24:02.697357 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-72501c/rate-test-model-72501c-kserve-5f8d857c9-pqx6q" Jul 22 03:24:02.751083 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:24:02.751002 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6d898f3d-5f7c-4df7-8348-164cee485f36-dshm\") pod \"6d898f3d-5f7c-4df7-8348-164cee485f36\" (UID: \"6d898f3d-5f7c-4df7-8348-164cee485f36\") " Jul 22 03:24:02.751083 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:24:02.751068 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6d898f3d-5f7c-4df7-8348-164cee485f36-model-cache\") pod \"6d898f3d-5f7c-4df7-8348-164cee485f36\" (UID: \"6d898f3d-5f7c-4df7-8348-164cee485f36\") " Jul 22 03:24:02.751318 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:24:02.751126 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6d898f3d-5f7c-4df7-8348-164cee485f36-home\") pod \"6d898f3d-5f7c-4df7-8348-164cee485f36\" (UID: \"6d898f3d-5f7c-4df7-8348-164cee485f36\") " Jul 22 03:24:02.751318 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:24:02.751146 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f5rcg\" (UniqueName: \"kubernetes.io/projected/6d898f3d-5f7c-4df7-8348-164cee485f36-kube-api-access-f5rcg\") pod \"6d898f3d-5f7c-4df7-8348-164cee485f36\" (UID: \"6d898f3d-5f7c-4df7-8348-164cee485f36\") " Jul 22 03:24:02.751318 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:24:02.751177 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6d898f3d-5f7c-4df7-8348-164cee485f36-tls-certs\") pod \"6d898f3d-5f7c-4df7-8348-164cee485f36\" (UID: \"6d898f3d-5f7c-4df7-8348-164cee485f36\") " Jul 22 03:24:02.751318 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:24:02.751214 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6d898f3d-5f7c-4df7-8348-164cee485f36-tmp-dir\") pod \"6d898f3d-5f7c-4df7-8348-164cee485f36\" (UID: \"6d898f3d-5f7c-4df7-8348-164cee485f36\") " Jul 22 03:24:02.751507 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:24:02.751329 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d898f3d-5f7c-4df7-8348-164cee485f36-home" (OuterVolumeSpecName: "home") pod "6d898f3d-5f7c-4df7-8348-164cee485f36" (UID: "6d898f3d-5f7c-4df7-8348-164cee485f36"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 03:24:02.751507 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:24:02.751351 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d898f3d-5f7c-4df7-8348-164cee485f36-model-cache" (OuterVolumeSpecName: "model-cache") pod "6d898f3d-5f7c-4df7-8348-164cee485f36" (UID: "6d898f3d-5f7c-4df7-8348-164cee485f36"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 03:24:02.751650 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:24:02.751525 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d898f3d-5f7c-4df7-8348-164cee485f36-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6d898f3d-5f7c-4df7-8348-164cee485f36" (UID: "6d898f3d-5f7c-4df7-8348-164cee485f36"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 03:24:02.751650 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:24:02.751602 2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6d898f3d-5f7c-4df7-8348-164cee485f36-model-cache\") on node \"ip-10-0-133-31.ec2.internal\" DevicePath \"\"" Jul 22 03:24:02.751650 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:24:02.751622 2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6d898f3d-5f7c-4df7-8348-164cee485f36-home\") on node \"ip-10-0-133-31.ec2.internal\" DevicePath \"\"" Jul 22 03:24:02.753289 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:24:02.753258 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d898f3d-5f7c-4df7-8348-164cee485f36-dshm" (OuterVolumeSpecName: "dshm") pod "6d898f3d-5f7c-4df7-8348-164cee485f36" (UID: "6d898f3d-5f7c-4df7-8348-164cee485f36"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 03:24:02.753289 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:24:02.753263 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6d898f3d-5f7c-4df7-8348-164cee485f36-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6d898f3d-5f7c-4df7-8348-164cee485f36" (UID: "6d898f3d-5f7c-4df7-8348-164cee485f36"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 03:24:02.753419 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:24:02.753338 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d898f3d-5f7c-4df7-8348-164cee485f36-kube-api-access-f5rcg" (OuterVolumeSpecName: "kube-api-access-f5rcg") pod "6d898f3d-5f7c-4df7-8348-164cee485f36" (UID: "6d898f3d-5f7c-4df7-8348-164cee485f36"). InnerVolumeSpecName "kube-api-access-f5rcg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 03:24:02.852980 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:24:02.852944 2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6d898f3d-5f7c-4df7-8348-164cee485f36-dshm\") on node \"ip-10-0-133-31.ec2.internal\" DevicePath \"\"" Jul 22 03:24:02.852980 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:24:02.852973 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f5rcg\" (UniqueName: \"kubernetes.io/projected/6d898f3d-5f7c-4df7-8348-164cee485f36-kube-api-access-f5rcg\") on node \"ip-10-0-133-31.ec2.internal\" DevicePath \"\"" Jul 22 03:24:02.852980 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:24:02.852983 2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6d898f3d-5f7c-4df7-8348-164cee485f36-tls-certs\") on node \"ip-10-0-133-31.ec2.internal\" DevicePath \"\"" Jul 22 03:24:02.852980 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:24:02.852992 2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6d898f3d-5f7c-4df7-8348-164cee485f36-tmp-dir\") on node \"ip-10-0-133-31.ec2.internal\" DevicePath \"\"" Jul 22 03:24:03.338461 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:24:03.338422 2578 generic.go:356] "Generic (PLEG): container finished" podID="6d898f3d-5f7c-4df7-8348-164cee485f36" containerID="d99f26001dc63961a33eee0b6d11a6159dc1088ecb5849deff4c0bf10fd83178" exitCode=0 Jul 22 03:24:03.338691 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:24:03.338493 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-72501c/rate-test-model-72501c-kserve-5f8d857c9-pqx6q" Jul 22 03:24:03.338691 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:24:03.338513 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-72501c/rate-test-model-72501c-kserve-5f8d857c9-pqx6q" event={"ID":"6d898f3d-5f7c-4df7-8348-164cee485f36","Type":"ContainerDied","Data":"d99f26001dc63961a33eee0b6d11a6159dc1088ecb5849deff4c0bf10fd83178"} Jul 22 03:24:03.338691 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:24:03.338581 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-72501c/rate-test-model-72501c-kserve-5f8d857c9-pqx6q" event={"ID":"6d898f3d-5f7c-4df7-8348-164cee485f36","Type":"ContainerDied","Data":"ce0e64be7cf16e4b3b64ca51a073e5cc3aed32dc760042047723f0d3d86d8374"} Jul 22 03:24:03.338691 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:24:03.338603 2578 scope.go:117] "RemoveContainer" containerID="d99f26001dc63961a33eee0b6d11a6159dc1088ecb5849deff4c0bf10fd83178" Jul 22 03:24:03.347904 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:24:03.347884 2578 scope.go:117] "RemoveContainer" containerID="d99f26001dc63961a33eee0b6d11a6159dc1088ecb5849deff4c0bf10fd83178" Jul 22 03:24:03.348186 ip-10-0-133-31 kubenswrapper[2578]: E0722 03:24:03.348167 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d99f26001dc63961a33eee0b6d11a6159dc1088ecb5849deff4c0bf10fd83178\": container with ID starting with d99f26001dc63961a33eee0b6d11a6159dc1088ecb5849deff4c0bf10fd83178 not found: ID does not exist" containerID="d99f26001dc63961a33eee0b6d11a6159dc1088ecb5849deff4c0bf10fd83178" Jul 22 03:24:03.348239 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:24:03.348194 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d99f26001dc63961a33eee0b6d11a6159dc1088ecb5849deff4c0bf10fd83178"} err="failed to get container status \"d99f26001dc63961a33eee0b6d11a6159dc1088ecb5849deff4c0bf10fd83178\": rpc error: code = NotFound desc = could not find container \"d99f26001dc63961a33eee0b6d11a6159dc1088ecb5849deff4c0bf10fd83178\": container with ID starting with d99f26001dc63961a33eee0b6d11a6159dc1088ecb5849deff4c0bf10fd83178 not found: ID does not exist" Jul 22 03:24:03.361388 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:24:03.361365 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-72501c/rate-test-model-72501c-kserve-5f8d857c9-pqx6q"] Jul 22 03:24:03.370040 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:24:03.370019 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-72501c/rate-test-model-72501c-kserve-5f8d857c9-pqx6q"] Jul 22 03:24:04.143072 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:24:04.143042 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6d898f3d-5f7c-4df7-8348-164cee485f36" path="/var/lib/kubelet/pods/6d898f3d-5f7c-4df7-8348-164cee485f36/volumes" Jul 22 03:24:29.425686 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:24:29.425657 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-72501c-openshift-default-7f7d685fff-dvkn2"] Jul 22 03:24:29.426153 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:24:29.425860 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-72501c-openshift-default-7f7d685fff-dvkn2" podUID="c52d4866-0bd6-4fdf-ab45-166f1778ba19" containerName="istio-proxy" containerID="cri-o://0ef2bbfe527acbe8a98853f85b223abb50d78ce954557f20aaee8dc8d2fdcdcb" gracePeriod=30 Jul 22 03:24:33.019722 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:24:33.019681 2578 patch_prober.go:28] interesting pod/e2e-shared-a-72501c-openshift-default-7f7d685fff-dvkn2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 22 03:24:33.020105 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:24:33.019750 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-72501c-openshift-default-7f7d685fff-dvkn2" podUID="c52d4866-0bd6-4fdf-ab45-166f1778ba19" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 22 03:24:34.682132 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:24:34.682108 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-72501c-openshift-default-7f7d685fff-dvkn2" Jul 22 03:24:34.838947 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:24:34.838922 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c52d4866-0bd6-4fdf-ab45-166f1778ba19-istio-token\") pod \"c52d4866-0bd6-4fdf-ab45-166f1778ba19\" (UID: \"c52d4866-0bd6-4fdf-ab45-166f1778ba19\") " Jul 22 03:24:34.838947 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:24:34.838950 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c52d4866-0bd6-4fdf-ab45-166f1778ba19-workload-socket\") pod \"c52d4866-0bd6-4fdf-ab45-166f1778ba19\" (UID: \"c52d4866-0bd6-4fdf-ab45-166f1778ba19\") " Jul 22 03:24:34.839217 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:24:34.838970 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ssmm2\" (UniqueName: \"kubernetes.io/projected/c52d4866-0bd6-4fdf-ab45-166f1778ba19-kube-api-access-ssmm2\") pod \"c52d4866-0bd6-4fdf-ab45-166f1778ba19\" (UID: \"c52d4866-0bd6-4fdf-ab45-166f1778ba19\") " Jul 22 03:24:34.839217 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:24:34.838989 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c52d4866-0bd6-4fdf-ab45-166f1778ba19-istiod-ca-cert\") pod \"c52d4866-0bd6-4fdf-ab45-166f1778ba19\" (UID: \"c52d4866-0bd6-4fdf-ab45-166f1778ba19\") " Jul 22 03:24:34.839217 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:24:34.839161 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c52d4866-0bd6-4fdf-ab45-166f1778ba19-istio-data\") pod \"c52d4866-0bd6-4fdf-ab45-166f1778ba19\" (UID: \"c52d4866-0bd6-4fdf-ab45-166f1778ba19\") " Jul 22 03:24:34.839373 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:24:34.839252 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c52d4866-0bd6-4fdf-ab45-166f1778ba19-workload-certs\") pod \"c52d4866-0bd6-4fdf-ab45-166f1778ba19\" (UID: \"c52d4866-0bd6-4fdf-ab45-166f1778ba19\") " Jul 22 03:24:34.839373 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:24:34.839279 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c52d4866-0bd6-4fdf-ab45-166f1778ba19-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "c52d4866-0bd6-4fdf-ab45-166f1778ba19" (UID: "c52d4866-0bd6-4fdf-ab45-166f1778ba19"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 03:24:34.839373 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:24:34.839287 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c52d4866-0bd6-4fdf-ab45-166f1778ba19-istio-podinfo\") pod \"c52d4866-0bd6-4fdf-ab45-166f1778ba19\" (UID: \"c52d4866-0bd6-4fdf-ab45-166f1778ba19\") " Jul 22 03:24:34.839533 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:24:34.839377 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c52d4866-0bd6-4fdf-ab45-166f1778ba19-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "c52d4866-0bd6-4fdf-ab45-166f1778ba19" (UID: "c52d4866-0bd6-4fdf-ab45-166f1778ba19"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 03:24:34.839533 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:24:34.839389 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c52d4866-0bd6-4fdf-ab45-166f1778ba19-istio-envoy\") pod \"c52d4866-0bd6-4fdf-ab45-166f1778ba19\" (UID: \"c52d4866-0bd6-4fdf-ab45-166f1778ba19\") " Jul 22 03:24:34.839533 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:24:34.839445 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c52d4866-0bd6-4fdf-ab45-166f1778ba19-credential-socket\") pod \"c52d4866-0bd6-4fdf-ab45-166f1778ba19\" (UID: \"c52d4866-0bd6-4fdf-ab45-166f1778ba19\") " Jul 22 03:24:34.839533 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:24:34.839495 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c52d4866-0bd6-4fdf-ab45-166f1778ba19-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "c52d4866-0bd6-4fdf-ab45-166f1778ba19" (UID: "c52d4866-0bd6-4fdf-ab45-166f1778ba19"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 03:24:34.840166 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:24:34.839806 2578 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c52d4866-0bd6-4fdf-ab45-166f1778ba19-workload-socket\") on node \"ip-10-0-133-31.ec2.internal\" DevicePath \"\"" Jul 22 03:24:34.840166 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:24:34.839828 2578 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c52d4866-0bd6-4fdf-ab45-166f1778ba19-istiod-ca-cert\") on node \"ip-10-0-133-31.ec2.internal\" DevicePath \"\"" Jul 22 03:24:34.840166 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:24:34.839843 2578 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c52d4866-0bd6-4fdf-ab45-166f1778ba19-workload-certs\") on node \"ip-10-0-133-31.ec2.internal\" DevicePath \"\"" Jul 22 03:24:34.840166 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:24:34.839912 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c52d4866-0bd6-4fdf-ab45-166f1778ba19-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "c52d4866-0bd6-4fdf-ab45-166f1778ba19" (UID: "c52d4866-0bd6-4fdf-ab45-166f1778ba19"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 03:24:34.840461 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:24:34.840195 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c52d4866-0bd6-4fdf-ab45-166f1778ba19-istio-data" (OuterVolumeSpecName: "istio-data") pod "c52d4866-0bd6-4fdf-ab45-166f1778ba19" (UID: "c52d4866-0bd6-4fdf-ab45-166f1778ba19"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 03:24:34.841627 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:24:34.841592 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c52d4866-0bd6-4fdf-ab45-166f1778ba19-istio-token" (OuterVolumeSpecName: "istio-token") pod "c52d4866-0bd6-4fdf-ab45-166f1778ba19" (UID: "c52d4866-0bd6-4fdf-ab45-166f1778ba19"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 03:24:34.841627 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:24:34.841594 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c52d4866-0bd6-4fdf-ab45-166f1778ba19-kube-api-access-ssmm2" (OuterVolumeSpecName: "kube-api-access-ssmm2") pod "c52d4866-0bd6-4fdf-ab45-166f1778ba19" (UID: "c52d4866-0bd6-4fdf-ab45-166f1778ba19"). InnerVolumeSpecName "kube-api-access-ssmm2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 03:24:34.841856 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:24:34.841834 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c52d4866-0bd6-4fdf-ab45-166f1778ba19-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "c52d4866-0bd6-4fdf-ab45-166f1778ba19" (UID: "c52d4866-0bd6-4fdf-ab45-166f1778ba19"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 22 03:24:34.841927 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:24:34.841865 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c52d4866-0bd6-4fdf-ab45-166f1778ba19-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "c52d4866-0bd6-4fdf-ab45-166f1778ba19" (UID: "c52d4866-0bd6-4fdf-ab45-166f1778ba19"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 03:24:34.940906 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:24:34.940868 2578 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c52d4866-0bd6-4fdf-ab45-166f1778ba19-istio-token\") on node \"ip-10-0-133-31.ec2.internal\" DevicePath \"\"" Jul 22 03:24:34.940906 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:24:34.940899 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ssmm2\" (UniqueName: \"kubernetes.io/projected/c52d4866-0bd6-4fdf-ab45-166f1778ba19-kube-api-access-ssmm2\") on node \"ip-10-0-133-31.ec2.internal\" DevicePath \"\"" Jul 22 03:24:34.940906 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:24:34.940907 2578 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c52d4866-0bd6-4fdf-ab45-166f1778ba19-istio-data\") on node \"ip-10-0-133-31.ec2.internal\" DevicePath \"\"" Jul 22 03:24:34.940906 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:24:34.940915 2578 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c52d4866-0bd6-4fdf-ab45-166f1778ba19-istio-podinfo\") on node \"ip-10-0-133-31.ec2.internal\" DevicePath \"\"" Jul 22 03:24:34.941238 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:24:34.940922 2578 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c52d4866-0bd6-4fdf-ab45-166f1778ba19-istio-envoy\") on node \"ip-10-0-133-31.ec2.internal\" DevicePath \"\"" Jul 22 03:24:34.941238 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:24:34.940930 2578 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c52d4866-0bd6-4fdf-ab45-166f1778ba19-credential-socket\") on node \"ip-10-0-133-31.ec2.internal\" DevicePath \"\"" Jul 22 03:24:35.514052 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:24:35.514014 2578 generic.go:356] "Generic (PLEG): container finished" podID="c52d4866-0bd6-4fdf-ab45-166f1778ba19" containerID="0ef2bbfe527acbe8a98853f85b223abb50d78ce954557f20aaee8dc8d2fdcdcb" exitCode=0 Jul 22 03:24:35.514229 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:24:35.514064 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-72501c-openshift-default-7f7d685fff-dvkn2" Jul 22 03:24:35.514229 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:24:35.514099 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-72501c-openshift-default-7f7d685fff-dvkn2" event={"ID":"c52d4866-0bd6-4fdf-ab45-166f1778ba19","Type":"ContainerDied","Data":"0ef2bbfe527acbe8a98853f85b223abb50d78ce954557f20aaee8dc8d2fdcdcb"} Jul 22 03:24:35.514229 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:24:35.514156 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-72501c-openshift-default-7f7d685fff-dvkn2" event={"ID":"c52d4866-0bd6-4fdf-ab45-166f1778ba19","Type":"ContainerDied","Data":"b6503724ca321b83be53e16f588c022a534494af83c832a76e070409b42d86ff"} Jul 22 03:24:35.514229 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:24:35.514180 2578 scope.go:117] "RemoveContainer" containerID="0ef2bbfe527acbe8a98853f85b223abb50d78ce954557f20aaee8dc8d2fdcdcb" Jul 22 03:24:35.523492 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:24:35.523472 2578 scope.go:117] "RemoveContainer" containerID="ff0dfc1c872ff0dc11924df062fed85c0796e5823060a10d0ee0c86ce046e1f7" Jul 22 03:24:35.531927 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:24:35.531907 2578 scope.go:117] "RemoveContainer" containerID="0ef2bbfe527acbe8a98853f85b223abb50d78ce954557f20aaee8dc8d2fdcdcb" Jul 22 03:24:35.532198 ip-10-0-133-31 kubenswrapper[2578]: E0722 03:24:35.532180 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ef2bbfe527acbe8a98853f85b223abb50d78ce954557f20aaee8dc8d2fdcdcb\": container with ID starting with 0ef2bbfe527acbe8a98853f85b223abb50d78ce954557f20aaee8dc8d2fdcdcb not found: ID does not exist" containerID="0ef2bbfe527acbe8a98853f85b223abb50d78ce954557f20aaee8dc8d2fdcdcb" Jul 22 03:24:35.532241 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:24:35.532209 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ef2bbfe527acbe8a98853f85b223abb50d78ce954557f20aaee8dc8d2fdcdcb"} err="failed to get container status \"0ef2bbfe527acbe8a98853f85b223abb50d78ce954557f20aaee8dc8d2fdcdcb\": rpc error: code = NotFound desc = could not find container \"0ef2bbfe527acbe8a98853f85b223abb50d78ce954557f20aaee8dc8d2fdcdcb\": container with ID starting with 0ef2bbfe527acbe8a98853f85b223abb50d78ce954557f20aaee8dc8d2fdcdcb not found: ID does not exist" Jul 22 03:24:35.532241 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:24:35.532226 2578 scope.go:117] "RemoveContainer" containerID="ff0dfc1c872ff0dc11924df062fed85c0796e5823060a10d0ee0c86ce046e1f7" Jul 22 03:24:35.532477 ip-10-0-133-31 kubenswrapper[2578]: E0722 03:24:35.532460 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff0dfc1c872ff0dc11924df062fed85c0796e5823060a10d0ee0c86ce046e1f7\": container with ID starting with ff0dfc1c872ff0dc11924df062fed85c0796e5823060a10d0ee0c86ce046e1f7 not found: ID does not exist" containerID="ff0dfc1c872ff0dc11924df062fed85c0796e5823060a10d0ee0c86ce046e1f7" Jul 22 03:24:35.532526 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:24:35.532482 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff0dfc1c872ff0dc11924df062fed85c0796e5823060a10d0ee0c86ce046e1f7"} err="failed to get container status \"ff0dfc1c872ff0dc11924df062fed85c0796e5823060a10d0ee0c86ce046e1f7\": rpc error: code = NotFound desc = could not find container \"ff0dfc1c872ff0dc11924df062fed85c0796e5823060a10d0ee0c86ce046e1f7\": container with ID starting with ff0dfc1c872ff0dc11924df062fed85c0796e5823060a10d0ee0c86ce046e1f7 not found: ID does not exist" Jul 22 03:24:35.538641 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:24:35.538618 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-72501c-openshift-default-7f7d685fff-dvkn2"] Jul 22 03:24:35.547794 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:24:35.547772 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-72501c-openshift-default-7f7d685fff-dvkn2"] Jul 22 03:24:36.142147 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:24:36.142106 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c52d4866-0bd6-4fdf-ab45-166f1778ba19" path="/var/lib/kubelet/pods/c52d4866-0bd6-4fdf-ab45-166f1778ba19/volumes" Jul 22 03:24:52.041337 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:24:52.041301 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-e9ef33-openshift-default-76c5696ddc-5g9gv"] Jul 22 03:24:52.041809 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:24:52.041578 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-e9ef33-openshift-default-76c5696ddc-5g9gv" podUID="b8649024-1788-46ee-959c-d521b00975f4" containerName="istio-proxy" containerID="cri-o://8b5fbc9196b074164d88522c300a6994df416fe1cc27f511b15aae80115862c6" gracePeriod=30 Jul 22 03:24:55.787799 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:24:55.787753 2578 patch_prober.go:28] interesting pod/e2e-shared-b-e9ef33-openshift-default-76c5696ddc-5g9gv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 22 03:24:55.788163 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:24:55.787836 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-e9ef33-openshift-default-76c5696ddc-5g9gv" podUID="b8649024-1788-46ee-959c-d521b00975f4" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 22 03:24:57.295350 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:24:57.295324 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-e9ef33-openshift-default-76c5696ddc-5g9gv" Jul 22 03:24:57.347684 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:24:57.347658 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8649024-1788-46ee-959c-d521b00975f4-istio-envoy\") pod \"b8649024-1788-46ee-959c-d521b00975f4\" (UID: \"b8649024-1788-46ee-959c-d521b00975f4\") " Jul 22 03:24:57.347866 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:24:57.347693 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8649024-1788-46ee-959c-d521b00975f4-workload-socket\") pod \"b8649024-1788-46ee-959c-d521b00975f4\" (UID: \"b8649024-1788-46ee-959c-d521b00975f4\") " Jul 22 03:24:57.347866 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:24:57.347714 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8649024-1788-46ee-959c-d521b00975f4-workload-certs\") pod \"b8649024-1788-46ee-959c-d521b00975f4\" (UID: \"b8649024-1788-46ee-959c-d521b00975f4\") " Jul 22 03:24:57.347866 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:24:57.347755 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8649024-1788-46ee-959c-d521b00975f4-istio-data\") pod \"b8649024-1788-46ee-959c-d521b00975f4\" (UID: \"b8649024-1788-46ee-959c-d521b00975f4\") " Jul 22 03:24:57.347866 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:24:57.347790 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8649024-1788-46ee-959c-d521b00975f4-istiod-ca-cert\") pod \"b8649024-1788-46ee-959c-d521b00975f4\" (UID: \"b8649024-1788-46ee-959c-d521b00975f4\") " Jul 22 03:24:57.348084 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:24:57.347885 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-chlpc\" (UniqueName: \"kubernetes.io/projected/b8649024-1788-46ee-959c-d521b00975f4-kube-api-access-chlpc\") pod \"b8649024-1788-46ee-959c-d521b00975f4\" (UID: \"b8649024-1788-46ee-959c-d521b00975f4\") " Jul 22 03:24:57.348084 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:24:57.347915 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8649024-1788-46ee-959c-d521b00975f4-istio-podinfo\") pod \"b8649024-1788-46ee-959c-d521b00975f4\" (UID: \"b8649024-1788-46ee-959c-d521b00975f4\") " Jul 22 03:24:57.348084 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:24:57.347938 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b8649024-1788-46ee-959c-d521b00975f4-istio-token\") pod \"b8649024-1788-46ee-959c-d521b00975f4\" (UID: \"b8649024-1788-46ee-959c-d521b00975f4\") " Jul 22 03:24:57.348084 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:24:57.347963 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8649024-1788-46ee-959c-d521b00975f4-credential-socket\") pod \"b8649024-1788-46ee-959c-d521b00975f4\" (UID: \"b8649024-1788-46ee-959c-d521b00975f4\") " Jul 22 03:24:57.348277 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:24:57.348071 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8649024-1788-46ee-959c-d521b00975f4-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "b8649024-1788-46ee-959c-d521b00975f4" (UID: "b8649024-1788-46ee-959c-d521b00975f4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 03:24:57.348277 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:24:57.348272 2578 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8649024-1788-46ee-959c-d521b00975f4-workload-certs\") on node \"ip-10-0-133-31.ec2.internal\" DevicePath \"\"" Jul 22 03:24:57.348406 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:24:57.348317 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8649024-1788-46ee-959c-d521b00975f4-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "b8649024-1788-46ee-959c-d521b00975f4" (UID: "b8649024-1788-46ee-959c-d521b00975f4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 03:24:57.348406 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:24:57.348323 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8649024-1788-46ee-959c-d521b00975f4-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "b8649024-1788-46ee-959c-d521b00975f4" (UID: "b8649024-1788-46ee-959c-d521b00975f4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 03:24:57.348708 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:24:57.348686 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8649024-1788-46ee-959c-d521b00975f4-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "b8649024-1788-46ee-959c-d521b00975f4" (UID: "b8649024-1788-46ee-959c-d521b00975f4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 03:24:57.348936 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:24:57.348833 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8649024-1788-46ee-959c-d521b00975f4-istio-data" (OuterVolumeSpecName: "istio-data") pod "b8649024-1788-46ee-959c-d521b00975f4" (UID: "b8649024-1788-46ee-959c-d521b00975f4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 03:24:57.350395 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:24:57.350370 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8649024-1788-46ee-959c-d521b00975f4-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "b8649024-1788-46ee-959c-d521b00975f4" (UID: "b8649024-1788-46ee-959c-d521b00975f4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 03:24:57.350493 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:24:57.350371 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b8649024-1788-46ee-959c-d521b00975f4-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "b8649024-1788-46ee-959c-d521b00975f4" (UID: "b8649024-1788-46ee-959c-d521b00975f4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 22 03:24:57.350959 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:24:57.350927 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8649024-1788-46ee-959c-d521b00975f4-istio-token" (OuterVolumeSpecName: "istio-token") pod "b8649024-1788-46ee-959c-d521b00975f4" (UID: "b8649024-1788-46ee-959c-d521b00975f4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 03:24:57.350959 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:24:57.350933 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8649024-1788-46ee-959c-d521b00975f4-kube-api-access-chlpc" (OuterVolumeSpecName: "kube-api-access-chlpc") pod "b8649024-1788-46ee-959c-d521b00975f4" (UID: "b8649024-1788-46ee-959c-d521b00975f4"). InnerVolumeSpecName "kube-api-access-chlpc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 03:24:57.449138 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:24:57.449096 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-chlpc\" (UniqueName: \"kubernetes.io/projected/b8649024-1788-46ee-959c-d521b00975f4-kube-api-access-chlpc\") on node \"ip-10-0-133-31.ec2.internal\" DevicePath \"\"" Jul 22 03:24:57.449138 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:24:57.449131 2578 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8649024-1788-46ee-959c-d521b00975f4-istio-podinfo\") on node \"ip-10-0-133-31.ec2.internal\" DevicePath \"\"" Jul 22 03:24:57.449138 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:24:57.449144 2578 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b8649024-1788-46ee-959c-d521b00975f4-istio-token\") on node \"ip-10-0-133-31.ec2.internal\" DevicePath \"\"" Jul 22 03:24:57.449378 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:24:57.449154 2578 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8649024-1788-46ee-959c-d521b00975f4-credential-socket\") on node \"ip-10-0-133-31.ec2.internal\" DevicePath \"\"" Jul 22 03:24:57.449378 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:24:57.449166 2578 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8649024-1788-46ee-959c-d521b00975f4-istio-envoy\") on node \"ip-10-0-133-31.ec2.internal\" DevicePath \"\"" Jul 22 03:24:57.449378 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:24:57.449179 2578 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8649024-1788-46ee-959c-d521b00975f4-workload-socket\") on node \"ip-10-0-133-31.ec2.internal\" DevicePath \"\"" Jul 22 03:24:57.449378 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:24:57.449189 2578 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8649024-1788-46ee-959c-d521b00975f4-istio-data\") on node \"ip-10-0-133-31.ec2.internal\" DevicePath \"\"" Jul 22 03:24:57.449378 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:24:57.449199 2578 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8649024-1788-46ee-959c-d521b00975f4-istiod-ca-cert\") on node \"ip-10-0-133-31.ec2.internal\" DevicePath \"\"" Jul 22 03:24:57.634101 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:24:57.634013 2578 generic.go:356] "Generic (PLEG): container finished" podID="b8649024-1788-46ee-959c-d521b00975f4" containerID="8b5fbc9196b074164d88522c300a6994df416fe1cc27f511b15aae80115862c6" exitCode=0 Jul 22 03:24:57.634101 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:24:57.634066 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-e9ef33-openshift-default-76c5696ddc-5g9gv" Jul 22 03:24:57.634101 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:24:57.634088 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-e9ef33-openshift-default-76c5696ddc-5g9gv" event={"ID":"b8649024-1788-46ee-959c-d521b00975f4","Type":"ContainerDied","Data":"8b5fbc9196b074164d88522c300a6994df416fe1cc27f511b15aae80115862c6"} Jul 22 03:24:57.634365 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:24:57.634146 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-e9ef33-openshift-default-76c5696ddc-5g9gv" event={"ID":"b8649024-1788-46ee-959c-d521b00975f4","Type":"ContainerDied","Data":"b3f44d6ef1c1f66f633686b72d6512394ddd448509f43a1d9ba5d0ca7453d1b2"} Jul 22 03:24:57.634365 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:24:57.634175 2578 scope.go:117] "RemoveContainer" containerID="8b5fbc9196b074164d88522c300a6994df416fe1cc27f511b15aae80115862c6" Jul 22 03:24:57.643775 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:24:57.643757 2578 scope.go:117] "RemoveContainer" containerID="694b178fd7e3f0d6187134a6ee626bd2c40a132cda4646f785ea20b6775db2a0" Jul 22 03:24:57.652386 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:24:57.652367 2578 scope.go:117] "RemoveContainer" containerID="8b5fbc9196b074164d88522c300a6994df416fe1cc27f511b15aae80115862c6" Jul 22 03:24:57.652647 ip-10-0-133-31 kubenswrapper[2578]: E0722 03:24:57.652628 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8b5fbc9196b074164d88522c300a6994df416fe1cc27f511b15aae80115862c6\": container with ID starting with 8b5fbc9196b074164d88522c300a6994df416fe1cc27f511b15aae80115862c6 not found: ID does not exist" containerID="8b5fbc9196b074164d88522c300a6994df416fe1cc27f511b15aae80115862c6" Jul 22 03:24:57.652694 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:24:57.652655 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b5fbc9196b074164d88522c300a6994df416fe1cc27f511b15aae80115862c6"} err="failed to get container status \"8b5fbc9196b074164d88522c300a6994df416fe1cc27f511b15aae80115862c6\": rpc error: code = NotFound desc = could not find container \"8b5fbc9196b074164d88522c300a6994df416fe1cc27f511b15aae80115862c6\": container with ID starting with 8b5fbc9196b074164d88522c300a6994df416fe1cc27f511b15aae80115862c6 not found: ID does not exist" Jul 22 03:24:57.652694 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:24:57.652672 2578 scope.go:117] "RemoveContainer" containerID="694b178fd7e3f0d6187134a6ee626bd2c40a132cda4646f785ea20b6775db2a0" Jul 22 03:24:57.652907 ip-10-0-133-31 kubenswrapper[2578]: E0722 03:24:57.652891 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"694b178fd7e3f0d6187134a6ee626bd2c40a132cda4646f785ea20b6775db2a0\": container with ID starting with 694b178fd7e3f0d6187134a6ee626bd2c40a132cda4646f785ea20b6775db2a0 not found: ID does not exist" containerID="694b178fd7e3f0d6187134a6ee626bd2c40a132cda4646f785ea20b6775db2a0" Jul 22 03:24:57.652962 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:24:57.652910 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"694b178fd7e3f0d6187134a6ee626bd2c40a132cda4646f785ea20b6775db2a0"} err="failed to get container status \"694b178fd7e3f0d6187134a6ee626bd2c40a132cda4646f785ea20b6775db2a0\": rpc error: code = NotFound desc = could not find container \"694b178fd7e3f0d6187134a6ee626bd2c40a132cda4646f785ea20b6775db2a0\": container with ID starting with 694b178fd7e3f0d6187134a6ee626bd2c40a132cda4646f785ea20b6775db2a0 not found: ID does not exist" Jul 22 03:24:57.657959 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:24:57.657936 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-e9ef33-openshift-default-76c5696ddc-5g9gv"] Jul 22 03:24:57.661890 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:24:57.661871 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-e9ef33-openshift-default-76c5696ddc-5g9gv"] Jul 22 03:24:58.142457 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:24:58.142424 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b8649024-1788-46ee-959c-d521b00975f4" path="/var/lib/kubelet/pods/b8649024-1788-46ee-959c-d521b00975f4/volumes" Jul 22 03:24:58.216415 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:24:58.216381 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-g5rrk_e61437db-765c-4abf-bf6b-2df9bf59faf4/manager/0.log" Jul 22 03:24:58.328453 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:24:58.328421 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-f6p2m_bab76e8d-039e-45e2-8122-e402782e7cd1/manager/0.log" Jul 22 03:24:58.587952 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:24:58.587927 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-4fz8t_27c81423-5417-4cb0-87f8-2edaa3b345d5/server/0.log" Jul 22 03:24:58.956205 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:24:58.956115 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5b96c6dbc5-t7fj2_f60d3c35-191d-441a-8caa-95913b917e57/copy-manifests/0.log" Jul 22 03:24:58.963137 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:24:58.963112 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5b96c6dbc5-t7fj2_f60d3c35-191d-441a-8caa-95913b917e57/manager/0.log" Jul 22 03:25:00.561055 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:25:00.561018 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7598m48r_969f440a-4dbb-4982-8817-e27fde2b3f19/pull/0.log" Jul 22 03:25:00.568171 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:25:00.568147 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7598m48r_969f440a-4dbb-4982-8817-e27fde2b3f19/extract/0.log" Jul 22 03:25:00.574711 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:25:00.574685 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7598m48r_969f440a-4dbb-4982-8817-e27fde2b3f19/util/0.log" Jul 22 03:25:00.693476 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:25:00.693445 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0gkpq6_74b87c7d-f11f-4cf4-88f4-9f6bf42b6d59/util/0.log" Jul 22 03:25:00.699264 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:25:00.699240 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0gkpq6_74b87c7d-f11f-4cf4-88f4-9f6bf42b6d59/pull/0.log" Jul 22 03:25:00.705421 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:25:00.705400 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0gkpq6_74b87c7d-f11f-4cf4-88f4-9f6bf42b6d59/extract/0.log" Jul 22 03:25:00.816859 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:25:00.816780 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735h76m_122e8d30-1f3c-4d5f-a322-54896dff729a/util/0.log" Jul 22 03:25:00.822866 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:25:00.822847 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735h76m_122e8d30-1f3c-4d5f-a322-54896dff729a/pull/0.log" Jul 22 03:25:00.828361 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:25:00.828344 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735h76m_122e8d30-1f3c-4d5f-a322-54896dff729a/extract/0.log" Jul 22 03:25:00.948284 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:25:00.948256 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1n794h_f47eb2cd-4a19-4bec-b095-0dd6f0409ff2/util/0.log" Jul 22 03:25:00.955914 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:25:00.955889 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1n794h_f47eb2cd-4a19-4bec-b095-0dd6f0409ff2/pull/0.log" Jul 22 03:25:00.962605 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:25:00.962582 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1n794h_f47eb2cd-4a19-4bec-b095-0dd6f0409ff2/extract/0.log" Jul 22 03:25:01.206850 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:25:01.206822 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-v2m7g_9d127701-d2c6-4681-b40e-66ed377d5ef3/manager/0.log" Jul 22 03:25:01.320376 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:25:01.320345 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-dzltm_08542239-2a2d-463c-8b80-c3d5b90651b4/manager/0.log" Jul 22 03:25:01.432783 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:25:01.432754 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-ttwjf_3979efe4-f663-44f9-ad69-c243fb7ea8fe/kuadrant-console-plugin/0.log" Jul 22 03:25:01.556299 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:25:01.556221 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-wfxc7_81b36d89-3471-4dea-9a7d-0e2e8bdab2bc/registry-server/0.log" Jul 22 03:25:01.686736 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:25:01.686704 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-7md5z_e01f4154-c4a7-4994-9685-8111f41adff0/manager/0.log" Jul 22 03:25:01.927380 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:25:01.927351 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-pjpdf_44dc2356-c6f8-4a3a-b81e-c0e84ee68812/manager/0.log" Jul 22 03:25:02.297662 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:25:02.297589 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf46vpxp_07b23859-7ee5-4299-965e-b1592f48e054/istio-proxy/0.log" Jul 22 03:25:02.662535 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:25:02.662508 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7885ddc578-vv7tg_ef04119c-df6f-4ad7-8619-fb9bc10be49b/kube-auth-proxy/0.log" Jul 22 03:25:02.781114 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:25:02.781086 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-n8zpf_1021ed34-ba20-48c2-a67f-6bd304cfd0cb/istio-proxy/1.log" Jul 22 03:25:03.608227 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:25:03.608198 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-fws8d_c7548ae4-2b73-49a3-9d98-421a214bb792/main/0.log" Jul 22 03:25:03.836918 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:25:03.836885 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fztprq_62dc2edb-a885-4562-8b80-597c1b43926c/main/0.log" Jul 22 03:25:04.065532 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:25:04.065457 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-57fc66f697-bjwkm_9f685069-4532-491e-a7f7-e11cf0816f72/main/0.log" Jul 22 03:25:12.495426 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:25:12.495391 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-rb9sf_684d87e0-1cec-4247-9446-881aff58a246/global-pull-secret-syncer/0.log" Jul 22 03:25:12.607911 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:25:12.607877 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-bxg22_e56db148-7293-423e-b9fb-0ebd5db740a3/konnectivity-agent/0.log" Jul 22 03:25:12.701359 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:25:12.701327 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-133-31.ec2.internal_47b7203f116608cac74abbb19cda28d8/haproxy/0.log" Jul 22 03:25:17.571281 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:25:17.571246 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7598m48r_969f440a-4dbb-4982-8817-e27fde2b3f19/extract/0.log" Jul 22 03:25:17.594739 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:25:17.594712 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7598m48r_969f440a-4dbb-4982-8817-e27fde2b3f19/util/0.log" Jul 22 03:25:17.623826 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:25:17.623798 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7598m48r_969f440a-4dbb-4982-8817-e27fde2b3f19/pull/0.log" Jul 22 03:25:17.650847 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:25:17.650820 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0gkpq6_74b87c7d-f11f-4cf4-88f4-9f6bf42b6d59/extract/0.log" Jul 22 03:25:17.684712 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:25:17.684676 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0gkpq6_74b87c7d-f11f-4cf4-88f4-9f6bf42b6d59/util/0.log" Jul 22 03:25:17.707673 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:25:17.707649 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0gkpq6_74b87c7d-f11f-4cf4-88f4-9f6bf42b6d59/pull/0.log" Jul 22 03:25:17.736980 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:25:17.736955 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735h76m_122e8d30-1f3c-4d5f-a322-54896dff729a/extract/0.log" Jul 22 03:25:17.760457 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:25:17.760432 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735h76m_122e8d30-1f3c-4d5f-a322-54896dff729a/util/0.log" Jul 22 03:25:17.782734 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:25:17.782706 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735h76m_122e8d30-1f3c-4d5f-a322-54896dff729a/pull/0.log" Jul 22 03:25:17.835195 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:25:17.835159 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1n794h_f47eb2cd-4a19-4bec-b095-0dd6f0409ff2/extract/0.log" Jul 22 03:25:17.859866 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:25:17.859836 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1n794h_f47eb2cd-4a19-4bec-b095-0dd6f0409ff2/util/0.log" Jul 22 03:25:17.882752 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:25:17.882723 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1n794h_f47eb2cd-4a19-4bec-b095-0dd6f0409ff2/pull/0.log" Jul 22 03:25:18.632897 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:25:18.632869 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-v2m7g_9d127701-d2c6-4681-b40e-66ed377d5ef3/manager/0.log" Jul 22 03:25:18.669680 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:25:18.669651 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-dzltm_08542239-2a2d-463c-8b80-c3d5b90651b4/manager/0.log" Jul 22 03:25:18.710511 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:25:18.710485 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-ttwjf_3979efe4-f663-44f9-ad69-c243fb7ea8fe/kuadrant-console-plugin/0.log" Jul 22 03:25:18.754868 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:25:18.754837 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-wfxc7_81b36d89-3471-4dea-9a7d-0e2e8bdab2bc/registry-server/0.log" Jul 22 03:25:18.937045 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:25:18.936961 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-7md5z_e01f4154-c4a7-4994-9685-8111f41adff0/manager/0.log" Jul 22 03:25:19.056620 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:25:19.056586 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-pjpdf_44dc2356-c6f8-4a3a-b81e-c0e84ee68812/manager/0.log" Jul 22 03:25:21.065724 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:25:21.065692 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-7495b9d698-s64vm_b9b81360-170a-4e01-af3e-8fd2947ec5a6/metrics-server/0.log" Jul 22 03:25:21.138241 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:25:21.138216 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-bq2jp_04566074-17d2-481d-bf22-d5d63462ad91/node-exporter/0.log" Jul 22 03:25:21.163170 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:25:21.163144 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-bq2jp_04566074-17d2-481d-bf22-d5d63462ad91/kube-rbac-proxy/0.log" Jul 22 03:25:21.190760 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:25:21.190730 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-bq2jp_04566074-17d2-481d-bf22-d5d63462ad91/init-textfile/0.log" Jul 22 03:25:23.160580 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:25:23.160531 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-7bc7489cb5-gxllf_35b750f1-f674-4130-b170-8adeb7a3fe91/networking-console-plugin/0.log" Jul 22 03:25:23.380016 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:25:23.379986 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-gzxjk/perf-node-gather-daemonset-drrpj"] Jul 22 03:25:23.380353 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:25:23.380338 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b8649024-1788-46ee-959c-d521b00975f4" containerName="istio-proxy" Jul 22 03:25:23.380353 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:25:23.380352 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="b8649024-1788-46ee-959c-d521b00975f4" containerName="istio-proxy" Jul 22 03:25:23.380531 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:25:23.380364 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6d898f3d-5f7c-4df7-8348-164cee485f36" containerName="main" Jul 22 03:25:23.380531 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:25:23.380371 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="6d898f3d-5f7c-4df7-8348-164cee485f36" containerName="main" Jul 22 03:25:23.380531 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:25:23.380392 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c52d4866-0bd6-4fdf-ab45-166f1778ba19" containerName="istio-proxy" Jul 22 03:25:23.380531 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:25:23.380398 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="c52d4866-0bd6-4fdf-ab45-166f1778ba19" containerName="istio-proxy" Jul 22 03:25:23.380531 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:25:23.380404 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e4b99269-7b0c-48e4-8617-82aae55cbdd0" containerName="main" Jul 22 03:25:23.380531 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:25:23.380409 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="e4b99269-7b0c-48e4-8617-82aae55cbdd0" containerName="main" Jul 22 03:25:23.380531 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:25:23.380422 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c52d4866-0bd6-4fdf-ab45-166f1778ba19" containerName="istio-proxy" Jul 22 03:25:23.380531 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:25:23.380427 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="c52d4866-0bd6-4fdf-ab45-166f1778ba19" containerName="istio-proxy" Jul 22 03:25:23.380531 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:25:23.380498 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="c52d4866-0bd6-4fdf-ab45-166f1778ba19" containerName="istio-proxy" Jul 22 03:25:23.380531 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:25:23.380512 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="c52d4866-0bd6-4fdf-ab45-166f1778ba19" containerName="istio-proxy" Jul 22 03:25:23.380531 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:25:23.380521 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="b8649024-1788-46ee-959c-d521b00975f4" containerName="istio-proxy" Jul 22 03:25:23.380531 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:25:23.380528 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="b8649024-1788-46ee-959c-d521b00975f4" containerName="istio-proxy" Jul 22 03:25:23.380531 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:25:23.380536 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="e4b99269-7b0c-48e4-8617-82aae55cbdd0" containerName="main" Jul 22 03:25:23.380976 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:25:23.380545 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="6d898f3d-5f7c-4df7-8348-164cee485f36" containerName="main" Jul 22 03:25:23.383374 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:25:23.383354 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-gzxjk/perf-node-gather-daemonset-drrpj" Jul 22 03:25:23.387552 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:25:23.387532 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-gzxjk\"/\"kube-root-ca.crt\"" Jul 22 03:25:23.388402 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:25:23.388388 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-gzxjk\"/\"default-dockercfg-k4hhd\"" Jul 22 03:25:23.388402 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:25:23.388397 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-gzxjk\"/\"openshift-service-ca.crt\"" Jul 22 03:25:23.393388 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:25:23.393364 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-gzxjk/perf-node-gather-daemonset-drrpj"] Jul 22 03:25:23.494666 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:25:23.494556 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/b71acc45-7684-464b-934c-4bb5b6304eac-proc\") pod \"perf-node-gather-daemonset-drrpj\" (UID: \"b71acc45-7684-464b-934c-4bb5b6304eac\") " pod="openshift-must-gather-gzxjk/perf-node-gather-daemonset-drrpj" Jul 22 03:25:23.494843 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:25:23.494671 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/b71acc45-7684-464b-934c-4bb5b6304eac-podres\") pod \"perf-node-gather-daemonset-drrpj\" (UID: \"b71acc45-7684-464b-934c-4bb5b6304eac\") " pod="openshift-must-gather-gzxjk/perf-node-gather-daemonset-drrpj" Jul 22 03:25:23.494843 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:25:23.494692 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b71acc45-7684-464b-934c-4bb5b6304eac-lib-modules\") pod \"perf-node-gather-daemonset-drrpj\" (UID: \"b71acc45-7684-464b-934c-4bb5b6304eac\") " pod="openshift-must-gather-gzxjk/perf-node-gather-daemonset-drrpj" Jul 22 03:25:23.494843 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:25:23.494718 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b71acc45-7684-464b-934c-4bb5b6304eac-sys\") pod \"perf-node-gather-daemonset-drrpj\" (UID: \"b71acc45-7684-464b-934c-4bb5b6304eac\") " pod="openshift-must-gather-gzxjk/perf-node-gather-daemonset-drrpj" Jul 22 03:25:23.494843 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:25:23.494734 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x8m6s\" (UniqueName: \"kubernetes.io/projected/b71acc45-7684-464b-934c-4bb5b6304eac-kube-api-access-x8m6s\") pod \"perf-node-gather-daemonset-drrpj\" (UID: \"b71acc45-7684-464b-934c-4bb5b6304eac\") " pod="openshift-must-gather-gzxjk/perf-node-gather-daemonset-drrpj" Jul 22 03:25:23.595677 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:25:23.595643 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/b71acc45-7684-464b-934c-4bb5b6304eac-proc\") pod \"perf-node-gather-daemonset-drrpj\" (UID: \"b71acc45-7684-464b-934c-4bb5b6304eac\") " pod="openshift-must-gather-gzxjk/perf-node-gather-daemonset-drrpj" Jul 22 03:25:23.595874 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:25:23.595728 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/b71acc45-7684-464b-934c-4bb5b6304eac-podres\") pod \"perf-node-gather-daemonset-drrpj\" (UID: \"b71acc45-7684-464b-934c-4bb5b6304eac\") " pod="openshift-must-gather-gzxjk/perf-node-gather-daemonset-drrpj" Jul 22 03:25:23.595874 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:25:23.595746 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b71acc45-7684-464b-934c-4bb5b6304eac-lib-modules\") pod \"perf-node-gather-daemonset-drrpj\" (UID: \"b71acc45-7684-464b-934c-4bb5b6304eac\") " pod="openshift-must-gather-gzxjk/perf-node-gather-daemonset-drrpj" Jul 22 03:25:23.595874 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:25:23.595770 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b71acc45-7684-464b-934c-4bb5b6304eac-sys\") pod \"perf-node-gather-daemonset-drrpj\" (UID: \"b71acc45-7684-464b-934c-4bb5b6304eac\") " pod="openshift-must-gather-gzxjk/perf-node-gather-daemonset-drrpj" Jul 22 03:25:23.595874 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:25:23.595771 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/b71acc45-7684-464b-934c-4bb5b6304eac-proc\") pod \"perf-node-gather-daemonset-drrpj\" (UID: \"b71acc45-7684-464b-934c-4bb5b6304eac\") " pod="openshift-must-gather-gzxjk/perf-node-gather-daemonset-drrpj" Jul 22 03:25:23.595874 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:25:23.595788 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x8m6s\" (UniqueName: \"kubernetes.io/projected/b71acc45-7684-464b-934c-4bb5b6304eac-kube-api-access-x8m6s\") pod \"perf-node-gather-daemonset-drrpj\" (UID: \"b71acc45-7684-464b-934c-4bb5b6304eac\") " pod="openshift-must-gather-gzxjk/perf-node-gather-daemonset-drrpj" Jul 22 03:25:23.596066 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:25:23.595876 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b71acc45-7684-464b-934c-4bb5b6304eac-sys\") pod \"perf-node-gather-daemonset-drrpj\" (UID: \"b71acc45-7684-464b-934c-4bb5b6304eac\") " pod="openshift-must-gather-gzxjk/perf-node-gather-daemonset-drrpj" Jul 22 03:25:23.596066 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:25:23.595928 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b71acc45-7684-464b-934c-4bb5b6304eac-lib-modules\") pod \"perf-node-gather-daemonset-drrpj\" (UID: \"b71acc45-7684-464b-934c-4bb5b6304eac\") " pod="openshift-must-gather-gzxjk/perf-node-gather-daemonset-drrpj" Jul 22 03:25:23.596066 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:25:23.595928 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/b71acc45-7684-464b-934c-4bb5b6304eac-podres\") pod \"perf-node-gather-daemonset-drrpj\" (UID: \"b71acc45-7684-464b-934c-4bb5b6304eac\") " pod="openshift-must-gather-gzxjk/perf-node-gather-daemonset-drrpj" Jul 22 03:25:23.605876 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:25:23.605849 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x8m6s\" (UniqueName: \"kubernetes.io/projected/b71acc45-7684-464b-934c-4bb5b6304eac-kube-api-access-x8m6s\") pod \"perf-node-gather-daemonset-drrpj\" (UID: \"b71acc45-7684-464b-934c-4bb5b6304eac\") " pod="openshift-must-gather-gzxjk/perf-node-gather-daemonset-drrpj" Jul 22 03:25:23.694602 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:25:23.694543 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-gzxjk/perf-node-gather-daemonset-drrpj" Jul 22 03:25:23.835187 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:25:23.835161 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-gzxjk/perf-node-gather-daemonset-drrpj"] Jul 22 03:25:23.836516 ip-10-0-133-31 kubenswrapper[2578]: W0722 03:25:23.836491 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podb71acc45_7684_464b_934c_4bb5b6304eac.slice/crio-f07a7602aaccce2bd7a8b142307ce6d830a581ad2ce9594307e03be99d09310c WatchSource:0}: Error finding container f07a7602aaccce2bd7a8b142307ce6d830a581ad2ce9594307e03be99d09310c: Status 404 returned error can't find the container with id f07a7602aaccce2bd7a8b142307ce6d830a581ad2ce9594307e03be99d09310c Jul 22 03:25:24.481926 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:25:24.481887 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-9c4f7966-xxjsg_52d794b1-1afa-4ee1-868e-abf89dc7a91f/console/0.log" Jul 22 03:25:24.782290 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:25:24.782193 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-gzxjk/perf-node-gather-daemonset-drrpj" event={"ID":"b71acc45-7684-464b-934c-4bb5b6304eac","Type":"ContainerStarted","Data":"855dbe1577d7b75ca13470cc2e559a0c0245eb6d5354eb8ef433b0a86ae508f2"} Jul 22 03:25:24.782290 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:25:24.782249 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-gzxjk/perf-node-gather-daemonset-drrpj" event={"ID":"b71acc45-7684-464b-934c-4bb5b6304eac","Type":"ContainerStarted","Data":"f07a7602aaccce2bd7a8b142307ce6d830a581ad2ce9594307e03be99d09310c"} Jul 22 03:25:24.782500 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:25:24.782398 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-gzxjk/perf-node-gather-daemonset-drrpj" Jul 22 03:25:24.804279 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:25:24.804223 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-gzxjk/perf-node-gather-daemonset-drrpj" podStartSLOduration=1.804206145 podStartE2EDuration="1.804206145s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 03:25:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 03:25:24.802520162 +0000 UTC m=+2891.121695462" watchObservedRunningTime="2026-07-22 03:25:24.804206145 +0000 UTC m=+2891.123381445" Jul 22 03:25:26.459482 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:25:26.459448 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-trlg7_26d80b89-26cd-4736-9a65-753db280e9fd/dns/0.log" Jul 22 03:25:26.488011 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:25:26.487981 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-trlg7_26d80b89-26cd-4736-9a65-753db280e9fd/kube-rbac-proxy/0.log" Jul 22 03:25:26.517398 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:25:26.517372 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-9dhrs_dea9d10b-9320-471c-9efd-544fabc16854/dns-node-resolver/0.log" Jul 22 03:25:27.191401 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:25:27.191371 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-7cf654f645-clg89_9e580eb6-1fce-40b4-a59a-446daa0e0398/registry/0.log" Jul 22 03:25:27.242856 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:25:27.242828 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-8kmfc_d7953894-cdfc-48a3-afe8-89503dfc3c57/node-ca/0.log" Jul 22 03:25:28.392330 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:25:28.392299 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf46vpxp_07b23859-7ee5-4299-965e-b1592f48e054/istio-proxy/0.log" Jul 22 03:25:28.679310 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:25:28.679230 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7885ddc578-vv7tg_ef04119c-df6f-4ad7-8619-fb9bc10be49b/kube-auth-proxy/0.log" Jul 22 03:25:28.708201 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:25:28.708169 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-n8zpf_1021ed34-ba20-48c2-a67f-6bd304cfd0cb/istio-proxy/0.log" Jul 22 03:25:28.732647 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:25:28.732603 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-n8zpf_1021ed34-ba20-48c2-a67f-6bd304cfd0cb/istio-proxy/1.log" Jul 22 03:25:29.765453 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:25:29.765405 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-8fzdj_2bb410f4-fbcd-43d0-bfa1-e0cf2eaf4289/serve-healthcheck-canary/0.log" Jul 22 03:25:30.396928 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:25:30.396898 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-gvwx6_a0f7cb8b-7cc6-441e-accc-8c2061b4baab/kube-rbac-proxy/0.log" Jul 22 03:25:30.422654 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:25:30.422629 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-gvwx6_a0f7cb8b-7cc6-441e-accc-8c2061b4baab/exporter/0.log" Jul 22 03:25:30.447623 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:25:30.447592 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-gvwx6_a0f7cb8b-7cc6-441e-accc-8c2061b4baab/extractor/0.log" Jul 22 03:25:30.797971 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:25:30.797903 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-gzxjk/perf-node-gather-daemonset-drrpj" Jul 22 03:25:33.033962 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:25:33.033929 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-g5rrk_e61437db-765c-4abf-bf6b-2df9bf59faf4/manager/0.log" Jul 22 03:25:33.210914 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:25:33.210882 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-f6p2m_bab76e8d-039e-45e2-8122-e402782e7cd1/manager/0.log" Jul 22 03:25:33.491634 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:25:33.491597 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-4fz8t_27c81423-5417-4cb0-87f8-2edaa3b345d5/server/0.log" Jul 22 03:25:33.730213 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:25:33.730117 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5b96c6dbc5-t7fj2_f60d3c35-191d-441a-8caa-95913b917e57/manager/0.log" Jul 22 03:25:33.755413 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:25:33.755389 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5b96c6dbc5-t7fj2_f60d3c35-191d-441a-8caa-95913b917e57/copy-manifests/0.log" Jul 22 03:25:34.342621 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:25:34.342592 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf46vpxp_07b23859-7ee5-4299-965e-b1592f48e054/istio-proxy/0.log" Jul 22 03:25:34.604478 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:25:34.604394 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7885ddc578-vv7tg_ef04119c-df6f-4ad7-8619-fb9bc10be49b/kube-auth-proxy/0.log" Jul 22 03:25:34.623236 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:25:34.623209 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-n8zpf_1021ed34-ba20-48c2-a67f-6bd304cfd0cb/istio-proxy/0.log" Jul 22 03:25:34.648058 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:25:34.648023 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-n8zpf_1021ed34-ba20-48c2-a67f-6bd304cfd0cb/istio-proxy/1.log" Jul 22 03:25:35.985289 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:25:35.985262 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf46vpxp_07b23859-7ee5-4299-965e-b1592f48e054/istio-proxy/0.log" Jul 22 03:25:36.245510 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:25:36.245429 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7885ddc578-vv7tg_ef04119c-df6f-4ad7-8619-fb9bc10be49b/kube-auth-proxy/0.log" Jul 22 03:25:36.272986 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:25:36.272947 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-n8zpf_1021ed34-ba20-48c2-a67f-6bd304cfd0cb/istio-proxy/0.log" Jul 22 03:25:36.302447 ip-10-0-133-31 kubenswrapper[2578]: I0722 03:25:36.302418 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-n8zpf_1021ed34-ba20-48c2-a67f-6bd304cfd0cb/istio-proxy/1.log"