Jul 16 16:25:41.384734 ip-10-0-142-101 systemd[1]: Starting Kubernetes Kubelet... Jul 16 16:25:41.780793 ip-10-0-142-101 kubenswrapper[2577]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 16 16:25:41.780793 ip-10-0-142-101 kubenswrapper[2577]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 16 16:25:41.780793 ip-10-0-142-101 kubenswrapper[2577]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 16 16:25:41.780793 ip-10-0-142-101 kubenswrapper[2577]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 16 16:25:41.780793 ip-10-0-142-101 kubenswrapper[2577]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 16 16:25:41.781549 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:41.780819 2577 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 16 16:25:41.786978 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.786956 2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 16 16:25:41.786978 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.786976 2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 16 16:25:41.787055 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.786980 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 16 16:25:41.787055 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.786984 2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 16 16:25:41.787055 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.786988 2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 16 16:25:41.787055 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.786991 2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 16 16:25:41.787055 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.786994 2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 16 16:25:41.787055 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.786996 2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 16 16:25:41.787055 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.787001 2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 16 16:25:41.787055 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.787004 2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 16 16:25:41.787055 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.787007 2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 16 16:25:41.787055 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.787010 2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 16 16:25:41.787055 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.787013 2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 16 16:25:41.787055 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.787015 2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 16 16:25:41.787055 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.787018 2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 16 16:25:41.787055 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.787021 2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 16 16:25:41.787055 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.787024 2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 16 16:25:41.787055 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.787027 2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 16 16:25:41.787055 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.787029 2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 16 16:25:41.787055 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.787032 2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 16 16:25:41.787055 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.787035 2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 16 16:25:41.787055 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.787038 2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 16 16:25:41.787561 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.787042 2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 16 16:25:41.787561 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.787045 2577 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 16 16:25:41.787561 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.787048 2577 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 16 16:25:41.787561 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.787051 2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 16 16:25:41.787561 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.787054 2577 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 16 16:25:41.787561 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.787057 2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 16 16:25:41.787561 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.787060 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 16 16:25:41.787561 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.787063 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 16 16:25:41.787561 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.787066 2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 16 16:25:41.787561 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.787069 2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 16 16:25:41.787561 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.787072 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 16 16:25:41.787561 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.787074 2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 16 16:25:41.787561 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.787077 2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 16 16:25:41.787561 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.787080 2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 16 16:25:41.787561 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.787083 2577 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 16 16:25:41.787561 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.787086 2577 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 16 16:25:41.787561 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.787089 2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 16 16:25:41.787561 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.787092 2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 16 16:25:41.787561 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.787095 2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 16 16:25:41.787561 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.787097 2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 16 16:25:41.788102 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.787100 2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 16 16:25:41.788102 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.787103 2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 16 16:25:41.788102 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.787107 2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 16 16:25:41.788102 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.787111 2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 16 16:25:41.788102 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.787115 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 16 16:25:41.788102 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.787118 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 16 16:25:41.788102 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.787121 2577 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 16 16:25:41.788102 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.787124 2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 16 16:25:41.788102 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.787128 2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 16 16:25:41.788102 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.787131 2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 16 16:25:41.788102 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.787135 2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 16 16:25:41.788102 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.787138 2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 16 16:25:41.788102 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.787142 2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 16 16:25:41.788102 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.787148 2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 16 16:25:41.788102 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.787151 2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 16 16:25:41.788102 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.787155 2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 16 16:25:41.788102 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.787157 2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 16 16:25:41.788102 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.787160 2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 16 16:25:41.788102 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.787163 2577 feature_gate.go:328] unrecognized feature gate: Example2 Jul 16 16:25:41.788605 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.787166 2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 16 16:25:41.788605 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.787168 2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 16 16:25:41.788605 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.787171 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 16 16:25:41.788605 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.787174 2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 16 16:25:41.788605 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.787176 2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 16 16:25:41.788605 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.787179 2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 16 16:25:41.788605 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.787182 2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 16 16:25:41.788605 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.787184 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 16 16:25:41.788605 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.787187 2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 16 16:25:41.788605 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.787193 2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 16 16:25:41.788605 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.787198 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 16 16:25:41.788605 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.787201 2577 feature_gate.go:328] unrecognized feature gate: Example Jul 16 16:25:41.788605 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.787204 2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 16 16:25:41.788605 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.787207 2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 16 16:25:41.788605 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.787211 2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 16 16:25:41.788605 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.787214 2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 16 16:25:41.788605 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.787217 2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 16 16:25:41.788605 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.787219 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 16 16:25:41.788605 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.787222 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 16 16:25:41.788605 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.787225 2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 16 16:25:41.789119 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.787228 2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 16 16:25:41.789119 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.787232 2577 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 16 16:25:41.789119 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.787235 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 16 16:25:41.789119 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.787237 2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 16 16:25:41.789119 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.787240 2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 16 16:25:41.789119 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.787244 2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 16 16:25:41.789119 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.787247 2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 16 16:25:41.789119 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.787250 2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 16 16:25:41.789119 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.787254 2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 16 16:25:41.789119 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.787257 2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 16 16:25:41.789119 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.787259 2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 16 16:25:41.789119 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.787262 2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 16 16:25:41.789119 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.787265 2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 16 16:25:41.789119 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.787267 2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 16 16:25:41.789119 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.787270 2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 16 16:25:41.789119 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.787273 2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 16 16:25:41.789119 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.787276 2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 16 16:25:41.789119 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.787278 2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 16 16:25:41.789119 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.788099 2577 feature_gate.go:328] unrecognized feature gate: Example2 Jul 16 16:25:41.789119 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.788107 2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 16 16:25:41.789716 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.788110 2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 16 16:25:41.789716 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.788113 2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 16 16:25:41.789716 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.788116 2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 16 16:25:41.789716 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.788120 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 16 16:25:41.789716 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.788123 2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 16 16:25:41.789716 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.788126 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 16 16:25:41.789716 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.788129 2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 16 16:25:41.789716 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.788132 2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 16 16:25:41.789716 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.788135 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 16 16:25:41.789716 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.788137 2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 16 16:25:41.789716 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.788140 2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 16 16:25:41.789716 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.788143 2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 16 16:25:41.789716 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.788146 2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 16 16:25:41.789716 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.788149 2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 16 16:25:41.789716 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.788152 2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 16 16:25:41.789716 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.788155 2577 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 16 16:25:41.789716 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.788158 2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 16 16:25:41.789716 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.788162 2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 16 16:25:41.789716 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.788166 2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 16 16:25:41.789716 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.788168 2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 16 16:25:41.790283 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.788171 2577 feature_gate.go:328] unrecognized feature gate: Example Jul 16 16:25:41.790283 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.788174 2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 16 16:25:41.790283 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.788177 2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 16 16:25:41.790283 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.788180 2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 16 16:25:41.790283 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.788184 2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 16 16:25:41.790283 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.788186 2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 16 16:25:41.790283 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.788189 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 16 16:25:41.790283 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.788192 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 16 16:25:41.790283 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.788194 2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 16 16:25:41.790283 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.788198 2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 16 16:25:41.790283 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.788201 2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 16 16:25:41.790283 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.788204 2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 16 16:25:41.790283 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.788207 2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 16 16:25:41.790283 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.788209 2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 16 16:25:41.790283 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.788212 2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 16 16:25:41.790283 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.788215 2577 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 16 16:25:41.790283 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.788223 2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 16 16:25:41.790283 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.788226 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 16 16:25:41.790283 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.788229 2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 16 16:25:41.790283 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.788233 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 16 16:25:41.790827 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.788236 2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 16 16:25:41.790827 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.788238 2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 16 16:25:41.790827 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.788241 2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 16 16:25:41.790827 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.788244 2577 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 16 16:25:41.790827 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.788247 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 16 16:25:41.790827 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.788249 2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 16 16:25:41.790827 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.788252 2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 16 16:25:41.790827 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.788255 2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 16 16:25:41.790827 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.788257 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 16 16:25:41.790827 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.788260 2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 16 16:25:41.790827 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.788264 2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 16 16:25:41.790827 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.788267 2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 16 16:25:41.790827 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.788270 2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 16 16:25:41.790827 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.788273 2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 16 16:25:41.790827 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.788276 2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 16 16:25:41.790827 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.788279 2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 16 16:25:41.790827 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.788281 2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 16 16:25:41.790827 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.788284 2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 16 16:25:41.790827 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.788287 2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 16 16:25:41.790827 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.788290 2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 16 16:25:41.791374 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.788293 2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 16 16:25:41.791374 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.788295 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 16 16:25:41.791374 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.788298 2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 16 16:25:41.791374 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.788301 2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 16 16:25:41.791374 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.788304 2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 16 16:25:41.791374 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.788307 2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 16 16:25:41.791374 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.788310 2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 16 16:25:41.791374 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.788312 2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 16 16:25:41.791374 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.788315 2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 16 16:25:41.791374 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.788324 2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 16 16:25:41.791374 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.788327 2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 16 16:25:41.791374 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.788330 2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 16 16:25:41.791374 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.788333 2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 16 16:25:41.791374 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.788336 2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 16 16:25:41.791374 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.788338 2577 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 16 16:25:41.791374 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.788343 2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 16 16:25:41.791374 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.788345 2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 16 16:25:41.791374 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.788348 2577 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 16 16:25:41.791374 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.788351 2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 16 16:25:41.791885 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.788354 2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 16 16:25:41.791885 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.788357 2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 16 16:25:41.791885 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.788359 2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 16 16:25:41.791885 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.788362 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 16 16:25:41.791885 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.788365 2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 16 16:25:41.791885 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.788368 2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 16 16:25:41.791885 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.788370 2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 16 16:25:41.791885 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.788373 2577 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 16 16:25:41.791885 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.788377 2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 16 16:25:41.791885 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.788381 2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 16 16:25:41.791885 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.788384 2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 16 16:25:41.791885 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.788387 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 16 16:25:41.791885 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.788390 2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 16 16:25:41.791885 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.788393 2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 16 16:25:41.791885 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.788397 2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 16 16:25:41.791885 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.788400 2577 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 16 16:25:41.791885 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.788403 2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 16 16:25:41.791885 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.788406 2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 16 16:25:41.791885 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.788869 2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 16 16:25:41.792384 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.788874 2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 16 16:25:41.792384 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.788879 2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 16 16:25:41.792384 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.788882 2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 16 16:25:41.792384 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.788885 2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 16 16:25:41.792384 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.788889 2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 16 16:25:41.792384 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.788892 2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 16 16:25:41.792384 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.788895 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 16 16:25:41.792384 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.788898 2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 16 16:25:41.792384 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.788900 2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 16 16:25:41.792384 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.788904 2577 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 16 16:25:41.792384 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.788908 2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 16 16:25:41.792384 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.788912 2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 16 16:25:41.792384 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.788915 2577 feature_gate.go:328] unrecognized feature gate: Example2 Jul 16 16:25:41.792384 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.788919 2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 16 16:25:41.792384 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.788922 2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 16 16:25:41.792384 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.788925 2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 16 16:25:41.792384 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.788929 2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 16 16:25:41.792384 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.788932 2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 16 16:25:41.792384 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.788935 2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 16 16:25:41.792384 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.788938 2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 16 16:25:41.792926 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.788941 2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 16 16:25:41.792926 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.788944 2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 16 16:25:41.792926 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.788947 2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 16 16:25:41.792926 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.788950 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 16 16:25:41.792926 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.788953 2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 16 16:25:41.792926 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.788956 2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 16 16:25:41.792926 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.788959 2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 16 16:25:41.792926 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.788961 2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 16 16:25:41.792926 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.788964 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 16 16:25:41.792926 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.788967 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 16 16:25:41.792926 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.788970 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 16 16:25:41.792926 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.788972 2577 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 16 16:25:41.792926 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.788975 2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 16 16:25:41.792926 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.788978 2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 16 16:25:41.792926 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.788981 2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 16 16:25:41.792926 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.788984 2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 16 16:25:41.792926 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.788988 2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 16 16:25:41.792926 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.788990 2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 16 16:25:41.792926 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.788994 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 16 16:25:41.792926 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.788997 2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 16 16:25:41.793440 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.789000 2577 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 16 16:25:41.793440 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.789002 2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 16 16:25:41.793440 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.789006 2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 16 16:25:41.793440 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.789010 2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 16 16:25:41.793440 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.789012 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 16 16:25:41.793440 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.789016 2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 16 16:25:41.793440 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.789019 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 16 16:25:41.793440 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.789022 2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 16 16:25:41.793440 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.789025 2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 16 16:25:41.793440 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.789028 2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 16 16:25:41.793440 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.789031 2577 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 16 16:25:41.793440 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.789034 2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 16 16:25:41.793440 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.789036 2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 16 16:25:41.793440 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.789040 2577 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 16 16:25:41.793440 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.789042 2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 16 16:25:41.793440 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.789045 2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 16 16:25:41.793440 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.789048 2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 16 16:25:41.793440 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.789051 2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 16 16:25:41.793440 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.789053 2577 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 16 16:25:41.793440 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.789056 2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 16 16:25:41.793994 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.789059 2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 16 16:25:41.793994 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.789062 2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 16 16:25:41.793994 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.789065 2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 16 16:25:41.793994 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.789067 2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 16 16:25:41.793994 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.789070 2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 16 16:25:41.793994 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.789073 2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 16 16:25:41.793994 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.789076 2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 16 16:25:41.793994 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.789078 2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 16 16:25:41.793994 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.789081 2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 16 16:25:41.793994 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.789085 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 16 16:25:41.793994 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.789087 2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 16 16:25:41.793994 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.789091 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 16 16:25:41.793994 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.789094 2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 16 16:25:41.793994 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.789097 2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 16 16:25:41.793994 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.789100 2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 16 16:25:41.793994 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.789102 2577 feature_gate.go:328] unrecognized feature gate: Example Jul 16 16:25:41.793994 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.789105 2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 16 16:25:41.793994 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.789108 2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 16 16:25:41.793994 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.789111 2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 16 16:25:41.794494 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.789113 2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 16 16:25:41.794494 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.789119 2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 16 16:25:41.794494 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.789123 2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 16 16:25:41.794494 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.789126 2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 16 16:25:41.794494 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.789129 2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 16 16:25:41.794494 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.789132 2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 16 16:25:41.794494 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.789135 2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 16 16:25:41.794494 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.789140 2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 16 16:25:41.794494 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.789143 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 16 16:25:41.794494 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.789147 2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 16 16:25:41.794494 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.789150 2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 16 16:25:41.794494 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.789153 2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 16 16:25:41.794494 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.789156 2577 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 16 16:25:41.794494 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.789159 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 16 16:25:41.794494 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.789162 2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 16 16:25:41.794494 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.789164 2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 16 16:25:41.794494 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.789167 2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 16 16:25:41.794494 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.789169 2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 16 16:25:41.794494 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.789172 2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 16 16:25:41.794494 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:41.789249 2577 flags.go:64] FLAG: --address="0.0.0.0" Jul 16 16:25:41.795014 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:41.789257 2577 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 16 16:25:41.795014 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:41.789265 2577 flags.go:64] FLAG: --anonymous-auth="true" Jul 16 16:25:41.795014 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:41.789270 2577 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 16 16:25:41.795014 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:41.789276 2577 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 16 16:25:41.795014 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:41.789280 2577 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 16 16:25:41.795014 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:41.789285 2577 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 16 16:25:41.795014 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:41.789290 2577 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 16 16:25:41.795014 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:41.789294 2577 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 16 16:25:41.795014 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:41.789297 2577 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 16 16:25:41.795014 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:41.789303 2577 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 16 16:25:41.795014 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:41.789307 2577 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 16 16:25:41.795014 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:41.789311 2577 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 16 16:25:41.795014 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:41.789314 2577 flags.go:64] FLAG: --cgroup-root="" Jul 16 16:25:41.795014 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:41.789319 2577 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 16 16:25:41.795014 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:41.789323 2577 flags.go:64] FLAG: --client-ca-file="" Jul 16 16:25:41.795014 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:41.789326 2577 flags.go:64] FLAG: --cloud-provider="external" Jul 16 16:25:41.795014 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:41.789329 2577 flags.go:64] FLAG: --cluster-dns="[]" Jul 16 16:25:41.795014 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:41.789334 2577 flags.go:64] FLAG: --cluster-domain="" Jul 16 16:25:41.795014 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:41.789337 2577 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 16 16:25:41.795014 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:41.789341 2577 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 16 16:25:41.795014 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:41.789344 2577 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 16 16:25:41.795014 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:41.789348 2577 flags.go:64] FLAG: --container-log-max-files="5" Jul 16 16:25:41.795014 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:41.789352 2577 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 16 16:25:41.795014 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:41.789356 2577 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 16 16:25:41.795651 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:41.789360 2577 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 16 16:25:41.795651 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:41.789364 2577 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 16 16:25:41.795651 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:41.789367 2577 flags.go:64] FLAG: --contention-profiling="false" Jul 16 16:25:41.795651 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:41.789371 2577 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 16 16:25:41.795651 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:41.789374 2577 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 16 16:25:41.795651 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:41.789377 2577 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 16 16:25:41.795651 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:41.789380 2577 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 16 16:25:41.795651 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:41.789385 2577 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 16 16:25:41.795651 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:41.789388 2577 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 16 16:25:41.795651 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:41.789391 2577 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 16 16:25:41.795651 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:41.789394 2577 flags.go:64] FLAG: --enable-load-reader="false" Jul 16 16:25:41.795651 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:41.789397 2577 flags.go:64] FLAG: --enable-server="true" Jul 16 16:25:41.795651 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:41.789406 2577 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 16 16:25:41.795651 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:41.789411 2577 flags.go:64] FLAG: --event-burst="100" Jul 16 16:25:41.795651 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:41.789415 2577 flags.go:64] FLAG: --event-qps="50" Jul 16 16:25:41.795651 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:41.789419 2577 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 16 16:25:41.795651 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:41.789423 2577 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 16 16:25:41.795651 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:41.789428 2577 flags.go:64] FLAG: --eviction-hard="" Jul 16 16:25:41.795651 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:41.789436 2577 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 16 16:25:41.795651 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:41.789439 2577 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 16 16:25:41.795651 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:41.789443 2577 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 16 16:25:41.795651 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:41.789448 2577 flags.go:64] FLAG: --eviction-soft="" Jul 16 16:25:41.795651 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:41.789451 2577 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 16 16:25:41.795651 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:41.789455 2577 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 16 16:25:41.795651 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:41.789458 2577 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 16 16:25:41.796379 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:41.789476 2577 flags.go:64] FLAG: --experimental-mounter-path="" Jul 16 16:25:41.796379 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:41.789480 2577 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 16 16:25:41.796379 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:41.789483 2577 flags.go:64] FLAG: --fail-swap-on="true" Jul 16 16:25:41.796379 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:41.789487 2577 flags.go:64] FLAG: --feature-gates="" Jul 16 16:25:41.796379 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:41.789491 2577 flags.go:64] FLAG: --file-check-frequency="20s" Jul 16 16:25:41.796379 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:41.789495 2577 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 16 16:25:41.796379 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:41.789498 2577 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 16 16:25:41.796379 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:41.789501 2577 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 16 16:25:41.796379 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:41.789505 2577 flags.go:64] FLAG: --healthz-port="10248" Jul 16 16:25:41.796379 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:41.789508 2577 flags.go:64] FLAG: --help="false" Jul 16 16:25:41.796379 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:41.789511 2577 flags.go:64] FLAG: --hostname-override="ip-10-0-142-101.ec2.internal" Jul 16 16:25:41.796379 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:41.789514 2577 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 16 16:25:41.796379 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:41.789517 2577 flags.go:64] FLAG: --http-check-frequency="20s" Jul 16 16:25:41.796379 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:41.789520 2577 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 16 16:25:41.796379 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:41.789524 2577 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 16 16:25:41.796379 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:41.789528 2577 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 16 16:25:41.796379 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:41.789531 2577 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 16 16:25:41.796379 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:41.789534 2577 flags.go:64] FLAG: --image-service-endpoint="" Jul 16 16:25:41.796379 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:41.789537 2577 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 16 16:25:41.796379 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:41.789540 2577 flags.go:64] FLAG: --kube-api-burst="100" Jul 16 16:25:41.796379 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:41.789544 2577 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 16 16:25:41.796379 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:41.789547 2577 flags.go:64] FLAG: --kube-api-qps="50" Jul 16 16:25:41.796379 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:41.789550 2577 flags.go:64] FLAG: --kube-reserved="" Jul 16 16:25:41.796379 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:41.789553 2577 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 16 16:25:41.797041 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:41.789558 2577 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 16 16:25:41.797041 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:41.789561 2577 flags.go:64] FLAG: --kubelet-cgroups="" Jul 16 16:25:41.797041 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:41.789564 2577 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 16 16:25:41.797041 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:41.789567 2577 flags.go:64] FLAG: --lock-file="" Jul 16 16:25:41.797041 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:41.789573 2577 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 16 16:25:41.797041 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:41.789576 2577 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 16 16:25:41.797041 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:41.789579 2577 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 16 16:25:41.797041 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:41.789585 2577 flags.go:64] FLAG: --log-json-split-stream="false" Jul 16 16:25:41.797041 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:41.789588 2577 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 16 16:25:41.797041 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:41.789591 2577 flags.go:64] FLAG: --log-text-split-stream="false" Jul 16 16:25:41.797041 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:41.789594 2577 flags.go:64] FLAG: --logging-format="text" Jul 16 16:25:41.797041 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:41.789597 2577 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 16 16:25:41.797041 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:41.789601 2577 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 16 16:25:41.797041 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:41.789604 2577 flags.go:64] FLAG: --manifest-url="" Jul 16 16:25:41.797041 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:41.789607 2577 flags.go:64] FLAG: --manifest-url-header="" Jul 16 16:25:41.797041 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:41.789612 2577 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 16 16:25:41.797041 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:41.789616 2577 flags.go:64] FLAG: --max-open-files="1000000" Jul 16 16:25:41.797041 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:41.789621 2577 flags.go:64] FLAG: --max-pods="110" Jul 16 16:25:41.797041 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:41.789624 2577 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 16 16:25:41.797041 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:41.789627 2577 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 16 16:25:41.797041 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:41.789630 2577 flags.go:64] FLAG: --memory-manager-policy="None" Jul 16 16:25:41.797041 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:41.789634 2577 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 16 16:25:41.797041 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:41.789637 2577 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 16 16:25:41.797041 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:41.789640 2577 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 16 16:25:41.797041 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:41.789643 2577 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 16 16:25:41.797761 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:41.789652 2577 flags.go:64] FLAG: --node-status-max-images="50" Jul 16 16:25:41.797761 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:41.789656 2577 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 16 16:25:41.797761 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:41.789659 2577 flags.go:64] FLAG: --oom-score-adj="-999" Jul 16 16:25:41.797761 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:41.789662 2577 flags.go:64] FLAG: --pod-cidr="" Jul 16 16:25:41.797761 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:41.789666 2577 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b04c0b1f73af4d021dd8f0b51bfeb302c24482c093de5f2c43654ab56c506926" Jul 16 16:25:41.797761 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:41.789673 2577 flags.go:64] FLAG: --pod-manifest-path="" Jul 16 16:25:41.797761 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:41.789676 2577 flags.go:64] FLAG: --pod-max-pids="-1" Jul 16 16:25:41.797761 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:41.789681 2577 flags.go:64] FLAG: --pods-per-core="0" Jul 16 16:25:41.797761 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:41.789700 2577 flags.go:64] FLAG: --port="10250" Jul 16 16:25:41.797761 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:41.789704 2577 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 16 16:25:41.797761 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:41.789708 2577 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0d9158e88be5b2146" Jul 16 16:25:41.797761 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:41.789712 2577 flags.go:64] FLAG: --qos-reserved="" Jul 16 16:25:41.797761 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:41.789717 2577 flags.go:64] FLAG: --read-only-port="10255" Jul 16 16:25:41.797761 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:41.789720 2577 flags.go:64] FLAG: --register-node="true" Jul 16 16:25:41.797761 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:41.789723 2577 flags.go:64] FLAG: --register-with-taints="" Jul 16 16:25:41.797761 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:41.789728 2577 flags.go:64] FLAG: --registry-burst="10" Jul 16 16:25:41.797761 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:41.789732 2577 flags.go:64] FLAG: --registry-qps="5" Jul 16 16:25:41.797761 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:41.789735 2577 flags.go:64] FLAG: --reserved-cpus="" Jul 16 16:25:41.797761 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:41.789738 2577 flags.go:64] FLAG: --reserved-memory="" Jul 16 16:25:41.797761 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:41.789742 2577 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 16 16:25:41.797761 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:41.789745 2577 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 16 16:25:41.797761 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:41.789749 2577 flags.go:64] FLAG: --rotate-certificates="false" Jul 16 16:25:41.797761 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:41.789753 2577 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 16 16:25:41.797761 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:41.789756 2577 flags.go:64] FLAG: --runonce="false" Jul 16 16:25:41.797761 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:41.789759 2577 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 16 16:25:41.798412 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:41.789762 2577 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 16 16:25:41.798412 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:41.789766 2577 flags.go:64] FLAG: --seccomp-default="false" Jul 16 16:25:41.798412 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:41.789769 2577 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 16 16:25:41.798412 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:41.789772 2577 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 16 16:25:41.798412 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:41.789775 2577 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 16 16:25:41.798412 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:41.789778 2577 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 16 16:25:41.798412 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:41.789782 2577 flags.go:64] FLAG: --storage-driver-password="root" Jul 16 16:25:41.798412 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:41.789785 2577 flags.go:64] FLAG: --storage-driver-secure="false" Jul 16 16:25:41.798412 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:41.789788 2577 flags.go:64] FLAG: --storage-driver-table="stats" Jul 16 16:25:41.798412 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:41.789791 2577 flags.go:64] FLAG: --storage-driver-user="root" Jul 16 16:25:41.798412 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:41.789794 2577 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 16 16:25:41.798412 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:41.789798 2577 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 16 16:25:41.798412 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:41.789802 2577 flags.go:64] FLAG: --system-cgroups="" Jul 16 16:25:41.798412 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:41.789805 2577 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 16 16:25:41.798412 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:41.789813 2577 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 16 16:25:41.798412 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:41.789816 2577 flags.go:64] FLAG: --tls-cert-file="" Jul 16 16:25:41.798412 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:41.789819 2577 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 16 16:25:41.798412 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:41.789824 2577 flags.go:64] FLAG: --tls-min-version="" Jul 16 16:25:41.798412 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:41.789827 2577 flags.go:64] FLAG: --tls-private-key-file="" Jul 16 16:25:41.798412 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:41.789831 2577 flags.go:64] FLAG: --topology-manager-policy="none" Jul 16 16:25:41.798412 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:41.789834 2577 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 16 16:25:41.798412 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:41.789837 2577 flags.go:64] FLAG: --topology-manager-scope="container" Jul 16 16:25:41.798412 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:41.789840 2577 flags.go:64] FLAG: --v="2" Jul 16 16:25:41.798412 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:41.789844 2577 flags.go:64] FLAG: --version="false" Jul 16 16:25:41.798412 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:41.789848 2577 flags.go:64] FLAG: --vmodule="" Jul 16 16:25:41.799078 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:41.789856 2577 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 16 16:25:41.799078 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:41.789859 2577 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 16 16:25:41.799078 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.790007 2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 16 16:25:41.799078 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.790012 2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 16 16:25:41.799078 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.790015 2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 16 16:25:41.799078 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.790019 2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 16 16:25:41.799078 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.790022 2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 16 16:25:41.799078 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.790025 2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 16 16:25:41.799078 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.790028 2577 feature_gate.go:328] unrecognized feature gate: Example Jul 16 16:25:41.799078 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.790031 2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 16 16:25:41.799078 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.790034 2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 16 16:25:41.799078 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.790037 2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 16 16:25:41.799078 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.790040 2577 feature_gate.go:328] unrecognized feature gate: Example2 Jul 16 16:25:41.799078 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.790042 2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 16 16:25:41.799078 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.790045 2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 16 16:25:41.799078 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.790048 2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 16 16:25:41.799078 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.790051 2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 16 16:25:41.799078 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.790053 2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 16 16:25:41.799078 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.790056 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 16 16:25:41.799597 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.790059 2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 16 16:25:41.799597 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.790063 2577 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 16 16:25:41.799597 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.790067 2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 16 16:25:41.799597 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.790070 2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 16 16:25:41.799597 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.790073 2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 16 16:25:41.799597 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.790076 2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 16 16:25:41.799597 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.790078 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 16 16:25:41.799597 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.790081 2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 16 16:25:41.799597 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.790084 2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 16 16:25:41.799597 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.790087 2577 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 16 16:25:41.799597 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.790091 2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 16 16:25:41.799597 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.790095 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 16 16:25:41.799597 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.790097 2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 16 16:25:41.799597 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.790100 2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 16 16:25:41.799597 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.790103 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 16 16:25:41.799597 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.790107 2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 16 16:25:41.799597 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.790111 2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 16 16:25:41.799597 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.790114 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 16 16:25:41.799597 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.790117 2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 16 16:25:41.800151 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.790120 2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 16 16:25:41.800151 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.790123 2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 16 16:25:41.800151 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.790126 2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 16 16:25:41.800151 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.790129 2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 16 16:25:41.800151 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.790132 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 16 16:25:41.800151 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.790134 2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 16 16:25:41.800151 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.790137 2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 16 16:25:41.800151 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.790140 2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 16 16:25:41.800151 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.790143 2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 16 16:25:41.800151 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.790146 2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 16 16:25:41.800151 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.790148 2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 16 16:25:41.800151 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.790151 2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 16 16:25:41.800151 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.790154 2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 16 16:25:41.800151 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.790157 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 16 16:25:41.800151 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.790160 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 16 16:25:41.800151 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.790165 2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 16 16:25:41.800151 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.790168 2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 16 16:25:41.800151 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.790171 2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 16 16:25:41.800151 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.790174 2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 16 16:25:41.800151 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.790177 2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 16 16:25:41.800764 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.790180 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 16 16:25:41.800764 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.790183 2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 16 16:25:41.800764 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.790185 2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 16 16:25:41.800764 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.790188 2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 16 16:25:41.800764 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.790191 2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 16 16:25:41.800764 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.790194 2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 16 16:25:41.800764 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.790197 2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 16 16:25:41.800764 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.790199 2577 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 16 16:25:41.800764 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.790202 2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 16 16:25:41.800764 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.790205 2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 16 16:25:41.800764 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.790207 2577 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 16 16:25:41.800764 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.790210 2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 16 16:25:41.800764 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.790213 2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 16 16:25:41.800764 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.790216 2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 16 16:25:41.800764 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.790219 2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 16 16:25:41.800764 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.790221 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 16 16:25:41.800764 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.790224 2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 16 16:25:41.800764 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.790227 2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 16 16:25:41.800764 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.790230 2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 16 16:25:41.800764 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.790232 2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 16 16:25:41.801286 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.790235 2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 16 16:25:41.801286 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.790238 2577 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 16 16:25:41.801286 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.790240 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 16 16:25:41.801286 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.790243 2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 16 16:25:41.801286 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.790246 2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 16 16:25:41.801286 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.790248 2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 16 16:25:41.801286 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.790251 2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 16 16:25:41.801286 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.790255 2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 16 16:25:41.801286 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.790259 2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 16 16:25:41.801286 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.790262 2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 16 16:25:41.801286 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.790265 2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 16 16:25:41.801286 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.790267 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 16 16:25:41.801286 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.790270 2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 16 16:25:41.801286 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.790273 2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 16 16:25:41.801286 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.790276 2577 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 16 16:25:41.801286 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.790279 2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 16 16:25:41.801286 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.790282 2577 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 16 16:25:41.801286 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.790284 2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 16 16:25:41.801286 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.790287 2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 16 16:25:41.801286 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.790290 2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 16 16:25:41.801825 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.790295 2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 16 16:25:41.801825 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.790298 2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 16 16:25:41.801825 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.790302 2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 16 16:25:41.801825 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:41.790892 2577 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 16 16:25:41.801825 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:41.798577 2577 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 16 16:25:41.801825 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:41.798598 2577 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 16 16:25:41.801825 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:41.800297 2577 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 16 16:25:41.801825 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:41.800325 2577 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 16 16:25:41.801825 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.800398 2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 16 16:25:41.801825 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.800404 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 16 16:25:41.801825 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.800408 2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 16 16:25:41.801825 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.800411 2577 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 16 16:25:41.801825 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.800414 2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 16 16:25:41.801825 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.800418 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 16 16:25:41.801825 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.800421 2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 16 16:25:41.802235 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.800424 2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 16 16:25:41.802235 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.800427 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 16 16:25:41.802235 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.800430 2577 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 16 16:25:41.802235 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.800433 2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 16 16:25:41.802235 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.800436 2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 16 16:25:41.802235 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.800439 2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 16 16:25:41.802235 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.800442 2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 16 16:25:41.802235 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.800447 2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 16 16:25:41.802235 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.800452 2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 16 16:25:41.802235 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.800455 2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 16 16:25:41.802235 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.800471 2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 16 16:25:41.802235 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.800474 2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 16 16:25:41.802235 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.800477 2577 feature_gate.go:328] unrecognized feature gate: Example Jul 16 16:25:41.802235 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.800480 2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 16 16:25:41.802235 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.800484 2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 16 16:25:41.802235 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.800489 2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 16 16:25:41.802235 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.800492 2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 16 16:25:41.802235 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.800496 2577 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 16 16:25:41.802235 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.800499 2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 16 16:25:41.802748 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.800502 2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 16 16:25:41.802748 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.800505 2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 16 16:25:41.802748 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.800508 2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 16 16:25:41.802748 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.800519 2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 16 16:25:41.802748 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.800522 2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 16 16:25:41.802748 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.800525 2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 16 16:25:41.802748 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.800528 2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 16 16:25:41.802748 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.800530 2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 16 16:25:41.802748 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.800533 2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 16 16:25:41.802748 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.800536 2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 16 16:25:41.802748 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.800539 2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 16 16:25:41.802748 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.800542 2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 16 16:25:41.802748 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.800545 2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 16 16:25:41.802748 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.800548 2577 feature_gate.go:328] unrecognized feature gate: Example2 Jul 16 16:25:41.802748 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.800551 2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 16 16:25:41.802748 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.800554 2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 16 16:25:41.802748 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.800557 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 16 16:25:41.802748 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.800560 2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 16 16:25:41.802748 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.800563 2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 16 16:25:41.802748 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.800566 2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 16 16:25:41.803497 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.800568 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 16 16:25:41.803497 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.800571 2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 16 16:25:41.803497 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.800574 2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 16 16:25:41.803497 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.800577 2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 16 16:25:41.803497 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.800579 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 16 16:25:41.803497 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.800582 2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 16 16:25:41.803497 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.800585 2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 16 16:25:41.803497 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.800588 2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 16 16:25:41.803497 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.800590 2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 16 16:25:41.803497 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.800594 2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 16 16:25:41.803497 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.800597 2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 16 16:25:41.803497 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.800600 2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 16 16:25:41.803497 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.800603 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 16 16:25:41.803497 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.800605 2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 16 16:25:41.803497 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.800608 2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 16 16:25:41.803497 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.800611 2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 16 16:25:41.803497 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.800620 2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 16 16:25:41.803497 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.800623 2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 16 16:25:41.803497 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.800628 2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 16 16:25:41.804413 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.800631 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 16 16:25:41.804413 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.800634 2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 16 16:25:41.804413 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.800638 2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 16 16:25:41.804413 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.800641 2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 16 16:25:41.804413 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.800644 2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 16 16:25:41.804413 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.800646 2577 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 16 16:25:41.804413 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.800649 2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 16 16:25:41.804413 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.800652 2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 16 16:25:41.804413 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.800654 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 16 16:25:41.804413 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.800657 2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 16 16:25:41.804413 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.800660 2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 16 16:25:41.804413 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.800662 2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 16 16:25:41.804413 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.800665 2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 16 16:25:41.804413 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.800668 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 16 16:25:41.804413 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.800671 2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 16 16:25:41.804413 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.800673 2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 16 16:25:41.804413 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.800676 2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 16 16:25:41.804413 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.800678 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 16 16:25:41.804413 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.800681 2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 16 16:25:41.804413 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.800684 2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 16 16:25:41.805113 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.800687 2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 16 16:25:41.805113 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.800689 2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 16 16:25:41.805113 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.800692 2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 16 16:25:41.805113 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.800695 2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 16 16:25:41.805113 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.800698 2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 16 16:25:41.805113 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.800701 2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 16 16:25:41.805113 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.800704 2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 16 16:25:41.805113 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.800707 2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 16 16:25:41.805113 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.800710 2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 16 16:25:41.805113 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.800713 2577 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 16 16:25:41.805113 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.800723 2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 16 16:25:41.805113 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.800726 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 16 16:25:41.805113 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.800729 2577 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 16 16:25:41.805113 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.800732 2577 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 16 16:25:41.805113 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:41.800737 2577 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 16 16:25:41.805880 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.800911 2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 16 16:25:41.805880 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.800918 2577 feature_gate.go:328] unrecognized feature gate: Example Jul 16 16:25:41.805880 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.800921 2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 16 16:25:41.805880 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.800924 2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 16 16:25:41.805880 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.800927 2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 16 16:25:41.805880 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.800930 2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 16 16:25:41.805880 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.800933 2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 16 16:25:41.805880 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.800937 2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 16 16:25:41.805880 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.800940 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 16 16:25:41.805880 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.800943 2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 16 16:25:41.805880 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.800946 2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 16 16:25:41.805880 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.800949 2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 16 16:25:41.805880 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.800952 2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 16 16:25:41.805880 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.800954 2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 16 16:25:41.805880 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.800957 2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 16 16:25:41.805880 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.800960 2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 16 16:25:41.805880 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.800963 2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 16 16:25:41.805880 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.800965 2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 16 16:25:41.805880 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.800968 2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 16 16:25:41.805880 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.800971 2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 16 16:25:41.806519 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.800975 2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 16 16:25:41.806519 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.800978 2577 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 16 16:25:41.806519 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.800980 2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 16 16:25:41.806519 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.800983 2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 16 16:25:41.806519 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.800986 2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 16 16:25:41.806519 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.800989 2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 16 16:25:41.806519 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.800992 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 16 16:25:41.806519 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.800995 2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 16 16:25:41.806519 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.801005 2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 16 16:25:41.806519 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.801008 2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 16 16:25:41.806519 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.801010 2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 16 16:25:41.806519 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.801013 2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 16 16:25:41.806519 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.801016 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 16 16:25:41.806519 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.801019 2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 16 16:25:41.806519 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.801021 2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 16 16:25:41.806519 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.801024 2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 16 16:25:41.806519 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.801027 2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 16 16:25:41.806519 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.801030 2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 16 16:25:41.806519 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.801032 2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 16 16:25:41.806519 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.801035 2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 16 16:25:41.807129 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.801038 2577 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 16 16:25:41.807129 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.801040 2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 16 16:25:41.807129 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.801043 2577 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 16 16:25:41.807129 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.801047 2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 16 16:25:41.807129 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.801050 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 16 16:25:41.807129 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.801053 2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 16 16:25:41.807129 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.801056 2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 16 16:25:41.807129 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.801059 2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 16 16:25:41.807129 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.801061 2577 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 16 16:25:41.807129 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.801064 2577 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 16 16:25:41.807129 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.801067 2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 16 16:25:41.807129 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.801069 2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 16 16:25:41.807129 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.801072 2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 16 16:25:41.807129 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.801075 2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 16 16:25:41.807129 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.801078 2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 16 16:25:41.807129 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.801081 2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 16 16:25:41.807129 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.801084 2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 16 16:25:41.807129 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.801089 2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 16 16:25:41.807129 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.801092 2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 16 16:25:41.807654 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.801096 2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 16 16:25:41.807654 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.801099 2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 16 16:25:41.807654 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.801108 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 16 16:25:41.807654 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.801111 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 16 16:25:41.807654 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.801114 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 16 16:25:41.807654 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.801118 2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 16 16:25:41.807654 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.801121 2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 16 16:25:41.807654 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.801124 2577 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 16 16:25:41.807654 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.801127 2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 16 16:25:41.807654 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.801130 2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 16 16:25:41.807654 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.801133 2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 16 16:25:41.807654 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.801135 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 16 16:25:41.807654 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.801138 2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 16 16:25:41.807654 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.801141 2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 16 16:25:41.807654 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.801144 2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 16 16:25:41.807654 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.801146 2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 16 16:25:41.807654 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.801149 2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 16 16:25:41.807654 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.801152 2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 16 16:25:41.807654 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.801154 2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 16 16:25:41.807654 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.801157 2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 16 16:25:41.808238 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.801160 2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 16 16:25:41.808238 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.801163 2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 16 16:25:41.808238 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.801166 2577 feature_gate.go:328] unrecognized feature gate: Example2 Jul 16 16:25:41.808238 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.801169 2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 16 16:25:41.808238 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.801171 2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 16 16:25:41.808238 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.801174 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 16 16:25:41.808238 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.801177 2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 16 16:25:41.808238 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.801180 2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 16 16:25:41.808238 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.801183 2577 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 16 16:25:41.808238 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.801185 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 16 16:25:41.808238 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.801188 2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 16 16:25:41.808238 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.801191 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 16 16:25:41.808238 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.801194 2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 16 16:25:41.808238 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.801197 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 16 16:25:41.808238 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.801199 2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 16 16:25:41.808238 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.801208 2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 16 16:25:41.808238 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.801211 2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 16 16:25:41.808238 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.801213 2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 16 16:25:41.808238 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.801217 2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 16 16:25:41.808238 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:41.801221 2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 16 16:25:41.808858 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:41.801227 2577 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 16 16:25:41.808858 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:41.801396 2577 server.go:962] "Client rotation is on, will bootstrap in background" Jul 16 16:25:41.808858 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:41.806412 2577 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 16 16:25:41.808858 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:41.807700 2577 server.go:1019] "Starting client certificate rotation" Jul 16 16:25:41.808858 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:41.807813 2577 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 16 16:25:41.808858 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:41.807848 2577 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 16 16:25:41.810472 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:41.810435 2577 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 16 16:25:41.811789 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:41.811764 2577 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 16 16:25:41.823348 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:41.823325 2577 log.go:25] "Validated CRI v1 runtime API" Jul 16 16:25:41.828004 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:41.827987 2577 log.go:25] "Validated CRI v1 image API" Jul 16 16:25:41.829749 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:41.829731 2577 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 16 16:25:41.834571 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:41.834543 2577 fs.go:135] Filesystem UUIDs: map[459139d9-343f-4335-8c50-55d708ba1138:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 8b76933c-b60c-4656-8564-572849e80f04:/dev/nvme0n1p3] Jul 16 16:25:41.834571 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:41.834564 2577 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 16 16:25:41.841242 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:41.841112 2577 manager.go:216] Machine: {Timestamp:2026-07-16 16:25:41.83910359 +0000 UTC m=+0.343161434 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3098917 MemoryCapacity:33164496896 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2e86dfe1970e5956dae8e9fa78d2a1 SystemUUID:ec2e86df-e197-0e59-56da-e8e9fa78d2a1 BootID:fab6bce5-c830-46c5-b354-5ffa20aae5c4 Filesystems:[{Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:8a:36:e2:4c:db Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:8a:36:e2:4c:db Speed:0 Mtu:9001} {Name:ovs-system MacAddress:c2:e7:de:ab:28:c0 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164496896 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 16 16:25:41.841242 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:41.841233 2577 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 16 16:25:41.841378 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:41.841333 2577 manager.go:232] Version: {KernelVersion:5.14.0-570.126.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260702-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 16 16:25:41.843761 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:41.843730 2577 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 16 16:25:41.843942 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:41.843765 2577 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-142-101.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 16 16:25:41.844496 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:41.844484 2577 topology_manager.go:138] "Creating topology manager with none policy" Jul 16 16:25:41.844539 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:41.844498 2577 container_manager_linux.go:309] "Creating device plugin manager" Jul 16 16:25:41.844539 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:41.844507 2577 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 16 16:25:41.845180 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:41.845170 2577 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 16 16:25:41.846104 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:41.846093 2577 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 16 16:25:41.849207 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:41.849191 2577 state_mem.go:36] "Initialized new in-memory state store" Jul 16 16:25:41.849366 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:41.849354 2577 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 16 16:25:41.850026 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:41.850014 2577 kubelet.go:486] "Attempting to sync node with API server" Jul 16 16:25:41.850088 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:41.850038 2577 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 16 16:25:41.850088 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:41.850060 2577 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 16 16:25:41.850088 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:41.850072 2577 kubelet.go:398] "Adding apiserver pod source" Jul 16 16:25:41.850088 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:41.850084 2577 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 16 16:25:41.850638 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:41.850621 2577 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 16 16:25:41.850724 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:41.850657 2577 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 16 16:25:41.853844 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:41.853820 2577 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 16 16:25:41.855086 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:41.855064 2577 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 16 16:25:41.855888 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:41.855873 2577 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 16 16:25:41.855944 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:41.855906 2577 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 16 16:25:41.856662 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:41.856649 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 16 16:25:41.856707 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:41.856671 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 16 16:25:41.856707 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:41.856679 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 16 16:25:41.856707 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:41.856686 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 16 16:25:41.856707 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:41.856695 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 16 16:25:41.856707 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:41.856701 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 16 16:25:41.856707 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:41.856710 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 16 16:25:41.856865 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:41.856716 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 16 16:25:41.856865 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:41.856722 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 16 16:25:41.856865 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:41.856728 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 16 16:25:41.856865 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:41.856737 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 16 16:25:41.856865 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:41.856743 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 16 16:25:41.857448 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:41.857438 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 16 16:25:41.857504 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:41.857449 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 16 16:25:41.860926 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:41.860905 2577 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-142-101.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 16 16:25:41.860926 ip-10-0-142-101 kubenswrapper[2577]: E0716 16:25:41.860913 2577 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-142-101.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 16 16:25:41.861227 ip-10-0-142-101 kubenswrapper[2577]: E0716 16:25:41.861206 2577 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 16 16:25:41.861313 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:41.861233 2577 server.go:1268] "Started kubelet" Jul 16 16:25:41.861313 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:41.861289 2577 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 16 16:25:41.861594 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:41.861475 2577 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 16 16:25:41.861672 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:41.861625 2577 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 16 16:25:41.862231 ip-10-0-142-101 systemd[1]: Started Kubernetes Kubelet. Jul 16 16:25:41.863568 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:41.863529 2577 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 16 16:25:41.863781 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:41.863766 2577 server.go:310] "Adding debug handlers to kubelet server" Jul 16 16:25:41.867725 ip-10-0-142-101 kubenswrapper[2577]: E0716 16:25:41.866805 2577 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-142-101.ec2.internal.18c2d1db3816e56c default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-142-101.ec2.internal,UID:ip-10-0-142-101.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-142-101.ec2.internal,},FirstTimestamp:2026-07-16 16:25:41.861205356 +0000 UTC m=+0.365263197,LastTimestamp:2026-07-16 16:25:41.861205356 +0000 UTC m=+0.365263197,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-142-101.ec2.internal,}" Jul 16 16:25:41.868374 ip-10-0-142-101 kubenswrapper[2577]: E0716 16:25:41.868345 2577 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 16 16:25:41.869690 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:41.869665 2577 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 16 16:25:41.869690 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:41.869682 2577 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 16 16:25:41.870014 ip-10-0-142-101 kubenswrapper[2577]: E0716 16:25:41.869994 2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-101.ec2.internal\" not found" Jul 16 16:25:41.870430 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:41.870398 2577 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 16 16:25:41.870430 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:41.870412 2577 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 16 16:25:41.870430 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:41.870423 2577 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 16 16:25:41.870609 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:41.870517 2577 reconstruct.go:97] "Volume reconstruction finished" Jul 16 16:25:41.870609 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:41.870525 2577 reconciler.go:29] "Reconciler: start to sync state" Jul 16 16:25:41.870825 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:41.870806 2577 factory.go:55] Registering systemd factory Jul 16 16:25:41.870825 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:41.870826 2577 factory.go:223] Registration of the systemd container factory successfully Jul 16 16:25:41.871120 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:41.871102 2577 factory.go:153] Registering CRI-O factory Jul 16 16:25:41.871182 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:41.871127 2577 factory.go:223] Registration of the crio container factory successfully Jul 16 16:25:41.871247 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:41.871235 2577 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 16 16:25:41.871284 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:41.871266 2577 factory.go:103] Registering Raw factory Jul 16 16:25:41.871332 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:41.871299 2577 manager.go:1196] Started watching for new ooms in manager Jul 16 16:25:41.871825 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:41.871769 2577 manager.go:349] Starting recovery of all containers Jul 16 16:25:41.872667 ip-10-0-142-101 kubenswrapper[2577]: E0716 16:25:41.872643 2577 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 16 16:25:41.872920 ip-10-0-142-101 kubenswrapper[2577]: E0716 16:25:41.872889 2577 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-142-101.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 16 16:25:41.884949 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:41.884926 2577 manager.go:354] Recovery completed Jul 16 16:25:41.889453 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:41.889440 2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 16 16:25:41.891665 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:41.891640 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-101.ec2.internal" event="NodeHasSufficientMemory" Jul 16 16:25:41.891765 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:41.891682 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-101.ec2.internal" event="NodeHasNoDiskPressure" Jul 16 16:25:41.891765 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:41.891695 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-101.ec2.internal" event="NodeHasSufficientPID" Jul 16 16:25:41.892373 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:41.892357 2577 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 16 16:25:41.892420 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:41.892373 2577 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 16 16:25:41.892420 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:41.892390 2577 state_mem.go:36] "Initialized new in-memory state store" Jul 16 16:25:41.892420 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:41.892410 2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 16 16:25:41.893215 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:41.893197 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-101.ec2.internal" event="NodeHasSufficientMemory" Jul 16 16:25:41.893302 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:41.893229 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-101.ec2.internal" event="NodeHasNoDiskPressure" Jul 16 16:25:41.893302 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:41.893240 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-101.ec2.internal" event="NodeHasSufficientPID" Jul 16 16:25:41.893744 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:41.893731 2577 policy_none.go:49] "None policy: Start" Jul 16 16:25:41.893797 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:41.893759 2577 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 16 16:25:41.893797 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:41.893772 2577 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 16 16:25:41.894772 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:41.894761 2577 policy_none.go:47] "Start" Jul 16 16:25:41.902898 ip-10-0-142-101 kubenswrapper[2577]: E0716 16:25:41.902654 2577 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-142-101.ec2.internal.18c2d1db39e7a844 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-142-101.ec2.internal,UID:ip-10-0-142-101.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-142-101.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-142-101.ec2.internal,},FirstTimestamp:2026-07-16 16:25:41.89166394 +0000 UTC m=+0.395721789,LastTimestamp:2026-07-16 16:25:41.89166394 +0000 UTC m=+0.395721789,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-142-101.ec2.internal,}" Jul 16 16:25:41.903710 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:41.903682 2577 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-qjznx" Jul 16 16:25:41.913806 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:41.913777 2577 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-qjznx" Jul 16 16:25:41.914329 ip-10-0-142-101 kubenswrapper[2577]: E0716 16:25:41.914247 2577 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-142-101.ec2.internal.18c2d1db39e807af default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-142-101.ec2.internal,UID:ip-10-0-142-101.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-142-101.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-142-101.ec2.internal,},FirstTimestamp:2026-07-16 16:25:41.891688367 +0000 UTC m=+0.395746212,LastTimestamp:2026-07-16 16:25:41.891688367 +0000 UTC m=+0.395746212,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-142-101.ec2.internal,}" Jul 16 16:25:41.935185 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:41.935164 2577 manager.go:337] "Starting Device Plugin manager" Jul 16 16:25:41.935285 ip-10-0-142-101 kubenswrapper[2577]: E0716 16:25:41.935214 2577 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 16 16:25:41.935285 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:41.935229 2577 server.go:87] "Starting device plugin registration server" Jul 16 16:25:41.935581 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:41.935565 2577 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 16 16:25:41.935638 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:41.935587 2577 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 16 16:25:41.935727 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:41.935703 2577 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 16 16:25:41.935853 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:41.935785 2577 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 16 16:25:41.935853 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:41.935793 2577 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 16 16:25:41.936476 ip-10-0-142-101 kubenswrapper[2577]: E0716 16:25:41.936455 2577 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 16 16:25:41.936567 ip-10-0-142-101 kubenswrapper[2577]: E0716 16:25:41.936513 2577 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-142-101.ec2.internal\" not found" Jul 16 16:25:42.009774 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:42.009725 2577 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 16 16:25:42.011145 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:42.011076 2577 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 16 16:25:42.011145 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:42.011118 2577 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 16 16:25:42.011145 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:42.011145 2577 kubelet.go:2448] "Starting kubelet main sync loop" Jul 16 16:25:42.011316 ip-10-0-142-101 kubenswrapper[2577]: E0716 16:25:42.011182 2577 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 16 16:25:42.017023 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:42.016998 2577 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 16 16:25:42.036432 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:42.036343 2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 16 16:25:42.037473 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:42.037454 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-101.ec2.internal" event="NodeHasSufficientMemory" Jul 16 16:25:42.037545 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:42.037506 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-101.ec2.internal" event="NodeHasNoDiskPressure" Jul 16 16:25:42.037545 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:42.037523 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-101.ec2.internal" event="NodeHasSufficientPID" Jul 16 16:25:42.037629 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:42.037550 2577 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-142-101.ec2.internal" Jul 16 16:25:42.057939 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:42.057912 2577 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-142-101.ec2.internal" Jul 16 16:25:42.058016 ip-10-0-142-101 kubenswrapper[2577]: E0716 16:25:42.057944 2577 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-142-101.ec2.internal\": node \"ip-10-0-142-101.ec2.internal\" not found" Jul 16 16:25:42.111315 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:42.111265 2577 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-101.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-142-101.ec2.internal"] Jul 16 16:25:42.111547 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:42.111365 2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 16 16:25:42.112698 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:42.112673 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-101.ec2.internal" event="NodeHasSufficientMemory" Jul 16 16:25:42.112889 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:42.112713 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-101.ec2.internal" event="NodeHasNoDiskPressure" Jul 16 16:25:42.112889 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:42.112727 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-101.ec2.internal" event="NodeHasSufficientPID" Jul 16 16:25:42.114346 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:42.114327 2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 16 16:25:42.114490 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:42.114417 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-101.ec2.internal" Jul 16 16:25:42.114490 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:42.114452 2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 16 16:25:42.116187 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:42.116171 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-101.ec2.internal" event="NodeHasSufficientMemory" Jul 16 16:25:42.116288 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:42.116201 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-101.ec2.internal" event="NodeHasNoDiskPressure" Jul 16 16:25:42.116288 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:42.116212 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-101.ec2.internal" event="NodeHasSufficientPID" Jul 16 16:25:42.116288 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:42.116170 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-101.ec2.internal" event="NodeHasSufficientMemory" Jul 16 16:25:42.116288 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:42.116279 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-101.ec2.internal" event="NodeHasNoDiskPressure" Jul 16 16:25:42.116288 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:42.116289 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-101.ec2.internal" event="NodeHasSufficientPID" Jul 16 16:25:42.117900 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:42.117884 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-101.ec2.internal" Jul 16 16:25:42.117948 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:42.117914 2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 16 16:25:42.118785 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:42.118770 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-101.ec2.internal" event="NodeHasSufficientMemory" Jul 16 16:25:42.118844 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:42.118801 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-101.ec2.internal" event="NodeHasNoDiskPressure" Jul 16 16:25:42.118844 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:42.118812 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-101.ec2.internal" event="NodeHasSufficientPID" Jul 16 16:25:42.131665 ip-10-0-142-101 kubenswrapper[2577]: E0716 16:25:42.131640 2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-101.ec2.internal\" not found" Jul 16 16:25:42.144464 ip-10-0-142-101 kubenswrapper[2577]: E0716 16:25:42.144435 2577 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-142-101.ec2.internal\" not found" node="ip-10-0-142-101.ec2.internal" Jul 16 16:25:42.149122 ip-10-0-142-101 kubenswrapper[2577]: E0716 16:25:42.149102 2577 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-142-101.ec2.internal\" not found" node="ip-10-0-142-101.ec2.internal" Jul 16 16:25:42.171831 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:42.171795 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f0ff4a720fce2411438eb201c85e2744-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-101.ec2.internal\" (UID: \"f0ff4a720fce2411438eb201c85e2744\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-101.ec2.internal" Jul 16 16:25:42.171831 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:42.171832 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f0ff4a720fce2411438eb201c85e2744-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-101.ec2.internal\" (UID: \"f0ff4a720fce2411438eb201c85e2744\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-101.ec2.internal" Jul 16 16:25:42.172045 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:42.171849 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/077fca01a64486b516cac217b95ed2fa-config\") pod \"kube-apiserver-proxy-ip-10-0-142-101.ec2.internal\" (UID: \"077fca01a64486b516cac217b95ed2fa\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-101.ec2.internal" Jul 16 16:25:42.232248 ip-10-0-142-101 kubenswrapper[2577]: E0716 16:25:42.232216 2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-101.ec2.internal\" not found" Jul 16 16:25:42.272545 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:42.272513 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f0ff4a720fce2411438eb201c85e2744-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-101.ec2.internal\" (UID: \"f0ff4a720fce2411438eb201c85e2744\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-101.ec2.internal" Jul 16 16:25:42.272622 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:42.272477 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f0ff4a720fce2411438eb201c85e2744-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-101.ec2.internal\" (UID: \"f0ff4a720fce2411438eb201c85e2744\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-101.ec2.internal" Jul 16 16:25:42.272622 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:42.272600 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f0ff4a720fce2411438eb201c85e2744-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-101.ec2.internal\" (UID: \"f0ff4a720fce2411438eb201c85e2744\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-101.ec2.internal" Jul 16 16:25:42.272714 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:42.272622 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/077fca01a64486b516cac217b95ed2fa-config\") pod \"kube-apiserver-proxy-ip-10-0-142-101.ec2.internal\" (UID: \"077fca01a64486b516cac217b95ed2fa\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-101.ec2.internal" Jul 16 16:25:42.272714 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:42.272649 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/077fca01a64486b516cac217b95ed2fa-config\") pod \"kube-apiserver-proxy-ip-10-0-142-101.ec2.internal\" (UID: \"077fca01a64486b516cac217b95ed2fa\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-101.ec2.internal" Jul 16 16:25:42.272714 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:42.272675 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f0ff4a720fce2411438eb201c85e2744-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-101.ec2.internal\" (UID: \"f0ff4a720fce2411438eb201c85e2744\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-101.ec2.internal" Jul 16 16:25:42.332887 ip-10-0-142-101 kubenswrapper[2577]: E0716 16:25:42.332796 2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-101.ec2.internal\" not found" Jul 16 16:25:42.434099 ip-10-0-142-101 kubenswrapper[2577]: E0716 16:25:42.434058 2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-101.ec2.internal\" not found" Jul 16 16:25:42.446146 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:42.446111 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-101.ec2.internal" Jul 16 16:25:42.452472 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:42.452441 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-101.ec2.internal" Jul 16 16:25:42.534502 ip-10-0-142-101 kubenswrapper[2577]: E0716 16:25:42.534443 2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-101.ec2.internal\" not found" Jul 16 16:25:42.638615 ip-10-0-142-101 kubenswrapper[2577]: E0716 16:25:42.638509 2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-101.ec2.internal\" not found" Jul 16 16:25:42.739020 ip-10-0-142-101 kubenswrapper[2577]: E0716 16:25:42.738972 2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-101.ec2.internal\" not found" Jul 16 16:25:42.803529 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:42.803492 2577 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 16 16:25:42.807180 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:42.807156 2577 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 16 16:25:42.807322 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:42.807294 2577 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 16 16:25:42.807322 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:42.807309 2577 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 16 16:25:42.850851 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:42.850819 2577 apiserver.go:52] "Watching apiserver" Jul 16 16:25:42.869835 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:42.869790 2577 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 16 16:25:42.869835 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:42.869832 2577 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-101.ec2.internal" Jul 16 16:25:42.887292 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:42.887258 2577 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 16 16:25:42.887608 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:42.887589 2577 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 16 16:25:42.888737 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:42.888668 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-nrzr6","openshift-network-diagnostics/network-check-target-ntktc","openshift-ovn-kubernetes/ovnkube-node-9bjst","kube-system/konnectivity-agent-xljrg","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t4w26","openshift-cluster-node-tuning-operator/tuned-8pl5s","openshift-dns/node-resolver-6sc67","openshift-image-registry/node-ca-hwq4x","openshift-multus/network-metrics-daemon-bcb7f","openshift-network-operator/iptables-alerter-cnft7","kube-system/global-pull-secret-syncer-rqzn4","openshift-multus/multus-additional-cni-plugins-xz7fp"] Jul 16 16:25:42.890358 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:42.890335 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-nrzr6" Jul 16 16:25:42.891656 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:42.891627 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ntktc" Jul 16 16:25:42.891765 ip-10-0-142-101 kubenswrapper[2577]: E0716 16:25:42.891734 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ntktc" podUID="84cc596d-4f15-4c84-8625-c7f37ed9dae6" Jul 16 16:25:42.893177 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:42.893158 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-9bjst" Jul 16 16:25:42.893669 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:42.893648 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-nzn5f\"" Jul 16 16:25:42.893861 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:42.893848 2577 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 16 16:25:42.893913 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:42.893868 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 16 16:25:42.893948 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:42.893936 2577 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-101.ec2.internal" Jul 16 16:25:42.894079 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:42.894058 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 16 16:25:42.894177 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:42.894161 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 16 16:25:42.894260 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:42.894246 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 16 16:25:42.894490 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:42.894472 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-xljrg" Jul 16 16:25:42.895893 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:42.895875 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t4w26" Jul 16 16:25:42.897254 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:42.897231 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-8pl5s" Jul 16 16:25:42.898558 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:42.898536 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-6sc67" Jul 16 16:25:42.900058 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:42.900039 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-hwq4x" Jul 16 16:25:42.901551 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:42.901532 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bcb7f" Jul 16 16:25:42.901684 ip-10-0-142-101 kubenswrapper[2577]: E0716 16:25:42.901601 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bcb7f" podUID="94ec201c-7285-4751-be16-d23d2f2c0290" Jul 16 16:25:42.904374 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:42.904336 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-cnft7" Jul 16 16:25:42.904672 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:42.904638 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 16 16:25:42.904781 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:42.904763 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 16 16:25:42.905201 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:42.905045 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 16 16:25:42.905201 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:42.905121 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 16 16:25:42.905354 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:42.905212 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 16 16:25:42.905787 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:42.905593 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 16 16:25:42.905787 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:42.905635 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 16 16:25:42.905787 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:42.905641 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 16 16:25:42.905787 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:42.905665 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 16 16:25:42.905787 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:42.905683 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 16 16:25:42.905787 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:42.905714 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-r4z2l\"" Jul 16 16:25:42.905787 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:42.905601 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-gpjqx\"" Jul 16 16:25:42.906092 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:42.906031 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 16 16:25:42.906092 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:42.906054 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 16 16:25:42.906092 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:42.906061 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 16 16:25:42.906092 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:42.906089 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 16 16:25:42.906328 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:42.906308 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 16 16:25:42.906390 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:42.906339 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-768cx\"" Jul 16 16:25:42.906390 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:42.906360 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-xcll2\"" Jul 16 16:25:42.906514 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:42.906361 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 16 16:25:42.906514 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:42.906438 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-2zpt9\"" Jul 16 16:25:42.906514 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:42.906363 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 16 16:25:42.906705 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:42.906682 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 16 16:25:42.906944 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:42.906926 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 16 16:25:42.907009 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:42.906968 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 16 16:25:42.907083 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:42.907055 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 16 16:25:42.907145 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:42.907105 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-s2fd4\"" Jul 16 16:25:42.907257 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:42.907239 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-bgmcf\"" Jul 16 16:25:42.908421 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:42.908040 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rqzn4" Jul 16 16:25:42.908421 ip-10-0-142-101 kubenswrapper[2577]: E0716 16:25:42.908251 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rqzn4" podUID="e2d675d8-edf6-4765-a240-4b723cdd2d38" Jul 16 16:25:42.912025 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:42.912003 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-101.ec2.internal"] Jul 16 16:25:42.912134 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:42.912114 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-xz7fp" Jul 16 16:25:42.915371 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:42.915349 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 16 16:25:42.916475 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:42.916444 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 16 16:25:42.916828 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:42.916812 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-tg6vv\"" Jul 16 16:25:42.917894 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:42.917857 2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-15 16:20:42 +0000 UTC" deadline="2028-02-07 11:20:02.605208139 +0000 UTC" Jul 16 16:25:42.917894 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:42.917893 2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13698h54m19.687317415s" Jul 16 16:25:42.920148 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:42.920129 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-142-101.ec2.internal"] Jul 16 16:25:42.924023 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:42.923999 2577 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 16 16:25:42.960338 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:42.960314 2577 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-cxtvs" Jul 16 16:25:42.972113 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:42.972085 2577 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 16 16:25:42.975296 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:42.975275 2577 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-cxtvs" Jul 16 16:25:42.976881 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:42.976861 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8f44aa64-f94f-4a4e-a24f-b6a231b6df4b-cnibin\") pod \"multus-nrzr6\" (UID: \"8f44aa64-f94f-4a4e-a24f-b6a231b6df4b\") " pod="openshift-multus/multus-nrzr6" Jul 16 16:25:42.976975 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:42.976888 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/b20b154f-1a4c-4b25-86db-aebe2d4ffc08-ovnkube-script-lib\") pod \"ovnkube-node-9bjst\" (UID: \"b20b154f-1a4c-4b25-86db-aebe2d4ffc08\") " pod="openshift-ovn-kubernetes/ovnkube-node-9bjst" Jul 16 16:25:42.976975 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:42.976906 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/aa5f6bf8-3599-45f1-a09d-ef1976f22840-registration-dir\") pod \"aws-ebs-csi-driver-node-t4w26\" (UID: \"aa5f6bf8-3599-45f1-a09d-ef1976f22840\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t4w26" Jul 16 16:25:42.976975 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:42.976922 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/aa5f6bf8-3599-45f1-a09d-ef1976f22840-etc-selinux\") pod \"aws-ebs-csi-driver-node-t4w26\" (UID: \"aa5f6bf8-3599-45f1-a09d-ef1976f22840\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t4w26" Jul 16 16:25:42.976975 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:42.976939 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/93f30808-1a75-44b5-801f-8faeceb72afb-tmp-dir\") pod \"node-resolver-6sc67\" (UID: \"93f30808-1a75-44b5-801f-8faeceb72afb\") " pod="openshift-dns/node-resolver-6sc67" Jul 16 16:25:42.976975 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:42.976966 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7bbtp\" (UniqueName: \"kubernetes.io/projected/cbd027ac-97d9-4769-a867-47b7b8256cd0-kube-api-access-7bbtp\") pod \"node-ca-hwq4x\" (UID: \"cbd027ac-97d9-4769-a867-47b7b8256cd0\") " pod="openshift-image-registry/node-ca-hwq4x" Jul 16 16:25:42.977251 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:42.977005 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/b2410943-db6a-4668-845d-db0ae081a389-os-release\") pod \"multus-additional-cni-plugins-xz7fp\" (UID: \"b2410943-db6a-4668-845d-db0ae081a389\") " pod="openshift-multus/multus-additional-cni-plugins-xz7fp" Jul 16 16:25:42.977251 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:42.977060 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8f44aa64-f94f-4a4e-a24f-b6a231b6df4b-host-var-lib-cni-bin\") pod \"multus-nrzr6\" (UID: \"8f44aa64-f94f-4a4e-a24f-b6a231b6df4b\") " pod="openshift-multus/multus-nrzr6" Jul 16 16:25:42.977251 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:42.977083 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/8f44aa64-f94f-4a4e-a24f-b6a231b6df4b-host-var-lib-cni-multus\") pod \"multus-nrzr6\" (UID: \"8f44aa64-f94f-4a4e-a24f-b6a231b6df4b\") " pod="openshift-multus/multus-nrzr6" Jul 16 16:25:42.977251 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:42.977108 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/8f44aa64-f94f-4a4e-a24f-b6a231b6df4b-host-run-multus-certs\") pod \"multus-nrzr6\" (UID: \"8f44aa64-f94f-4a4e-a24f-b6a231b6df4b\") " pod="openshift-multus/multus-nrzr6" Jul 16 16:25:42.977251 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:42.977164 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ba81ab34-5cc0-4f3a-856a-ff794d0dc0e5-etc-modprobe-d\") pod \"tuned-8pl5s\" (UID: \"ba81ab34-5cc0-4f3a-856a-ff794d0dc0e5\") " pod="openshift-cluster-node-tuning-operator/tuned-8pl5s" Jul 16 16:25:42.977251 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:42.977206 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ba81ab34-5cc0-4f3a-856a-ff794d0dc0e5-sys\") pod \"tuned-8pl5s\" (UID: \"ba81ab34-5cc0-4f3a-856a-ff794d0dc0e5\") " pod="openshift-cluster-node-tuning-operator/tuned-8pl5s" Jul 16 16:25:42.977251 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:42.977235 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ba81ab34-5cc0-4f3a-856a-ff794d0dc0e5-etc-tuned\") pod \"tuned-8pl5s\" (UID: \"ba81ab34-5cc0-4f3a-856a-ff794d0dc0e5\") " pod="openshift-cluster-node-tuning-operator/tuned-8pl5s" Jul 16 16:25:42.977627 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:42.977262 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/93f30808-1a75-44b5-801f-8faeceb72afb-hosts-file\") pod \"node-resolver-6sc67\" (UID: \"93f30808-1a75-44b5-801f-8faeceb72afb\") " pod="openshift-dns/node-resolver-6sc67" Jul 16 16:25:42.977627 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:42.977287 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b2410943-db6a-4668-845d-db0ae081a389-tuning-conf-dir\") pod \"multus-additional-cni-plugins-xz7fp\" (UID: \"b2410943-db6a-4668-845d-db0ae081a389\") " pod="openshift-multus/multus-additional-cni-plugins-xz7fp" Jul 16 16:25:42.977627 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:42.977315 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8f44aa64-f94f-4a4e-a24f-b6a231b6df4b-multus-conf-dir\") pod \"multus-nrzr6\" (UID: \"8f44aa64-f94f-4a4e-a24f-b6a231b6df4b\") " pod="openshift-multus/multus-nrzr6" Jul 16 16:25:42.977627 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:42.977338 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8f44aa64-f94f-4a4e-a24f-b6a231b6df4b-etc-kubernetes\") pod \"multus-nrzr6\" (UID: \"8f44aa64-f94f-4a4e-a24f-b6a231b6df4b\") " pod="openshift-multus/multus-nrzr6" Jul 16 16:25:42.977627 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:42.977362 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b20b154f-1a4c-4b25-86db-aebe2d4ffc08-var-lib-openvswitch\") pod \"ovnkube-node-9bjst\" (UID: \"b20b154f-1a4c-4b25-86db-aebe2d4ffc08\") " pod="openshift-ovn-kubernetes/ovnkube-node-9bjst" Jul 16 16:25:42.977627 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:42.977385 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e2d675d8-edf6-4765-a240-4b723cdd2d38-kubelet-config\") pod \"global-pull-secret-syncer-rqzn4\" (UID: \"e2d675d8-edf6-4765-a240-4b723cdd2d38\") " pod="kube-system/global-pull-secret-syncer-rqzn4" Jul 16 16:25:42.977627 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:42.977408 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b2410943-db6a-4668-845d-db0ae081a389-cnibin\") pod \"multus-additional-cni-plugins-xz7fp\" (UID: \"b2410943-db6a-4668-845d-db0ae081a389\") " pod="openshift-multus/multus-additional-cni-plugins-xz7fp" Jul 16 16:25:42.977627 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:42.977448 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8f44aa64-f94f-4a4e-a24f-b6a231b6df4b-system-cni-dir\") pod \"multus-nrzr6\" (UID: \"8f44aa64-f94f-4a4e-a24f-b6a231b6df4b\") " pod="openshift-multus/multus-nrzr6" Jul 16 16:25:42.977627 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:42.977482 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b20b154f-1a4c-4b25-86db-aebe2d4ffc08-host-cni-bin\") pod \"ovnkube-node-9bjst\" (UID: \"b20b154f-1a4c-4b25-86db-aebe2d4ffc08\") " pod="openshift-ovn-kubernetes/ovnkube-node-9bjst" Jul 16 16:25:42.977627 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:42.977512 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b20b154f-1a4c-4b25-86db-aebe2d4ffc08-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-9bjst\" (UID: \"b20b154f-1a4c-4b25-86db-aebe2d4ffc08\") " pod="openshift-ovn-kubernetes/ovnkube-node-9bjst" Jul 16 16:25:42.977627 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:42.977539 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ba81ab34-5cc0-4f3a-856a-ff794d0dc0e5-etc-sysctl-d\") pod \"tuned-8pl5s\" (UID: \"ba81ab34-5cc0-4f3a-856a-ff794d0dc0e5\") " pod="openshift-cluster-node-tuning-operator/tuned-8pl5s" Jul 16 16:25:42.977627 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:42.977576 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ba81ab34-5cc0-4f3a-856a-ff794d0dc0e5-lib-modules\") pod \"tuned-8pl5s\" (UID: \"ba81ab34-5cc0-4f3a-856a-ff794d0dc0e5\") " pod="openshift-cluster-node-tuning-operator/tuned-8pl5s" Jul 16 16:25:42.977627 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:42.977602 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8f44aa64-f94f-4a4e-a24f-b6a231b6df4b-host-var-lib-kubelet\") pod \"multus-nrzr6\" (UID: \"8f44aa64-f94f-4a4e-a24f-b6a231b6df4b\") " pod="openshift-multus/multus-nrzr6" Jul 16 16:25:42.977627 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:42.977626 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/8f44aa64-f94f-4a4e-a24f-b6a231b6df4b-multus-daemon-config\") pod \"multus-nrzr6\" (UID: \"8f44aa64-f94f-4a4e-a24f-b6a231b6df4b\") " pod="openshift-multus/multus-nrzr6" Jul 16 16:25:42.978293 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:42.977655 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w9lm2\" (UniqueName: \"kubernetes.io/projected/8f44aa64-f94f-4a4e-a24f-b6a231b6df4b-kube-api-access-w9lm2\") pod \"multus-nrzr6\" (UID: \"8f44aa64-f94f-4a4e-a24f-b6a231b6df4b\") " pod="openshift-multus/multus-nrzr6" Jul 16 16:25:42.978293 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:42.977681 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b20b154f-1a4c-4b25-86db-aebe2d4ffc08-host-slash\") pod \"ovnkube-node-9bjst\" (UID: \"b20b154f-1a4c-4b25-86db-aebe2d4ffc08\") " pod="openshift-ovn-kubernetes/ovnkube-node-9bjst" Jul 16 16:25:42.978293 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:42.977704 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b20b154f-1a4c-4b25-86db-aebe2d4ffc08-host-cni-netd\") pod \"ovnkube-node-9bjst\" (UID: \"b20b154f-1a4c-4b25-86db-aebe2d4ffc08\") " pod="openshift-ovn-kubernetes/ovnkube-node-9bjst" Jul 16 16:25:42.978293 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:42.977736 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8f44aa64-f94f-4a4e-a24f-b6a231b6df4b-host-run-netns\") pod \"multus-nrzr6\" (UID: \"8f44aa64-f94f-4a4e-a24f-b6a231b6df4b\") " pod="openshift-multus/multus-nrzr6" Jul 16 16:25:42.978293 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:42.977759 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ba81ab34-5cc0-4f3a-856a-ff794d0dc0e5-run\") pod \"tuned-8pl5s\" (UID: \"ba81ab34-5cc0-4f3a-856a-ff794d0dc0e5\") " pod="openshift-cluster-node-tuning-operator/tuned-8pl5s" Jul 16 16:25:42.978293 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:42.977783 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/cbd027ac-97d9-4769-a867-47b7b8256cd0-serviceca\") pod \"node-ca-hwq4x\" (UID: \"cbd027ac-97d9-4769-a867-47b7b8256cd0\") " pod="openshift-image-registry/node-ca-hwq4x" Jul 16 16:25:42.978293 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:42.977843 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3062d489-a9b7-43fb-b1d8-4e39d30acdec-host-slash\") pod \"iptables-alerter-cnft7\" (UID: \"3062d489-a9b7-43fb-b1d8-4e39d30acdec\") " pod="openshift-network-operator/iptables-alerter-cnft7" Jul 16 16:25:42.978293 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:42.977871 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b2410943-db6a-4668-845d-db0ae081a389-system-cni-dir\") pod \"multus-additional-cni-plugins-xz7fp\" (UID: \"b2410943-db6a-4668-845d-db0ae081a389\") " pod="openshift-multus/multus-additional-cni-plugins-xz7fp" Jul 16 16:25:42.978293 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:42.977899 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j27b9\" (UniqueName: \"kubernetes.io/projected/b20b154f-1a4c-4b25-86db-aebe2d4ffc08-kube-api-access-j27b9\") pod \"ovnkube-node-9bjst\" (UID: \"b20b154f-1a4c-4b25-86db-aebe2d4ffc08\") " pod="openshift-ovn-kubernetes/ovnkube-node-9bjst" Jul 16 16:25:42.978293 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:42.977924 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ba81ab34-5cc0-4f3a-856a-ff794d0dc0e5-etc-sysctl-conf\") pod \"tuned-8pl5s\" (UID: \"ba81ab34-5cc0-4f3a-856a-ff794d0dc0e5\") " pod="openshift-cluster-node-tuning-operator/tuned-8pl5s" Jul 16 16:25:42.978293 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:42.977952 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e2d675d8-edf6-4765-a240-4b723cdd2d38-original-pull-secret\") pod \"global-pull-secret-syncer-rqzn4\" (UID: \"e2d675d8-edf6-4765-a240-4b723cdd2d38\") " pod="kube-system/global-pull-secret-syncer-rqzn4" Jul 16 16:25:42.978293 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:42.977984 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/8f44aa64-f94f-4a4e-a24f-b6a231b6df4b-os-release\") pod \"multus-nrzr6\" (UID: \"8f44aa64-f94f-4a4e-a24f-b6a231b6df4b\") " pod="openshift-multus/multus-nrzr6" Jul 16 16:25:42.978293 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:42.978025 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b20b154f-1a4c-4b25-86db-aebe2d4ffc08-host-kubelet\") pod \"ovnkube-node-9bjst\" (UID: \"b20b154f-1a4c-4b25-86db-aebe2d4ffc08\") " pod="openshift-ovn-kubernetes/ovnkube-node-9bjst" Jul 16 16:25:42.978293 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:42.978063 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b20b154f-1a4c-4b25-86db-aebe2d4ffc08-host-run-netns\") pod \"ovnkube-node-9bjst\" (UID: \"b20b154f-1a4c-4b25-86db-aebe2d4ffc08\") " pod="openshift-ovn-kubernetes/ovnkube-node-9bjst" Jul 16 16:25:42.978293 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:42.978098 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b20b154f-1a4c-4b25-86db-aebe2d4ffc08-run-openvswitch\") pod \"ovnkube-node-9bjst\" (UID: \"b20b154f-1a4c-4b25-86db-aebe2d4ffc08\") " pod="openshift-ovn-kubernetes/ovnkube-node-9bjst" Jul 16 16:25:42.978293 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:42.978125 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b20b154f-1a4c-4b25-86db-aebe2d4ffc08-run-ovn\") pod \"ovnkube-node-9bjst\" (UID: \"b20b154f-1a4c-4b25-86db-aebe2d4ffc08\") " pod="openshift-ovn-kubernetes/ovnkube-node-9bjst" Jul 16 16:25:42.978293 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:42.978150 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/b20b154f-1a4c-4b25-86db-aebe2d4ffc08-ovnkube-config\") pod \"ovnkube-node-9bjst\" (UID: \"b20b154f-1a4c-4b25-86db-aebe2d4ffc08\") " pod="openshift-ovn-kubernetes/ovnkube-node-9bjst" Jul 16 16:25:42.979112 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:42.978175 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/8f44aa64-f94f-4a4e-a24f-b6a231b6df4b-host-run-k8s-cni-cncf-io\") pod \"multus-nrzr6\" (UID: \"8f44aa64-f94f-4a4e-a24f-b6a231b6df4b\") " pod="openshift-multus/multus-nrzr6" Jul 16 16:25:42.979112 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:42.978206 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/aa5f6bf8-3599-45f1-a09d-ef1976f22840-socket-dir\") pod \"aws-ebs-csi-driver-node-t4w26\" (UID: \"aa5f6bf8-3599-45f1-a09d-ef1976f22840\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t4w26" Jul 16 16:25:42.979112 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:42.978233 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/aa5f6bf8-3599-45f1-a09d-ef1976f22840-device-dir\") pod \"aws-ebs-csi-driver-node-t4w26\" (UID: \"aa5f6bf8-3599-45f1-a09d-ef1976f22840\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t4w26" Jul 16 16:25:42.979112 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:42.978260 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ffgxf\" (UniqueName: \"kubernetes.io/projected/93f30808-1a75-44b5-801f-8faeceb72afb-kube-api-access-ffgxf\") pod \"node-resolver-6sc67\" (UID: \"93f30808-1a75-44b5-801f-8faeceb72afb\") " pod="openshift-dns/node-resolver-6sc67" Jul 16 16:25:42.979112 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:42.978289 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/b2410943-db6a-4668-845d-db0ae081a389-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-xz7fp\" (UID: \"b2410943-db6a-4668-845d-db0ae081a389\") " pod="openshift-multus/multus-additional-cni-plugins-xz7fp" Jul 16 16:25:42.979112 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:42.978314 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/b20b154f-1a4c-4b25-86db-aebe2d4ffc08-ovn-node-metrics-cert\") pod \"ovnkube-node-9bjst\" (UID: \"b20b154f-1a4c-4b25-86db-aebe2d4ffc08\") " pod="openshift-ovn-kubernetes/ovnkube-node-9bjst" Jul 16 16:25:42.979112 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:42.978348 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ba81ab34-5cc0-4f3a-856a-ff794d0dc0e5-host\") pod \"tuned-8pl5s\" (UID: \"ba81ab34-5cc0-4f3a-856a-ff794d0dc0e5\") " pod="openshift-cluster-node-tuning-operator/tuned-8pl5s" Jul 16 16:25:42.979112 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:42.978374 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b2410943-db6a-4668-845d-db0ae081a389-cni-binary-copy\") pod \"multus-additional-cni-plugins-xz7fp\" (UID: \"b2410943-db6a-4668-845d-db0ae081a389\") " pod="openshift-multus/multus-additional-cni-plugins-xz7fp" Jul 16 16:25:42.979112 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:42.978398 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8f44aa64-f94f-4a4e-a24f-b6a231b6df4b-cni-binary-copy\") pod \"multus-nrzr6\" (UID: \"8f44aa64-f94f-4a4e-a24f-b6a231b6df4b\") " pod="openshift-multus/multus-nrzr6" Jul 16 16:25:42.979112 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:42.978447 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ba81ab34-5cc0-4f3a-856a-ff794d0dc0e5-etc-systemd\") pod \"tuned-8pl5s\" (UID: \"ba81ab34-5cc0-4f3a-856a-ff794d0dc0e5\") " pod="openshift-cluster-node-tuning-operator/tuned-8pl5s" Jul 16 16:25:42.979112 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:42.978472 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8f44aa64-f94f-4a4e-a24f-b6a231b6df4b-multus-cni-dir\") pod \"multus-nrzr6\" (UID: \"8f44aa64-f94f-4a4e-a24f-b6a231b6df4b\") " pod="openshift-multus/multus-nrzr6" Jul 16 16:25:42.979112 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:42.978494 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/aa5f6bf8-3599-45f1-a09d-ef1976f22840-kubelet-dir\") pod \"aws-ebs-csi-driver-node-t4w26\" (UID: \"aa5f6bf8-3599-45f1-a09d-ef1976f22840\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t4w26" Jul 16 16:25:42.979112 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:42.978517 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-46nv8\" (UniqueName: \"kubernetes.io/projected/b2410943-db6a-4668-845d-db0ae081a389-kube-api-access-46nv8\") pod \"multus-additional-cni-plugins-xz7fp\" (UID: \"b2410943-db6a-4668-845d-db0ae081a389\") " pod="openshift-multus/multus-additional-cni-plugins-xz7fp" Jul 16 16:25:42.979112 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:42.978554 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b20b154f-1a4c-4b25-86db-aebe2d4ffc08-systemd-units\") pod \"ovnkube-node-9bjst\" (UID: \"b20b154f-1a4c-4b25-86db-aebe2d4ffc08\") " pod="openshift-ovn-kubernetes/ovnkube-node-9bjst" Jul 16 16:25:42.979112 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:42.978580 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b20b154f-1a4c-4b25-86db-aebe2d4ffc08-etc-openvswitch\") pod \"ovnkube-node-9bjst\" (UID: \"b20b154f-1a4c-4b25-86db-aebe2d4ffc08\") " pod="openshift-ovn-kubernetes/ovnkube-node-9bjst" Jul 16 16:25:42.979112 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:42.978653 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zr76g\" (UniqueName: \"kubernetes.io/projected/aa5f6bf8-3599-45f1-a09d-ef1976f22840-kube-api-access-zr76g\") pod \"aws-ebs-csi-driver-node-t4w26\" (UID: \"aa5f6bf8-3599-45f1-a09d-ef1976f22840\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t4w26" Jul 16 16:25:42.979906 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:42.978705 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4ht8t\" (UniqueName: \"kubernetes.io/projected/ba81ab34-5cc0-4f3a-856a-ff794d0dc0e5-kube-api-access-4ht8t\") pod \"tuned-8pl5s\" (UID: \"ba81ab34-5cc0-4f3a-856a-ff794d0dc0e5\") " pod="openshift-cluster-node-tuning-operator/tuned-8pl5s" Jul 16 16:25:42.979906 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:42.978740 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/8f44aa64-f94f-4a4e-a24f-b6a231b6df4b-hostroot\") pod \"multus-nrzr6\" (UID: \"8f44aa64-f94f-4a4e-a24f-b6a231b6df4b\") " pod="openshift-multus/multus-nrzr6" Jul 16 16:25:42.979906 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:42.978764 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/cbd027ac-97d9-4769-a867-47b7b8256cd0-host\") pod \"node-ca-hwq4x\" (UID: \"cbd027ac-97d9-4769-a867-47b7b8256cd0\") " pod="openshift-image-registry/node-ca-hwq4x" Jul 16 16:25:42.979906 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:42.978793 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e2d675d8-edf6-4765-a240-4b723cdd2d38-dbus\") pod \"global-pull-secret-syncer-rqzn4\" (UID: \"e2d675d8-edf6-4765-a240-4b723cdd2d38\") " pod="kube-system/global-pull-secret-syncer-rqzn4" Jul 16 16:25:42.979906 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:42.978813 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b20b154f-1a4c-4b25-86db-aebe2d4ffc08-node-log\") pod \"ovnkube-node-9bjst\" (UID: \"b20b154f-1a4c-4b25-86db-aebe2d4ffc08\") " pod="openshift-ovn-kubernetes/ovnkube-node-9bjst" Jul 16 16:25:42.979906 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:42.978830 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/b2410943-db6a-4668-845d-db0ae081a389-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-xz7fp\" (UID: \"b2410943-db6a-4668-845d-db0ae081a389\") " pod="openshift-multus/multus-additional-cni-plugins-xz7fp" Jul 16 16:25:42.979906 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:42.978853 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ba81ab34-5cc0-4f3a-856a-ff794d0dc0e5-etc-sysconfig\") pod \"tuned-8pl5s\" (UID: \"ba81ab34-5cc0-4f3a-856a-ff794d0dc0e5\") " pod="openshift-cluster-node-tuning-operator/tuned-8pl5s" Jul 16 16:25:42.979906 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:42.978871 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/a2cb45f0-0fb3-4761-98b0-a1797a00e647-konnectivity-ca\") pod \"konnectivity-agent-xljrg\" (UID: \"a2cb45f0-0fb3-4761-98b0-a1797a00e647\") " pod="kube-system/konnectivity-agent-xljrg" Jul 16 16:25:42.979906 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:42.978888 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ba81ab34-5cc0-4f3a-856a-ff794d0dc0e5-etc-kubernetes\") pod \"tuned-8pl5s\" (UID: \"ba81ab34-5cc0-4f3a-856a-ff794d0dc0e5\") " pod="openshift-cluster-node-tuning-operator/tuned-8pl5s" Jul 16 16:25:42.979906 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:42.978906 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ba81ab34-5cc0-4f3a-856a-ff794d0dc0e5-var-lib-kubelet\") pod \"tuned-8pl5s\" (UID: \"ba81ab34-5cc0-4f3a-856a-ff794d0dc0e5\") " pod="openshift-cluster-node-tuning-operator/tuned-8pl5s" Jul 16 16:25:42.979906 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:42.978931 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/94ec201c-7285-4751-be16-d23d2f2c0290-metrics-certs\") pod \"network-metrics-daemon-bcb7f\" (UID: \"94ec201c-7285-4751-be16-d23d2f2c0290\") " pod="openshift-multus/network-metrics-daemon-bcb7f" Jul 16 16:25:42.979906 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:42.978948 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b20b154f-1a4c-4b25-86db-aebe2d4ffc08-run-systemd\") pod \"ovnkube-node-9bjst\" (UID: \"b20b154f-1a4c-4b25-86db-aebe2d4ffc08\") " pod="openshift-ovn-kubernetes/ovnkube-node-9bjst" Jul 16 16:25:42.979906 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:42.978969 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/b20b154f-1a4c-4b25-86db-aebe2d4ffc08-env-overrides\") pod \"ovnkube-node-9bjst\" (UID: \"b20b154f-1a4c-4b25-86db-aebe2d4ffc08\") " pod="openshift-ovn-kubernetes/ovnkube-node-9bjst" Jul 16 16:25:42.979906 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:42.978991 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/aa5f6bf8-3599-45f1-a09d-ef1976f22840-sys-fs\") pod \"aws-ebs-csi-driver-node-t4w26\" (UID: \"aa5f6bf8-3599-45f1-a09d-ef1976f22840\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t4w26" Jul 16 16:25:42.979906 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:42.979014 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ba81ab34-5cc0-4f3a-856a-ff794d0dc0e5-tmp\") pod \"tuned-8pl5s\" (UID: \"ba81ab34-5cc0-4f3a-856a-ff794d0dc0e5\") " pod="openshift-cluster-node-tuning-operator/tuned-8pl5s" Jul 16 16:25:42.979906 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:42.979033 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/3062d489-a9b7-43fb-b1d8-4e39d30acdec-iptables-alerter-script\") pod \"iptables-alerter-cnft7\" (UID: \"3062d489-a9b7-43fb-b1d8-4e39d30acdec\") " pod="openshift-network-operator/iptables-alerter-cnft7" Jul 16 16:25:42.980758 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:42.979070 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fbz6z\" (UniqueName: \"kubernetes.io/projected/3062d489-a9b7-43fb-b1d8-4e39d30acdec-kube-api-access-fbz6z\") pod \"iptables-alerter-cnft7\" (UID: \"3062d489-a9b7-43fb-b1d8-4e39d30acdec\") " pod="openshift-network-operator/iptables-alerter-cnft7" Jul 16 16:25:42.980758 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:42.979110 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s8dcf\" (UniqueName: \"kubernetes.io/projected/84cc596d-4f15-4c84-8625-c7f37ed9dae6-kube-api-access-s8dcf\") pod \"network-check-target-ntktc\" (UID: \"84cc596d-4f15-4c84-8625-c7f37ed9dae6\") " pod="openshift-network-diagnostics/network-check-target-ntktc" Jul 16 16:25:42.980758 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:42.979135 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b20b154f-1a4c-4b25-86db-aebe2d4ffc08-log-socket\") pod \"ovnkube-node-9bjst\" (UID: \"b20b154f-1a4c-4b25-86db-aebe2d4ffc08\") " pod="openshift-ovn-kubernetes/ovnkube-node-9bjst" Jul 16 16:25:42.980758 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:42.979160 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b20b154f-1a4c-4b25-86db-aebe2d4ffc08-host-run-ovn-kubernetes\") pod \"ovnkube-node-9bjst\" (UID: \"b20b154f-1a4c-4b25-86db-aebe2d4ffc08\") " pod="openshift-ovn-kubernetes/ovnkube-node-9bjst" Jul 16 16:25:42.980758 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:42.979182 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/8f44aa64-f94f-4a4e-a24f-b6a231b6df4b-multus-socket-dir-parent\") pod \"multus-nrzr6\" (UID: \"8f44aa64-f94f-4a4e-a24f-b6a231b6df4b\") " pod="openshift-multus/multus-nrzr6" Jul 16 16:25:42.980758 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:42.979198 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/a2cb45f0-0fb3-4761-98b0-a1797a00e647-agent-certs\") pod \"konnectivity-agent-xljrg\" (UID: \"a2cb45f0-0fb3-4761-98b0-a1797a00e647\") " pod="kube-system/konnectivity-agent-xljrg" Jul 16 16:25:42.980758 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:42.979221 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g4lrp\" (UniqueName: \"kubernetes.io/projected/94ec201c-7285-4751-be16-d23d2f2c0290-kube-api-access-g4lrp\") pod \"network-metrics-daemon-bcb7f\" (UID: \"94ec201c-7285-4751-be16-d23d2f2c0290\") " pod="openshift-multus/network-metrics-daemon-bcb7f" Jul 16 16:25:43.072634 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:43.072598 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod077fca01a64486b516cac217b95ed2fa.slice/crio-7e9f458b7462f6d989326a3d998b561ee29ef4d219ed00e40aa9ef83905caf38 WatchSource:0}: Error finding container 7e9f458b7462f6d989326a3d998b561ee29ef4d219ed00e40aa9ef83905caf38: Status 404 returned error can't find the container with id 7e9f458b7462f6d989326a3d998b561ee29ef4d219ed00e40aa9ef83905caf38 Jul 16 16:25:43.077088 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:43.076989 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0ff4a720fce2411438eb201c85e2744.slice/crio-67f89ffd44e5fb87862e941e87ad81dcbfa617699f28f2cdc7fea7264c7d028f WatchSource:0}: Error finding container 67f89ffd44e5fb87862e941e87ad81dcbfa617699f28f2cdc7fea7264c7d028f: Status 404 returned error can't find the container with id 67f89ffd44e5fb87862e941e87ad81dcbfa617699f28f2cdc7fea7264c7d028f Jul 16 16:25:43.079603 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:43.079473 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/b2410943-db6a-4668-845d-db0ae081a389-os-release\") pod \"multus-additional-cni-plugins-xz7fp\" (UID: \"b2410943-db6a-4668-845d-db0ae081a389\") " pod="openshift-multus/multus-additional-cni-plugins-xz7fp" Jul 16 16:25:43.079603 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:43.079499 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8f44aa64-f94f-4a4e-a24f-b6a231b6df4b-host-var-lib-cni-bin\") pod \"multus-nrzr6\" (UID: \"8f44aa64-f94f-4a4e-a24f-b6a231b6df4b\") " pod="openshift-multus/multus-nrzr6" Jul 16 16:25:43.079603 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:43.079515 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/8f44aa64-f94f-4a4e-a24f-b6a231b6df4b-host-var-lib-cni-multus\") pod \"multus-nrzr6\" (UID: \"8f44aa64-f94f-4a4e-a24f-b6a231b6df4b\") " pod="openshift-multus/multus-nrzr6" Jul 16 16:25:43.079603 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:43.079532 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/8f44aa64-f94f-4a4e-a24f-b6a231b6df4b-host-run-multus-certs\") pod \"multus-nrzr6\" (UID: \"8f44aa64-f94f-4a4e-a24f-b6a231b6df4b\") " pod="openshift-multus/multus-nrzr6" Jul 16 16:25:43.079603 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:43.079552 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ba81ab34-5cc0-4f3a-856a-ff794d0dc0e5-etc-modprobe-d\") pod \"tuned-8pl5s\" (UID: \"ba81ab34-5cc0-4f3a-856a-ff794d0dc0e5\") " pod="openshift-cluster-node-tuning-operator/tuned-8pl5s" Jul 16 16:25:43.079603 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:43.079577 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ba81ab34-5cc0-4f3a-856a-ff794d0dc0e5-sys\") pod \"tuned-8pl5s\" (UID: \"ba81ab34-5cc0-4f3a-856a-ff794d0dc0e5\") " pod="openshift-cluster-node-tuning-operator/tuned-8pl5s" Jul 16 16:25:43.079603 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:43.079599 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ba81ab34-5cc0-4f3a-856a-ff794d0dc0e5-etc-tuned\") pod \"tuned-8pl5s\" (UID: \"ba81ab34-5cc0-4f3a-856a-ff794d0dc0e5\") " pod="openshift-cluster-node-tuning-operator/tuned-8pl5s" Jul 16 16:25:43.079970 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:43.079606 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8f44aa64-f94f-4a4e-a24f-b6a231b6df4b-host-var-lib-cni-bin\") pod \"multus-nrzr6\" (UID: \"8f44aa64-f94f-4a4e-a24f-b6a231b6df4b\") " pod="openshift-multus/multus-nrzr6" Jul 16 16:25:43.079970 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:43.079623 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/93f30808-1a75-44b5-801f-8faeceb72afb-hosts-file\") pod \"node-resolver-6sc67\" (UID: \"93f30808-1a75-44b5-801f-8faeceb72afb\") " pod="openshift-dns/node-resolver-6sc67" Jul 16 16:25:43.079970 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:43.079610 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/8f44aa64-f94f-4a4e-a24f-b6a231b6df4b-host-var-lib-cni-multus\") pod \"multus-nrzr6\" (UID: \"8f44aa64-f94f-4a4e-a24f-b6a231b6df4b\") " pod="openshift-multus/multus-nrzr6" Jul 16 16:25:43.079970 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:43.079623 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/8f44aa64-f94f-4a4e-a24f-b6a231b6df4b-host-run-multus-certs\") pod \"multus-nrzr6\" (UID: \"8f44aa64-f94f-4a4e-a24f-b6a231b6df4b\") " pod="openshift-multus/multus-nrzr6" Jul 16 16:25:43.079970 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:43.079613 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/b2410943-db6a-4668-845d-db0ae081a389-os-release\") pod \"multus-additional-cni-plugins-xz7fp\" (UID: \"b2410943-db6a-4668-845d-db0ae081a389\") " pod="openshift-multus/multus-additional-cni-plugins-xz7fp" Jul 16 16:25:43.079970 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:43.079652 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b2410943-db6a-4668-845d-db0ae081a389-tuning-conf-dir\") pod \"multus-additional-cni-plugins-xz7fp\" (UID: \"b2410943-db6a-4668-845d-db0ae081a389\") " pod="openshift-multus/multus-additional-cni-plugins-xz7fp" Jul 16 16:25:43.079970 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:43.079671 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ba81ab34-5cc0-4f3a-856a-ff794d0dc0e5-sys\") pod \"tuned-8pl5s\" (UID: \"ba81ab34-5cc0-4f3a-856a-ff794d0dc0e5\") " pod="openshift-cluster-node-tuning-operator/tuned-8pl5s" Jul 16 16:25:43.079970 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:43.079675 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8f44aa64-f94f-4a4e-a24f-b6a231b6df4b-multus-conf-dir\") pod \"multus-nrzr6\" (UID: \"8f44aa64-f94f-4a4e-a24f-b6a231b6df4b\") " pod="openshift-multus/multus-nrzr6" Jul 16 16:25:43.079970 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:43.079713 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8f44aa64-f94f-4a4e-a24f-b6a231b6df4b-multus-conf-dir\") pod \"multus-nrzr6\" (UID: \"8f44aa64-f94f-4a4e-a24f-b6a231b6df4b\") " pod="openshift-multus/multus-nrzr6" Jul 16 16:25:43.079970 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:43.079718 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/93f30808-1a75-44b5-801f-8faeceb72afb-hosts-file\") pod \"node-resolver-6sc67\" (UID: \"93f30808-1a75-44b5-801f-8faeceb72afb\") " pod="openshift-dns/node-resolver-6sc67" Jul 16 16:25:43.079970 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:43.079760 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8f44aa64-f94f-4a4e-a24f-b6a231b6df4b-etc-kubernetes\") pod \"multus-nrzr6\" (UID: \"8f44aa64-f94f-4a4e-a24f-b6a231b6df4b\") " pod="openshift-multus/multus-nrzr6" Jul 16 16:25:43.079970 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:43.079765 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ba81ab34-5cc0-4f3a-856a-ff794d0dc0e5-etc-modprobe-d\") pod \"tuned-8pl5s\" (UID: \"ba81ab34-5cc0-4f3a-856a-ff794d0dc0e5\") " pod="openshift-cluster-node-tuning-operator/tuned-8pl5s" Jul 16 16:25:43.079970 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:43.079807 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b20b154f-1a4c-4b25-86db-aebe2d4ffc08-var-lib-openvswitch\") pod \"ovnkube-node-9bjst\" (UID: \"b20b154f-1a4c-4b25-86db-aebe2d4ffc08\") " pod="openshift-ovn-kubernetes/ovnkube-node-9bjst" Jul 16 16:25:43.079970 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:43.079806 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8f44aa64-f94f-4a4e-a24f-b6a231b6df4b-etc-kubernetes\") pod \"multus-nrzr6\" (UID: \"8f44aa64-f94f-4a4e-a24f-b6a231b6df4b\") " pod="openshift-multus/multus-nrzr6" Jul 16 16:25:43.079970 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:43.079863 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b2410943-db6a-4668-845d-db0ae081a389-tuning-conf-dir\") pod \"multus-additional-cni-plugins-xz7fp\" (UID: \"b2410943-db6a-4668-845d-db0ae081a389\") " pod="openshift-multus/multus-additional-cni-plugins-xz7fp" Jul 16 16:25:43.079970 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:43.079865 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e2d675d8-edf6-4765-a240-4b723cdd2d38-kubelet-config\") pod \"global-pull-secret-syncer-rqzn4\" (UID: \"e2d675d8-edf6-4765-a240-4b723cdd2d38\") " pod="kube-system/global-pull-secret-syncer-rqzn4" Jul 16 16:25:43.079970 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:43.079905 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b2410943-db6a-4668-845d-db0ae081a389-cnibin\") pod \"multus-additional-cni-plugins-xz7fp\" (UID: \"b2410943-db6a-4668-845d-db0ae081a389\") " pod="openshift-multus/multus-additional-cni-plugins-xz7fp" Jul 16 16:25:43.079970 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:43.079910 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b20b154f-1a4c-4b25-86db-aebe2d4ffc08-var-lib-openvswitch\") pod \"ovnkube-node-9bjst\" (UID: \"b20b154f-1a4c-4b25-86db-aebe2d4ffc08\") " pod="openshift-ovn-kubernetes/ovnkube-node-9bjst" Jul 16 16:25:43.081131 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:43.079944 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b2410943-db6a-4668-845d-db0ae081a389-cnibin\") pod \"multus-additional-cni-plugins-xz7fp\" (UID: \"b2410943-db6a-4668-845d-db0ae081a389\") " pod="openshift-multus/multus-additional-cni-plugins-xz7fp" Jul 16 16:25:43.081131 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:43.079951 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8f44aa64-f94f-4a4e-a24f-b6a231b6df4b-system-cni-dir\") pod \"multus-nrzr6\" (UID: \"8f44aa64-f94f-4a4e-a24f-b6a231b6df4b\") " pod="openshift-multus/multus-nrzr6" Jul 16 16:25:43.081131 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:43.079946 2577 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 16 16:25:43.081131 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:43.079994 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b20b154f-1a4c-4b25-86db-aebe2d4ffc08-host-cni-bin\") pod \"ovnkube-node-9bjst\" (UID: \"b20b154f-1a4c-4b25-86db-aebe2d4ffc08\") " pod="openshift-ovn-kubernetes/ovnkube-node-9bjst" Jul 16 16:25:43.081131 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:43.079970 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e2d675d8-edf6-4765-a240-4b723cdd2d38-kubelet-config\") pod \"global-pull-secret-syncer-rqzn4\" (UID: \"e2d675d8-edf6-4765-a240-4b723cdd2d38\") " pod="kube-system/global-pull-secret-syncer-rqzn4" Jul 16 16:25:43.081131 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:43.080007 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8f44aa64-f94f-4a4e-a24f-b6a231b6df4b-system-cni-dir\") pod \"multus-nrzr6\" (UID: \"8f44aa64-f94f-4a4e-a24f-b6a231b6df4b\") " pod="openshift-multus/multus-nrzr6" Jul 16 16:25:43.081131 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:43.080005 2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 16 16:25:43.081131 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:43.080027 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b20b154f-1a4c-4b25-86db-aebe2d4ffc08-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-9bjst\" (UID: \"b20b154f-1a4c-4b25-86db-aebe2d4ffc08\") " pod="openshift-ovn-kubernetes/ovnkube-node-9bjst" Jul 16 16:25:43.081131 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:43.080052 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b20b154f-1a4c-4b25-86db-aebe2d4ffc08-host-cni-bin\") pod \"ovnkube-node-9bjst\" (UID: \"b20b154f-1a4c-4b25-86db-aebe2d4ffc08\") " pod="openshift-ovn-kubernetes/ovnkube-node-9bjst" Jul 16 16:25:43.081131 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:43.080085 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ba81ab34-5cc0-4f3a-856a-ff794d0dc0e5-etc-sysctl-d\") pod \"tuned-8pl5s\" (UID: \"ba81ab34-5cc0-4f3a-856a-ff794d0dc0e5\") " pod="openshift-cluster-node-tuning-operator/tuned-8pl5s" Jul 16 16:25:43.081131 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:43.080097 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b20b154f-1a4c-4b25-86db-aebe2d4ffc08-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-9bjst\" (UID: \"b20b154f-1a4c-4b25-86db-aebe2d4ffc08\") " pod="openshift-ovn-kubernetes/ovnkube-node-9bjst" Jul 16 16:25:43.081131 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:43.080132 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ba81ab34-5cc0-4f3a-856a-ff794d0dc0e5-lib-modules\") pod \"tuned-8pl5s\" (UID: \"ba81ab34-5cc0-4f3a-856a-ff794d0dc0e5\") " pod="openshift-cluster-node-tuning-operator/tuned-8pl5s" Jul 16 16:25:43.081131 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:43.080157 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8f44aa64-f94f-4a4e-a24f-b6a231b6df4b-host-var-lib-kubelet\") pod \"multus-nrzr6\" (UID: \"8f44aa64-f94f-4a4e-a24f-b6a231b6df4b\") " pod="openshift-multus/multus-nrzr6" Jul 16 16:25:43.081131 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:43.080198 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8f44aa64-f94f-4a4e-a24f-b6a231b6df4b-host-var-lib-kubelet\") pod \"multus-nrzr6\" (UID: \"8f44aa64-f94f-4a4e-a24f-b6a231b6df4b\") " pod="openshift-multus/multus-nrzr6" Jul 16 16:25:43.081131 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:43.080203 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/8f44aa64-f94f-4a4e-a24f-b6a231b6df4b-multus-daemon-config\") pod \"multus-nrzr6\" (UID: \"8f44aa64-f94f-4a4e-a24f-b6a231b6df4b\") " pod="openshift-multus/multus-nrzr6" Jul 16 16:25:43.081131 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:43.080226 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ba81ab34-5cc0-4f3a-856a-ff794d0dc0e5-etc-sysctl-d\") pod \"tuned-8pl5s\" (UID: \"ba81ab34-5cc0-4f3a-856a-ff794d0dc0e5\") " pod="openshift-cluster-node-tuning-operator/tuned-8pl5s" Jul 16 16:25:43.081131 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:43.080241 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w9lm2\" (UniqueName: \"kubernetes.io/projected/8f44aa64-f94f-4a4e-a24f-b6a231b6df4b-kube-api-access-w9lm2\") pod \"multus-nrzr6\" (UID: \"8f44aa64-f94f-4a4e-a24f-b6a231b6df4b\") " pod="openshift-multus/multus-nrzr6" Jul 16 16:25:43.081131 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:43.080243 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ba81ab34-5cc0-4f3a-856a-ff794d0dc0e5-lib-modules\") pod \"tuned-8pl5s\" (UID: \"ba81ab34-5cc0-4f3a-856a-ff794d0dc0e5\") " pod="openshift-cluster-node-tuning-operator/tuned-8pl5s" Jul 16 16:25:43.082114 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:43.080268 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b20b154f-1a4c-4b25-86db-aebe2d4ffc08-host-slash\") pod \"ovnkube-node-9bjst\" (UID: \"b20b154f-1a4c-4b25-86db-aebe2d4ffc08\") " pod="openshift-ovn-kubernetes/ovnkube-node-9bjst" Jul 16 16:25:43.082114 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:43.080284 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b20b154f-1a4c-4b25-86db-aebe2d4ffc08-host-cni-netd\") pod \"ovnkube-node-9bjst\" (UID: \"b20b154f-1a4c-4b25-86db-aebe2d4ffc08\") " pod="openshift-ovn-kubernetes/ovnkube-node-9bjst" Jul 16 16:25:43.082114 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:43.080299 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8f44aa64-f94f-4a4e-a24f-b6a231b6df4b-host-run-netns\") pod \"multus-nrzr6\" (UID: \"8f44aa64-f94f-4a4e-a24f-b6a231b6df4b\") " pod="openshift-multus/multus-nrzr6" Jul 16 16:25:43.082114 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:43.080308 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b20b154f-1a4c-4b25-86db-aebe2d4ffc08-host-slash\") pod \"ovnkube-node-9bjst\" (UID: \"b20b154f-1a4c-4b25-86db-aebe2d4ffc08\") " pod="openshift-ovn-kubernetes/ovnkube-node-9bjst" Jul 16 16:25:43.082114 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:43.080319 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ba81ab34-5cc0-4f3a-856a-ff794d0dc0e5-run\") pod \"tuned-8pl5s\" (UID: \"ba81ab34-5cc0-4f3a-856a-ff794d0dc0e5\") " pod="openshift-cluster-node-tuning-operator/tuned-8pl5s" Jul 16 16:25:43.082114 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:43.080344 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/cbd027ac-97d9-4769-a867-47b7b8256cd0-serviceca\") pod \"node-ca-hwq4x\" (UID: \"cbd027ac-97d9-4769-a867-47b7b8256cd0\") " pod="openshift-image-registry/node-ca-hwq4x" Jul 16 16:25:43.082114 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:43.080368 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3062d489-a9b7-43fb-b1d8-4e39d30acdec-host-slash\") pod \"iptables-alerter-cnft7\" (UID: \"3062d489-a9b7-43fb-b1d8-4e39d30acdec\") " pod="openshift-network-operator/iptables-alerter-cnft7" Jul 16 16:25:43.082114 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:43.080393 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b2410943-db6a-4668-845d-db0ae081a389-system-cni-dir\") pod \"multus-additional-cni-plugins-xz7fp\" (UID: \"b2410943-db6a-4668-845d-db0ae081a389\") " pod="openshift-multus/multus-additional-cni-plugins-xz7fp" Jul 16 16:25:43.082114 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:43.080437 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j27b9\" (UniqueName: \"kubernetes.io/projected/b20b154f-1a4c-4b25-86db-aebe2d4ffc08-kube-api-access-j27b9\") pod \"ovnkube-node-9bjst\" (UID: \"b20b154f-1a4c-4b25-86db-aebe2d4ffc08\") " pod="openshift-ovn-kubernetes/ovnkube-node-9bjst" Jul 16 16:25:43.082114 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:43.080484 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ba81ab34-5cc0-4f3a-856a-ff794d0dc0e5-run\") pod \"tuned-8pl5s\" (UID: \"ba81ab34-5cc0-4f3a-856a-ff794d0dc0e5\") " pod="openshift-cluster-node-tuning-operator/tuned-8pl5s" Jul 16 16:25:43.082114 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:43.080512 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ba81ab34-5cc0-4f3a-856a-ff794d0dc0e5-etc-sysctl-conf\") pod \"tuned-8pl5s\" (UID: \"ba81ab34-5cc0-4f3a-856a-ff794d0dc0e5\") " pod="openshift-cluster-node-tuning-operator/tuned-8pl5s" Jul 16 16:25:43.082114 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:43.080536 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3062d489-a9b7-43fb-b1d8-4e39d30acdec-host-slash\") pod \"iptables-alerter-cnft7\" (UID: \"3062d489-a9b7-43fb-b1d8-4e39d30acdec\") " pod="openshift-network-operator/iptables-alerter-cnft7" Jul 16 16:25:43.082114 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:43.080574 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b20b154f-1a4c-4b25-86db-aebe2d4ffc08-host-cni-netd\") pod \"ovnkube-node-9bjst\" (UID: \"b20b154f-1a4c-4b25-86db-aebe2d4ffc08\") " pod="openshift-ovn-kubernetes/ovnkube-node-9bjst" Jul 16 16:25:43.082114 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:43.080612 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b2410943-db6a-4668-845d-db0ae081a389-system-cni-dir\") pod \"multus-additional-cni-plugins-xz7fp\" (UID: \"b2410943-db6a-4668-845d-db0ae081a389\") " pod="openshift-multus/multus-additional-cni-plugins-xz7fp" Jul 16 16:25:43.082114 ip-10-0-142-101 kubenswrapper[2577]: E0716 16:25:43.080617 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 16 16:25:43.082114 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:43.080633 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8f44aa64-f94f-4a4e-a24f-b6a231b6df4b-host-run-netns\") pod \"multus-nrzr6\" (UID: \"8f44aa64-f94f-4a4e-a24f-b6a231b6df4b\") " pod="openshift-multus/multus-nrzr6" Jul 16 16:25:43.082114 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:43.080539 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e2d675d8-edf6-4765-a240-4b723cdd2d38-original-pull-secret\") pod \"global-pull-secret-syncer-rqzn4\" (UID: \"e2d675d8-edf6-4765-a240-4b723cdd2d38\") " pod="kube-system/global-pull-secret-syncer-rqzn4" Jul 16 16:25:43.083140 ip-10-0-142-101 kubenswrapper[2577]: E0716 16:25:43.080675 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e2d675d8-edf6-4765-a240-4b723cdd2d38-original-pull-secret podName:e2d675d8-edf6-4765-a240-4b723cdd2d38 nodeName:}" failed. No retries permitted until 2026-07-16 16:25:43.580653816 +0000 UTC m=+2.084711664 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e2d675d8-edf6-4765-a240-4b723cdd2d38-original-pull-secret") pod "global-pull-secret-syncer-rqzn4" (UID: "e2d675d8-edf6-4765-a240-4b723cdd2d38") : object "kube-system"/"original-pull-secret" not registered Jul 16 16:25:43.083140 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:43.080705 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/8f44aa64-f94f-4a4e-a24f-b6a231b6df4b-os-release\") pod \"multus-nrzr6\" (UID: \"8f44aa64-f94f-4a4e-a24f-b6a231b6df4b\") " pod="openshift-multus/multus-nrzr6" Jul 16 16:25:43.083140 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:43.080730 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b20b154f-1a4c-4b25-86db-aebe2d4ffc08-host-kubelet\") pod \"ovnkube-node-9bjst\" (UID: \"b20b154f-1a4c-4b25-86db-aebe2d4ffc08\") " pod="openshift-ovn-kubernetes/ovnkube-node-9bjst" Jul 16 16:25:43.083140 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:43.080737 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ba81ab34-5cc0-4f3a-856a-ff794d0dc0e5-etc-sysctl-conf\") pod \"tuned-8pl5s\" (UID: \"ba81ab34-5cc0-4f3a-856a-ff794d0dc0e5\") " pod="openshift-cluster-node-tuning-operator/tuned-8pl5s" Jul 16 16:25:43.083140 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:43.080756 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b20b154f-1a4c-4b25-86db-aebe2d4ffc08-host-run-netns\") pod \"ovnkube-node-9bjst\" (UID: \"b20b154f-1a4c-4b25-86db-aebe2d4ffc08\") " pod="openshift-ovn-kubernetes/ovnkube-node-9bjst" Jul 16 16:25:43.083140 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:43.080777 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b20b154f-1a4c-4b25-86db-aebe2d4ffc08-host-kubelet\") pod \"ovnkube-node-9bjst\" (UID: \"b20b154f-1a4c-4b25-86db-aebe2d4ffc08\") " pod="openshift-ovn-kubernetes/ovnkube-node-9bjst" Jul 16 16:25:43.083140 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:43.080781 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b20b154f-1a4c-4b25-86db-aebe2d4ffc08-run-openvswitch\") pod \"ovnkube-node-9bjst\" (UID: \"b20b154f-1a4c-4b25-86db-aebe2d4ffc08\") " pod="openshift-ovn-kubernetes/ovnkube-node-9bjst" Jul 16 16:25:43.083140 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:43.080794 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/8f44aa64-f94f-4a4e-a24f-b6a231b6df4b-os-release\") pod \"multus-nrzr6\" (UID: \"8f44aa64-f94f-4a4e-a24f-b6a231b6df4b\") " pod="openshift-multus/multus-nrzr6" Jul 16 16:25:43.083140 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:43.080805 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b20b154f-1a4c-4b25-86db-aebe2d4ffc08-run-ovn\") pod \"ovnkube-node-9bjst\" (UID: \"b20b154f-1a4c-4b25-86db-aebe2d4ffc08\") " pod="openshift-ovn-kubernetes/ovnkube-node-9bjst" Jul 16 16:25:43.083140 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:43.080813 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b20b154f-1a4c-4b25-86db-aebe2d4ffc08-host-run-netns\") pod \"ovnkube-node-9bjst\" (UID: \"b20b154f-1a4c-4b25-86db-aebe2d4ffc08\") " pod="openshift-ovn-kubernetes/ovnkube-node-9bjst" Jul 16 16:25:43.083140 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:43.080838 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/b20b154f-1a4c-4b25-86db-aebe2d4ffc08-ovnkube-config\") pod \"ovnkube-node-9bjst\" (UID: \"b20b154f-1a4c-4b25-86db-aebe2d4ffc08\") " pod="openshift-ovn-kubernetes/ovnkube-node-9bjst" Jul 16 16:25:43.083140 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:43.080846 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b20b154f-1a4c-4b25-86db-aebe2d4ffc08-run-ovn\") pod \"ovnkube-node-9bjst\" (UID: \"b20b154f-1a4c-4b25-86db-aebe2d4ffc08\") " pod="openshift-ovn-kubernetes/ovnkube-node-9bjst" Jul 16 16:25:43.083140 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:43.080852 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b20b154f-1a4c-4b25-86db-aebe2d4ffc08-run-openvswitch\") pod \"ovnkube-node-9bjst\" (UID: \"b20b154f-1a4c-4b25-86db-aebe2d4ffc08\") " pod="openshift-ovn-kubernetes/ovnkube-node-9bjst" Jul 16 16:25:43.083140 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:43.080856 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/8f44aa64-f94f-4a4e-a24f-b6a231b6df4b-multus-daemon-config\") pod \"multus-nrzr6\" (UID: \"8f44aa64-f94f-4a4e-a24f-b6a231b6df4b\") " pod="openshift-multus/multus-nrzr6" Jul 16 16:25:43.083140 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:43.080867 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/8f44aa64-f94f-4a4e-a24f-b6a231b6df4b-host-run-k8s-cni-cncf-io\") pod \"multus-nrzr6\" (UID: \"8f44aa64-f94f-4a4e-a24f-b6a231b6df4b\") " pod="openshift-multus/multus-nrzr6" Jul 16 16:25:43.083140 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:43.080896 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/aa5f6bf8-3599-45f1-a09d-ef1976f22840-socket-dir\") pod \"aws-ebs-csi-driver-node-t4w26\" (UID: \"aa5f6bf8-3599-45f1-a09d-ef1976f22840\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t4w26" Jul 16 16:25:43.083140 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:43.080949 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/aa5f6bf8-3599-45f1-a09d-ef1976f22840-device-dir\") pod \"aws-ebs-csi-driver-node-t4w26\" (UID: \"aa5f6bf8-3599-45f1-a09d-ef1976f22840\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t4w26" Jul 16 16:25:43.084072 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:43.080960 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/8f44aa64-f94f-4a4e-a24f-b6a231b6df4b-host-run-k8s-cni-cncf-io\") pod \"multus-nrzr6\" (UID: \"8f44aa64-f94f-4a4e-a24f-b6a231b6df4b\") " pod="openshift-multus/multus-nrzr6" Jul 16 16:25:43.084072 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:43.080980 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ffgxf\" (UniqueName: \"kubernetes.io/projected/93f30808-1a75-44b5-801f-8faeceb72afb-kube-api-access-ffgxf\") pod \"node-resolver-6sc67\" (UID: \"93f30808-1a75-44b5-801f-8faeceb72afb\") " pod="openshift-dns/node-resolver-6sc67" Jul 16 16:25:43.084072 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:43.080988 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/aa5f6bf8-3599-45f1-a09d-ef1976f22840-socket-dir\") pod \"aws-ebs-csi-driver-node-t4w26\" (UID: \"aa5f6bf8-3599-45f1-a09d-ef1976f22840\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t4w26" Jul 16 16:25:43.084072 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:43.081006 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/aa5f6bf8-3599-45f1-a09d-ef1976f22840-device-dir\") pod \"aws-ebs-csi-driver-node-t4w26\" (UID: \"aa5f6bf8-3599-45f1-a09d-ef1976f22840\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t4w26" Jul 16 16:25:43.084072 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:43.081008 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/b2410943-db6a-4668-845d-db0ae081a389-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-xz7fp\" (UID: \"b2410943-db6a-4668-845d-db0ae081a389\") " pod="openshift-multus/multus-additional-cni-plugins-xz7fp" Jul 16 16:25:43.084072 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:43.081034 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/b20b154f-1a4c-4b25-86db-aebe2d4ffc08-ovn-node-metrics-cert\") pod \"ovnkube-node-9bjst\" (UID: \"b20b154f-1a4c-4b25-86db-aebe2d4ffc08\") " pod="openshift-ovn-kubernetes/ovnkube-node-9bjst" Jul 16 16:25:43.084072 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:43.081057 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ba81ab34-5cc0-4f3a-856a-ff794d0dc0e5-host\") pod \"tuned-8pl5s\" (UID: \"ba81ab34-5cc0-4f3a-856a-ff794d0dc0e5\") " pod="openshift-cluster-node-tuning-operator/tuned-8pl5s" Jul 16 16:25:43.084072 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:43.081083 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b2410943-db6a-4668-845d-db0ae081a389-cni-binary-copy\") pod \"multus-additional-cni-plugins-xz7fp\" (UID: \"b2410943-db6a-4668-845d-db0ae081a389\") " pod="openshift-multus/multus-additional-cni-plugins-xz7fp" Jul 16 16:25:43.084072 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:43.081109 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8f44aa64-f94f-4a4e-a24f-b6a231b6df4b-cni-binary-copy\") pod \"multus-nrzr6\" (UID: \"8f44aa64-f94f-4a4e-a24f-b6a231b6df4b\") " pod="openshift-multus/multus-nrzr6" Jul 16 16:25:43.084072 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:43.081136 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ba81ab34-5cc0-4f3a-856a-ff794d0dc0e5-etc-systemd\") pod \"tuned-8pl5s\" (UID: \"ba81ab34-5cc0-4f3a-856a-ff794d0dc0e5\") " pod="openshift-cluster-node-tuning-operator/tuned-8pl5s" Jul 16 16:25:43.084072 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:43.081160 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8f44aa64-f94f-4a4e-a24f-b6a231b6df4b-multus-cni-dir\") pod \"multus-nrzr6\" (UID: \"8f44aa64-f94f-4a4e-a24f-b6a231b6df4b\") " pod="openshift-multus/multus-nrzr6" Jul 16 16:25:43.084072 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:43.081171 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ba81ab34-5cc0-4f3a-856a-ff794d0dc0e5-host\") pod \"tuned-8pl5s\" (UID: \"ba81ab34-5cc0-4f3a-856a-ff794d0dc0e5\") " pod="openshift-cluster-node-tuning-operator/tuned-8pl5s" Jul 16 16:25:43.084072 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:43.081107 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/cbd027ac-97d9-4769-a867-47b7b8256cd0-serviceca\") pod \"node-ca-hwq4x\" (UID: \"cbd027ac-97d9-4769-a867-47b7b8256cd0\") " pod="openshift-image-registry/node-ca-hwq4x" Jul 16 16:25:43.084072 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:43.081201 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/aa5f6bf8-3599-45f1-a09d-ef1976f22840-kubelet-dir\") pod \"aws-ebs-csi-driver-node-t4w26\" (UID: \"aa5f6bf8-3599-45f1-a09d-ef1976f22840\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t4w26" Jul 16 16:25:43.084072 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:43.081253 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ba81ab34-5cc0-4f3a-856a-ff794d0dc0e5-etc-systemd\") pod \"tuned-8pl5s\" (UID: \"ba81ab34-5cc0-4f3a-856a-ff794d0dc0e5\") " pod="openshift-cluster-node-tuning-operator/tuned-8pl5s" Jul 16 16:25:43.084072 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:43.081279 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-46nv8\" (UniqueName: \"kubernetes.io/projected/b2410943-db6a-4668-845d-db0ae081a389-kube-api-access-46nv8\") pod \"multus-additional-cni-plugins-xz7fp\" (UID: \"b2410943-db6a-4668-845d-db0ae081a389\") " pod="openshift-multus/multus-additional-cni-plugins-xz7fp" Jul 16 16:25:43.084072 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:43.081303 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b20b154f-1a4c-4b25-86db-aebe2d4ffc08-systemd-units\") pod \"ovnkube-node-9bjst\" (UID: \"b20b154f-1a4c-4b25-86db-aebe2d4ffc08\") " pod="openshift-ovn-kubernetes/ovnkube-node-9bjst" Jul 16 16:25:43.085004 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:43.081328 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b20b154f-1a4c-4b25-86db-aebe2d4ffc08-etc-openvswitch\") pod \"ovnkube-node-9bjst\" (UID: \"b20b154f-1a4c-4b25-86db-aebe2d4ffc08\") " pod="openshift-ovn-kubernetes/ovnkube-node-9bjst" Jul 16 16:25:43.085004 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:43.081344 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8f44aa64-f94f-4a4e-a24f-b6a231b6df4b-multus-cni-dir\") pod \"multus-nrzr6\" (UID: \"8f44aa64-f94f-4a4e-a24f-b6a231b6df4b\") " pod="openshift-multus/multus-nrzr6" Jul 16 16:25:43.085004 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:43.081359 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zr76g\" (UniqueName: \"kubernetes.io/projected/aa5f6bf8-3599-45f1-a09d-ef1976f22840-kube-api-access-zr76g\") pod \"aws-ebs-csi-driver-node-t4w26\" (UID: \"aa5f6bf8-3599-45f1-a09d-ef1976f22840\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t4w26" Jul 16 16:25:43.085004 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:43.081387 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4ht8t\" (UniqueName: \"kubernetes.io/projected/ba81ab34-5cc0-4f3a-856a-ff794d0dc0e5-kube-api-access-4ht8t\") pod \"tuned-8pl5s\" (UID: \"ba81ab34-5cc0-4f3a-856a-ff794d0dc0e5\") " pod="openshift-cluster-node-tuning-operator/tuned-8pl5s" Jul 16 16:25:43.085004 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:43.081410 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/8f44aa64-f94f-4a4e-a24f-b6a231b6df4b-hostroot\") pod \"multus-nrzr6\" (UID: \"8f44aa64-f94f-4a4e-a24f-b6a231b6df4b\") " pod="openshift-multus/multus-nrzr6" Jul 16 16:25:43.085004 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:43.081435 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/b20b154f-1a4c-4b25-86db-aebe2d4ffc08-ovnkube-config\") pod \"ovnkube-node-9bjst\" (UID: \"b20b154f-1a4c-4b25-86db-aebe2d4ffc08\") " pod="openshift-ovn-kubernetes/ovnkube-node-9bjst" Jul 16 16:25:43.085004 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:43.081502 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/cbd027ac-97d9-4769-a867-47b7b8256cd0-host\") pod \"node-ca-hwq4x\" (UID: \"cbd027ac-97d9-4769-a867-47b7b8256cd0\") " pod="openshift-image-registry/node-ca-hwq4x" Jul 16 16:25:43.085004 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:43.081529 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e2d675d8-edf6-4765-a240-4b723cdd2d38-dbus\") pod \"global-pull-secret-syncer-rqzn4\" (UID: \"e2d675d8-edf6-4765-a240-4b723cdd2d38\") " pod="kube-system/global-pull-secret-syncer-rqzn4" Jul 16 16:25:43.085004 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:43.081554 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b20b154f-1a4c-4b25-86db-aebe2d4ffc08-node-log\") pod \"ovnkube-node-9bjst\" (UID: \"b20b154f-1a4c-4b25-86db-aebe2d4ffc08\") " pod="openshift-ovn-kubernetes/ovnkube-node-9bjst" Jul 16 16:25:43.085004 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:43.081605 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b20b154f-1a4c-4b25-86db-aebe2d4ffc08-node-log\") pod \"ovnkube-node-9bjst\" (UID: \"b20b154f-1a4c-4b25-86db-aebe2d4ffc08\") " pod="openshift-ovn-kubernetes/ovnkube-node-9bjst" Jul 16 16:25:43.085004 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:43.081663 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/8f44aa64-f94f-4a4e-a24f-b6a231b6df4b-hostroot\") pod \"multus-nrzr6\" (UID: \"8f44aa64-f94f-4a4e-a24f-b6a231b6df4b\") " pod="openshift-multus/multus-nrzr6" Jul 16 16:25:43.085004 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:43.081687 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/b2410943-db6a-4668-845d-db0ae081a389-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-xz7fp\" (UID: \"b2410943-db6a-4668-845d-db0ae081a389\") " pod="openshift-multus/multus-additional-cni-plugins-xz7fp" Jul 16 16:25:43.085004 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:43.081735 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/cbd027ac-97d9-4769-a867-47b7b8256cd0-host\") pod \"node-ca-hwq4x\" (UID: \"cbd027ac-97d9-4769-a867-47b7b8256cd0\") " pod="openshift-image-registry/node-ca-hwq4x" Jul 16 16:25:43.085004 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:43.081752 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8f44aa64-f94f-4a4e-a24f-b6a231b6df4b-cni-binary-copy\") pod \"multus-nrzr6\" (UID: \"8f44aa64-f94f-4a4e-a24f-b6a231b6df4b\") " pod="openshift-multus/multus-nrzr6" Jul 16 16:25:43.085004 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:43.081757 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b20b154f-1a4c-4b25-86db-aebe2d4ffc08-etc-openvswitch\") pod \"ovnkube-node-9bjst\" (UID: \"b20b154f-1a4c-4b25-86db-aebe2d4ffc08\") " pod="openshift-ovn-kubernetes/ovnkube-node-9bjst" Jul 16 16:25:43.085004 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:43.081793 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b20b154f-1a4c-4b25-86db-aebe2d4ffc08-systemd-units\") pod \"ovnkube-node-9bjst\" (UID: \"b20b154f-1a4c-4b25-86db-aebe2d4ffc08\") " pod="openshift-ovn-kubernetes/ovnkube-node-9bjst" Jul 16 16:25:43.085004 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:43.081795 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e2d675d8-edf6-4765-a240-4b723cdd2d38-dbus\") pod \"global-pull-secret-syncer-rqzn4\" (UID: \"e2d675d8-edf6-4765-a240-4b723cdd2d38\") " pod="kube-system/global-pull-secret-syncer-rqzn4" Jul 16 16:25:43.085004 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:43.081820 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/b2410943-db6a-4668-845d-db0ae081a389-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-xz7fp\" (UID: \"b2410943-db6a-4668-845d-db0ae081a389\") " pod="openshift-multus/multus-additional-cni-plugins-xz7fp" Jul 16 16:25:43.085650 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:43.081840 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/aa5f6bf8-3599-45f1-a09d-ef1976f22840-kubelet-dir\") pod \"aws-ebs-csi-driver-node-t4w26\" (UID: \"aa5f6bf8-3599-45f1-a09d-ef1976f22840\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t4w26" Jul 16 16:25:43.085650 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:43.081850 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ba81ab34-5cc0-4f3a-856a-ff794d0dc0e5-etc-sysconfig\") pod \"tuned-8pl5s\" (UID: \"ba81ab34-5cc0-4f3a-856a-ff794d0dc0e5\") " pod="openshift-cluster-node-tuning-operator/tuned-8pl5s" Jul 16 16:25:43.085650 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:43.081889 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/a2cb45f0-0fb3-4761-98b0-a1797a00e647-konnectivity-ca\") pod \"konnectivity-agent-xljrg\" (UID: \"a2cb45f0-0fb3-4761-98b0-a1797a00e647\") " pod="kube-system/konnectivity-agent-xljrg" Jul 16 16:25:43.085650 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:43.081927 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ba81ab34-5cc0-4f3a-856a-ff794d0dc0e5-etc-kubernetes\") pod \"tuned-8pl5s\" (UID: \"ba81ab34-5cc0-4f3a-856a-ff794d0dc0e5\") " pod="openshift-cluster-node-tuning-operator/tuned-8pl5s" Jul 16 16:25:43.085650 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:43.081952 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b2410943-db6a-4668-845d-db0ae081a389-cni-binary-copy\") pod \"multus-additional-cni-plugins-xz7fp\" (UID: \"b2410943-db6a-4668-845d-db0ae081a389\") " pod="openshift-multus/multus-additional-cni-plugins-xz7fp" Jul 16 16:25:43.085650 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:43.081966 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ba81ab34-5cc0-4f3a-856a-ff794d0dc0e5-var-lib-kubelet\") pod \"tuned-8pl5s\" (UID: \"ba81ab34-5cc0-4f3a-856a-ff794d0dc0e5\") " pod="openshift-cluster-node-tuning-operator/tuned-8pl5s" Jul 16 16:25:43.085650 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:43.081994 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/94ec201c-7285-4751-be16-d23d2f2c0290-metrics-certs\") pod \"network-metrics-daemon-bcb7f\" (UID: \"94ec201c-7285-4751-be16-d23d2f2c0290\") " pod="openshift-multus/network-metrics-daemon-bcb7f" Jul 16 16:25:43.085650 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:43.082035 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b20b154f-1a4c-4b25-86db-aebe2d4ffc08-run-systemd\") pod \"ovnkube-node-9bjst\" (UID: \"b20b154f-1a4c-4b25-86db-aebe2d4ffc08\") " pod="openshift-ovn-kubernetes/ovnkube-node-9bjst" Jul 16 16:25:43.085650 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:43.082038 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ba81ab34-5cc0-4f3a-856a-ff794d0dc0e5-etc-kubernetes\") pod \"tuned-8pl5s\" (UID: \"ba81ab34-5cc0-4f3a-856a-ff794d0dc0e5\") " pod="openshift-cluster-node-tuning-operator/tuned-8pl5s" Jul 16 16:25:43.085650 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:43.082096 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ba81ab34-5cc0-4f3a-856a-ff794d0dc0e5-var-lib-kubelet\") pod \"tuned-8pl5s\" (UID: \"ba81ab34-5cc0-4f3a-856a-ff794d0dc0e5\") " pod="openshift-cluster-node-tuning-operator/tuned-8pl5s" Jul 16 16:25:43.085650 ip-10-0-142-101 kubenswrapper[2577]: E0716 16:25:43.082135 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 16:25:43.085650 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:43.082157 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b20b154f-1a4c-4b25-86db-aebe2d4ffc08-run-systemd\") pod \"ovnkube-node-9bjst\" (UID: \"b20b154f-1a4c-4b25-86db-aebe2d4ffc08\") " pod="openshift-ovn-kubernetes/ovnkube-node-9bjst" Jul 16 16:25:43.085650 ip-10-0-142-101 kubenswrapper[2577]: E0716 16:25:43.082186 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/94ec201c-7285-4751-be16-d23d2f2c0290-metrics-certs podName:94ec201c-7285-4751-be16-d23d2f2c0290 nodeName:}" failed. No retries permitted until 2026-07-16 16:25:43.582165735 +0000 UTC m=+2.086223583 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/94ec201c-7285-4751-be16-d23d2f2c0290-metrics-certs") pod "network-metrics-daemon-bcb7f" (UID: "94ec201c-7285-4751-be16-d23d2f2c0290") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 16:25:43.085650 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:43.082205 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/b20b154f-1a4c-4b25-86db-aebe2d4ffc08-env-overrides\") pod \"ovnkube-node-9bjst\" (UID: \"b20b154f-1a4c-4b25-86db-aebe2d4ffc08\") " pod="openshift-ovn-kubernetes/ovnkube-node-9bjst" Jul 16 16:25:43.085650 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:43.082220 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ba81ab34-5cc0-4f3a-856a-ff794d0dc0e5-etc-sysconfig\") pod \"tuned-8pl5s\" (UID: \"ba81ab34-5cc0-4f3a-856a-ff794d0dc0e5\") " pod="openshift-cluster-node-tuning-operator/tuned-8pl5s" Jul 16 16:25:43.085650 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:43.082235 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/aa5f6bf8-3599-45f1-a09d-ef1976f22840-sys-fs\") pod \"aws-ebs-csi-driver-node-t4w26\" (UID: \"aa5f6bf8-3599-45f1-a09d-ef1976f22840\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t4w26" Jul 16 16:25:43.085650 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:43.082265 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ba81ab34-5cc0-4f3a-856a-ff794d0dc0e5-tmp\") pod \"tuned-8pl5s\" (UID: \"ba81ab34-5cc0-4f3a-856a-ff794d0dc0e5\") " pod="openshift-cluster-node-tuning-operator/tuned-8pl5s" Jul 16 16:25:43.086346 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:43.082293 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/3062d489-a9b7-43fb-b1d8-4e39d30acdec-iptables-alerter-script\") pod \"iptables-alerter-cnft7\" (UID: \"3062d489-a9b7-43fb-b1d8-4e39d30acdec\") " pod="openshift-network-operator/iptables-alerter-cnft7" Jul 16 16:25:43.086346 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:43.082333 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fbz6z\" (UniqueName: \"kubernetes.io/projected/3062d489-a9b7-43fb-b1d8-4e39d30acdec-kube-api-access-fbz6z\") pod \"iptables-alerter-cnft7\" (UID: \"3062d489-a9b7-43fb-b1d8-4e39d30acdec\") " pod="openshift-network-operator/iptables-alerter-cnft7" Jul 16 16:25:43.086346 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:43.082355 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/b2410943-db6a-4668-845d-db0ae081a389-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-xz7fp\" (UID: \"b2410943-db6a-4668-845d-db0ae081a389\") " pod="openshift-multus/multus-additional-cni-plugins-xz7fp" Jul 16 16:25:43.086346 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:43.082416 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/aa5f6bf8-3599-45f1-a09d-ef1976f22840-sys-fs\") pod \"aws-ebs-csi-driver-node-t4w26\" (UID: \"aa5f6bf8-3599-45f1-a09d-ef1976f22840\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t4w26" Jul 16 16:25:43.086346 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:43.082359 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s8dcf\" (UniqueName: \"kubernetes.io/projected/84cc596d-4f15-4c84-8625-c7f37ed9dae6-kube-api-access-s8dcf\") pod \"network-check-target-ntktc\" (UID: \"84cc596d-4f15-4c84-8625-c7f37ed9dae6\") " pod="openshift-network-diagnostics/network-check-target-ntktc" Jul 16 16:25:43.086346 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:43.082629 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b20b154f-1a4c-4b25-86db-aebe2d4ffc08-log-socket\") pod \"ovnkube-node-9bjst\" (UID: \"b20b154f-1a4c-4b25-86db-aebe2d4ffc08\") " pod="openshift-ovn-kubernetes/ovnkube-node-9bjst" Jul 16 16:25:43.086346 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:43.082633 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/a2cb45f0-0fb3-4761-98b0-a1797a00e647-konnectivity-ca\") pod \"konnectivity-agent-xljrg\" (UID: \"a2cb45f0-0fb3-4761-98b0-a1797a00e647\") " pod="kube-system/konnectivity-agent-xljrg" Jul 16 16:25:43.086346 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:43.082696 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b20b154f-1a4c-4b25-86db-aebe2d4ffc08-host-run-ovn-kubernetes\") pod \"ovnkube-node-9bjst\" (UID: \"b20b154f-1a4c-4b25-86db-aebe2d4ffc08\") " pod="openshift-ovn-kubernetes/ovnkube-node-9bjst" Jul 16 16:25:43.086346 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:43.082728 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/8f44aa64-f94f-4a4e-a24f-b6a231b6df4b-multus-socket-dir-parent\") pod \"multus-nrzr6\" (UID: \"8f44aa64-f94f-4a4e-a24f-b6a231b6df4b\") " pod="openshift-multus/multus-nrzr6" Jul 16 16:25:43.086346 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:43.082754 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/a2cb45f0-0fb3-4761-98b0-a1797a00e647-agent-certs\") pod \"konnectivity-agent-xljrg\" (UID: \"a2cb45f0-0fb3-4761-98b0-a1797a00e647\") " pod="kube-system/konnectivity-agent-xljrg" Jul 16 16:25:43.086346 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:43.082753 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/b20b154f-1a4c-4b25-86db-aebe2d4ffc08-env-overrides\") pod \"ovnkube-node-9bjst\" (UID: \"b20b154f-1a4c-4b25-86db-aebe2d4ffc08\") " pod="openshift-ovn-kubernetes/ovnkube-node-9bjst" Jul 16 16:25:43.086346 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:43.082783 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g4lrp\" (UniqueName: \"kubernetes.io/projected/94ec201c-7285-4751-be16-d23d2f2c0290-kube-api-access-g4lrp\") pod \"network-metrics-daemon-bcb7f\" (UID: \"94ec201c-7285-4751-be16-d23d2f2c0290\") " pod="openshift-multus/network-metrics-daemon-bcb7f" Jul 16 16:25:43.086346 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:43.082819 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8f44aa64-f94f-4a4e-a24f-b6a231b6df4b-cnibin\") pod \"multus-nrzr6\" (UID: \"8f44aa64-f94f-4a4e-a24f-b6a231b6df4b\") " pod="openshift-multus/multus-nrzr6" Jul 16 16:25:43.086346 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:43.082819 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b20b154f-1a4c-4b25-86db-aebe2d4ffc08-host-run-ovn-kubernetes\") pod \"ovnkube-node-9bjst\" (UID: \"b20b154f-1a4c-4b25-86db-aebe2d4ffc08\") " pod="openshift-ovn-kubernetes/ovnkube-node-9bjst" Jul 16 16:25:43.086346 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:43.082874 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b20b154f-1a4c-4b25-86db-aebe2d4ffc08-log-socket\") pod \"ovnkube-node-9bjst\" (UID: \"b20b154f-1a4c-4b25-86db-aebe2d4ffc08\") " pod="openshift-ovn-kubernetes/ovnkube-node-9bjst" Jul 16 16:25:43.086346 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:43.082913 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/8f44aa64-f94f-4a4e-a24f-b6a231b6df4b-multus-socket-dir-parent\") pod \"multus-nrzr6\" (UID: \"8f44aa64-f94f-4a4e-a24f-b6a231b6df4b\") " pod="openshift-multus/multus-nrzr6" Jul 16 16:25:43.086346 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:43.082929 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/b20b154f-1a4c-4b25-86db-aebe2d4ffc08-ovnkube-script-lib\") pod \"ovnkube-node-9bjst\" (UID: \"b20b154f-1a4c-4b25-86db-aebe2d4ffc08\") " pod="openshift-ovn-kubernetes/ovnkube-node-9bjst" Jul 16 16:25:43.086922 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:43.082958 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/aa5f6bf8-3599-45f1-a09d-ef1976f22840-registration-dir\") pod \"aws-ebs-csi-driver-node-t4w26\" (UID: \"aa5f6bf8-3599-45f1-a09d-ef1976f22840\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t4w26" Jul 16 16:25:43.086922 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:43.083006 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/aa5f6bf8-3599-45f1-a09d-ef1976f22840-etc-selinux\") pod \"aws-ebs-csi-driver-node-t4w26\" (UID: \"aa5f6bf8-3599-45f1-a09d-ef1976f22840\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t4w26" Jul 16 16:25:43.086922 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:43.083031 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/93f30808-1a75-44b5-801f-8faeceb72afb-tmp-dir\") pod \"node-resolver-6sc67\" (UID: \"93f30808-1a75-44b5-801f-8faeceb72afb\") " pod="openshift-dns/node-resolver-6sc67" Jul 16 16:25:43.086922 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:43.083056 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7bbtp\" (UniqueName: \"kubernetes.io/projected/cbd027ac-97d9-4769-a867-47b7b8256cd0-kube-api-access-7bbtp\") pod \"node-ca-hwq4x\" (UID: \"cbd027ac-97d9-4769-a867-47b7b8256cd0\") " pod="openshift-image-registry/node-ca-hwq4x" Jul 16 16:25:43.086922 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:43.083130 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/aa5f6bf8-3599-45f1-a09d-ef1976f22840-registration-dir\") pod \"aws-ebs-csi-driver-node-t4w26\" (UID: \"aa5f6bf8-3599-45f1-a09d-ef1976f22840\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t4w26" Jul 16 16:25:43.086922 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:43.083188 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8f44aa64-f94f-4a4e-a24f-b6a231b6df4b-cnibin\") pod \"multus-nrzr6\" (UID: \"8f44aa64-f94f-4a4e-a24f-b6a231b6df4b\") " pod="openshift-multus/multus-nrzr6" Jul 16 16:25:43.086922 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:43.083204 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/3062d489-a9b7-43fb-b1d8-4e39d30acdec-iptables-alerter-script\") pod \"iptables-alerter-cnft7\" (UID: \"3062d489-a9b7-43fb-b1d8-4e39d30acdec\") " pod="openshift-network-operator/iptables-alerter-cnft7" Jul 16 16:25:43.086922 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:43.083344 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/aa5f6bf8-3599-45f1-a09d-ef1976f22840-etc-selinux\") pod \"aws-ebs-csi-driver-node-t4w26\" (UID: \"aa5f6bf8-3599-45f1-a09d-ef1976f22840\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t4w26" Jul 16 16:25:43.086922 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:43.083619 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/93f30808-1a75-44b5-801f-8faeceb72afb-tmp-dir\") pod \"node-resolver-6sc67\" (UID: \"93f30808-1a75-44b5-801f-8faeceb72afb\") " pod="openshift-dns/node-resolver-6sc67" Jul 16 16:25:43.086922 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:43.083732 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/b20b154f-1a4c-4b25-86db-aebe2d4ffc08-ovnkube-script-lib\") pod \"ovnkube-node-9bjst\" (UID: \"b20b154f-1a4c-4b25-86db-aebe2d4ffc08\") " pod="openshift-ovn-kubernetes/ovnkube-node-9bjst" Jul 16 16:25:43.086922 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:43.084182 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ba81ab34-5cc0-4f3a-856a-ff794d0dc0e5-etc-tuned\") pod \"tuned-8pl5s\" (UID: \"ba81ab34-5cc0-4f3a-856a-ff794d0dc0e5\") " pod="openshift-cluster-node-tuning-operator/tuned-8pl5s" Jul 16 16:25:43.086922 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:43.084248 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/b20b154f-1a4c-4b25-86db-aebe2d4ffc08-ovn-node-metrics-cert\") pod \"ovnkube-node-9bjst\" (UID: \"b20b154f-1a4c-4b25-86db-aebe2d4ffc08\") " pod="openshift-ovn-kubernetes/ovnkube-node-9bjst" Jul 16 16:25:43.086922 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:43.084898 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ba81ab34-5cc0-4f3a-856a-ff794d0dc0e5-tmp\") pod \"tuned-8pl5s\" (UID: \"ba81ab34-5cc0-4f3a-856a-ff794d0dc0e5\") " pod="openshift-cluster-node-tuning-operator/tuned-8pl5s" Jul 16 16:25:43.086922 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:43.086137 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/a2cb45f0-0fb3-4761-98b0-a1797a00e647-agent-certs\") pod \"konnectivity-agent-xljrg\" (UID: \"a2cb45f0-0fb3-4761-98b0-a1797a00e647\") " pod="kube-system/konnectivity-agent-xljrg" Jul 16 16:25:43.113424 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:43.113391 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w9lm2\" (UniqueName: \"kubernetes.io/projected/8f44aa64-f94f-4a4e-a24f-b6a231b6df4b-kube-api-access-w9lm2\") pod \"multus-nrzr6\" (UID: \"8f44aa64-f94f-4a4e-a24f-b6a231b6df4b\") " pod="openshift-multus/multus-nrzr6" Jul 16 16:25:43.114164 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:43.114139 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zr76g\" (UniqueName: \"kubernetes.io/projected/aa5f6bf8-3599-45f1-a09d-ef1976f22840-kube-api-access-zr76g\") pod \"aws-ebs-csi-driver-node-t4w26\" (UID: \"aa5f6bf8-3599-45f1-a09d-ef1976f22840\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t4w26" Jul 16 16:25:43.114273 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:43.114201 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ffgxf\" (UniqueName: \"kubernetes.io/projected/93f30808-1a75-44b5-801f-8faeceb72afb-kube-api-access-ffgxf\") pod \"node-resolver-6sc67\" (UID: \"93f30808-1a75-44b5-801f-8faeceb72afb\") " pod="openshift-dns/node-resolver-6sc67" Jul 16 16:25:43.117828 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:43.117810 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j27b9\" (UniqueName: \"kubernetes.io/projected/b20b154f-1a4c-4b25-86db-aebe2d4ffc08-kube-api-access-j27b9\") pod \"ovnkube-node-9bjst\" (UID: \"b20b154f-1a4c-4b25-86db-aebe2d4ffc08\") " pod="openshift-ovn-kubernetes/ovnkube-node-9bjst" Jul 16 16:25:43.141020 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:43.140942 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-46nv8\" (UniqueName: \"kubernetes.io/projected/b2410943-db6a-4668-845d-db0ae081a389-kube-api-access-46nv8\") pod \"multus-additional-cni-plugins-xz7fp\" (UID: \"b2410943-db6a-4668-845d-db0ae081a389\") " pod="openshift-multus/multus-additional-cni-plugins-xz7fp" Jul 16 16:25:43.143346 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:43.143314 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fbz6z\" (UniqueName: \"kubernetes.io/projected/3062d489-a9b7-43fb-b1d8-4e39d30acdec-kube-api-access-fbz6z\") pod \"iptables-alerter-cnft7\" (UID: \"3062d489-a9b7-43fb-b1d8-4e39d30acdec\") " pod="openshift-network-operator/iptables-alerter-cnft7" Jul 16 16:25:43.150038 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:43.150009 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g4lrp\" (UniqueName: \"kubernetes.io/projected/94ec201c-7285-4751-be16-d23d2f2c0290-kube-api-access-g4lrp\") pod \"network-metrics-daemon-bcb7f\" (UID: \"94ec201c-7285-4751-be16-d23d2f2c0290\") " pod="openshift-multus/network-metrics-daemon-bcb7f" Jul 16 16:25:43.150352 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:43.150332 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4ht8t\" (UniqueName: \"kubernetes.io/projected/ba81ab34-5cc0-4f3a-856a-ff794d0dc0e5-kube-api-access-4ht8t\") pod \"tuned-8pl5s\" (UID: \"ba81ab34-5cc0-4f3a-856a-ff794d0dc0e5\") " pod="openshift-cluster-node-tuning-operator/tuned-8pl5s" Jul 16 16:25:43.151764 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:43.151738 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7bbtp\" (UniqueName: \"kubernetes.io/projected/cbd027ac-97d9-4769-a867-47b7b8256cd0-kube-api-access-7bbtp\") pod \"node-ca-hwq4x\" (UID: \"cbd027ac-97d9-4769-a867-47b7b8256cd0\") " pod="openshift-image-registry/node-ca-hwq4x" Jul 16 16:25:43.153857 ip-10-0-142-101 kubenswrapper[2577]: E0716 16:25:43.153838 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 16 16:25:43.153912 ip-10-0-142-101 kubenswrapper[2577]: E0716 16:25:43.153862 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 16 16:25:43.153912 ip-10-0-142-101 kubenswrapper[2577]: E0716 16:25:43.153877 2577 projected.go:196] Error preparing data for projected volume kube-api-access-s8dcf for pod openshift-network-diagnostics/network-check-target-ntktc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 16:25:43.153987 ip-10-0-142-101 kubenswrapper[2577]: E0716 16:25:43.153936 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/84cc596d-4f15-4c84-8625-c7f37ed9dae6-kube-api-access-s8dcf podName:84cc596d-4f15-4c84-8625-c7f37ed9dae6 nodeName:}" failed. No retries permitted until 2026-07-16 16:25:43.65392018 +0000 UTC m=+2.157978011 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-s8dcf" (UniqueName: "kubernetes.io/projected/84cc596d-4f15-4c84-8625-c7f37ed9dae6-kube-api-access-s8dcf") pod "network-check-target-ntktc" (UID: "84cc596d-4f15-4c84-8625-c7f37ed9dae6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 16:25:43.225372 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:43.225334 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-nrzr6" Jul 16 16:25:43.236134 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:43.236107 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f44aa64_f94f_4a4e_a24f_b6a231b6df4b.slice/crio-abc11892040a9073b213a717e46c41f34a71ba17e4a175bee2d269635b46c086 WatchSource:0}: Error finding container abc11892040a9073b213a717e46c41f34a71ba17e4a175bee2d269635b46c086: Status 404 returned error can't find the container with id abc11892040a9073b213a717e46c41f34a71ba17e4a175bee2d269635b46c086 Jul 16 16:25:43.246653 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:43.246629 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-9bjst" Jul 16 16:25:43.251897 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:43.251866 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-xljrg" Jul 16 16:25:43.254133 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:43.254114 2577 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 16 16:25:43.256223 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:43.256203 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb20b154f_1a4c_4b25_86db_aebe2d4ffc08.slice/crio-43db29fd20103f0ca7ee5e814ced88d3a04553b9e9fb6392dca265d63963cdad WatchSource:0}: Error finding container 43db29fd20103f0ca7ee5e814ced88d3a04553b9e9fb6392dca265d63963cdad: Status 404 returned error can't find the container with id 43db29fd20103f0ca7ee5e814ced88d3a04553b9e9fb6392dca265d63963cdad Jul 16 16:25:43.261765 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:43.261738 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2cb45f0_0fb3_4761_98b0_a1797a00e647.slice/crio-d642e318c05a71d23e0967f5cd2f7aeb946dbfcff3d2dedd2242ab2d95772d35 WatchSource:0}: Error finding container d642e318c05a71d23e0967f5cd2f7aeb946dbfcff3d2dedd2242ab2d95772d35: Status 404 returned error can't find the container with id d642e318c05a71d23e0967f5cd2f7aeb946dbfcff3d2dedd2242ab2d95772d35 Jul 16 16:25:43.274257 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:43.274239 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t4w26" Jul 16 16:25:43.276235 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:43.276210 2577 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 16 16:25:43.284615 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:43.284588 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa5f6bf8_3599_45f1_a09d_ef1976f22840.slice/crio-90e6163497f787cff17e2e7f8ccb344a343254fca8633b9e9340f1c84b91c840 WatchSource:0}: Error finding container 90e6163497f787cff17e2e7f8ccb344a343254fca8633b9e9340f1c84b91c840: Status 404 returned error can't find the container with id 90e6163497f787cff17e2e7f8ccb344a343254fca8633b9e9340f1c84b91c840 Jul 16 16:25:43.285777 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:43.285757 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-8pl5s" Jul 16 16:25:43.294358 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:43.294333 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba81ab34_5cc0_4f3a_856a_ff794d0dc0e5.slice/crio-53db9d12fb5ad390ca9f456e8d676d13fb35f06605b3737f6ecd88ecb5b1d7ac WatchSource:0}: Error finding container 53db9d12fb5ad390ca9f456e8d676d13fb35f06605b3737f6ecd88ecb5b1d7ac: Status 404 returned error can't find the container with id 53db9d12fb5ad390ca9f456e8d676d13fb35f06605b3737f6ecd88ecb5b1d7ac Jul 16 16:25:43.303389 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:43.303368 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-6sc67" Jul 16 16:25:43.312548 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:43.312522 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93f30808_1a75_44b5_801f_8faeceb72afb.slice/crio-15e969c4238576f5d95d61a47d77197121b367ce119694cfce1e7fe32702763f WatchSource:0}: Error finding container 15e969c4238576f5d95d61a47d77197121b367ce119694cfce1e7fe32702763f: Status 404 returned error can't find the container with id 15e969c4238576f5d95d61a47d77197121b367ce119694cfce1e7fe32702763f Jul 16 16:25:43.322153 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:43.322133 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-hwq4x" Jul 16 16:25:43.340107 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:43.340083 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-cnft7" Jul 16 16:25:43.345016 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:43.344991 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-xz7fp" Jul 16 16:25:43.350618 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:43.350593 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3062d489_a9b7_43fb_b1d8_4e39d30acdec.slice/crio-2eccb94a84aa858ecf38a9357957afd90f11bb56f468968edc9969850634c5bd WatchSource:0}: Error finding container 2eccb94a84aa858ecf38a9357957afd90f11bb56f468968edc9969850634c5bd: Status 404 returned error can't find the container with id 2eccb94a84aa858ecf38a9357957afd90f11bb56f468968edc9969850634c5bd Jul 16 16:25:43.355356 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:25:43.355333 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2410943_db6a_4668_845d_db0ae081a389.slice/crio-cf7330fc64af1aee8b073e33f53c015eec3a1dfc05c12acf8e8f78ce1f0aeaa2 WatchSource:0}: Error finding container cf7330fc64af1aee8b073e33f53c015eec3a1dfc05c12acf8e8f78ce1f0aeaa2: Status 404 returned error can't find the container with id cf7330fc64af1aee8b073e33f53c015eec3a1dfc05c12acf8e8f78ce1f0aeaa2 Jul 16 16:25:43.586703 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:43.586590 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/94ec201c-7285-4751-be16-d23d2f2c0290-metrics-certs\") pod \"network-metrics-daemon-bcb7f\" (UID: \"94ec201c-7285-4751-be16-d23d2f2c0290\") " pod="openshift-multus/network-metrics-daemon-bcb7f" Jul 16 16:25:43.586703 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:43.586660 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e2d675d8-edf6-4765-a240-4b723cdd2d38-original-pull-secret\") pod \"global-pull-secret-syncer-rqzn4\" (UID: \"e2d675d8-edf6-4765-a240-4b723cdd2d38\") " pod="kube-system/global-pull-secret-syncer-rqzn4" Jul 16 16:25:43.586957 ip-10-0-142-101 kubenswrapper[2577]: E0716 16:25:43.586772 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 16:25:43.586957 ip-10-0-142-101 kubenswrapper[2577]: E0716 16:25:43.586813 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 16 16:25:43.586957 ip-10-0-142-101 kubenswrapper[2577]: E0716 16:25:43.586854 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/94ec201c-7285-4751-be16-d23d2f2c0290-metrics-certs podName:94ec201c-7285-4751-be16-d23d2f2c0290 nodeName:}" failed. No retries permitted until 2026-07-16 16:25:44.586834 +0000 UTC m=+3.090891830 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/94ec201c-7285-4751-be16-d23d2f2c0290-metrics-certs") pod "network-metrics-daemon-bcb7f" (UID: "94ec201c-7285-4751-be16-d23d2f2c0290") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 16:25:43.586957 ip-10-0-142-101 kubenswrapper[2577]: E0716 16:25:43.586874 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e2d675d8-edf6-4765-a240-4b723cdd2d38-original-pull-secret podName:e2d675d8-edf6-4765-a240-4b723cdd2d38 nodeName:}" failed. No retries permitted until 2026-07-16 16:25:44.586864944 +0000 UTC m=+3.090922775 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e2d675d8-edf6-4765-a240-4b723cdd2d38-original-pull-secret") pod "global-pull-secret-syncer-rqzn4" (UID: "e2d675d8-edf6-4765-a240-4b723cdd2d38") : object "kube-system"/"original-pull-secret" not registered Jul 16 16:25:43.687713 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:43.687678 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s8dcf\" (UniqueName: \"kubernetes.io/projected/84cc596d-4f15-4c84-8625-c7f37ed9dae6-kube-api-access-s8dcf\") pod \"network-check-target-ntktc\" (UID: \"84cc596d-4f15-4c84-8625-c7f37ed9dae6\") " pod="openshift-network-diagnostics/network-check-target-ntktc" Jul 16 16:25:43.688176 ip-10-0-142-101 kubenswrapper[2577]: E0716 16:25:43.688146 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 16 16:25:43.688344 ip-10-0-142-101 kubenswrapper[2577]: E0716 16:25:43.688183 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 16 16:25:43.688344 ip-10-0-142-101 kubenswrapper[2577]: E0716 16:25:43.688208 2577 projected.go:196] Error preparing data for projected volume kube-api-access-s8dcf for pod openshift-network-diagnostics/network-check-target-ntktc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 16:25:43.688344 ip-10-0-142-101 kubenswrapper[2577]: E0716 16:25:43.688289 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/84cc596d-4f15-4c84-8625-c7f37ed9dae6-kube-api-access-s8dcf podName:84cc596d-4f15-4c84-8625-c7f37ed9dae6 nodeName:}" failed. No retries permitted until 2026-07-16 16:25:44.688256558 +0000 UTC m=+3.192314408 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-s8dcf" (UniqueName: "kubernetes.io/projected/84cc596d-4f15-4c84-8625-c7f37ed9dae6-kube-api-access-s8dcf") pod "network-check-target-ntktc" (UID: "84cc596d-4f15-4c84-8625-c7f37ed9dae6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 16:25:43.975070 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:43.974703 2577 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 16 16:25:43.978260 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:43.978193 2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-15 16:20:43 +0000 UTC" deadline="2028-03-28 08:20:59.867575606 +0000 UTC" Jul 16 16:25:43.978260 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:43.978260 2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14895h55m15.889320194s" Jul 16 16:25:44.013889 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:44.013117 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bcb7f" Jul 16 16:25:44.013889 ip-10-0-142-101 kubenswrapper[2577]: E0716 16:25:44.013282 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bcb7f" podUID="94ec201c-7285-4751-be16-d23d2f2c0290" Jul 16 16:25:44.023142 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:44.023077 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-hwq4x" event={"ID":"cbd027ac-97d9-4769-a867-47b7b8256cd0","Type":"ContainerStarted","Data":"8f65d0355a9d2b93328ab11787985bf1dee53ef3eb083a150d14af4782209102"} Jul 16 16:25:44.028242 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:44.028177 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-8pl5s" event={"ID":"ba81ab34-5cc0-4f3a-856a-ff794d0dc0e5","Type":"ContainerStarted","Data":"53db9d12fb5ad390ca9f456e8d676d13fb35f06605b3737f6ecd88ecb5b1d7ac"} Jul 16 16:25:44.037327 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:44.037286 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t4w26" event={"ID":"aa5f6bf8-3599-45f1-a09d-ef1976f22840","Type":"ContainerStarted","Data":"90e6163497f787cff17e2e7f8ccb344a343254fca8633b9e9340f1c84b91c840"} Jul 16 16:25:44.047971 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:44.047858 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9bjst" event={"ID":"b20b154f-1a4c-4b25-86db-aebe2d4ffc08","Type":"ContainerStarted","Data":"43db29fd20103f0ca7ee5e814ced88d3a04553b9e9fb6392dca265d63963cdad"} Jul 16 16:25:44.051804 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:44.051713 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-nrzr6" event={"ID":"8f44aa64-f94f-4a4e-a24f-b6a231b6df4b","Type":"ContainerStarted","Data":"abc11892040a9073b213a717e46c41f34a71ba17e4a175bee2d269635b46c086"} Jul 16 16:25:44.058339 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:44.058298 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-101.ec2.internal" event={"ID":"f0ff4a720fce2411438eb201c85e2744","Type":"ContainerStarted","Data":"67f89ffd44e5fb87862e941e87ad81dcbfa617699f28f2cdc7fea7264c7d028f"} Jul 16 16:25:44.063176 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:44.063129 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-101.ec2.internal" event={"ID":"077fca01a64486b516cac217b95ed2fa","Type":"ContainerStarted","Data":"7e9f458b7462f6d989326a3d998b561ee29ef4d219ed00e40aa9ef83905caf38"} Jul 16 16:25:44.068730 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:44.068664 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xz7fp" event={"ID":"b2410943-db6a-4668-845d-db0ae081a389","Type":"ContainerStarted","Data":"cf7330fc64af1aee8b073e33f53c015eec3a1dfc05c12acf8e8f78ce1f0aeaa2"} Jul 16 16:25:44.076684 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:44.076639 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-cnft7" event={"ID":"3062d489-a9b7-43fb-b1d8-4e39d30acdec","Type":"ContainerStarted","Data":"2eccb94a84aa858ecf38a9357957afd90f11bb56f468968edc9969850634c5bd"} Jul 16 16:25:44.084782 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:44.084741 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-6sc67" event={"ID":"93f30808-1a75-44b5-801f-8faeceb72afb","Type":"ContainerStarted","Data":"15e969c4238576f5d95d61a47d77197121b367ce119694cfce1e7fe32702763f"} Jul 16 16:25:44.087326 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:44.087263 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-xljrg" event={"ID":"a2cb45f0-0fb3-4761-98b0-a1797a00e647","Type":"ContainerStarted","Data":"d642e318c05a71d23e0967f5cd2f7aeb946dbfcff3d2dedd2242ab2d95772d35"} Jul 16 16:25:44.593929 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:44.593885 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/94ec201c-7285-4751-be16-d23d2f2c0290-metrics-certs\") pod \"network-metrics-daemon-bcb7f\" (UID: \"94ec201c-7285-4751-be16-d23d2f2c0290\") " pod="openshift-multus/network-metrics-daemon-bcb7f" Jul 16 16:25:44.594131 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:44.593993 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e2d675d8-edf6-4765-a240-4b723cdd2d38-original-pull-secret\") pod \"global-pull-secret-syncer-rqzn4\" (UID: \"e2d675d8-edf6-4765-a240-4b723cdd2d38\") " pod="kube-system/global-pull-secret-syncer-rqzn4" Jul 16 16:25:44.594131 ip-10-0-142-101 kubenswrapper[2577]: E0716 16:25:44.594072 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 16:25:44.594131 ip-10-0-142-101 kubenswrapper[2577]: E0716 16:25:44.594118 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 16 16:25:44.594320 ip-10-0-142-101 kubenswrapper[2577]: E0716 16:25:44.594154 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/94ec201c-7285-4751-be16-d23d2f2c0290-metrics-certs podName:94ec201c-7285-4751-be16-d23d2f2c0290 nodeName:}" failed. No retries permitted until 2026-07-16 16:25:46.594130651 +0000 UTC m=+5.098188482 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/94ec201c-7285-4751-be16-d23d2f2c0290-metrics-certs") pod "network-metrics-daemon-bcb7f" (UID: "94ec201c-7285-4751-be16-d23d2f2c0290") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 16:25:44.594320 ip-10-0-142-101 kubenswrapper[2577]: E0716 16:25:44.594176 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e2d675d8-edf6-4765-a240-4b723cdd2d38-original-pull-secret podName:e2d675d8-edf6-4765-a240-4b723cdd2d38 nodeName:}" failed. No retries permitted until 2026-07-16 16:25:46.594166405 +0000 UTC m=+5.098224236 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e2d675d8-edf6-4765-a240-4b723cdd2d38-original-pull-secret") pod "global-pull-secret-syncer-rqzn4" (UID: "e2d675d8-edf6-4765-a240-4b723cdd2d38") : object "kube-system"/"original-pull-secret" not registered Jul 16 16:25:44.694474 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:44.694398 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s8dcf\" (UniqueName: \"kubernetes.io/projected/84cc596d-4f15-4c84-8625-c7f37ed9dae6-kube-api-access-s8dcf\") pod \"network-check-target-ntktc\" (UID: \"84cc596d-4f15-4c84-8625-c7f37ed9dae6\") " pod="openshift-network-diagnostics/network-check-target-ntktc" Jul 16 16:25:44.694673 ip-10-0-142-101 kubenswrapper[2577]: E0716 16:25:44.694640 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 16 16:25:44.694673 ip-10-0-142-101 kubenswrapper[2577]: E0716 16:25:44.694664 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 16 16:25:44.694795 ip-10-0-142-101 kubenswrapper[2577]: E0716 16:25:44.694678 2577 projected.go:196] Error preparing data for projected volume kube-api-access-s8dcf for pod openshift-network-diagnostics/network-check-target-ntktc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 16:25:44.694795 ip-10-0-142-101 kubenswrapper[2577]: E0716 16:25:44.694744 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/84cc596d-4f15-4c84-8625-c7f37ed9dae6-kube-api-access-s8dcf podName:84cc596d-4f15-4c84-8625-c7f37ed9dae6 nodeName:}" failed. No retries permitted until 2026-07-16 16:25:46.694724046 +0000 UTC m=+5.198781901 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-s8dcf" (UniqueName: "kubernetes.io/projected/84cc596d-4f15-4c84-8625-c7f37ed9dae6-kube-api-access-s8dcf") pod "network-check-target-ntktc" (UID: "84cc596d-4f15-4c84-8625-c7f37ed9dae6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 16:25:44.978964 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:44.978916 2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-15 16:20:43 +0000 UTC" deadline="2028-01-31 22:41:04.821453667 +0000 UTC" Jul 16 16:25:44.978964 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:44.978962 2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13542h15m19.842496897s" Jul 16 16:25:45.012213 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:45.012015 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rqzn4" Jul 16 16:25:45.012213 ip-10-0-142-101 kubenswrapper[2577]: E0716 16:25:45.012162 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rqzn4" podUID="e2d675d8-edf6-4765-a240-4b723cdd2d38" Jul 16 16:25:45.012618 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:45.012599 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ntktc" Jul 16 16:25:45.012721 ip-10-0-142-101 kubenswrapper[2577]: E0716 16:25:45.012700 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ntktc" podUID="84cc596d-4f15-4c84-8625-c7f37ed9dae6" Jul 16 16:25:45.688903 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:45.688870 2577 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 16 16:25:46.012541 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:46.012067 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bcb7f" Jul 16 16:25:46.012541 ip-10-0-142-101 kubenswrapper[2577]: E0716 16:25:46.012226 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bcb7f" podUID="94ec201c-7285-4751-be16-d23d2f2c0290" Jul 16 16:25:46.612382 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:46.612304 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/94ec201c-7285-4751-be16-d23d2f2c0290-metrics-certs\") pod \"network-metrics-daemon-bcb7f\" (UID: \"94ec201c-7285-4751-be16-d23d2f2c0290\") " pod="openshift-multus/network-metrics-daemon-bcb7f" Jul 16 16:25:46.612602 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:46.612396 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e2d675d8-edf6-4765-a240-4b723cdd2d38-original-pull-secret\") pod \"global-pull-secret-syncer-rqzn4\" (UID: \"e2d675d8-edf6-4765-a240-4b723cdd2d38\") " pod="kube-system/global-pull-secret-syncer-rqzn4" Jul 16 16:25:46.612602 ip-10-0-142-101 kubenswrapper[2577]: E0716 16:25:46.612485 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 16:25:46.612602 ip-10-0-142-101 kubenswrapper[2577]: E0716 16:25:46.612523 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 16 16:25:46.612602 ip-10-0-142-101 kubenswrapper[2577]: E0716 16:25:46.612570 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/94ec201c-7285-4751-be16-d23d2f2c0290-metrics-certs podName:94ec201c-7285-4751-be16-d23d2f2c0290 nodeName:}" failed. No retries permitted until 2026-07-16 16:25:50.61254662 +0000 UTC m=+9.116604454 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/94ec201c-7285-4751-be16-d23d2f2c0290-metrics-certs") pod "network-metrics-daemon-bcb7f" (UID: "94ec201c-7285-4751-be16-d23d2f2c0290") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 16:25:46.612602 ip-10-0-142-101 kubenswrapper[2577]: E0716 16:25:46.612595 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e2d675d8-edf6-4765-a240-4b723cdd2d38-original-pull-secret podName:e2d675d8-edf6-4765-a240-4b723cdd2d38 nodeName:}" failed. No retries permitted until 2026-07-16 16:25:50.612583408 +0000 UTC m=+9.116641240 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e2d675d8-edf6-4765-a240-4b723cdd2d38-original-pull-secret") pod "global-pull-secret-syncer-rqzn4" (UID: "e2d675d8-edf6-4765-a240-4b723cdd2d38") : object "kube-system"/"original-pull-secret" not registered Jul 16 16:25:46.713643 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:46.713604 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s8dcf\" (UniqueName: \"kubernetes.io/projected/84cc596d-4f15-4c84-8625-c7f37ed9dae6-kube-api-access-s8dcf\") pod \"network-check-target-ntktc\" (UID: \"84cc596d-4f15-4c84-8625-c7f37ed9dae6\") " pod="openshift-network-diagnostics/network-check-target-ntktc" Jul 16 16:25:46.713855 ip-10-0-142-101 kubenswrapper[2577]: E0716 16:25:46.713833 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 16 16:25:46.713855 ip-10-0-142-101 kubenswrapper[2577]: E0716 16:25:46.713853 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 16 16:25:46.713976 ip-10-0-142-101 kubenswrapper[2577]: E0716 16:25:46.713867 2577 projected.go:196] Error preparing data for projected volume kube-api-access-s8dcf for pod openshift-network-diagnostics/network-check-target-ntktc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 16:25:46.713976 ip-10-0-142-101 kubenswrapper[2577]: E0716 16:25:46.713947 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/84cc596d-4f15-4c84-8625-c7f37ed9dae6-kube-api-access-s8dcf podName:84cc596d-4f15-4c84-8625-c7f37ed9dae6 nodeName:}" failed. No retries permitted until 2026-07-16 16:25:50.713917229 +0000 UTC m=+9.217975082 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-s8dcf" (UniqueName: "kubernetes.io/projected/84cc596d-4f15-4c84-8625-c7f37ed9dae6-kube-api-access-s8dcf") pod "network-check-target-ntktc" (UID: "84cc596d-4f15-4c84-8625-c7f37ed9dae6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 16:25:47.011504 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:47.011410 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ntktc" Jul 16 16:25:47.011690 ip-10-0-142-101 kubenswrapper[2577]: E0716 16:25:47.011565 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ntktc" podUID="84cc596d-4f15-4c84-8625-c7f37ed9dae6" Jul 16 16:25:47.011690 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:47.011633 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rqzn4" Jul 16 16:25:47.011817 ip-10-0-142-101 kubenswrapper[2577]: E0716 16:25:47.011713 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rqzn4" podUID="e2d675d8-edf6-4765-a240-4b723cdd2d38" Jul 16 16:25:48.013024 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:48.012987 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bcb7f" Jul 16 16:25:48.013558 ip-10-0-142-101 kubenswrapper[2577]: E0716 16:25:48.013163 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bcb7f" podUID="94ec201c-7285-4751-be16-d23d2f2c0290" Jul 16 16:25:49.012320 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:49.011979 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rqzn4" Jul 16 16:25:49.012320 ip-10-0-142-101 kubenswrapper[2577]: E0716 16:25:49.012141 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rqzn4" podUID="e2d675d8-edf6-4765-a240-4b723cdd2d38" Jul 16 16:25:49.012320 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:49.012301 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ntktc" Jul 16 16:25:49.012676 ip-10-0-142-101 kubenswrapper[2577]: E0716 16:25:49.012438 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ntktc" podUID="84cc596d-4f15-4c84-8625-c7f37ed9dae6" Jul 16 16:25:50.011623 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:50.011536 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bcb7f" Jul 16 16:25:50.012146 ip-10-0-142-101 kubenswrapper[2577]: E0716 16:25:50.011720 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bcb7f" podUID="94ec201c-7285-4751-be16-d23d2f2c0290" Jul 16 16:25:50.648125 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:50.647624 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e2d675d8-edf6-4765-a240-4b723cdd2d38-original-pull-secret\") pod \"global-pull-secret-syncer-rqzn4\" (UID: \"e2d675d8-edf6-4765-a240-4b723cdd2d38\") " pod="kube-system/global-pull-secret-syncer-rqzn4" Jul 16 16:25:50.648125 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:50.647702 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/94ec201c-7285-4751-be16-d23d2f2c0290-metrics-certs\") pod \"network-metrics-daemon-bcb7f\" (UID: \"94ec201c-7285-4751-be16-d23d2f2c0290\") " pod="openshift-multus/network-metrics-daemon-bcb7f" Jul 16 16:25:50.648125 ip-10-0-142-101 kubenswrapper[2577]: E0716 16:25:50.647853 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 16:25:50.648125 ip-10-0-142-101 kubenswrapper[2577]: E0716 16:25:50.647924 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/94ec201c-7285-4751-be16-d23d2f2c0290-metrics-certs podName:94ec201c-7285-4751-be16-d23d2f2c0290 nodeName:}" failed. No retries permitted until 2026-07-16 16:25:58.647902893 +0000 UTC m=+17.151960726 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/94ec201c-7285-4751-be16-d23d2f2c0290-metrics-certs") pod "network-metrics-daemon-bcb7f" (UID: "94ec201c-7285-4751-be16-d23d2f2c0290") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 16:25:50.648125 ip-10-0-142-101 kubenswrapper[2577]: E0716 16:25:50.647976 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 16 16:25:50.648125 ip-10-0-142-101 kubenswrapper[2577]: E0716 16:25:50.648023 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e2d675d8-edf6-4765-a240-4b723cdd2d38-original-pull-secret podName:e2d675d8-edf6-4765-a240-4b723cdd2d38 nodeName:}" failed. No retries permitted until 2026-07-16 16:25:58.648008544 +0000 UTC m=+17.152066383 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e2d675d8-edf6-4765-a240-4b723cdd2d38-original-pull-secret") pod "global-pull-secret-syncer-rqzn4" (UID: "e2d675d8-edf6-4765-a240-4b723cdd2d38") : object "kube-system"/"original-pull-secret" not registered Jul 16 16:25:50.748692 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:50.748641 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s8dcf\" (UniqueName: \"kubernetes.io/projected/84cc596d-4f15-4c84-8625-c7f37ed9dae6-kube-api-access-s8dcf\") pod \"network-check-target-ntktc\" (UID: \"84cc596d-4f15-4c84-8625-c7f37ed9dae6\") " pod="openshift-network-diagnostics/network-check-target-ntktc" Jul 16 16:25:50.748882 ip-10-0-142-101 kubenswrapper[2577]: E0716 16:25:50.748844 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 16 16:25:50.748882 ip-10-0-142-101 kubenswrapper[2577]: E0716 16:25:50.748871 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 16 16:25:50.748882 ip-10-0-142-101 kubenswrapper[2577]: E0716 16:25:50.748884 2577 projected.go:196] Error preparing data for projected volume kube-api-access-s8dcf for pod openshift-network-diagnostics/network-check-target-ntktc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 16:25:50.749032 ip-10-0-142-101 kubenswrapper[2577]: E0716 16:25:50.748953 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/84cc596d-4f15-4c84-8625-c7f37ed9dae6-kube-api-access-s8dcf podName:84cc596d-4f15-4c84-8625-c7f37ed9dae6 nodeName:}" failed. No retries permitted until 2026-07-16 16:25:58.748934496 +0000 UTC m=+17.252992339 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-s8dcf" (UniqueName: "kubernetes.io/projected/84cc596d-4f15-4c84-8625-c7f37ed9dae6-kube-api-access-s8dcf") pod "network-check-target-ntktc" (UID: "84cc596d-4f15-4c84-8625-c7f37ed9dae6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 16:25:51.011526 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:51.011359 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ntktc" Jul 16 16:25:51.011526 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:51.011417 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rqzn4" Jul 16 16:25:51.012048 ip-10-0-142-101 kubenswrapper[2577]: E0716 16:25:51.011517 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ntktc" podUID="84cc596d-4f15-4c84-8625-c7f37ed9dae6" Jul 16 16:25:51.012048 ip-10-0-142-101 kubenswrapper[2577]: E0716 16:25:51.011652 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rqzn4" podUID="e2d675d8-edf6-4765-a240-4b723cdd2d38" Jul 16 16:25:52.013250 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:52.013216 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bcb7f" Jul 16 16:25:52.013736 ip-10-0-142-101 kubenswrapper[2577]: E0716 16:25:52.013348 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bcb7f" podUID="94ec201c-7285-4751-be16-d23d2f2c0290" Jul 16 16:25:53.012366 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:53.012330 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rqzn4" Jul 16 16:25:53.012555 ip-10-0-142-101 kubenswrapper[2577]: E0716 16:25:53.012487 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rqzn4" podUID="e2d675d8-edf6-4765-a240-4b723cdd2d38" Jul 16 16:25:53.012555 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:53.012549 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ntktc" Jul 16 16:25:53.012692 ip-10-0-142-101 kubenswrapper[2577]: E0716 16:25:53.012651 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ntktc" podUID="84cc596d-4f15-4c84-8625-c7f37ed9dae6" Jul 16 16:25:54.012105 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:54.012070 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bcb7f" Jul 16 16:25:54.012524 ip-10-0-142-101 kubenswrapper[2577]: E0716 16:25:54.012218 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bcb7f" podUID="94ec201c-7285-4751-be16-d23d2f2c0290" Jul 16 16:25:55.012157 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:55.012124 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ntktc" Jul 16 16:25:55.012157 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:55.012165 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rqzn4" Jul 16 16:25:55.012614 ip-10-0-142-101 kubenswrapper[2577]: E0716 16:25:55.012263 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ntktc" podUID="84cc596d-4f15-4c84-8625-c7f37ed9dae6" Jul 16 16:25:55.012614 ip-10-0-142-101 kubenswrapper[2577]: E0716 16:25:55.012396 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rqzn4" podUID="e2d675d8-edf6-4765-a240-4b723cdd2d38" Jul 16 16:25:56.011724 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:56.011701 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bcb7f" Jul 16 16:25:56.011873 ip-10-0-142-101 kubenswrapper[2577]: E0716 16:25:56.011849 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bcb7f" podUID="94ec201c-7285-4751-be16-d23d2f2c0290" Jul 16 16:25:57.011397 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:57.011342 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ntktc" Jul 16 16:25:57.011845 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:57.011397 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rqzn4" Jul 16 16:25:57.011845 ip-10-0-142-101 kubenswrapper[2577]: E0716 16:25:57.011497 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ntktc" podUID="84cc596d-4f15-4c84-8625-c7f37ed9dae6" Jul 16 16:25:57.011845 ip-10-0-142-101 kubenswrapper[2577]: E0716 16:25:57.011620 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rqzn4" podUID="e2d675d8-edf6-4765-a240-4b723cdd2d38" Jul 16 16:25:58.011709 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:58.011671 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bcb7f" Jul 16 16:25:58.012078 ip-10-0-142-101 kubenswrapper[2577]: E0716 16:25:58.011824 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bcb7f" podUID="94ec201c-7285-4751-be16-d23d2f2c0290" Jul 16 16:25:58.709302 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:58.709268 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e2d675d8-edf6-4765-a240-4b723cdd2d38-original-pull-secret\") pod \"global-pull-secret-syncer-rqzn4\" (UID: \"e2d675d8-edf6-4765-a240-4b723cdd2d38\") " pod="kube-system/global-pull-secret-syncer-rqzn4" Jul 16 16:25:58.709530 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:58.709330 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/94ec201c-7285-4751-be16-d23d2f2c0290-metrics-certs\") pod \"network-metrics-daemon-bcb7f\" (UID: \"94ec201c-7285-4751-be16-d23d2f2c0290\") " pod="openshift-multus/network-metrics-daemon-bcb7f" Jul 16 16:25:58.709530 ip-10-0-142-101 kubenswrapper[2577]: E0716 16:25:58.709460 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 16:25:58.709530 ip-10-0-142-101 kubenswrapper[2577]: E0716 16:25:58.709466 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 16 16:25:58.709530 ip-10-0-142-101 kubenswrapper[2577]: E0716 16:25:58.709521 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/94ec201c-7285-4751-be16-d23d2f2c0290-metrics-certs podName:94ec201c-7285-4751-be16-d23d2f2c0290 nodeName:}" failed. No retries permitted until 2026-07-16 16:26:14.709500183 +0000 UTC m=+33.213558033 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/94ec201c-7285-4751-be16-d23d2f2c0290-metrics-certs") pod "network-metrics-daemon-bcb7f" (UID: "94ec201c-7285-4751-be16-d23d2f2c0290") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 16:25:58.709698 ip-10-0-142-101 kubenswrapper[2577]: E0716 16:25:58.709539 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e2d675d8-edf6-4765-a240-4b723cdd2d38-original-pull-secret podName:e2d675d8-edf6-4765-a240-4b723cdd2d38 nodeName:}" failed. No retries permitted until 2026-07-16 16:26:14.709529935 +0000 UTC m=+33.213587767 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e2d675d8-edf6-4765-a240-4b723cdd2d38-original-pull-secret") pod "global-pull-secret-syncer-rqzn4" (UID: "e2d675d8-edf6-4765-a240-4b723cdd2d38") : object "kube-system"/"original-pull-secret" not registered Jul 16 16:25:58.809897 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:58.809858 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s8dcf\" (UniqueName: \"kubernetes.io/projected/84cc596d-4f15-4c84-8625-c7f37ed9dae6-kube-api-access-s8dcf\") pod \"network-check-target-ntktc\" (UID: \"84cc596d-4f15-4c84-8625-c7f37ed9dae6\") " pod="openshift-network-diagnostics/network-check-target-ntktc" Jul 16 16:25:58.810102 ip-10-0-142-101 kubenswrapper[2577]: E0716 16:25:58.810015 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 16 16:25:58.810102 ip-10-0-142-101 kubenswrapper[2577]: E0716 16:25:58.810039 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 16 16:25:58.810102 ip-10-0-142-101 kubenswrapper[2577]: E0716 16:25:58.810052 2577 projected.go:196] Error preparing data for projected volume kube-api-access-s8dcf for pod openshift-network-diagnostics/network-check-target-ntktc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 16:25:58.810262 ip-10-0-142-101 kubenswrapper[2577]: E0716 16:25:58.810121 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/84cc596d-4f15-4c84-8625-c7f37ed9dae6-kube-api-access-s8dcf podName:84cc596d-4f15-4c84-8625-c7f37ed9dae6 nodeName:}" failed. No retries permitted until 2026-07-16 16:26:14.810102353 +0000 UTC m=+33.314160183 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-s8dcf" (UniqueName: "kubernetes.io/projected/84cc596d-4f15-4c84-8625-c7f37ed9dae6-kube-api-access-s8dcf") pod "network-check-target-ntktc" (UID: "84cc596d-4f15-4c84-8625-c7f37ed9dae6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 16:25:59.012320 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:59.012231 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rqzn4" Jul 16 16:25:59.012320 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:25:59.012283 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ntktc" Jul 16 16:25:59.012851 ip-10-0-142-101 kubenswrapper[2577]: E0716 16:25:59.012345 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rqzn4" podUID="e2d675d8-edf6-4765-a240-4b723cdd2d38" Jul 16 16:25:59.012851 ip-10-0-142-101 kubenswrapper[2577]: E0716 16:25:59.012458 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ntktc" podUID="84cc596d-4f15-4c84-8625-c7f37ed9dae6" Jul 16 16:26:00.012655 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:00.012376 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bcb7f" Jul 16 16:26:00.013129 ip-10-0-142-101 kubenswrapper[2577]: E0716 16:26:00.012781 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bcb7f" podUID="94ec201c-7285-4751-be16-d23d2f2c0290" Jul 16 16:26:01.011631 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:01.011591 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rqzn4" Jul 16 16:26:01.011836 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:01.011591 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ntktc" Jul 16 16:26:01.011836 ip-10-0-142-101 kubenswrapper[2577]: E0716 16:26:01.011731 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rqzn4" podUID="e2d675d8-edf6-4765-a240-4b723cdd2d38" Jul 16 16:26:01.011836 ip-10-0-142-101 kubenswrapper[2577]: E0716 16:26:01.011775 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ntktc" podUID="84cc596d-4f15-4c84-8625-c7f37ed9dae6" Jul 16 16:26:02.012418 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:02.012367 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bcb7f" Jul 16 16:26:02.012879 ip-10-0-142-101 kubenswrapper[2577]: E0716 16:26:02.012514 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bcb7f" podUID="94ec201c-7285-4751-be16-d23d2f2c0290" Jul 16 16:26:03.012078 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:03.012045 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rqzn4" Jul 16 16:26:03.012265 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:03.012046 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ntktc" Jul 16 16:26:03.012265 ip-10-0-142-101 kubenswrapper[2577]: E0716 16:26:03.012161 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rqzn4" podUID="e2d675d8-edf6-4765-a240-4b723cdd2d38" Jul 16 16:26:03.012265 ip-10-0-142-101 kubenswrapper[2577]: E0716 16:26:03.012225 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ntktc" podUID="84cc596d-4f15-4c84-8625-c7f37ed9dae6" Jul 16 16:26:04.011806 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:04.011629 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bcb7f" Jul 16 16:26:04.011806 ip-10-0-142-101 kubenswrapper[2577]: E0716 16:26:04.011789 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bcb7f" podUID="94ec201c-7285-4751-be16-d23d2f2c0290" Jul 16 16:26:04.153559 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:04.152993 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-8pl5s" event={"ID":"ba81ab34-5cc0-4f3a-856a-ff794d0dc0e5","Type":"ContainerStarted","Data":"aab9ed979d27d4c5609daddfdf1b5f4516d02fe27faecf8e9fd563cff1b874f7"} Jul 16 16:26:04.162563 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:04.162521 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-101.ec2.internal" event={"ID":"077fca01a64486b516cac217b95ed2fa","Type":"ContainerStarted","Data":"e65262efd437a9c13602371b672e6176ae050803e49c292a649163da740771a1"} Jul 16 16:26:04.215946 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:04.215884 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-101.ec2.internal" podStartSLOduration=21.215862739 podStartE2EDuration="21.215862739s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 16:25:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 16:26:04.215776911 +0000 UTC m=+22.719834763" watchObservedRunningTime="2026-07-16 16:26:04.215862739 +0000 UTC m=+22.719920592" Jul 16 16:26:04.216214 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:04.216177 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-8pl5s" podStartSLOduration=1.498552692 podStartE2EDuration="22.216168242s" totalImagesPullingTime="20.71761555s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 16:25:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 16:26:04.179299894 +0000 UTC m=+22.683357748" watchObservedRunningTime="2026-07-16 16:26:04.216168242 +0000 UTC m=+22.720226094" Jul 16 16:26:05.011876 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:05.011583 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rqzn4" Jul 16 16:26:05.012506 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:05.011600 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ntktc" Jul 16 16:26:05.012506 ip-10-0-142-101 kubenswrapper[2577]: E0716 16:26:05.011959 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rqzn4" podUID="e2d675d8-edf6-4765-a240-4b723cdd2d38" Jul 16 16:26:05.012506 ip-10-0-142-101 kubenswrapper[2577]: E0716 16:26:05.012026 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ntktc" podUID="84cc596d-4f15-4c84-8625-c7f37ed9dae6" Jul 16 16:26:05.166094 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:05.166060 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-hwq4x" event={"ID":"cbd027ac-97d9-4769-a867-47b7b8256cd0","Type":"ContainerStarted","Data":"14e7a4be42d0de53b5cf9f4c988230879c2848b2121a0f910623d2977a917078"} Jul 16 16:26:05.167597 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:05.167569 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t4w26" event={"ID":"aa5f6bf8-3599-45f1-a09d-ef1976f22840","Type":"ContainerStarted","Data":"919560ff6a7aa8a5caae95eec0d843f26a8971e640b86ee79b469848b64bc190"} Jul 16 16:26:05.170161 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:05.170141 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9bjst_b20b154f-1a4c-4b25-86db-aebe2d4ffc08/ovn-acl-logging/0.log" Jul 16 16:26:05.170495 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:05.170474 2577 generic.go:356] "Generic (PLEG): container finished" podID="b20b154f-1a4c-4b25-86db-aebe2d4ffc08" containerID="1372e87b45202a62719944f6ff2a4bd963958aafeb13aca3caac20cc654ea3a9" exitCode=1 Jul 16 16:26:05.170568 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:05.170545 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9bjst" event={"ID":"b20b154f-1a4c-4b25-86db-aebe2d4ffc08","Type":"ContainerStarted","Data":"625306e6f013d6150d46448a2306f9886d57e0b75233f526bc9764c60dea848a"} Jul 16 16:26:05.170646 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:05.170625 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9bjst" event={"ID":"b20b154f-1a4c-4b25-86db-aebe2d4ffc08","Type":"ContainerStarted","Data":"181ea102adff390019554389953aa3914898b229fae1e8bf5fb4d8117aed13e3"} Jul 16 16:26:05.170765 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:05.170684 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9bjst" event={"ID":"b20b154f-1a4c-4b25-86db-aebe2d4ffc08","Type":"ContainerStarted","Data":"a56aeccd7c25876376ce1d5e0c821f3cc84bc064018a3564a7268b78610511f4"} Jul 16 16:26:05.170765 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:05.170738 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9bjst" event={"ID":"b20b154f-1a4c-4b25-86db-aebe2d4ffc08","Type":"ContainerStarted","Data":"7b48ef23e1b76621f4eed6a65021da4a2f031fb5592619cd15988c7bcd386d06"} Jul 16 16:26:05.170867 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:05.170773 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9bjst" event={"ID":"b20b154f-1a4c-4b25-86db-aebe2d4ffc08","Type":"ContainerStarted","Data":"25d70ec007f950eaf29806f20c4021e2e64d8601b8a2945d19fd2ada15d4c182"} Jul 16 16:26:05.170867 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:05.170818 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9bjst" event={"ID":"b20b154f-1a4c-4b25-86db-aebe2d4ffc08","Type":"ContainerDied","Data":"1372e87b45202a62719944f6ff2a4bd963958aafeb13aca3caac20cc654ea3a9"} Jul 16 16:26:05.171935 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:05.171918 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-nrzr6" event={"ID":"8f44aa64-f94f-4a4e-a24f-b6a231b6df4b","Type":"ContainerStarted","Data":"9981a997d92081f968cd939c4ba6f1b94fdde8fbdb431d4f8caa87c3870921fb"} Jul 16 16:26:05.173276 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:05.173256 2577 generic.go:356] "Generic (PLEG): container finished" podID="f0ff4a720fce2411438eb201c85e2744" containerID="c5dd093105669194194fd814464b3222fa8ee16f991a53c345df11ac34abc429" exitCode=0 Jul 16 16:26:05.173371 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:05.173324 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-101.ec2.internal" event={"ID":"f0ff4a720fce2411438eb201c85e2744","Type":"ContainerDied","Data":"c5dd093105669194194fd814464b3222fa8ee16f991a53c345df11ac34abc429"} Jul 16 16:26:05.174637 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:05.174618 2577 generic.go:356] "Generic (PLEG): container finished" podID="b2410943-db6a-4668-845d-db0ae081a389" containerID="746a1133bce5c459209e5c3841e403f0b1d53d2a025696fb7fd1c9f3ee740507" exitCode=0 Jul 16 16:26:05.174747 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:05.174678 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xz7fp" event={"ID":"b2410943-db6a-4668-845d-db0ae081a389","Type":"ContainerDied","Data":"746a1133bce5c459209e5c3841e403f0b1d53d2a025696fb7fd1c9f3ee740507"} Jul 16 16:26:05.176019 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:05.175950 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-6sc67" event={"ID":"93f30808-1a75-44b5-801f-8faeceb72afb","Type":"ContainerStarted","Data":"259c0b86390904e93fb3a7a42714a9c268f4a07cfeb6b0894135f713195286d3"} Jul 16 16:26:05.177359 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:05.177265 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-xljrg" event={"ID":"a2cb45f0-0fb3-4761-98b0-a1797a00e647","Type":"ContainerStarted","Data":"2c4257d2ba81ece0ccb481a3a0506b62aefd0ddbedad688c27b83f6733cab0ef"} Jul 16 16:26:05.188012 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:05.187962 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-hwq4x" podStartSLOduration=2.534263916 podStartE2EDuration="23.187944168s" totalImagesPullingTime="20.653680252s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 16:25:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 16:26:05.187351975 +0000 UTC m=+23.691409827" watchObservedRunningTime="2026-07-16 16:26:05.187944168 +0000 UTC m=+23.692002019" Jul 16 16:26:05.245178 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:05.245134 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-nrzr6" podStartSLOduration=2.438417093 podStartE2EDuration="23.245117032s" totalImagesPullingTime="20.806699939s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 16:25:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 16:26:05.244344967 +0000 UTC m=+23.748402815" watchObservedRunningTime="2026-07-16 16:26:05.245117032 +0000 UTC m=+23.749174883" Jul 16 16:26:05.262957 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:05.262871 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-xljrg" podStartSLOduration=2.53950033 podStartE2EDuration="23.262855268s" totalImagesPullingTime="20.723354938s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 16:25:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 16:26:05.262800297 +0000 UTC m=+23.766858149" watchObservedRunningTime="2026-07-16 16:26:05.262855268 +0000 UTC m=+23.766913118" Jul 16 16:26:05.279912 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:05.279861 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-6sc67" podStartSLOduration=2.607587091 podStartE2EDuration="23.279843858s" totalImagesPullingTime="20.672256767s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 16:25:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 16:26:05.279358222 +0000 UTC m=+23.783416073" watchObservedRunningTime="2026-07-16 16:26:05.279843858 +0000 UTC m=+23.783901709" Jul 16 16:26:06.011484 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:06.011446 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bcb7f" Jul 16 16:26:06.011701 ip-10-0-142-101 kubenswrapper[2577]: E0716 16:26:06.011569 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bcb7f" podUID="94ec201c-7285-4751-be16-d23d2f2c0290" Jul 16 16:26:06.181264 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:06.181223 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-cnft7" event={"ID":"3062d489-a9b7-43fb-b1d8-4e39d30acdec","Type":"ContainerStarted","Data":"8eaf4f15d089473e1d9599f1f157ebc8dbdce602897d242887ba572b4ca9e469"} Jul 16 16:26:06.216309 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:06.216260 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-cnft7" podStartSLOduration=3.580451964 podStartE2EDuration="24.216243072s" totalImagesPullingTime="20.635791108s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 16:25:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 16:26:06.215664906 +0000 UTC m=+24.719722758" watchObservedRunningTime="2026-07-16 16:26:06.216243072 +0000 UTC m=+24.720300922" Jul 16 16:26:06.225816 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:06.225790 2577 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 16 16:26:06.703725 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:06.703689 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-xljrg" Jul 16 16:26:06.704541 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:06.704509 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-xljrg" Jul 16 16:26:06.948407 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:06.948042 2577 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-16T16:26:06.225812776Z","UUID":"5d25a052-a5a7-4355-a142-c5bdf73a2b35","Handler":null,"Name":"","Endpoint":""} Jul 16 16:26:06.950056 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:06.950035 2577 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 16 16:26:06.950161 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:06.950072 2577 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 16 16:26:07.012272 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:07.012026 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rqzn4" Jul 16 16:26:07.012272 ip-10-0-142-101 kubenswrapper[2577]: E0716 16:26:07.012130 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rqzn4" podUID="e2d675d8-edf6-4765-a240-4b723cdd2d38" Jul 16 16:26:07.012272 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:07.012186 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ntktc" Jul 16 16:26:07.012580 ip-10-0-142-101 kubenswrapper[2577]: E0716 16:26:07.012301 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ntktc" podUID="84cc596d-4f15-4c84-8625-c7f37ed9dae6" Jul 16 16:26:07.188647 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:07.188610 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t4w26" event={"ID":"aa5f6bf8-3599-45f1-a09d-ef1976f22840","Type":"ContainerStarted","Data":"77b5f1d9eefc7060fde60bed37c00efa18f3a3c5b55bdc6c7582b4a5e70e0f59"} Jul 16 16:26:07.191931 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:07.191898 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9bjst_b20b154f-1a4c-4b25-86db-aebe2d4ffc08/ovn-acl-logging/0.log" Jul 16 16:26:07.192335 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:07.192308 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9bjst" event={"ID":"b20b154f-1a4c-4b25-86db-aebe2d4ffc08","Type":"ContainerStarted","Data":"d49aa5a221a2b16a3d88b3a1a8987382c2ccb418019be61215ccc0065c3a6ab9"} Jul 16 16:26:07.194247 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:07.194218 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-101.ec2.internal" event={"ID":"f0ff4a720fce2411438eb201c85e2744","Type":"ContainerStarted","Data":"c41175cd77707f6a0b4df2f150636b635faf89f58c5cf5ef060d7b92519af511"} Jul 16 16:26:07.194608 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:07.194586 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-xljrg" Jul 16 16:26:07.195134 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:07.195098 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-xljrg" Jul 16 16:26:07.215407 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:07.215338 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-101.ec2.internal" podStartSLOduration=24.215319365 podStartE2EDuration="24.215319365s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 16:25:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 16:26:07.215121434 +0000 UTC m=+25.719179285" watchObservedRunningTime="2026-07-16 16:26:07.215319365 +0000 UTC m=+25.719377219" Jul 16 16:26:08.011864 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:08.011827 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bcb7f" Jul 16 16:26:08.012091 ip-10-0-142-101 kubenswrapper[2577]: E0716 16:26:08.011976 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bcb7f" podUID="94ec201c-7285-4751-be16-d23d2f2c0290" Jul 16 16:26:08.199139 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:08.199100 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t4w26" event={"ID":"aa5f6bf8-3599-45f1-a09d-ef1976f22840","Type":"ContainerStarted","Data":"7e0f9e5a930a80bcb6981b89a79c6b04e130b9fddbf3bdfda46d4dffde334b33"} Jul 16 16:26:09.012134 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:09.012107 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rqzn4" Jul 16 16:26:09.012278 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:09.012110 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ntktc" Jul 16 16:26:09.012278 ip-10-0-142-101 kubenswrapper[2577]: E0716 16:26:09.012234 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rqzn4" podUID="e2d675d8-edf6-4765-a240-4b723cdd2d38" Jul 16 16:26:09.012374 ip-10-0-142-101 kubenswrapper[2577]: E0716 16:26:09.012315 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ntktc" podUID="84cc596d-4f15-4c84-8625-c7f37ed9dae6" Jul 16 16:26:10.012691 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:10.012501 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bcb7f" Jul 16 16:26:10.013614 ip-10-0-142-101 kubenswrapper[2577]: E0716 16:26:10.012778 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bcb7f" podUID="94ec201c-7285-4751-be16-d23d2f2c0290" Jul 16 16:26:10.205479 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:10.205404 2577 generic.go:356] "Generic (PLEG): container finished" podID="b2410943-db6a-4668-845d-db0ae081a389" containerID="260bcc00073d081da637db00a50c8d91e0774231ab86863368e3b622aa9850b7" exitCode=0 Jul 16 16:26:10.205639 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:10.205485 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xz7fp" event={"ID":"b2410943-db6a-4668-845d-db0ae081a389","Type":"ContainerDied","Data":"260bcc00073d081da637db00a50c8d91e0774231ab86863368e3b622aa9850b7"} Jul 16 16:26:10.209126 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:10.209109 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9bjst_b20b154f-1a4c-4b25-86db-aebe2d4ffc08/ovn-acl-logging/0.log" Jul 16 16:26:10.209479 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:10.209459 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9bjst" event={"ID":"b20b154f-1a4c-4b25-86db-aebe2d4ffc08","Type":"ContainerStarted","Data":"3e89631877e1e2820ba0589dd95cbcfba0464c89fc4f0f66e634218dd1e23ab8"} Jul 16 16:26:10.209701 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:10.209676 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-9bjst" Jul 16 16:26:10.209778 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:10.209735 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-9bjst" Jul 16 16:26:10.209853 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:10.209838 2577 scope.go:117] "RemoveContainer" containerID="1372e87b45202a62719944f6ff2a4bd963958aafeb13aca3caac20cc654ea3a9" Jul 16 16:26:10.226481 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:10.226455 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-9bjst" Jul 16 16:26:10.229341 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:10.229297 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t4w26" podStartSLOduration=4.906354752 podStartE2EDuration="28.229279476s" totalImagesPullingTime="23.322924724s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 16:25:42 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 16:26:08.241761592 +0000 UTC m=+26.745819443" watchObservedRunningTime="2026-07-16 16:26:10.229279476 +0000 UTC m=+28.733337328" Jul 16 16:26:11.011803 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:11.011527 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ntktc" Jul 16 16:26:11.011803 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:11.011790 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rqzn4" Jul 16 16:26:11.012099 ip-10-0-142-101 kubenswrapper[2577]: E0716 16:26:11.011903 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ntktc" podUID="84cc596d-4f15-4c84-8625-c7f37ed9dae6" Jul 16 16:26:11.012099 ip-10-0-142-101 kubenswrapper[2577]: E0716 16:26:11.012006 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rqzn4" podUID="e2d675d8-edf6-4765-a240-4b723cdd2d38" Jul 16 16:26:11.214999 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:11.214972 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9bjst_b20b154f-1a4c-4b25-86db-aebe2d4ffc08/ovn-acl-logging/0.log" Jul 16 16:26:11.215479 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:11.215298 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9bjst" event={"ID":"b20b154f-1a4c-4b25-86db-aebe2d4ffc08","Type":"ContainerStarted","Data":"feb91adcfbf7d587ffccb921af8c48ef024eb350763b496a065bda0c1d110e88"} Jul 16 16:26:11.215543 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:11.215479 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-9bjst" Jul 16 16:26:11.230396 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:11.230358 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-9bjst" Jul 16 16:26:11.345827 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:11.345752 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-9bjst" podStartSLOduration=8.550592677000001 podStartE2EDuration="29.34573359s" totalImagesPullingTime="20.795140913s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 16:25:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 16:26:11.291856762 +0000 UTC m=+29.795914614" watchObservedRunningTime="2026-07-16 16:26:11.34573359 +0000 UTC m=+29.849791441" Jul 16 16:26:12.012560 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:12.012525 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bcb7f" Jul 16 16:26:12.012745 ip-10-0-142-101 kubenswrapper[2577]: E0716 16:26:12.012622 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bcb7f" podUID="94ec201c-7285-4751-be16-d23d2f2c0290" Jul 16 16:26:12.134326 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:12.134295 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-bcb7f"] Jul 16 16:26:12.137772 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:12.137745 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-ntktc"] Jul 16 16:26:12.137951 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:12.137865 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ntktc" Jul 16 16:26:12.138014 ip-10-0-142-101 kubenswrapper[2577]: E0716 16:26:12.137988 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ntktc" podUID="84cc596d-4f15-4c84-8625-c7f37ed9dae6" Jul 16 16:26:12.138456 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:12.138429 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-rqzn4"] Jul 16 16:26:12.138589 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:12.138539 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rqzn4" Jul 16 16:26:12.138652 ip-10-0-142-101 kubenswrapper[2577]: E0716 16:26:12.138625 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rqzn4" podUID="e2d675d8-edf6-4765-a240-4b723cdd2d38" Jul 16 16:26:12.220347 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:12.220249 2577 generic.go:356] "Generic (PLEG): container finished" podID="b2410943-db6a-4668-845d-db0ae081a389" containerID="9b8b5a5ac3873cc408ec7c05cffe25e1ef0226a22349bedf2d0bc127c11bb7cf" exitCode=0 Jul 16 16:26:12.220834 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:12.220337 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xz7fp" event={"ID":"b2410943-db6a-4668-845d-db0ae081a389","Type":"ContainerDied","Data":"9b8b5a5ac3873cc408ec7c05cffe25e1ef0226a22349bedf2d0bc127c11bb7cf"} Jul 16 16:26:12.220834 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:12.220542 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bcb7f" Jul 16 16:26:12.220834 ip-10-0-142-101 kubenswrapper[2577]: E0716 16:26:12.220634 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bcb7f" podUID="94ec201c-7285-4751-be16-d23d2f2c0290" Jul 16 16:26:13.689610 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:13.689584 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-6sc67_93f30808-1a75-44b5-801f-8faeceb72afb/dns-node-resolver/0.log" Jul 16 16:26:14.012355 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:14.012272 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bcb7f" Jul 16 16:26:14.012355 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:14.012319 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ntktc" Jul 16 16:26:14.012555 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:14.012272 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rqzn4" Jul 16 16:26:14.012555 ip-10-0-142-101 kubenswrapper[2577]: E0716 16:26:14.012446 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bcb7f" podUID="94ec201c-7285-4751-be16-d23d2f2c0290" Jul 16 16:26:14.012555 ip-10-0-142-101 kubenswrapper[2577]: E0716 16:26:14.012546 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rqzn4" podUID="e2d675d8-edf6-4765-a240-4b723cdd2d38" Jul 16 16:26:14.012666 ip-10-0-142-101 kubenswrapper[2577]: E0716 16:26:14.012600 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ntktc" podUID="84cc596d-4f15-4c84-8625-c7f37ed9dae6" Jul 16 16:26:14.227576 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:14.227540 2577 generic.go:356] "Generic (PLEG): container finished" podID="b2410943-db6a-4668-845d-db0ae081a389" containerID="fb9158de5a4776a532567a245c44a45ebb8d4f08ce5be3056ceec62fb3b1011a" exitCode=0 Jul 16 16:26:14.227752 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:14.227591 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xz7fp" event={"ID":"b2410943-db6a-4668-845d-db0ae081a389","Type":"ContainerDied","Data":"fb9158de5a4776a532567a245c44a45ebb8d4f08ce5be3056ceec62fb3b1011a"} Jul 16 16:26:14.731297 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:14.731083 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/94ec201c-7285-4751-be16-d23d2f2c0290-metrics-certs\") pod \"network-metrics-daemon-bcb7f\" (UID: \"94ec201c-7285-4751-be16-d23d2f2c0290\") " pod="openshift-multus/network-metrics-daemon-bcb7f" Jul 16 16:26:14.731684 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:14.731345 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e2d675d8-edf6-4765-a240-4b723cdd2d38-original-pull-secret\") pod \"global-pull-secret-syncer-rqzn4\" (UID: \"e2d675d8-edf6-4765-a240-4b723cdd2d38\") " pod="kube-system/global-pull-secret-syncer-rqzn4" Jul 16 16:26:14.731684 ip-10-0-142-101 kubenswrapper[2577]: E0716 16:26:14.731233 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 16:26:14.731684 ip-10-0-142-101 kubenswrapper[2577]: E0716 16:26:14.731433 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/94ec201c-7285-4751-be16-d23d2f2c0290-metrics-certs podName:94ec201c-7285-4751-be16-d23d2f2c0290 nodeName:}" failed. No retries permitted until 2026-07-16 16:26:46.731414087 +0000 UTC m=+65.235471927 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/94ec201c-7285-4751-be16-d23d2f2c0290-metrics-certs") pod "network-metrics-daemon-bcb7f" (UID: "94ec201c-7285-4751-be16-d23d2f2c0290") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 16:26:14.731684 ip-10-0-142-101 kubenswrapper[2577]: E0716 16:26:14.731469 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 16 16:26:14.731684 ip-10-0-142-101 kubenswrapper[2577]: E0716 16:26:14.731503 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e2d675d8-edf6-4765-a240-4b723cdd2d38-original-pull-secret podName:e2d675d8-edf6-4765-a240-4b723cdd2d38 nodeName:}" failed. No retries permitted until 2026-07-16 16:26:46.731492149 +0000 UTC m=+65.235549992 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e2d675d8-edf6-4765-a240-4b723cdd2d38-original-pull-secret") pod "global-pull-secret-syncer-rqzn4" (UID: "e2d675d8-edf6-4765-a240-4b723cdd2d38") : object "kube-system"/"original-pull-secret" not registered Jul 16 16:26:14.832073 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:14.832030 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s8dcf\" (UniqueName: \"kubernetes.io/projected/84cc596d-4f15-4c84-8625-c7f37ed9dae6-kube-api-access-s8dcf\") pod \"network-check-target-ntktc\" (UID: \"84cc596d-4f15-4c84-8625-c7f37ed9dae6\") " pod="openshift-network-diagnostics/network-check-target-ntktc" Jul 16 16:26:14.832263 ip-10-0-142-101 kubenswrapper[2577]: E0716 16:26:14.832232 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 16 16:26:14.832263 ip-10-0-142-101 kubenswrapper[2577]: E0716 16:26:14.832257 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 16 16:26:14.832427 ip-10-0-142-101 kubenswrapper[2577]: E0716 16:26:14.832271 2577 projected.go:196] Error preparing data for projected volume kube-api-access-s8dcf for pod openshift-network-diagnostics/network-check-target-ntktc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 16:26:14.832427 ip-10-0-142-101 kubenswrapper[2577]: E0716 16:26:14.832346 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/84cc596d-4f15-4c84-8625-c7f37ed9dae6-kube-api-access-s8dcf podName:84cc596d-4f15-4c84-8625-c7f37ed9dae6 nodeName:}" failed. No retries permitted until 2026-07-16 16:26:46.832321445 +0000 UTC m=+65.336379275 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-s8dcf" (UniqueName: "kubernetes.io/projected/84cc596d-4f15-4c84-8625-c7f37ed9dae6-kube-api-access-s8dcf") pod "network-check-target-ntktc" (UID: "84cc596d-4f15-4c84-8625-c7f37ed9dae6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 16:26:15.061332 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:15.061260 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-hwq4x_cbd027ac-97d9-4769-a867-47b7b8256cd0/node-ca/0.log" Jul 16 16:26:16.011638 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:16.011607 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bcb7f" Jul 16 16:26:16.012102 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:16.011607 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rqzn4" Jul 16 16:26:16.012102 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:16.011607 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ntktc" Jul 16 16:26:16.012102 ip-10-0-142-101 kubenswrapper[2577]: E0716 16:26:16.011793 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bcb7f" podUID="94ec201c-7285-4751-be16-d23d2f2c0290" Jul 16 16:26:16.012102 ip-10-0-142-101 kubenswrapper[2577]: E0716 16:26:16.011877 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ntktc" podUID="84cc596d-4f15-4c84-8625-c7f37ed9dae6" Jul 16 16:26:16.012102 ip-10-0-142-101 kubenswrapper[2577]: E0716 16:26:16.011956 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rqzn4" podUID="e2d675d8-edf6-4765-a240-4b723cdd2d38" Jul 16 16:26:18.012226 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:18.012192 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bcb7f" Jul 16 16:26:18.012668 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:18.012238 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ntktc" Jul 16 16:26:18.012668 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:18.012193 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rqzn4" Jul 16 16:26:18.012668 ip-10-0-142-101 kubenswrapper[2577]: E0716 16:26:18.012336 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bcb7f" podUID="94ec201c-7285-4751-be16-d23d2f2c0290" Jul 16 16:26:18.012668 ip-10-0-142-101 kubenswrapper[2577]: E0716 16:26:18.012409 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ntktc" podUID="84cc596d-4f15-4c84-8625-c7f37ed9dae6" Jul 16 16:26:18.012668 ip-10-0-142-101 kubenswrapper[2577]: E0716 16:26:18.012487 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rqzn4" podUID="e2d675d8-edf6-4765-a240-4b723cdd2d38" Jul 16 16:26:19.241694 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:19.241657 2577 generic.go:356] "Generic (PLEG): container finished" podID="b2410943-db6a-4668-845d-db0ae081a389" containerID="0797b60f0c924b3a20a0df627cdb6bee02f35818c6194db0be6c5c2771f83225" exitCode=0 Jul 16 16:26:19.242065 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:19.241727 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xz7fp" event={"ID":"b2410943-db6a-4668-845d-db0ae081a389","Type":"ContainerDied","Data":"0797b60f0c924b3a20a0df627cdb6bee02f35818c6194db0be6c5c2771f83225"} Jul 16 16:26:20.011950 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:20.011913 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ntktc" Jul 16 16:26:20.012126 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:20.011927 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bcb7f" Jul 16 16:26:20.012126 ip-10-0-142-101 kubenswrapper[2577]: E0716 16:26:20.012028 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ntktc" podUID="84cc596d-4f15-4c84-8625-c7f37ed9dae6" Jul 16 16:26:20.012126 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:20.011930 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rqzn4" Jul 16 16:26:20.012126 ip-10-0-142-101 kubenswrapper[2577]: E0716 16:26:20.012099 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bcb7f" podUID="94ec201c-7285-4751-be16-d23d2f2c0290" Jul 16 16:26:20.012302 ip-10-0-142-101 kubenswrapper[2577]: E0716 16:26:20.012193 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rqzn4" podUID="e2d675d8-edf6-4765-a240-4b723cdd2d38" Jul 16 16:26:20.247437 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:20.247404 2577 generic.go:356] "Generic (PLEG): container finished" podID="b2410943-db6a-4668-845d-db0ae081a389" containerID="f734fc7d2daca0a32b07336ad7c64cef15b787c8f75e76d09778e97f37f576e7" exitCode=0 Jul 16 16:26:20.247798 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:20.247452 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xz7fp" event={"ID":"b2410943-db6a-4668-845d-db0ae081a389","Type":"ContainerDied","Data":"f734fc7d2daca0a32b07336ad7c64cef15b787c8f75e76d09778e97f37f576e7"} Jul 16 16:26:21.253276 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:21.253245 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xz7fp" event={"ID":"b2410943-db6a-4668-845d-db0ae081a389","Type":"ContainerStarted","Data":"fea44d110fb68a861215de454a6c26a5464d752a57b672bb12bfaf72c44dde4e"} Jul 16 16:26:22.013499 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:22.013379 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ntktc" Jul 16 16:26:22.013499 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:22.013492 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bcb7f" Jul 16 16:26:22.013724 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:22.013537 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rqzn4" Jul 16 16:26:22.013724 ip-10-0-142-101 kubenswrapper[2577]: E0716 16:26:22.013538 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ntktc" podUID="84cc596d-4f15-4c84-8625-c7f37ed9dae6" Jul 16 16:26:22.013724 ip-10-0-142-101 kubenswrapper[2577]: E0716 16:26:22.013610 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rqzn4" podUID="e2d675d8-edf6-4765-a240-4b723cdd2d38" Jul 16 16:26:22.013724 ip-10-0-142-101 kubenswrapper[2577]: E0716 16:26:22.013697 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bcb7f" podUID="94ec201c-7285-4751-be16-d23d2f2c0290" Jul 16 16:26:24.011720 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:24.011681 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bcb7f" Jul 16 16:26:24.011720 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:24.011709 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ntktc" Jul 16 16:26:24.011720 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:24.011721 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rqzn4" Jul 16 16:26:24.012290 ip-10-0-142-101 kubenswrapper[2577]: E0716 16:26:24.011802 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bcb7f" podUID="94ec201c-7285-4751-be16-d23d2f2c0290" Jul 16 16:26:24.012290 ip-10-0-142-101 kubenswrapper[2577]: E0716 16:26:24.011866 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ntktc" podUID="84cc596d-4f15-4c84-8625-c7f37ed9dae6" Jul 16 16:26:24.012290 ip-10-0-142-101 kubenswrapper[2577]: E0716 16:26:24.011955 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rqzn4" podUID="e2d675d8-edf6-4765-a240-4b723cdd2d38" Jul 16 16:26:26.011805 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:26.011770 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rqzn4" Jul 16 16:26:26.012225 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:26.011770 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bcb7f" Jul 16 16:26:26.012225 ip-10-0-142-101 kubenswrapper[2577]: E0716 16:26:26.011887 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rqzn4" podUID="e2d675d8-edf6-4765-a240-4b723cdd2d38" Jul 16 16:26:26.012225 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:26.011779 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ntktc" Jul 16 16:26:26.012225 ip-10-0-142-101 kubenswrapper[2577]: E0716 16:26:26.011960 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bcb7f" podUID="94ec201c-7285-4751-be16-d23d2f2c0290" Jul 16 16:26:26.012225 ip-10-0-142-101 kubenswrapper[2577]: E0716 16:26:26.012049 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ntktc" podUID="84cc596d-4f15-4c84-8625-c7f37ed9dae6" Jul 16 16:26:28.011759 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:28.011721 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ntktc" Jul 16 16:26:28.011759 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:28.011738 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bcb7f" Jul 16 16:26:28.011759 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:28.011748 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rqzn4" Jul 16 16:26:28.012406 ip-10-0-142-101 kubenswrapper[2577]: E0716 16:26:28.011829 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ntktc" podUID="84cc596d-4f15-4c84-8625-c7f37ed9dae6" Jul 16 16:26:28.012406 ip-10-0-142-101 kubenswrapper[2577]: E0716 16:26:28.011932 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bcb7f" podUID="94ec201c-7285-4751-be16-d23d2f2c0290" Jul 16 16:26:28.012406 ip-10-0-142-101 kubenswrapper[2577]: E0716 16:26:28.011981 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rqzn4" podUID="e2d675d8-edf6-4765-a240-4b723cdd2d38" Jul 16 16:26:30.011686 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:30.011655 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ntktc" Jul 16 16:26:30.012124 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:30.011691 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rqzn4" Jul 16 16:26:30.012124 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:30.011691 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bcb7f" Jul 16 16:26:30.012124 ip-10-0-142-101 kubenswrapper[2577]: E0716 16:26:30.011763 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ntktc" podUID="84cc596d-4f15-4c84-8625-c7f37ed9dae6" Jul 16 16:26:30.012124 ip-10-0-142-101 kubenswrapper[2577]: E0716 16:26:30.011872 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bcb7f" podUID="94ec201c-7285-4751-be16-d23d2f2c0290" Jul 16 16:26:30.012124 ip-10-0-142-101 kubenswrapper[2577]: E0716 16:26:30.011936 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rqzn4" podUID="e2d675d8-edf6-4765-a240-4b723cdd2d38" Jul 16 16:26:32.014075 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:32.014040 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rqzn4" Jul 16 16:26:32.014464 ip-10-0-142-101 kubenswrapper[2577]: E0716 16:26:32.014135 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rqzn4" podUID="e2d675d8-edf6-4765-a240-4b723cdd2d38" Jul 16 16:26:32.014464 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:32.014146 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ntktc" Jul 16 16:26:32.014464 ip-10-0-142-101 kubenswrapper[2577]: E0716 16:26:32.014224 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ntktc" podUID="84cc596d-4f15-4c84-8625-c7f37ed9dae6" Jul 16 16:26:32.014464 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:32.014248 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bcb7f" Jul 16 16:26:32.014464 ip-10-0-142-101 kubenswrapper[2577]: E0716 16:26:32.014310 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bcb7f" podUID="94ec201c-7285-4751-be16-d23d2f2c0290" Jul 16 16:26:34.011620 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:34.011582 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bcb7f" Jul 16 16:26:34.012019 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:34.011585 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ntktc" Jul 16 16:26:34.012019 ip-10-0-142-101 kubenswrapper[2577]: E0716 16:26:34.011730 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bcb7f" podUID="94ec201c-7285-4751-be16-d23d2f2c0290" Jul 16 16:26:34.012019 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:34.011585 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rqzn4" Jul 16 16:26:34.012019 ip-10-0-142-101 kubenswrapper[2577]: E0716 16:26:34.011776 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ntktc" podUID="84cc596d-4f15-4c84-8625-c7f37ed9dae6" Jul 16 16:26:34.012019 ip-10-0-142-101 kubenswrapper[2577]: E0716 16:26:34.011841 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rqzn4" podUID="e2d675d8-edf6-4765-a240-4b723cdd2d38" Jul 16 16:26:36.012021 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:36.011984 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rqzn4" Jul 16 16:26:36.012507 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:36.011984 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ntktc" Jul 16 16:26:36.012507 ip-10-0-142-101 kubenswrapper[2577]: E0716 16:26:36.012118 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rqzn4" podUID="e2d675d8-edf6-4765-a240-4b723cdd2d38" Jul 16 16:26:36.012507 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:36.011982 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bcb7f" Jul 16 16:26:36.012507 ip-10-0-142-101 kubenswrapper[2577]: E0716 16:26:36.012195 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ntktc" podUID="84cc596d-4f15-4c84-8625-c7f37ed9dae6" Jul 16 16:26:36.012507 ip-10-0-142-101 kubenswrapper[2577]: E0716 16:26:36.012252 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bcb7f" podUID="94ec201c-7285-4751-be16-d23d2f2c0290" Jul 16 16:26:38.011477 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:38.011436 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bcb7f" Jul 16 16:26:38.011951 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:38.011445 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ntktc" Jul 16 16:26:38.011951 ip-10-0-142-101 kubenswrapper[2577]: E0716 16:26:38.011544 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bcb7f" podUID="94ec201c-7285-4751-be16-d23d2f2c0290" Jul 16 16:26:38.011951 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:38.011446 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rqzn4" Jul 16 16:26:38.011951 ip-10-0-142-101 kubenswrapper[2577]: E0716 16:26:38.011614 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ntktc" podUID="84cc596d-4f15-4c84-8625-c7f37ed9dae6" Jul 16 16:26:38.011951 ip-10-0-142-101 kubenswrapper[2577]: E0716 16:26:38.011674 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rqzn4" podUID="e2d675d8-edf6-4765-a240-4b723cdd2d38" Jul 16 16:26:38.336615 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:38.336585 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-101.ec2.internal" event="NodeReady" Jul 16 16:26:38.336808 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:38.336705 2577 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 16 16:26:38.423930 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:38.423873 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-xz7fp" podStartSLOduration=32.574306865 podStartE2EDuration="56.423855173s" totalImagesPullingTime="23.849548308s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 16:25:42 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 16:26:21.291348197 +0000 UTC m=+39.795406049" watchObservedRunningTime="2026-07-16 16:26:38.423855173 +0000 UTC m=+56.927913024" Jul 16 16:26:38.424201 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:38.424187 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-qhj4x"] Jul 16 16:26:38.426884 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:38.426856 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-qhj4x" Jul 16 16:26:38.427211 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:38.427188 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-6jhs5"] Jul 16 16:26:38.428913 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:38.428897 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-6jhs5" Jul 16 16:26:38.432872 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:38.432849 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 16 16:26:38.432999 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:38.432852 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 16 16:26:38.432999 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:38.432964 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-2tpfl\"" Jul 16 16:26:38.436153 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:38.436126 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 16 16:26:38.436274 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:38.436160 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-qw68r\"" Jul 16 16:26:38.436274 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:38.436220 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 16 16:26:38.442461 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:38.442444 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 16 16:26:38.457136 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:38.457106 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-6jhs5"] Jul 16 16:26:38.509267 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:38.509236 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pdscb\" (UniqueName: \"kubernetes.io/projected/533bf5ba-0e8a-40a2-b11a-44584469eae2-kube-api-access-pdscb\") pod \"ingress-canary-6jhs5\" (UID: \"533bf5ba-0e8a-40a2-b11a-44584469eae2\") " pod="openshift-ingress-canary/ingress-canary-6jhs5" Jul 16 16:26:38.509267 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:38.509272 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/984479f9-02a8-4fe9-ad59-23f9e3946e3a-config-volume\") pod \"dns-default-qhj4x\" (UID: \"984479f9-02a8-4fe9-ad59-23f9e3946e3a\") " pod="openshift-dns/dns-default-qhj4x" Jul 16 16:26:38.509520 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:38.509350 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/984479f9-02a8-4fe9-ad59-23f9e3946e3a-metrics-tls\") pod \"dns-default-qhj4x\" (UID: \"984479f9-02a8-4fe9-ad59-23f9e3946e3a\") " pod="openshift-dns/dns-default-qhj4x" Jul 16 16:26:38.509520 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:38.509401 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/984479f9-02a8-4fe9-ad59-23f9e3946e3a-tmp-dir\") pod \"dns-default-qhj4x\" (UID: \"984479f9-02a8-4fe9-ad59-23f9e3946e3a\") " pod="openshift-dns/dns-default-qhj4x" Jul 16 16:26:38.509520 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:38.509434 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/533bf5ba-0e8a-40a2-b11a-44584469eae2-cert\") pod \"ingress-canary-6jhs5\" (UID: \"533bf5ba-0e8a-40a2-b11a-44584469eae2\") " pod="openshift-ingress-canary/ingress-canary-6jhs5" Jul 16 16:26:38.509520 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:38.509455 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rkw4f\" (UniqueName: \"kubernetes.io/projected/984479f9-02a8-4fe9-ad59-23f9e3946e3a-kube-api-access-rkw4f\") pod \"dns-default-qhj4x\" (UID: \"984479f9-02a8-4fe9-ad59-23f9e3946e3a\") " pod="openshift-dns/dns-default-qhj4x" Jul 16 16:26:38.509657 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:38.509577 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-lf2qz"] Jul 16 16:26:38.511420 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:38.511404 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-lf2qz" Jul 16 16:26:38.517110 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:38.517085 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-qhj4x"] Jul 16 16:26:38.517990 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:38.517974 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 16 16:26:38.518207 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:38.518194 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 16 16:26:38.518494 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:38.518478 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 16 16:26:38.518987 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:38.518964 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 16 16:26:38.519870 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:38.519850 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-jn64s\"" Jul 16 16:26:38.540103 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:38.540069 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-lf2qz"] Jul 16 16:26:38.610847 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:38.610746 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9964ff62-4edc-4638-adf2-bbc37c39d29c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lf2qz\" (UID: \"9964ff62-4edc-4638-adf2-bbc37c39d29c\") " pod="openshift-insights/insights-runtime-extractor-lf2qz" Jul 16 16:26:38.610847 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:38.610793 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/533bf5ba-0e8a-40a2-b11a-44584469eae2-cert\") pod \"ingress-canary-6jhs5\" (UID: \"533bf5ba-0e8a-40a2-b11a-44584469eae2\") " pod="openshift-ingress-canary/ingress-canary-6jhs5" Jul 16 16:26:38.610847 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:38.610821 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rkw4f\" (UniqueName: \"kubernetes.io/projected/984479f9-02a8-4fe9-ad59-23f9e3946e3a-kube-api-access-rkw4f\") pod \"dns-default-qhj4x\" (UID: \"984479f9-02a8-4fe9-ad59-23f9e3946e3a\") " pod="openshift-dns/dns-default-qhj4x" Jul 16 16:26:38.611134 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:38.610918 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pdscb\" (UniqueName: \"kubernetes.io/projected/533bf5ba-0e8a-40a2-b11a-44584469eae2-kube-api-access-pdscb\") pod \"ingress-canary-6jhs5\" (UID: \"533bf5ba-0e8a-40a2-b11a-44584469eae2\") " pod="openshift-ingress-canary/ingress-canary-6jhs5" Jul 16 16:26:38.611134 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:38.610949 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/9964ff62-4edc-4638-adf2-bbc37c39d29c-data-volume\") pod \"insights-runtime-extractor-lf2qz\" (UID: \"9964ff62-4edc-4638-adf2-bbc37c39d29c\") " pod="openshift-insights/insights-runtime-extractor-lf2qz" Jul 16 16:26:38.611134 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:38.610969 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/984479f9-02a8-4fe9-ad59-23f9e3946e3a-config-volume\") pod \"dns-default-qhj4x\" (UID: \"984479f9-02a8-4fe9-ad59-23f9e3946e3a\") " pod="openshift-dns/dns-default-qhj4x" Jul 16 16:26:38.611134 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:38.610989 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9964ff62-4edc-4638-adf2-bbc37c39d29c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lf2qz\" (UID: \"9964ff62-4edc-4638-adf2-bbc37c39d29c\") " pod="openshift-insights/insights-runtime-extractor-lf2qz" Jul 16 16:26:38.611134 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:38.611010 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9tr7s\" (UniqueName: \"kubernetes.io/projected/9964ff62-4edc-4638-adf2-bbc37c39d29c-kube-api-access-9tr7s\") pod \"insights-runtime-extractor-lf2qz\" (UID: \"9964ff62-4edc-4638-adf2-bbc37c39d29c\") " pod="openshift-insights/insights-runtime-extractor-lf2qz" Jul 16 16:26:38.611134 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:38.611074 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/9964ff62-4edc-4638-adf2-bbc37c39d29c-crio-socket\") pod \"insights-runtime-extractor-lf2qz\" (UID: \"9964ff62-4edc-4638-adf2-bbc37c39d29c\") " pod="openshift-insights/insights-runtime-extractor-lf2qz" Jul 16 16:26:38.611134 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:38.611098 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/984479f9-02a8-4fe9-ad59-23f9e3946e3a-metrics-tls\") pod \"dns-default-qhj4x\" (UID: \"984479f9-02a8-4fe9-ad59-23f9e3946e3a\") " pod="openshift-dns/dns-default-qhj4x" Jul 16 16:26:38.611134 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:38.611125 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/984479f9-02a8-4fe9-ad59-23f9e3946e3a-tmp-dir\") pod \"dns-default-qhj4x\" (UID: \"984479f9-02a8-4fe9-ad59-23f9e3946e3a\") " pod="openshift-dns/dns-default-qhj4x" Jul 16 16:26:38.611512 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:38.611490 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/984479f9-02a8-4fe9-ad59-23f9e3946e3a-tmp-dir\") pod \"dns-default-qhj4x\" (UID: \"984479f9-02a8-4fe9-ad59-23f9e3946e3a\") " pod="openshift-dns/dns-default-qhj4x" Jul 16 16:26:38.611625 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:38.611606 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/984479f9-02a8-4fe9-ad59-23f9e3946e3a-config-volume\") pod \"dns-default-qhj4x\" (UID: \"984479f9-02a8-4fe9-ad59-23f9e3946e3a\") " pod="openshift-dns/dns-default-qhj4x" Jul 16 16:26:38.614723 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:38.614701 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/984479f9-02a8-4fe9-ad59-23f9e3946e3a-metrics-tls\") pod \"dns-default-qhj4x\" (UID: \"984479f9-02a8-4fe9-ad59-23f9e3946e3a\") " pod="openshift-dns/dns-default-qhj4x" Jul 16 16:26:38.614816 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:38.614759 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/533bf5ba-0e8a-40a2-b11a-44584469eae2-cert\") pod \"ingress-canary-6jhs5\" (UID: \"533bf5ba-0e8a-40a2-b11a-44584469eae2\") " pod="openshift-ingress-canary/ingress-canary-6jhs5" Jul 16 16:26:38.620874 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:38.620851 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rkw4f\" (UniqueName: \"kubernetes.io/projected/984479f9-02a8-4fe9-ad59-23f9e3946e3a-kube-api-access-rkw4f\") pod \"dns-default-qhj4x\" (UID: \"984479f9-02a8-4fe9-ad59-23f9e3946e3a\") " pod="openshift-dns/dns-default-qhj4x" Jul 16 16:26:38.623475 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:38.623446 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pdscb\" (UniqueName: \"kubernetes.io/projected/533bf5ba-0e8a-40a2-b11a-44584469eae2-kube-api-access-pdscb\") pod \"ingress-canary-6jhs5\" (UID: \"533bf5ba-0e8a-40a2-b11a-44584469eae2\") " pod="openshift-ingress-canary/ingress-canary-6jhs5" Jul 16 16:26:38.712292 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:38.712250 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/9964ff62-4edc-4638-adf2-bbc37c39d29c-data-volume\") pod \"insights-runtime-extractor-lf2qz\" (UID: \"9964ff62-4edc-4638-adf2-bbc37c39d29c\") " pod="openshift-insights/insights-runtime-extractor-lf2qz" Jul 16 16:26:38.712292 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:38.712292 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9964ff62-4edc-4638-adf2-bbc37c39d29c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lf2qz\" (UID: \"9964ff62-4edc-4638-adf2-bbc37c39d29c\") " pod="openshift-insights/insights-runtime-extractor-lf2qz" Jul 16 16:26:38.712514 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:38.712311 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9tr7s\" (UniqueName: \"kubernetes.io/projected/9964ff62-4edc-4638-adf2-bbc37c39d29c-kube-api-access-9tr7s\") pod \"insights-runtime-extractor-lf2qz\" (UID: \"9964ff62-4edc-4638-adf2-bbc37c39d29c\") " pod="openshift-insights/insights-runtime-extractor-lf2qz" Jul 16 16:26:38.712550 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:38.712503 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/9964ff62-4edc-4638-adf2-bbc37c39d29c-crio-socket\") pod \"insights-runtime-extractor-lf2qz\" (UID: \"9964ff62-4edc-4638-adf2-bbc37c39d29c\") " pod="openshift-insights/insights-runtime-extractor-lf2qz" Jul 16 16:26:38.712594 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:38.712562 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9964ff62-4edc-4638-adf2-bbc37c39d29c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lf2qz\" (UID: \"9964ff62-4edc-4638-adf2-bbc37c39d29c\") " pod="openshift-insights/insights-runtime-extractor-lf2qz" Jul 16 16:26:38.712651 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:38.712625 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/9964ff62-4edc-4638-adf2-bbc37c39d29c-crio-socket\") pod \"insights-runtime-extractor-lf2qz\" (UID: \"9964ff62-4edc-4638-adf2-bbc37c39d29c\") " pod="openshift-insights/insights-runtime-extractor-lf2qz" Jul 16 16:26:38.712703 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:38.712683 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/9964ff62-4edc-4638-adf2-bbc37c39d29c-data-volume\") pod \"insights-runtime-extractor-lf2qz\" (UID: \"9964ff62-4edc-4638-adf2-bbc37c39d29c\") " pod="openshift-insights/insights-runtime-extractor-lf2qz" Jul 16 16:26:38.712932 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:38.712914 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9964ff62-4edc-4638-adf2-bbc37c39d29c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lf2qz\" (UID: \"9964ff62-4edc-4638-adf2-bbc37c39d29c\") " pod="openshift-insights/insights-runtime-extractor-lf2qz" Jul 16 16:26:38.714725 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:38.714708 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9964ff62-4edc-4638-adf2-bbc37c39d29c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lf2qz\" (UID: \"9964ff62-4edc-4638-adf2-bbc37c39d29c\") " pod="openshift-insights/insights-runtime-extractor-lf2qz" Jul 16 16:26:38.726968 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:38.726942 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9tr7s\" (UniqueName: \"kubernetes.io/projected/9964ff62-4edc-4638-adf2-bbc37c39d29c-kube-api-access-9tr7s\") pod \"insights-runtime-extractor-lf2qz\" (UID: \"9964ff62-4edc-4638-adf2-bbc37c39d29c\") " pod="openshift-insights/insights-runtime-extractor-lf2qz" Jul 16 16:26:38.736942 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:38.736916 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-qhj4x" Jul 16 16:26:38.741593 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:38.741570 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-6jhs5" Jul 16 16:26:38.823298 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:38.823268 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-lf2qz" Jul 16 16:26:38.904516 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:38.904000 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-qhj4x"] Jul 16 16:26:38.908320 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:38.908294 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-6jhs5"] Jul 16 16:26:38.911053 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:26:38.911023 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod984479f9_02a8_4fe9_ad59_23f9e3946e3a.slice/crio-94688a194e7c8145ddd107a133a2de4341cc847bc278ecea7688528a41018100 WatchSource:0}: Error finding container 94688a194e7c8145ddd107a133a2de4341cc847bc278ecea7688528a41018100: Status 404 returned error can't find the container with id 94688a194e7c8145ddd107a133a2de4341cc847bc278ecea7688528a41018100 Jul 16 16:26:38.911333 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:26:38.911298 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod533bf5ba_0e8a_40a2_b11a_44584469eae2.slice/crio-47ba2bb035b32bf77ed433b98212d6dd0ecb67bb6fb41c2a0350361e33301adb WatchSource:0}: Error finding container 47ba2bb035b32bf77ed433b98212d6dd0ecb67bb6fb41c2a0350361e33301adb: Status 404 returned error can't find the container with id 47ba2bb035b32bf77ed433b98212d6dd0ecb67bb6fb41c2a0350361e33301adb Jul 16 16:26:38.992758 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:38.992720 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-lf2qz"] Jul 16 16:26:38.996424 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:26:38.996374 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9964ff62_4edc_4638_adf2_bbc37c39d29c.slice/crio-b12378e884099cc6b03199cdfec0765351fba235e1371cd7752536ad93cd3e20 WatchSource:0}: Error finding container b12378e884099cc6b03199cdfec0765351fba235e1371cd7752536ad93cd3e20: Status 404 returned error can't find the container with id b12378e884099cc6b03199cdfec0765351fba235e1371cd7752536ad93cd3e20 Jul 16 16:26:39.296302 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:39.296213 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-lf2qz" event={"ID":"9964ff62-4edc-4638-adf2-bbc37c39d29c","Type":"ContainerStarted","Data":"b12378e884099cc6b03199cdfec0765351fba235e1371cd7752536ad93cd3e20"} Jul 16 16:26:39.297237 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:39.297213 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-6jhs5" event={"ID":"533bf5ba-0e8a-40a2-b11a-44584469eae2","Type":"ContainerStarted","Data":"47ba2bb035b32bf77ed433b98212d6dd0ecb67bb6fb41c2a0350361e33301adb"} Jul 16 16:26:39.298199 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:39.298181 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-qhj4x" event={"ID":"984479f9-02a8-4fe9-ad59-23f9e3946e3a","Type":"ContainerStarted","Data":"94688a194e7c8145ddd107a133a2de4341cc847bc278ecea7688528a41018100"} Jul 16 16:26:40.015302 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:40.015266 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bcb7f" Jul 16 16:26:40.015736 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:40.015720 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ntktc" Jul 16 16:26:40.016181 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:40.016164 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rqzn4" Jul 16 16:26:40.018510 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:40.018312 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 16 16:26:40.020174 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:40.019701 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 16 16:26:40.020174 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:40.019961 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 16 16:26:40.021427 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:40.021405 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 16 16:26:40.021961 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:40.021739 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-vjpd8\"" Jul 16 16:26:40.021961 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:40.021841 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-npp5l\"" Jul 16 16:26:40.815832 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:40.815792 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-68b7b95d9-dnh2n"] Jul 16 16:26:40.818285 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:40.818257 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-68b7b95d9-dnh2n" Jul 16 16:26:40.823278 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:40.822988 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 16 16:26:40.823278 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:40.823085 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 16 16:26:40.823278 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:40.823249 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 16 16:26:40.824084 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:40.823671 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 16 16:26:40.824084 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:40.823884 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 16 16:26:40.824240 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:40.824096 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 16 16:26:40.824240 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:40.824133 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 16 16:26:40.824240 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:40.824222 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-jx67x\"" Jul 16 16:26:40.841629 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:40.841596 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-68b7b95d9-dnh2n"] Jul 16 16:26:40.931876 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:40.931839 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/cabb304d-fabd-4944-87f1-b7deccca8616-console-config\") pod \"console-68b7b95d9-dnh2n\" (UID: \"cabb304d-fabd-4944-87f1-b7deccca8616\") " pod="openshift-console/console-68b7b95d9-dnh2n" Jul 16 16:26:40.931876 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:40.931878 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/cabb304d-fabd-4944-87f1-b7deccca8616-console-oauth-config\") pod \"console-68b7b95d9-dnh2n\" (UID: \"cabb304d-fabd-4944-87f1-b7deccca8616\") " pod="openshift-console/console-68b7b95d9-dnh2n" Jul 16 16:26:40.932122 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:40.931972 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h5spp\" (UniqueName: \"kubernetes.io/projected/cabb304d-fabd-4944-87f1-b7deccca8616-kube-api-access-h5spp\") pod \"console-68b7b95d9-dnh2n\" (UID: \"cabb304d-fabd-4944-87f1-b7deccca8616\") " pod="openshift-console/console-68b7b95d9-dnh2n" Jul 16 16:26:40.932122 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:40.932016 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/cabb304d-fabd-4944-87f1-b7deccca8616-service-ca\") pod \"console-68b7b95d9-dnh2n\" (UID: \"cabb304d-fabd-4944-87f1-b7deccca8616\") " pod="openshift-console/console-68b7b95d9-dnh2n" Jul 16 16:26:40.932122 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:40.932105 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/cabb304d-fabd-4944-87f1-b7deccca8616-oauth-serving-cert\") pod \"console-68b7b95d9-dnh2n\" (UID: \"cabb304d-fabd-4944-87f1-b7deccca8616\") " pod="openshift-console/console-68b7b95d9-dnh2n" Jul 16 16:26:40.932264 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:40.932150 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/cabb304d-fabd-4944-87f1-b7deccca8616-console-serving-cert\") pod \"console-68b7b95d9-dnh2n\" (UID: \"cabb304d-fabd-4944-87f1-b7deccca8616\") " pod="openshift-console/console-68b7b95d9-dnh2n" Jul 16 16:26:41.033661 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:41.033596 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/cabb304d-fabd-4944-87f1-b7deccca8616-console-config\") pod \"console-68b7b95d9-dnh2n\" (UID: \"cabb304d-fabd-4944-87f1-b7deccca8616\") " pod="openshift-console/console-68b7b95d9-dnh2n" Jul 16 16:26:41.033861 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:41.033746 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/cabb304d-fabd-4944-87f1-b7deccca8616-console-oauth-config\") pod \"console-68b7b95d9-dnh2n\" (UID: \"cabb304d-fabd-4944-87f1-b7deccca8616\") " pod="openshift-console/console-68b7b95d9-dnh2n" Jul 16 16:26:41.033861 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:41.033786 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h5spp\" (UniqueName: \"kubernetes.io/projected/cabb304d-fabd-4944-87f1-b7deccca8616-kube-api-access-h5spp\") pod \"console-68b7b95d9-dnh2n\" (UID: \"cabb304d-fabd-4944-87f1-b7deccca8616\") " pod="openshift-console/console-68b7b95d9-dnh2n" Jul 16 16:26:41.033861 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:41.033813 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/cabb304d-fabd-4944-87f1-b7deccca8616-service-ca\") pod \"console-68b7b95d9-dnh2n\" (UID: \"cabb304d-fabd-4944-87f1-b7deccca8616\") " pod="openshift-console/console-68b7b95d9-dnh2n" Jul 16 16:26:41.034022 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:41.033874 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/cabb304d-fabd-4944-87f1-b7deccca8616-oauth-serving-cert\") pod \"console-68b7b95d9-dnh2n\" (UID: \"cabb304d-fabd-4944-87f1-b7deccca8616\") " pod="openshift-console/console-68b7b95d9-dnh2n" Jul 16 16:26:41.034022 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:41.033918 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/cabb304d-fabd-4944-87f1-b7deccca8616-console-serving-cert\") pod \"console-68b7b95d9-dnh2n\" (UID: \"cabb304d-fabd-4944-87f1-b7deccca8616\") " pod="openshift-console/console-68b7b95d9-dnh2n" Jul 16 16:26:41.034494 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:41.034441 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/cabb304d-fabd-4944-87f1-b7deccca8616-console-config\") pod \"console-68b7b95d9-dnh2n\" (UID: \"cabb304d-fabd-4944-87f1-b7deccca8616\") " pod="openshift-console/console-68b7b95d9-dnh2n" Jul 16 16:26:41.034887 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:41.034850 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/cabb304d-fabd-4944-87f1-b7deccca8616-service-ca\") pod \"console-68b7b95d9-dnh2n\" (UID: \"cabb304d-fabd-4944-87f1-b7deccca8616\") " pod="openshift-console/console-68b7b95d9-dnh2n" Jul 16 16:26:41.034986 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:41.034895 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/cabb304d-fabd-4944-87f1-b7deccca8616-oauth-serving-cert\") pod \"console-68b7b95d9-dnh2n\" (UID: \"cabb304d-fabd-4944-87f1-b7deccca8616\") " pod="openshift-console/console-68b7b95d9-dnh2n" Jul 16 16:26:41.036947 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:41.036899 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/cabb304d-fabd-4944-87f1-b7deccca8616-console-serving-cert\") pod \"console-68b7b95d9-dnh2n\" (UID: \"cabb304d-fabd-4944-87f1-b7deccca8616\") " pod="openshift-console/console-68b7b95d9-dnh2n" Jul 16 16:26:41.037066 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:41.037043 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/cabb304d-fabd-4944-87f1-b7deccca8616-console-oauth-config\") pod \"console-68b7b95d9-dnh2n\" (UID: \"cabb304d-fabd-4944-87f1-b7deccca8616\") " pod="openshift-console/console-68b7b95d9-dnh2n" Jul 16 16:26:41.043901 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:41.043870 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h5spp\" (UniqueName: \"kubernetes.io/projected/cabb304d-fabd-4944-87f1-b7deccca8616-kube-api-access-h5spp\") pod \"console-68b7b95d9-dnh2n\" (UID: \"cabb304d-fabd-4944-87f1-b7deccca8616\") " pod="openshift-console/console-68b7b95d9-dnh2n" Jul 16 16:26:41.132778 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:41.132741 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-68b7b95d9-dnh2n" Jul 16 16:26:42.022906 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:42.022875 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-875t5"] Jul 16 16:26:42.027791 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:42.027760 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-875t5" Jul 16 16:26:42.030502 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:42.030470 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Jul 16 16:26:42.030502 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:42.030500 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-zzx2t\"" Jul 16 16:26:42.040301 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:42.040266 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-875t5"] Jul 16 16:26:42.141745 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:42.141705 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/7ca62a57-8939-42e8-a31d-d8fd4e1595bb-tls-certificates\") pod \"prometheus-operator-admission-webhook-77d98bc786-875t5\" (UID: \"7ca62a57-8939-42e8-a31d-d8fd4e1595bb\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-875t5" Jul 16 16:26:42.242429 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:42.242364 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/7ca62a57-8939-42e8-a31d-d8fd4e1595bb-tls-certificates\") pod \"prometheus-operator-admission-webhook-77d98bc786-875t5\" (UID: \"7ca62a57-8939-42e8-a31d-d8fd4e1595bb\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-875t5" Jul 16 16:26:42.242608 ip-10-0-142-101 kubenswrapper[2577]: E0716 16:26:42.242544 2577 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-admission-webhook-tls: secret "prometheus-operator-admission-webhook-tls" not found Jul 16 16:26:42.242679 ip-10-0-142-101 kubenswrapper[2577]: E0716 16:26:42.242624 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7ca62a57-8939-42e8-a31d-d8fd4e1595bb-tls-certificates podName:7ca62a57-8939-42e8-a31d-d8fd4e1595bb nodeName:}" failed. No retries permitted until 2026-07-16 16:26:42.7426052 +0000 UTC m=+61.246663053 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certificates" (UniqueName: "kubernetes.io/secret/7ca62a57-8939-42e8-a31d-d8fd4e1595bb-tls-certificates") pod "prometheus-operator-admission-webhook-77d98bc786-875t5" (UID: "7ca62a57-8939-42e8-a31d-d8fd4e1595bb") : secret "prometheus-operator-admission-webhook-tls" not found Jul 16 16:26:42.746172 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:42.746127 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/7ca62a57-8939-42e8-a31d-d8fd4e1595bb-tls-certificates\") pod \"prometheus-operator-admission-webhook-77d98bc786-875t5\" (UID: \"7ca62a57-8939-42e8-a31d-d8fd4e1595bb\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-875t5" Jul 16 16:26:42.748890 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:42.748858 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/7ca62a57-8939-42e8-a31d-d8fd4e1595bb-tls-certificates\") pod \"prometheus-operator-admission-webhook-77d98bc786-875t5\" (UID: \"7ca62a57-8939-42e8-a31d-d8fd4e1595bb\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-875t5" Jul 16 16:26:42.940700 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:42.940656 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-875t5" Jul 16 16:26:43.234513 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:43.234483 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-9bjst" Jul 16 16:26:44.977211 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:44.977177 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-875t5"] Jul 16 16:26:44.993708 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:44.993667 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-68b7b95d9-dnh2n"] Jul 16 16:26:45.207012 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:26:45.206981 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcabb304d_fabd_4944_87f1_b7deccca8616.slice/crio-c4305d6867f91dbcb47fc20684165eb81209439fb08f2f170bfe192b7c809deb WatchSource:0}: Error finding container c4305d6867f91dbcb47fc20684165eb81209439fb08f2f170bfe192b7c809deb: Status 404 returned error can't find the container with id c4305d6867f91dbcb47fc20684165eb81209439fb08f2f170bfe192b7c809deb Jul 16 16:26:45.208180 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:26:45.208093 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ca62a57_8939_42e8_a31d_d8fd4e1595bb.slice/crio-29a678250387872cda5539705ac6d126ca489228d68cecfdd88bb86f92c52a87 WatchSource:0}: Error finding container 29a678250387872cda5539705ac6d126ca489228d68cecfdd88bb86f92c52a87: Status 404 returned error can't find the container with id 29a678250387872cda5539705ac6d126ca489228d68cecfdd88bb86f92c52a87 Jul 16 16:26:45.317913 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:45.317875 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-68b7b95d9-dnh2n" event={"ID":"cabb304d-fabd-4944-87f1-b7deccca8616","Type":"ContainerStarted","Data":"c4305d6867f91dbcb47fc20684165eb81209439fb08f2f170bfe192b7c809deb"} Jul 16 16:26:45.324617 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:45.324575 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-qhj4x" event={"ID":"984479f9-02a8-4fe9-ad59-23f9e3946e3a","Type":"ContainerStarted","Data":"bbd895990828ba0a676a5cf468f57c55f3c06d10d67326780c31338a22e57a10"} Jul 16 16:26:45.326714 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:45.326681 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-6jhs5" event={"ID":"533bf5ba-0e8a-40a2-b11a-44584469eae2","Type":"ContainerStarted","Data":"d4eaad1bae0774f9d65a322d4ac5eb5393232f25f01f5851ddfd050c3ad7e198"} Jul 16 16:26:45.328953 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:45.328924 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-875t5" event={"ID":"7ca62a57-8939-42e8-a31d-d8fd4e1595bb","Type":"ContainerStarted","Data":"29a678250387872cda5539705ac6d126ca489228d68cecfdd88bb86f92c52a87"} Jul 16 16:26:45.344609 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:45.344551 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-6jhs5" podStartSLOduration=1.052526287 podStartE2EDuration="7.344529193s" totalImagesPullingTime="6.292002906s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 16:26:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 16:26:45.343713482 +0000 UTC m=+63.847771335" watchObservedRunningTime="2026-07-16 16:26:45.344529193 +0000 UTC m=+63.848587048" Jul 16 16:26:46.334610 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:46.334569 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-qhj4x" event={"ID":"984479f9-02a8-4fe9-ad59-23f9e3946e3a","Type":"ContainerStarted","Data":"d702a18fce79ea9d59fd6a069f986536ffdcfb729368c6d15bb57916cb43d479"} Jul 16 16:26:46.334998 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:46.334973 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-qhj4x" Jul 16 16:26:46.336915 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:46.336877 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-lf2qz" event={"ID":"9964ff62-4edc-4638-adf2-bbc37c39d29c","Type":"ContainerStarted","Data":"e3ca2b33a0d21948df6cd0a1a345fb26ec83583d046f09fd5b9399cfadba5499"} Jul 16 16:26:46.363432 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:46.363167 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-qhj4x" podStartSLOduration=2.071195192 podStartE2EDuration="8.363143178s" totalImagesPullingTime="6.291947986s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 16:26:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 16:26:46.362065275 +0000 UTC m=+64.866123127" watchObservedRunningTime="2026-07-16 16:26:46.363143178 +0000 UTC m=+64.867201024" Jul 16 16:26:46.781762 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:46.781661 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/94ec201c-7285-4751-be16-d23d2f2c0290-metrics-certs\") pod \"network-metrics-daemon-bcb7f\" (UID: \"94ec201c-7285-4751-be16-d23d2f2c0290\") " pod="openshift-multus/network-metrics-daemon-bcb7f" Jul 16 16:26:46.781945 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:46.781765 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e2d675d8-edf6-4765-a240-4b723cdd2d38-original-pull-secret\") pod \"global-pull-secret-syncer-rqzn4\" (UID: \"e2d675d8-edf6-4765-a240-4b723cdd2d38\") " pod="kube-system/global-pull-secret-syncer-rqzn4" Jul 16 16:26:46.785524 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:46.785496 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 16 16:26:46.785666 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:46.785553 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 16 16:26:46.795188 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:46.795155 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e2d675d8-edf6-4765-a240-4b723cdd2d38-original-pull-secret\") pod \"global-pull-secret-syncer-rqzn4\" (UID: \"e2d675d8-edf6-4765-a240-4b723cdd2d38\") " pod="kube-system/global-pull-secret-syncer-rqzn4" Jul 16 16:26:46.795331 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:46.795167 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/94ec201c-7285-4751-be16-d23d2f2c0290-metrics-certs\") pod \"network-metrics-daemon-bcb7f\" (UID: \"94ec201c-7285-4751-be16-d23d2f2c0290\") " pod="openshift-multus/network-metrics-daemon-bcb7f" Jul 16 16:26:46.883092 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:46.883046 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s8dcf\" (UniqueName: \"kubernetes.io/projected/84cc596d-4f15-4c84-8625-c7f37ed9dae6-kube-api-access-s8dcf\") pod \"network-check-target-ntktc\" (UID: \"84cc596d-4f15-4c84-8625-c7f37ed9dae6\") " pod="openshift-network-diagnostics/network-check-target-ntktc" Jul 16 16:26:46.886345 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:46.886312 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 16 16:26:46.896284 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:46.896254 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 16 16:26:46.906369 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:46.906338 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s8dcf\" (UniqueName: \"kubernetes.io/projected/84cc596d-4f15-4c84-8625-c7f37ed9dae6-kube-api-access-s8dcf\") pod \"network-check-target-ntktc\" (UID: \"84cc596d-4f15-4c84-8625-c7f37ed9dae6\") " pod="openshift-network-diagnostics/network-check-target-ntktc" Jul 16 16:26:46.939472 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:46.939249 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-npp5l\"" Jul 16 16:26:46.942307 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:46.942271 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bcb7f" Jul 16 16:26:46.946248 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:46.946223 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-vjpd8\"" Jul 16 16:26:46.949857 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:46.949835 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rqzn4" Jul 16 16:26:46.953628 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:46.953595 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ntktc" Jul 16 16:26:47.343722 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:47.343683 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-875t5" event={"ID":"7ca62a57-8939-42e8-a31d-d8fd4e1595bb","Type":"ContainerStarted","Data":"442bf9679f19f6312e77a2af83e3eefe704040e651d031b3a15cb3dcb4ec2fcb"} Jul 16 16:26:47.366056 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:47.365997 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-875t5" podStartSLOduration=4.125630318 podStartE2EDuration="5.365978664s" totalImagesPullingTime="1.240348346s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 16:26:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 16:26:47.365434661 +0000 UTC m=+65.869492507" watchObservedRunningTime="2026-07-16 16:26:47.365978664 +0000 UTC m=+65.870036493" Jul 16 16:26:48.346977 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:48.346942 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-875t5" Jul 16 16:26:48.351970 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:48.351936 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-875t5" Jul 16 16:26:49.105694 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:49.105652 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-64785b6675-vp8d9"] Jul 16 16:26:49.111510 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:49.111479 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-64785b6675-vp8d9" Jul 16 16:26:49.114485 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:49.114451 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 16 16:26:49.114641 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:49.114483 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Jul 16 16:26:49.114641 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:49.114496 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Jul 16 16:26:49.115463 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:49.115439 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 16 16:26:49.115606 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:49.115506 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 16 16:26:49.115761 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:49.115734 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-bbbqs\"" Jul 16 16:26:49.125557 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:49.125529 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-64785b6675-vp8d9"] Jul 16 16:26:49.201839 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:49.201800 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7fe140c1-9268-46ef-aa31-3283957b0fff-metrics-client-ca\") pod \"prometheus-operator-64785b6675-vp8d9\" (UID: \"7fe140c1-9268-46ef-aa31-3283957b0fff\") " pod="openshift-monitoring/prometheus-operator-64785b6675-vp8d9" Jul 16 16:26:49.202026 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:49.201851 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rqm89\" (UniqueName: \"kubernetes.io/projected/7fe140c1-9268-46ef-aa31-3283957b0fff-kube-api-access-rqm89\") pod \"prometheus-operator-64785b6675-vp8d9\" (UID: \"7fe140c1-9268-46ef-aa31-3283957b0fff\") " pod="openshift-monitoring/prometheus-operator-64785b6675-vp8d9" Jul 16 16:26:49.202026 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:49.201901 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/7fe140c1-9268-46ef-aa31-3283957b0fff-prometheus-operator-tls\") pod \"prometheus-operator-64785b6675-vp8d9\" (UID: \"7fe140c1-9268-46ef-aa31-3283957b0fff\") " pod="openshift-monitoring/prometheus-operator-64785b6675-vp8d9" Jul 16 16:26:49.202026 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:49.201926 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7fe140c1-9268-46ef-aa31-3283957b0fff-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-64785b6675-vp8d9\" (UID: \"7fe140c1-9268-46ef-aa31-3283957b0fff\") " pod="openshift-monitoring/prometheus-operator-64785b6675-vp8d9" Jul 16 16:26:49.303121 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:49.303085 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/7fe140c1-9268-46ef-aa31-3283957b0fff-prometheus-operator-tls\") pod \"prometheus-operator-64785b6675-vp8d9\" (UID: \"7fe140c1-9268-46ef-aa31-3283957b0fff\") " pod="openshift-monitoring/prometheus-operator-64785b6675-vp8d9" Jul 16 16:26:49.303300 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:49.303126 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7fe140c1-9268-46ef-aa31-3283957b0fff-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-64785b6675-vp8d9\" (UID: \"7fe140c1-9268-46ef-aa31-3283957b0fff\") " pod="openshift-monitoring/prometheus-operator-64785b6675-vp8d9" Jul 16 16:26:49.303300 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:49.303172 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7fe140c1-9268-46ef-aa31-3283957b0fff-metrics-client-ca\") pod \"prometheus-operator-64785b6675-vp8d9\" (UID: \"7fe140c1-9268-46ef-aa31-3283957b0fff\") " pod="openshift-monitoring/prometheus-operator-64785b6675-vp8d9" Jul 16 16:26:49.303300 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:49.303201 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rqm89\" (UniqueName: \"kubernetes.io/projected/7fe140c1-9268-46ef-aa31-3283957b0fff-kube-api-access-rqm89\") pod \"prometheus-operator-64785b6675-vp8d9\" (UID: \"7fe140c1-9268-46ef-aa31-3283957b0fff\") " pod="openshift-monitoring/prometheus-operator-64785b6675-vp8d9" Jul 16 16:26:49.303300 ip-10-0-142-101 kubenswrapper[2577]: E0716 16:26:49.303244 2577 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found Jul 16 16:26:49.303537 ip-10-0-142-101 kubenswrapper[2577]: E0716 16:26:49.303309 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7fe140c1-9268-46ef-aa31-3283957b0fff-prometheus-operator-tls podName:7fe140c1-9268-46ef-aa31-3283957b0fff nodeName:}" failed. No retries permitted until 2026-07-16 16:26:49.803291479 +0000 UTC m=+68.307349308 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/7fe140c1-9268-46ef-aa31-3283957b0fff-prometheus-operator-tls") pod "prometheus-operator-64785b6675-vp8d9" (UID: "7fe140c1-9268-46ef-aa31-3283957b0fff") : secret "prometheus-operator-tls" not found Jul 16 16:26:49.304043 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:49.303991 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7fe140c1-9268-46ef-aa31-3283957b0fff-metrics-client-ca\") pod \"prometheus-operator-64785b6675-vp8d9\" (UID: \"7fe140c1-9268-46ef-aa31-3283957b0fff\") " pod="openshift-monitoring/prometheus-operator-64785b6675-vp8d9" Jul 16 16:26:49.305802 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:49.305775 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7fe140c1-9268-46ef-aa31-3283957b0fff-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-64785b6675-vp8d9\" (UID: \"7fe140c1-9268-46ef-aa31-3283957b0fff\") " pod="openshift-monitoring/prometheus-operator-64785b6675-vp8d9" Jul 16 16:26:49.312726 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:49.312698 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rqm89\" (UniqueName: \"kubernetes.io/projected/7fe140c1-9268-46ef-aa31-3283957b0fff-kube-api-access-rqm89\") pod \"prometheus-operator-64785b6675-vp8d9\" (UID: \"7fe140c1-9268-46ef-aa31-3283957b0fff\") " pod="openshift-monitoring/prometheus-operator-64785b6675-vp8d9" Jul 16 16:26:49.806294 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:49.806257 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/7fe140c1-9268-46ef-aa31-3283957b0fff-prometheus-operator-tls\") pod \"prometheus-operator-64785b6675-vp8d9\" (UID: \"7fe140c1-9268-46ef-aa31-3283957b0fff\") " pod="openshift-monitoring/prometheus-operator-64785b6675-vp8d9" Jul 16 16:26:49.808538 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:49.808507 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/7fe140c1-9268-46ef-aa31-3283957b0fff-prometheus-operator-tls\") pod \"prometheus-operator-64785b6675-vp8d9\" (UID: \"7fe140c1-9268-46ef-aa31-3283957b0fff\") " pod="openshift-monitoring/prometheus-operator-64785b6675-vp8d9" Jul 16 16:26:50.023375 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:50.023114 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-64785b6675-vp8d9" Jul 16 16:26:50.096958 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:50.096860 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-bcb7f"] Jul 16 16:26:50.104745 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:26:50.104713 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94ec201c_7285_4751_be16_d23d2f2c0290.slice/crio-23254102ed30d4b934d8bee58caceda3c1d3336510e79fa6455284b58eb89159 WatchSource:0}: Error finding container 23254102ed30d4b934d8bee58caceda3c1d3336510e79fa6455284b58eb89159: Status 404 returned error can't find the container with id 23254102ed30d4b934d8bee58caceda3c1d3336510e79fa6455284b58eb89159 Jul 16 16:26:50.124139 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:50.124080 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-rqzn4"] Jul 16 16:26:50.128485 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:26:50.128454 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2d675d8_edf6_4765_a240_4b723cdd2d38.slice/crio-013a8d51998029df33d77a7d6e6fd1c5fa7fb80a48bd8c7c1b8a86e797f4e22e WatchSource:0}: Error finding container 013a8d51998029df33d77a7d6e6fd1c5fa7fb80a48bd8c7c1b8a86e797f4e22e: Status 404 returned error can't find the container with id 013a8d51998029df33d77a7d6e6fd1c5fa7fb80a48bd8c7c1b8a86e797f4e22e Jul 16 16:26:50.144202 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:50.144173 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-ntktc"] Jul 16 16:26:50.147567 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:26:50.147530 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84cc596d_4f15_4c84_8625_c7f37ed9dae6.slice/crio-198811370f526d3a610b697df32e08e59dc80399284596b5267ffb80c0e69896 WatchSource:0}: Error finding container 198811370f526d3a610b697df32e08e59dc80399284596b5267ffb80c0e69896: Status 404 returned error can't find the container with id 198811370f526d3a610b697df32e08e59dc80399284596b5267ffb80c0e69896 Jul 16 16:26:50.191134 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:50.191108 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-64785b6675-vp8d9"] Jul 16 16:26:50.193533 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:26:50.193505 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7fe140c1_9268_46ef_aa31_3283957b0fff.slice/crio-d137cb2de3dca02e9965a911dfd5c7ab2c541c9aed86a9c4e31f70344f3d992f WatchSource:0}: Error finding container d137cb2de3dca02e9965a911dfd5c7ab2c541c9aed86a9c4e31f70344f3d992f: Status 404 returned error can't find the container with id d137cb2de3dca02e9965a911dfd5c7ab2c541c9aed86a9c4e31f70344f3d992f Jul 16 16:26:50.355721 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:50.355623 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-68b7b95d9-dnh2n" event={"ID":"cabb304d-fabd-4944-87f1-b7deccca8616","Type":"ContainerStarted","Data":"704d5aada534d8e3c194bff401788a7021278ef9f8f99309cd4085a0ac99eeb1"} Jul 16 16:26:50.356941 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:50.356914 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-ntktc" event={"ID":"84cc596d-4f15-4c84-8625-c7f37ed9dae6","Type":"ContainerStarted","Data":"198811370f526d3a610b697df32e08e59dc80399284596b5267ffb80c0e69896"} Jul 16 16:26:50.357950 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:50.357930 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-bcb7f" event={"ID":"94ec201c-7285-4751-be16-d23d2f2c0290","Type":"ContainerStarted","Data":"23254102ed30d4b934d8bee58caceda3c1d3336510e79fa6455284b58eb89159"} Jul 16 16:26:50.359247 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:50.359193 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-rqzn4" event={"ID":"e2d675d8-edf6-4765-a240-4b723cdd2d38","Type":"ContainerStarted","Data":"013a8d51998029df33d77a7d6e6fd1c5fa7fb80a48bd8c7c1b8a86e797f4e22e"} Jul 16 16:26:50.361032 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:50.361011 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-lf2qz" event={"ID":"9964ff62-4edc-4638-adf2-bbc37c39d29c","Type":"ContainerStarted","Data":"d06a3abe9cf69db02a03069996da5a180a099dfe1d1895b5b41f681e61f89ef2"} Jul 16 16:26:50.362032 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:50.362012 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-64785b6675-vp8d9" event={"ID":"7fe140c1-9268-46ef-aa31-3283957b0fff","Type":"ContainerStarted","Data":"d137cb2de3dca02e9965a911dfd5c7ab2c541c9aed86a9c4e31f70344f3d992f"} Jul 16 16:26:50.376070 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:50.376016 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-68b7b95d9-dnh2n" podStartSLOduration=5.631274768 podStartE2EDuration="10.376001025s" totalImagesPullingTime="4.744726257s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 16:26:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 16:26:50.375454276 +0000 UTC m=+68.879512124" watchObservedRunningTime="2026-07-16 16:26:50.376001025 +0000 UTC m=+68.880058876" Jul 16 16:26:51.133900 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:51.133802 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-68b7b95d9-dnh2n" Jul 16 16:26:51.133900 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:51.133874 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-68b7b95d9-dnh2n" Jul 16 16:26:51.135548 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:51.135178 2577 patch_prober.go:28] interesting pod/console-68b7b95d9-dnh2n container/console namespace/openshift-console: Startup probe status=failure output="Get \"https://10.134.0.9:8443/health\": dial tcp 10.134.0.9:8443: connect: connection refused" start-of-body= Jul 16 16:26:51.135548 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:51.135257 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-console/console-68b7b95d9-dnh2n" podUID="cabb304d-fabd-4944-87f1-b7deccca8616" containerName="console" probeResult="failure" output="Get \"https://10.134.0.9:8443/health\": dial tcp 10.134.0.9:8443: connect: connection refused" Jul 16 16:26:51.363044 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:51.362746 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-76dcf44fd5-tx9p2"] Jul 16 16:26:51.386736 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:51.386616 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-76dcf44fd5-tx9p2"] Jul 16 16:26:51.386909 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:51.386765 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-76dcf44fd5-tx9p2" Jul 16 16:26:51.398327 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:51.398298 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 16 16:26:51.521735 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:51.521694 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/744cb445-c1cf-423d-9f10-7f00df75a709-console-serving-cert\") pod \"console-76dcf44fd5-tx9p2\" (UID: \"744cb445-c1cf-423d-9f10-7f00df75a709\") " pod="openshift-console/console-76dcf44fd5-tx9p2" Jul 16 16:26:51.521907 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:51.521749 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4p9nn\" (UniqueName: \"kubernetes.io/projected/744cb445-c1cf-423d-9f10-7f00df75a709-kube-api-access-4p9nn\") pod \"console-76dcf44fd5-tx9p2\" (UID: \"744cb445-c1cf-423d-9f10-7f00df75a709\") " pod="openshift-console/console-76dcf44fd5-tx9p2" Jul 16 16:26:51.521907 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:51.521777 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/744cb445-c1cf-423d-9f10-7f00df75a709-trusted-ca-bundle\") pod \"console-76dcf44fd5-tx9p2\" (UID: \"744cb445-c1cf-423d-9f10-7f00df75a709\") " pod="openshift-console/console-76dcf44fd5-tx9p2" Jul 16 16:26:51.521907 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:51.521816 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/744cb445-c1cf-423d-9f10-7f00df75a709-service-ca\") pod \"console-76dcf44fd5-tx9p2\" (UID: \"744cb445-c1cf-423d-9f10-7f00df75a709\") " pod="openshift-console/console-76dcf44fd5-tx9p2" Jul 16 16:26:51.521907 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:51.521857 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/744cb445-c1cf-423d-9f10-7f00df75a709-console-config\") pod \"console-76dcf44fd5-tx9p2\" (UID: \"744cb445-c1cf-423d-9f10-7f00df75a709\") " pod="openshift-console/console-76dcf44fd5-tx9p2" Jul 16 16:26:51.521907 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:51.521888 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/744cb445-c1cf-423d-9f10-7f00df75a709-oauth-serving-cert\") pod \"console-76dcf44fd5-tx9p2\" (UID: \"744cb445-c1cf-423d-9f10-7f00df75a709\") " pod="openshift-console/console-76dcf44fd5-tx9p2" Jul 16 16:26:51.522169 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:51.521943 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/744cb445-c1cf-423d-9f10-7f00df75a709-console-oauth-config\") pod \"console-76dcf44fd5-tx9p2\" (UID: \"744cb445-c1cf-423d-9f10-7f00df75a709\") " pod="openshift-console/console-76dcf44fd5-tx9p2" Jul 16 16:26:51.623018 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:51.622963 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/744cb445-c1cf-423d-9f10-7f00df75a709-console-config\") pod \"console-76dcf44fd5-tx9p2\" (UID: \"744cb445-c1cf-423d-9f10-7f00df75a709\") " pod="openshift-console/console-76dcf44fd5-tx9p2" Jul 16 16:26:51.623018 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:51.623019 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/744cb445-c1cf-423d-9f10-7f00df75a709-oauth-serving-cert\") pod \"console-76dcf44fd5-tx9p2\" (UID: \"744cb445-c1cf-423d-9f10-7f00df75a709\") " pod="openshift-console/console-76dcf44fd5-tx9p2" Jul 16 16:26:51.623254 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:51.623072 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/744cb445-c1cf-423d-9f10-7f00df75a709-console-oauth-config\") pod \"console-76dcf44fd5-tx9p2\" (UID: \"744cb445-c1cf-423d-9f10-7f00df75a709\") " pod="openshift-console/console-76dcf44fd5-tx9p2" Jul 16 16:26:51.623254 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:51.623112 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/744cb445-c1cf-423d-9f10-7f00df75a709-console-serving-cert\") pod \"console-76dcf44fd5-tx9p2\" (UID: \"744cb445-c1cf-423d-9f10-7f00df75a709\") " pod="openshift-console/console-76dcf44fd5-tx9p2" Jul 16 16:26:51.623254 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:51.623137 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4p9nn\" (UniqueName: \"kubernetes.io/projected/744cb445-c1cf-423d-9f10-7f00df75a709-kube-api-access-4p9nn\") pod \"console-76dcf44fd5-tx9p2\" (UID: \"744cb445-c1cf-423d-9f10-7f00df75a709\") " pod="openshift-console/console-76dcf44fd5-tx9p2" Jul 16 16:26:51.623254 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:51.623165 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/744cb445-c1cf-423d-9f10-7f00df75a709-trusted-ca-bundle\") pod \"console-76dcf44fd5-tx9p2\" (UID: \"744cb445-c1cf-423d-9f10-7f00df75a709\") " pod="openshift-console/console-76dcf44fd5-tx9p2" Jul 16 16:26:51.623254 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:51.623200 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/744cb445-c1cf-423d-9f10-7f00df75a709-service-ca\") pod \"console-76dcf44fd5-tx9p2\" (UID: \"744cb445-c1cf-423d-9f10-7f00df75a709\") " pod="openshift-console/console-76dcf44fd5-tx9p2" Jul 16 16:26:51.624006 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:51.623976 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/744cb445-c1cf-423d-9f10-7f00df75a709-service-ca\") pod \"console-76dcf44fd5-tx9p2\" (UID: \"744cb445-c1cf-423d-9f10-7f00df75a709\") " pod="openshift-console/console-76dcf44fd5-tx9p2" Jul 16 16:26:51.624943 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:51.624917 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/744cb445-c1cf-423d-9f10-7f00df75a709-console-config\") pod \"console-76dcf44fd5-tx9p2\" (UID: \"744cb445-c1cf-423d-9f10-7f00df75a709\") " pod="openshift-console/console-76dcf44fd5-tx9p2" Jul 16 16:26:51.625569 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:51.625525 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/744cb445-c1cf-423d-9f10-7f00df75a709-oauth-serving-cert\") pod \"console-76dcf44fd5-tx9p2\" (UID: \"744cb445-c1cf-423d-9f10-7f00df75a709\") " pod="openshift-console/console-76dcf44fd5-tx9p2" Jul 16 16:26:51.626053 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:51.625955 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/744cb445-c1cf-423d-9f10-7f00df75a709-trusted-ca-bundle\") pod \"console-76dcf44fd5-tx9p2\" (UID: \"744cb445-c1cf-423d-9f10-7f00df75a709\") " pod="openshift-console/console-76dcf44fd5-tx9p2" Jul 16 16:26:51.627636 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:51.627609 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/744cb445-c1cf-423d-9f10-7f00df75a709-console-oauth-config\") pod \"console-76dcf44fd5-tx9p2\" (UID: \"744cb445-c1cf-423d-9f10-7f00df75a709\") " pod="openshift-console/console-76dcf44fd5-tx9p2" Jul 16 16:26:51.627768 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:51.627698 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/744cb445-c1cf-423d-9f10-7f00df75a709-console-serving-cert\") pod \"console-76dcf44fd5-tx9p2\" (UID: \"744cb445-c1cf-423d-9f10-7f00df75a709\") " pod="openshift-console/console-76dcf44fd5-tx9p2" Jul 16 16:26:51.634025 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:51.633999 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4p9nn\" (UniqueName: \"kubernetes.io/projected/744cb445-c1cf-423d-9f10-7f00df75a709-kube-api-access-4p9nn\") pod \"console-76dcf44fd5-tx9p2\" (UID: \"744cb445-c1cf-423d-9f10-7f00df75a709\") " pod="openshift-console/console-76dcf44fd5-tx9p2" Jul 16 16:26:51.702068 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:51.701800 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-76dcf44fd5-tx9p2" Jul 16 16:26:51.888568 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:51.888438 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-76dcf44fd5-tx9p2"] Jul 16 16:26:51.978743 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:26:51.977516 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod744cb445_c1cf_423d_9f10_7f00df75a709.slice/crio-07f74b231b30f0acd150d2e4126367cc29ebabeee1554d9e3d92d8033981f773 WatchSource:0}: Error finding container 07f74b231b30f0acd150d2e4126367cc29ebabeee1554d9e3d92d8033981f773: Status 404 returned error can't find the container with id 07f74b231b30f0acd150d2e4126367cc29ebabeee1554d9e3d92d8033981f773 Jul 16 16:26:52.377195 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:52.376005 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-bcb7f" event={"ID":"94ec201c-7285-4751-be16-d23d2f2c0290","Type":"ContainerStarted","Data":"eaeaee54f3da2edabd95ea163009c55805f13b2fe83a4237db86f9eb81333a71"} Jul 16 16:26:52.377195 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:52.376113 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-bcb7f" event={"ID":"94ec201c-7285-4751-be16-d23d2f2c0290","Type":"ContainerStarted","Data":"d1e39453916e497c37d564d73ad158987194c1a0c4bcfa8787ec5ae561cbf875"} Jul 16 16:26:52.384060 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:52.383791 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-76dcf44fd5-tx9p2" event={"ID":"744cb445-c1cf-423d-9f10-7f00df75a709","Type":"ContainerStarted","Data":"bbfeeaa3bedd1cc9ca63fa6e08dd5157028e0aa1eddb4fd8c35a4e6564af8cde"} Jul 16 16:26:52.384060 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:52.383838 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-76dcf44fd5-tx9p2" event={"ID":"744cb445-c1cf-423d-9f10-7f00df75a709","Type":"ContainerStarted","Data":"07f74b231b30f0acd150d2e4126367cc29ebabeee1554d9e3d92d8033981f773"} Jul 16 16:26:52.419416 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:52.418726 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-bcb7f" podStartSLOduration=68.51781023 podStartE2EDuration="1m10.418702199s" totalImagesPullingTime="1.900891969s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 16:25:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 16:26:52.396784521 +0000 UTC m=+70.900842375" watchObservedRunningTime="2026-07-16 16:26:52.418702199 +0000 UTC m=+70.922760051" Jul 16 16:26:52.419416 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:52.419026 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-76dcf44fd5-tx9p2" podStartSLOduration=1.419019341 podStartE2EDuration="1.419019341s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 16:26:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 16:26:52.417823394 +0000 UTC m=+70.921881256" watchObservedRunningTime="2026-07-16 16:26:52.419019341 +0000 UTC m=+70.923077194" Jul 16 16:26:56.346873 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:56.346843 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-qhj4x" Jul 16 16:26:57.403212 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:57.403174 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-rqzn4" event={"ID":"e2d675d8-edf6-4765-a240-4b723cdd2d38","Type":"ContainerStarted","Data":"e98cc28bf511e36962912cf6228184803494eecc4a682498d0c6a867f53502ba"} Jul 16 16:26:57.405093 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:57.405067 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-lf2qz" event={"ID":"9964ff62-4edc-4638-adf2-bbc37c39d29c","Type":"ContainerStarted","Data":"3a7b1f7fcfe419d636387459ddc05dc2ebdaff1fc3338458c02bd044e73333be"} Jul 16 16:26:57.406911 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:57.406887 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-64785b6675-vp8d9" event={"ID":"7fe140c1-9268-46ef-aa31-3283957b0fff","Type":"ContainerStarted","Data":"28366e164dc439e9747d25baab4a0958c9b9ffdad5003ada825810a34b5c83b1"} Jul 16 16:26:57.407040 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:57.406933 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-64785b6675-vp8d9" event={"ID":"7fe140c1-9268-46ef-aa31-3283957b0fff","Type":"ContainerStarted","Data":"926b5ca9a00de7f076c1d2322b5b238aa59938ca1a421975bb068a0f0cee9089"} Jul 16 16:26:57.408257 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:57.408239 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-ntktc" event={"ID":"84cc596d-4f15-4c84-8625-c7f37ed9dae6","Type":"ContainerStarted","Data":"6b6d945dbd371027eaaa5048dac92cc10de4a1ca970687b83c080763f07f804f"} Jul 16 16:26:57.408361 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:57.408349 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-ntktc" Jul 16 16:26:57.425514 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:57.425469 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-rqzn4" podStartSLOduration=68.560693472 podStartE2EDuration="1m15.425454326s" totalImagesPullingTime="6.864760854s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 16:25:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 16:26:57.425131066 +0000 UTC m=+75.929188917" watchObservedRunningTime="2026-07-16 16:26:57.425454326 +0000 UTC m=+75.929512178" Jul 16 16:26:57.447675 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:57.447624 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-64785b6675-vp8d9" podStartSLOduration=1.6640099849999999 podStartE2EDuration="8.447607518s" totalImagesPullingTime="6.783597533s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 16:26:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 16:26:57.446882496 +0000 UTC m=+75.950940343" watchObservedRunningTime="2026-07-16 16:26:57.447607518 +0000 UTC m=+75.951665369" Jul 16 16:26:57.466674 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:57.466620 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-ntktc" podStartSLOduration=68.630167412 podStartE2EDuration="1m15.466604712s" totalImagesPullingTime="6.8364373s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 16:25:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 16:26:57.465720112 +0000 UTC m=+75.969777962" watchObservedRunningTime="2026-07-16 16:26:57.466604712 +0000 UTC m=+75.970662590" Jul 16 16:26:57.486123 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:57.486071 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-lf2qz" podStartSLOduration=1.674870997 podStartE2EDuration="19.486055432s" totalImagesPullingTime="17.811184435s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 16:26:38 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 16:26:57.485410259 +0000 UTC m=+75.989468111" watchObservedRunningTime="2026-07-16 16:26:57.486055432 +0000 UTC m=+75.990113301" Jul 16 16:26:59.506332 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:59.506299 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-84bd68b896-xfk8h"] Jul 16 16:26:59.509539 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:59.509523 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-xfk8h" Jul 16 16:26:59.513913 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:59.513884 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Jul 16 16:26:59.514030 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:59.513922 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-m7fqx\"" Jul 16 16:26:59.514030 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:59.513977 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Jul 16 16:26:59.519633 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:59.519611 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-hvwxd"] Jul 16 16:26:59.522578 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:59.522558 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-hvwxd" Jul 16 16:26:59.525586 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:59.525561 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 16 16:26:59.525833 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:59.525817 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 16 16:26:59.526673 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:59.526658 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 16 16:26:59.528912 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:59.528891 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-kngrv\"" Jul 16 16:26:59.530020 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:59.529991 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-84bd68b896-xfk8h"] Jul 16 16:26:59.531414 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:59.531373 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-785778ff74-vfxfr"] Jul 16 16:26:59.536727 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:59.536705 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-785778ff74-vfxfr" Jul 16 16:26:59.539471 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:59.539443 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Jul 16 16:26:59.539699 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:59.539670 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-ldlrq\"" Jul 16 16:26:59.539963 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:59.539919 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Jul 16 16:26:59.540455 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:59.540439 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Jul 16 16:26:59.547826 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:59.547803 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-785778ff74-vfxfr"] Jul 16 16:26:59.592508 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:59.592473 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6nz6p\" (UniqueName: \"kubernetes.io/projected/42c6e795-fc08-4367-8e7f-9c65cb84c814-kube-api-access-6nz6p\") pod \"node-exporter-hvwxd\" (UID: \"42c6e795-fc08-4367-8e7f-9c65cb84c814\") " pod="openshift-monitoring/node-exporter-hvwxd" Jul 16 16:26:59.592723 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:59.592523 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/42c6e795-fc08-4367-8e7f-9c65cb84c814-node-exporter-wtmp\") pod \"node-exporter-hvwxd\" (UID: \"42c6e795-fc08-4367-8e7f-9c65cb84c814\") " pod="openshift-monitoring/node-exporter-hvwxd" Jul 16 16:26:59.592723 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:59.592553 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3253b8ec-f532-448b-9a7e-6fcb71b0d3fd-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-84bd68b896-xfk8h\" (UID: \"3253b8ec-f532-448b-9a7e-6fcb71b0d3fd\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-xfk8h" Jul 16 16:26:59.592723 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:59.592570 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/42c6e795-fc08-4367-8e7f-9c65cb84c814-node-exporter-textfile\") pod \"node-exporter-hvwxd\" (UID: \"42c6e795-fc08-4367-8e7f-9c65cb84c814\") " pod="openshift-monitoring/node-exporter-hvwxd" Jul 16 16:26:59.592723 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:59.592621 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/42c6e795-fc08-4367-8e7f-9c65cb84c814-node-exporter-tls\") pod \"node-exporter-hvwxd\" (UID: \"42c6e795-fc08-4367-8e7f-9c65cb84c814\") " pod="openshift-monitoring/node-exporter-hvwxd" Jul 16 16:26:59.592723 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:59.592681 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3253b8ec-f532-448b-9a7e-6fcb71b0d3fd-metrics-client-ca\") pod \"openshift-state-metrics-84bd68b896-xfk8h\" (UID: \"3253b8ec-f532-448b-9a7e-6fcb71b0d3fd\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-xfk8h" Jul 16 16:26:59.592723 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:59.592707 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/42c6e795-fc08-4367-8e7f-9c65cb84c814-node-exporter-accelerators-collector-config\") pod \"node-exporter-hvwxd\" (UID: \"42c6e795-fc08-4367-8e7f-9c65cb84c814\") " pod="openshift-monitoring/node-exporter-hvwxd" Jul 16 16:26:59.592991 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:59.592754 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/3253b8ec-f532-448b-9a7e-6fcb71b0d3fd-openshift-state-metrics-tls\") pod \"openshift-state-metrics-84bd68b896-xfk8h\" (UID: \"3253b8ec-f532-448b-9a7e-6fcb71b0d3fd\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-xfk8h" Jul 16 16:26:59.592991 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:59.592783 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/42c6e795-fc08-4367-8e7f-9c65cb84c814-root\") pod \"node-exporter-hvwxd\" (UID: \"42c6e795-fc08-4367-8e7f-9c65cb84c814\") " pod="openshift-monitoring/node-exporter-hvwxd" Jul 16 16:26:59.592991 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:59.592809 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/42c6e795-fc08-4367-8e7f-9c65cb84c814-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-hvwxd\" (UID: \"42c6e795-fc08-4367-8e7f-9c65cb84c814\") " pod="openshift-monitoring/node-exporter-hvwxd" Jul 16 16:26:59.592991 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:59.592838 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g8ngt\" (UniqueName: \"kubernetes.io/projected/3253b8ec-f532-448b-9a7e-6fcb71b0d3fd-kube-api-access-g8ngt\") pod \"openshift-state-metrics-84bd68b896-xfk8h\" (UID: \"3253b8ec-f532-448b-9a7e-6fcb71b0d3fd\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-xfk8h" Jul 16 16:26:59.592991 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:59.592862 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/42c6e795-fc08-4367-8e7f-9c65cb84c814-sys\") pod \"node-exporter-hvwxd\" (UID: \"42c6e795-fc08-4367-8e7f-9c65cb84c814\") " pod="openshift-monitoring/node-exporter-hvwxd" Jul 16 16:26:59.592991 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:59.592883 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/42c6e795-fc08-4367-8e7f-9c65cb84c814-metrics-client-ca\") pod \"node-exporter-hvwxd\" (UID: \"42c6e795-fc08-4367-8e7f-9c65cb84c814\") " pod="openshift-monitoring/node-exporter-hvwxd" Jul 16 16:26:59.693774 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:59.693739 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/10bc3cb6-aa0e-4d0f-a5a9-75f494f7fffb-metrics-client-ca\") pod \"kube-state-metrics-785778ff74-vfxfr\" (UID: \"10bc3cb6-aa0e-4d0f-a5a9-75f494f7fffb\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-vfxfr" Jul 16 16:26:59.693983 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:59.693786 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/10bc3cb6-aa0e-4d0f-a5a9-75f494f7fffb-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-785778ff74-vfxfr\" (UID: \"10bc3cb6-aa0e-4d0f-a5a9-75f494f7fffb\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-vfxfr" Jul 16 16:26:59.693983 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:59.693824 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/42c6e795-fc08-4367-8e7f-9c65cb84c814-node-exporter-wtmp\") pod \"node-exporter-hvwxd\" (UID: \"42c6e795-fc08-4367-8e7f-9c65cb84c814\") " pod="openshift-monitoring/node-exporter-hvwxd" Jul 16 16:26:59.693983 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:59.693853 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3253b8ec-f532-448b-9a7e-6fcb71b0d3fd-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-84bd68b896-xfk8h\" (UID: \"3253b8ec-f532-448b-9a7e-6fcb71b0d3fd\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-xfk8h" Jul 16 16:26:59.693983 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:59.693876 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/42c6e795-fc08-4367-8e7f-9c65cb84c814-node-exporter-textfile\") pod \"node-exporter-hvwxd\" (UID: \"42c6e795-fc08-4367-8e7f-9c65cb84c814\") " pod="openshift-monitoring/node-exporter-hvwxd" Jul 16 16:26:59.693983 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:59.693898 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/42c6e795-fc08-4367-8e7f-9c65cb84c814-node-exporter-tls\") pod \"node-exporter-hvwxd\" (UID: \"42c6e795-fc08-4367-8e7f-9c65cb84c814\") " pod="openshift-monitoring/node-exporter-hvwxd" Jul 16 16:26:59.693983 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:59.693922 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3253b8ec-f532-448b-9a7e-6fcb71b0d3fd-metrics-client-ca\") pod \"openshift-state-metrics-84bd68b896-xfk8h\" (UID: \"3253b8ec-f532-448b-9a7e-6fcb71b0d3fd\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-xfk8h" Jul 16 16:26:59.693983 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:59.693947 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/42c6e795-fc08-4367-8e7f-9c65cb84c814-node-exporter-accelerators-collector-config\") pod \"node-exporter-hvwxd\" (UID: \"42c6e795-fc08-4367-8e7f-9c65cb84c814\") " pod="openshift-monitoring/node-exporter-hvwxd" Jul 16 16:26:59.693983 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:59.693969 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rpvss\" (UniqueName: \"kubernetes.io/projected/10bc3cb6-aa0e-4d0f-a5a9-75f494f7fffb-kube-api-access-rpvss\") pod \"kube-state-metrics-785778ff74-vfxfr\" (UID: \"10bc3cb6-aa0e-4d0f-a5a9-75f494f7fffb\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-vfxfr" Jul 16 16:26:59.694346 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:59.693998 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/10bc3cb6-aa0e-4d0f-a5a9-75f494f7fffb-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-785778ff74-vfxfr\" (UID: \"10bc3cb6-aa0e-4d0f-a5a9-75f494f7fffb\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-vfxfr" Jul 16 16:26:59.694346 ip-10-0-142-101 kubenswrapper[2577]: E0716 16:26:59.694000 2577 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Jul 16 16:26:59.694346 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:59.694038 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/3253b8ec-f532-448b-9a7e-6fcb71b0d3fd-openshift-state-metrics-tls\") pod \"openshift-state-metrics-84bd68b896-xfk8h\" (UID: \"3253b8ec-f532-448b-9a7e-6fcb71b0d3fd\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-xfk8h" Jul 16 16:26:59.694346 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:59.694006 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/42c6e795-fc08-4367-8e7f-9c65cb84c814-node-exporter-wtmp\") pod \"node-exporter-hvwxd\" (UID: \"42c6e795-fc08-4367-8e7f-9c65cb84c814\") " pod="openshift-monitoring/node-exporter-hvwxd" Jul 16 16:26:59.694346 ip-10-0-142-101 kubenswrapper[2577]: E0716 16:26:59.694075 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/42c6e795-fc08-4367-8e7f-9c65cb84c814-node-exporter-tls podName:42c6e795-fc08-4367-8e7f-9c65cb84c814 nodeName:}" failed. No retries permitted until 2026-07-16 16:27:00.194054944 +0000 UTC m=+78.698112773 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/42c6e795-fc08-4367-8e7f-9c65cb84c814-node-exporter-tls") pod "node-exporter-hvwxd" (UID: "42c6e795-fc08-4367-8e7f-9c65cb84c814") : secret "node-exporter-tls" not found Jul 16 16:26:59.694346 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:59.694099 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/42c6e795-fc08-4367-8e7f-9c65cb84c814-root\") pod \"node-exporter-hvwxd\" (UID: \"42c6e795-fc08-4367-8e7f-9c65cb84c814\") " pod="openshift-monitoring/node-exporter-hvwxd" Jul 16 16:26:59.694346 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:59.694134 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/42c6e795-fc08-4367-8e7f-9c65cb84c814-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-hvwxd\" (UID: \"42c6e795-fc08-4367-8e7f-9c65cb84c814\") " pod="openshift-monitoring/node-exporter-hvwxd" Jul 16 16:26:59.694346 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:59.694167 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g8ngt\" (UniqueName: \"kubernetes.io/projected/3253b8ec-f532-448b-9a7e-6fcb71b0d3fd-kube-api-access-g8ngt\") pod \"openshift-state-metrics-84bd68b896-xfk8h\" (UID: \"3253b8ec-f532-448b-9a7e-6fcb71b0d3fd\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-xfk8h" Jul 16 16:26:59.694346 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:59.694196 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/42c6e795-fc08-4367-8e7f-9c65cb84c814-sys\") pod \"node-exporter-hvwxd\" (UID: \"42c6e795-fc08-4367-8e7f-9c65cb84c814\") " pod="openshift-monitoring/node-exporter-hvwxd" Jul 16 16:26:59.694346 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:59.694222 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/42c6e795-fc08-4367-8e7f-9c65cb84c814-metrics-client-ca\") pod \"node-exporter-hvwxd\" (UID: \"42c6e795-fc08-4367-8e7f-9c65cb84c814\") " pod="openshift-monitoring/node-exporter-hvwxd" Jul 16 16:26:59.694346 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:59.694249 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/10bc3cb6-aa0e-4d0f-a5a9-75f494f7fffb-volume-directive-shadow\") pod \"kube-state-metrics-785778ff74-vfxfr\" (UID: \"10bc3cb6-aa0e-4d0f-a5a9-75f494f7fffb\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-vfxfr" Jul 16 16:26:59.694346 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:59.694276 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/10bc3cb6-aa0e-4d0f-a5a9-75f494f7fffb-kube-state-metrics-tls\") pod \"kube-state-metrics-785778ff74-vfxfr\" (UID: \"10bc3cb6-aa0e-4d0f-a5a9-75f494f7fffb\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-vfxfr" Jul 16 16:26:59.694346 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:59.694287 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/42c6e795-fc08-4367-8e7f-9c65cb84c814-node-exporter-textfile\") pod \"node-exporter-hvwxd\" (UID: \"42c6e795-fc08-4367-8e7f-9c65cb84c814\") " pod="openshift-monitoring/node-exporter-hvwxd" Jul 16 16:26:59.695025 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:59.694362 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6nz6p\" (UniqueName: \"kubernetes.io/projected/42c6e795-fc08-4367-8e7f-9c65cb84c814-kube-api-access-6nz6p\") pod \"node-exporter-hvwxd\" (UID: \"42c6e795-fc08-4367-8e7f-9c65cb84c814\") " pod="openshift-monitoring/node-exporter-hvwxd" Jul 16 16:26:59.695025 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:59.694467 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/42c6e795-fc08-4367-8e7f-9c65cb84c814-sys\") pod \"node-exporter-hvwxd\" (UID: \"42c6e795-fc08-4367-8e7f-9c65cb84c814\") " pod="openshift-monitoring/node-exporter-hvwxd" Jul 16 16:26:59.695025 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:59.694742 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/42c6e795-fc08-4367-8e7f-9c65cb84c814-node-exporter-accelerators-collector-config\") pod \"node-exporter-hvwxd\" (UID: \"42c6e795-fc08-4367-8e7f-9c65cb84c814\") " pod="openshift-monitoring/node-exporter-hvwxd" Jul 16 16:26:59.695025 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:59.694787 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3253b8ec-f532-448b-9a7e-6fcb71b0d3fd-metrics-client-ca\") pod \"openshift-state-metrics-84bd68b896-xfk8h\" (UID: \"3253b8ec-f532-448b-9a7e-6fcb71b0d3fd\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-xfk8h" Jul 16 16:26:59.695025 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:59.694850 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/42c6e795-fc08-4367-8e7f-9c65cb84c814-root\") pod \"node-exporter-hvwxd\" (UID: \"42c6e795-fc08-4367-8e7f-9c65cb84c814\") " pod="openshift-monitoring/node-exporter-hvwxd" Jul 16 16:26:59.695226 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:59.695101 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/42c6e795-fc08-4367-8e7f-9c65cb84c814-metrics-client-ca\") pod \"node-exporter-hvwxd\" (UID: \"42c6e795-fc08-4367-8e7f-9c65cb84c814\") " pod="openshift-monitoring/node-exporter-hvwxd" Jul 16 16:26:59.696578 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:59.696553 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/3253b8ec-f532-448b-9a7e-6fcb71b0d3fd-openshift-state-metrics-tls\") pod \"openshift-state-metrics-84bd68b896-xfk8h\" (UID: \"3253b8ec-f532-448b-9a7e-6fcb71b0d3fd\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-xfk8h" Jul 16 16:26:59.696714 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:59.696694 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/42c6e795-fc08-4367-8e7f-9c65cb84c814-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-hvwxd\" (UID: \"42c6e795-fc08-4367-8e7f-9c65cb84c814\") " pod="openshift-monitoring/node-exporter-hvwxd" Jul 16 16:26:59.696814 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:59.696794 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3253b8ec-f532-448b-9a7e-6fcb71b0d3fd-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-84bd68b896-xfk8h\" (UID: \"3253b8ec-f532-448b-9a7e-6fcb71b0d3fd\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-xfk8h" Jul 16 16:26:59.705814 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:59.705785 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g8ngt\" (UniqueName: \"kubernetes.io/projected/3253b8ec-f532-448b-9a7e-6fcb71b0d3fd-kube-api-access-g8ngt\") pod \"openshift-state-metrics-84bd68b896-xfk8h\" (UID: \"3253b8ec-f532-448b-9a7e-6fcb71b0d3fd\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-xfk8h" Jul 16 16:26:59.706214 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:59.706190 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6nz6p\" (UniqueName: \"kubernetes.io/projected/42c6e795-fc08-4367-8e7f-9c65cb84c814-kube-api-access-6nz6p\") pod \"node-exporter-hvwxd\" (UID: \"42c6e795-fc08-4367-8e7f-9c65cb84c814\") " pod="openshift-monitoring/node-exporter-hvwxd" Jul 16 16:26:59.795258 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:59.795163 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/10bc3cb6-aa0e-4d0f-a5a9-75f494f7fffb-metrics-client-ca\") pod \"kube-state-metrics-785778ff74-vfxfr\" (UID: \"10bc3cb6-aa0e-4d0f-a5a9-75f494f7fffb\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-vfxfr" Jul 16 16:26:59.795258 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:59.795209 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/10bc3cb6-aa0e-4d0f-a5a9-75f494f7fffb-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-785778ff74-vfxfr\" (UID: \"10bc3cb6-aa0e-4d0f-a5a9-75f494f7fffb\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-vfxfr" Jul 16 16:26:59.795518 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:59.795264 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rpvss\" (UniqueName: \"kubernetes.io/projected/10bc3cb6-aa0e-4d0f-a5a9-75f494f7fffb-kube-api-access-rpvss\") pod \"kube-state-metrics-785778ff74-vfxfr\" (UID: \"10bc3cb6-aa0e-4d0f-a5a9-75f494f7fffb\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-vfxfr" Jul 16 16:26:59.795518 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:59.795294 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/10bc3cb6-aa0e-4d0f-a5a9-75f494f7fffb-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-785778ff74-vfxfr\" (UID: \"10bc3cb6-aa0e-4d0f-a5a9-75f494f7fffb\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-vfxfr" Jul 16 16:26:59.795518 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:59.795347 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/10bc3cb6-aa0e-4d0f-a5a9-75f494f7fffb-volume-directive-shadow\") pod \"kube-state-metrics-785778ff74-vfxfr\" (UID: \"10bc3cb6-aa0e-4d0f-a5a9-75f494f7fffb\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-vfxfr" Jul 16 16:26:59.795518 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:59.795465 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/10bc3cb6-aa0e-4d0f-a5a9-75f494f7fffb-kube-state-metrics-tls\") pod \"kube-state-metrics-785778ff74-vfxfr\" (UID: \"10bc3cb6-aa0e-4d0f-a5a9-75f494f7fffb\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-vfxfr" Jul 16 16:26:59.795808 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:59.795778 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/10bc3cb6-aa0e-4d0f-a5a9-75f494f7fffb-volume-directive-shadow\") pod \"kube-state-metrics-785778ff74-vfxfr\" (UID: \"10bc3cb6-aa0e-4d0f-a5a9-75f494f7fffb\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-vfxfr" Jul 16 16:26:59.796011 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:59.795993 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/10bc3cb6-aa0e-4d0f-a5a9-75f494f7fffb-metrics-client-ca\") pod \"kube-state-metrics-785778ff74-vfxfr\" (UID: \"10bc3cb6-aa0e-4d0f-a5a9-75f494f7fffb\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-vfxfr" Jul 16 16:26:59.796080 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:59.796059 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/10bc3cb6-aa0e-4d0f-a5a9-75f494f7fffb-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-785778ff74-vfxfr\" (UID: \"10bc3cb6-aa0e-4d0f-a5a9-75f494f7fffb\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-vfxfr" Jul 16 16:26:59.797735 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:59.797712 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/10bc3cb6-aa0e-4d0f-a5a9-75f494f7fffb-kube-state-metrics-tls\") pod \"kube-state-metrics-785778ff74-vfxfr\" (UID: \"10bc3cb6-aa0e-4d0f-a5a9-75f494f7fffb\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-vfxfr" Jul 16 16:26:59.797894 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:59.797872 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/10bc3cb6-aa0e-4d0f-a5a9-75f494f7fffb-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-785778ff74-vfxfr\" (UID: \"10bc3cb6-aa0e-4d0f-a5a9-75f494f7fffb\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-vfxfr" Jul 16 16:26:59.808207 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:59.808181 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rpvss\" (UniqueName: \"kubernetes.io/projected/10bc3cb6-aa0e-4d0f-a5a9-75f494f7fffb-kube-api-access-rpvss\") pod \"kube-state-metrics-785778ff74-vfxfr\" (UID: \"10bc3cb6-aa0e-4d0f-a5a9-75f494f7fffb\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-vfxfr" Jul 16 16:26:59.818202 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:59.818172 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-xfk8h" Jul 16 16:26:59.848748 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:59.848716 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-785778ff74-vfxfr" Jul 16 16:26:59.963738 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:59.963681 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-84bd68b896-xfk8h"] Jul 16 16:26:59.967127 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:26:59.967098 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3253b8ec_f532_448b_9a7e_6fcb71b0d3fd.slice/crio-772309a90d1b2e006e5dcf4f63cc6d63fec8c95b41ffe07d891883bc2d0f61e8 WatchSource:0}: Error finding container 772309a90d1b2e006e5dcf4f63cc6d63fec8c95b41ffe07d891883bc2d0f61e8: Status 404 returned error can't find the container with id 772309a90d1b2e006e5dcf4f63cc6d63fec8c95b41ffe07d891883bc2d0f61e8 Jul 16 16:26:59.994991 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:26:59.994963 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-785778ff74-vfxfr"] Jul 16 16:26:59.998184 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:26:59.998162 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10bc3cb6_aa0e_4d0f_a5a9_75f494f7fffb.slice/crio-c56d06030a5e251f3d0987a01f339e748f30fd1434b8bc4c77fe1cc83a19c6ed WatchSource:0}: Error finding container c56d06030a5e251f3d0987a01f339e748f30fd1434b8bc4c77fe1cc83a19c6ed: Status 404 returned error can't find the container with id c56d06030a5e251f3d0987a01f339e748f30fd1434b8bc4c77fe1cc83a19c6ed Jul 16 16:27:00.199031 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:00.198997 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/42c6e795-fc08-4367-8e7f-9c65cb84c814-node-exporter-tls\") pod \"node-exporter-hvwxd\" (UID: \"42c6e795-fc08-4367-8e7f-9c65cb84c814\") " pod="openshift-monitoring/node-exporter-hvwxd" Jul 16 16:27:00.201431 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:00.201401 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/42c6e795-fc08-4367-8e7f-9c65cb84c814-node-exporter-tls\") pod \"node-exporter-hvwxd\" (UID: \"42c6e795-fc08-4367-8e7f-9c65cb84c814\") " pod="openshift-monitoring/node-exporter-hvwxd" Jul 16 16:27:00.422156 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:00.422114 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-785778ff74-vfxfr" event={"ID":"10bc3cb6-aa0e-4d0f-a5a9-75f494f7fffb","Type":"ContainerStarted","Data":"c56d06030a5e251f3d0987a01f339e748f30fd1434b8bc4c77fe1cc83a19c6ed"} Jul 16 16:27:00.424456 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:00.424422 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-xfk8h" event={"ID":"3253b8ec-f532-448b-9a7e-6fcb71b0d3fd","Type":"ContainerStarted","Data":"56f8bfe1f9a5a61b23a06fdf871651c20824a09eed8bf46780f19cd22d2e4eca"} Jul 16 16:27:00.424617 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:00.424483 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-xfk8h" event={"ID":"3253b8ec-f532-448b-9a7e-6fcb71b0d3fd","Type":"ContainerStarted","Data":"544096be9f1cf186ce181d1870d61582bc46ed610c3c43a5529ad1d3107d1db6"} Jul 16 16:27:00.424617 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:00.424511 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-xfk8h" event={"ID":"3253b8ec-f532-448b-9a7e-6fcb71b0d3fd","Type":"ContainerStarted","Data":"772309a90d1b2e006e5dcf4f63cc6d63fec8c95b41ffe07d891883bc2d0f61e8"} Jul 16 16:27:00.432717 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:00.432688 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-hvwxd" Jul 16 16:27:00.446596 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:27:00.446564 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42c6e795_fc08_4367_8e7f_9c65cb84c814.slice/crio-ea36657b9e76ef73e9af40109aa3adef7b979d90fd3814a7d80d8a91bf7317f6 WatchSource:0}: Error finding container ea36657b9e76ef73e9af40109aa3adef7b979d90fd3814a7d80d8a91bf7317f6: Status 404 returned error can't find the container with id ea36657b9e76ef73e9af40109aa3adef7b979d90fd3814a7d80d8a91bf7317f6 Jul 16 16:27:00.598257 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:00.598223 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 16 16:27:00.603042 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:00.603019 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 16 16:27:00.608161 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:00.608134 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Jul 16 16:27:00.608297 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:00.608207 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Jul 16 16:27:00.608832 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:00.608812 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Jul 16 16:27:00.608997 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:00.608832 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Jul 16 16:27:00.608997 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:00.608813 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Jul 16 16:27:00.608997 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:00.608884 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Jul 16 16:27:00.609177 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:00.609135 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Jul 16 16:27:00.609177 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:00.609173 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Jul 16 16:27:00.609290 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:00.609215 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-g2j5b\"" Jul 16 16:27:00.611609 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:00.611588 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Jul 16 16:27:00.643538 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:00.643419 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 16 16:27:00.704046 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:00.703939 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8c88b029-8a81-402c-90e4-77eff554183c-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"8c88b029-8a81-402c-90e4-77eff554183c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 16:27:00.704046 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:00.703992 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8c88b029-8a81-402c-90e4-77eff554183c-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"8c88b029-8a81-402c-90e4-77eff554183c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 16:27:00.704227 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:00.704076 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6dx2b\" (UniqueName: \"kubernetes.io/projected/8c88b029-8a81-402c-90e4-77eff554183c-kube-api-access-6dx2b\") pod \"alertmanager-main-0\" (UID: \"8c88b029-8a81-402c-90e4-77eff554183c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 16:27:00.704227 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:00.704145 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/8c88b029-8a81-402c-90e4-77eff554183c-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"8c88b029-8a81-402c-90e4-77eff554183c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 16:27:00.704227 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:00.704186 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8c88b029-8a81-402c-90e4-77eff554183c-config-out\") pod \"alertmanager-main-0\" (UID: \"8c88b029-8a81-402c-90e4-77eff554183c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 16:27:00.704227 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:00.704209 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/8c88b029-8a81-402c-90e4-77eff554183c-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"8c88b029-8a81-402c-90e4-77eff554183c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 16:27:00.704468 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:00.704234 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8c88b029-8a81-402c-90e4-77eff554183c-web-config\") pod \"alertmanager-main-0\" (UID: \"8c88b029-8a81-402c-90e4-77eff554183c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 16:27:00.704468 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:00.704320 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/8c88b029-8a81-402c-90e4-77eff554183c-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"8c88b029-8a81-402c-90e4-77eff554183c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 16:27:00.704468 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:00.704352 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/8c88b029-8a81-402c-90e4-77eff554183c-config-volume\") pod \"alertmanager-main-0\" (UID: \"8c88b029-8a81-402c-90e4-77eff554183c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 16:27:00.704468 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:00.704407 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/8c88b029-8a81-402c-90e4-77eff554183c-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"8c88b029-8a81-402c-90e4-77eff554183c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 16:27:00.704468 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:00.704457 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8c88b029-8a81-402c-90e4-77eff554183c-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"8c88b029-8a81-402c-90e4-77eff554183c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 16:27:00.704719 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:00.704492 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8c88b029-8a81-402c-90e4-77eff554183c-tls-assets\") pod \"alertmanager-main-0\" (UID: \"8c88b029-8a81-402c-90e4-77eff554183c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 16:27:00.704719 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:00.704532 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8c88b029-8a81-402c-90e4-77eff554183c-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"8c88b029-8a81-402c-90e4-77eff554183c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 16:27:00.805491 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:00.805446 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8c88b029-8a81-402c-90e4-77eff554183c-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"8c88b029-8a81-402c-90e4-77eff554183c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 16:27:00.805684 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:00.805510 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8c88b029-8a81-402c-90e4-77eff554183c-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"8c88b029-8a81-402c-90e4-77eff554183c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 16:27:00.805684 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:00.805544 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8c88b029-8a81-402c-90e4-77eff554183c-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"8c88b029-8a81-402c-90e4-77eff554183c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 16:27:00.805684 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:00.805570 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6dx2b\" (UniqueName: \"kubernetes.io/projected/8c88b029-8a81-402c-90e4-77eff554183c-kube-api-access-6dx2b\") pod \"alertmanager-main-0\" (UID: \"8c88b029-8a81-402c-90e4-77eff554183c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 16:27:00.805684 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:00.805608 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/8c88b029-8a81-402c-90e4-77eff554183c-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"8c88b029-8a81-402c-90e4-77eff554183c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 16:27:00.805684 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:00.805641 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8c88b029-8a81-402c-90e4-77eff554183c-config-out\") pod \"alertmanager-main-0\" (UID: \"8c88b029-8a81-402c-90e4-77eff554183c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 16:27:00.805684 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:00.805667 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/8c88b029-8a81-402c-90e4-77eff554183c-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"8c88b029-8a81-402c-90e4-77eff554183c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 16:27:00.805942 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:00.805688 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8c88b029-8a81-402c-90e4-77eff554183c-web-config\") pod \"alertmanager-main-0\" (UID: \"8c88b029-8a81-402c-90e4-77eff554183c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 16:27:00.805942 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:00.805737 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/8c88b029-8a81-402c-90e4-77eff554183c-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"8c88b029-8a81-402c-90e4-77eff554183c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 16:27:00.805942 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:00.805761 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/8c88b029-8a81-402c-90e4-77eff554183c-config-volume\") pod \"alertmanager-main-0\" (UID: \"8c88b029-8a81-402c-90e4-77eff554183c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 16:27:00.805942 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:00.805789 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/8c88b029-8a81-402c-90e4-77eff554183c-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"8c88b029-8a81-402c-90e4-77eff554183c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 16:27:00.805942 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:00.805820 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8c88b029-8a81-402c-90e4-77eff554183c-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"8c88b029-8a81-402c-90e4-77eff554183c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 16:27:00.805942 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:00.805878 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8c88b029-8a81-402c-90e4-77eff554183c-tls-assets\") pod \"alertmanager-main-0\" (UID: \"8c88b029-8a81-402c-90e4-77eff554183c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 16:27:00.806225 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:00.806063 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/8c88b029-8a81-402c-90e4-77eff554183c-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"8c88b029-8a81-402c-90e4-77eff554183c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 16:27:00.806983 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:00.806415 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8c88b029-8a81-402c-90e4-77eff554183c-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"8c88b029-8a81-402c-90e4-77eff554183c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 16:27:00.806983 ip-10-0-142-101 kubenswrapper[2577]: E0716 16:27:00.806820 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/8c88b029-8a81-402c-90e4-77eff554183c-alertmanager-trusted-ca-bundle podName:8c88b029-8a81-402c-90e4-77eff554183c nodeName:}" failed. No retries permitted until 2026-07-16 16:27:01.306799468 +0000 UTC m=+79.810857316 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "alertmanager-trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/8c88b029-8a81-402c-90e4-77eff554183c-alertmanager-trusted-ca-bundle") pod "alertmanager-main-0" (UID: "8c88b029-8a81-402c-90e4-77eff554183c") : configmap references non-existent config key: ca-bundle.crt Jul 16 16:27:00.809309 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:00.809279 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8c88b029-8a81-402c-90e4-77eff554183c-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"8c88b029-8a81-402c-90e4-77eff554183c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 16:27:00.809430 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:00.809338 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/8c88b029-8a81-402c-90e4-77eff554183c-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"8c88b029-8a81-402c-90e4-77eff554183c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 16:27:00.810368 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:00.810342 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8c88b029-8a81-402c-90e4-77eff554183c-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"8c88b029-8a81-402c-90e4-77eff554183c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 16:27:00.810906 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:00.810884 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8c88b029-8a81-402c-90e4-77eff554183c-config-out\") pod \"alertmanager-main-0\" (UID: \"8c88b029-8a81-402c-90e4-77eff554183c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 16:27:00.811511 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:00.811489 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/8c88b029-8a81-402c-90e4-77eff554183c-config-volume\") pod \"alertmanager-main-0\" (UID: \"8c88b029-8a81-402c-90e4-77eff554183c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 16:27:00.811511 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:00.811502 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8c88b029-8a81-402c-90e4-77eff554183c-tls-assets\") pod \"alertmanager-main-0\" (UID: \"8c88b029-8a81-402c-90e4-77eff554183c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 16:27:00.811680 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:00.811544 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/8c88b029-8a81-402c-90e4-77eff554183c-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"8c88b029-8a81-402c-90e4-77eff554183c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 16:27:00.812181 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:00.812157 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/8c88b029-8a81-402c-90e4-77eff554183c-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"8c88b029-8a81-402c-90e4-77eff554183c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 16:27:00.812282 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:00.812228 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8c88b029-8a81-402c-90e4-77eff554183c-web-config\") pod \"alertmanager-main-0\" (UID: \"8c88b029-8a81-402c-90e4-77eff554183c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 16:27:00.818849 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:00.818827 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6dx2b\" (UniqueName: \"kubernetes.io/projected/8c88b029-8a81-402c-90e4-77eff554183c-kube-api-access-6dx2b\") pod \"alertmanager-main-0\" (UID: \"8c88b029-8a81-402c-90e4-77eff554183c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 16:27:01.139301 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:01.139246 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-68b7b95d9-dnh2n" Jul 16 16:27:01.144679 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:01.144641 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-68b7b95d9-dnh2n" Jul 16 16:27:01.310922 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:01.310883 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8c88b029-8a81-402c-90e4-77eff554183c-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"8c88b029-8a81-402c-90e4-77eff554183c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 16:27:01.311660 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:01.311635 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8c88b029-8a81-402c-90e4-77eff554183c-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"8c88b029-8a81-402c-90e4-77eff554183c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 16:27:01.430331 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:01.430239 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-hvwxd" event={"ID":"42c6e795-fc08-4367-8e7f-9c65cb84c814","Type":"ContainerStarted","Data":"ea36657b9e76ef73e9af40109aa3adef7b979d90fd3814a7d80d8a91bf7317f6"} Jul 16 16:27:01.513354 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:01.513300 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 16 16:27:01.593091 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:01.593057 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-b957bd756-6cvr2"] Jul 16 16:27:01.597757 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:01.595798 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-b957bd756-6cvr2" Jul 16 16:27:01.602318 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:01.602296 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Jul 16 16:27:01.603322 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:01.603297 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Jul 16 16:27:01.603457 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:01.603300 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Jul 16 16:27:01.603457 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:01.603306 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Jul 16 16:27:01.603797 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:01.603673 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Jul 16 16:27:01.603876 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:01.603866 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-ducdchmnltu3s\"" Jul 16 16:27:01.604135 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:01.604123 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-v7rr6\"" Jul 16 16:27:01.636005 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:01.635974 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-b957bd756-6cvr2"] Jul 16 16:27:01.702216 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:01.702190 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-76dcf44fd5-tx9p2" Jul 16 16:27:01.702356 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:01.702296 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-76dcf44fd5-tx9p2" Jul 16 16:27:01.707713 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:01.707692 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-76dcf44fd5-tx9p2" Jul 16 16:27:01.715395 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:01.715349 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/2e58ee24-b3bc-4e3a-828c-6ac801ec7fd7-secret-grpc-tls\") pod \"thanos-querier-b957bd756-6cvr2\" (UID: \"2e58ee24-b3bc-4e3a-828c-6ac801ec7fd7\") " pod="openshift-monitoring/thanos-querier-b957bd756-6cvr2" Jul 16 16:27:01.715546 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:01.715422 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/2e58ee24-b3bc-4e3a-828c-6ac801ec7fd7-secret-thanos-querier-tls\") pod \"thanos-querier-b957bd756-6cvr2\" (UID: \"2e58ee24-b3bc-4e3a-828c-6ac801ec7fd7\") " pod="openshift-monitoring/thanos-querier-b957bd756-6cvr2" Jul 16 16:27:01.715546 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:01.715460 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/2e58ee24-b3bc-4e3a-828c-6ac801ec7fd7-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-b957bd756-6cvr2\" (UID: \"2e58ee24-b3bc-4e3a-828c-6ac801ec7fd7\") " pod="openshift-monitoring/thanos-querier-b957bd756-6cvr2" Jul 16 16:27:01.715675 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:01.715564 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/2e58ee24-b3bc-4e3a-828c-6ac801ec7fd7-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-b957bd756-6cvr2\" (UID: \"2e58ee24-b3bc-4e3a-828c-6ac801ec7fd7\") " pod="openshift-monitoring/thanos-querier-b957bd756-6cvr2" Jul 16 16:27:01.715675 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:01.715613 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2e58ee24-b3bc-4e3a-828c-6ac801ec7fd7-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-b957bd756-6cvr2\" (UID: \"2e58ee24-b3bc-4e3a-828c-6ac801ec7fd7\") " pod="openshift-monitoring/thanos-querier-b957bd756-6cvr2" Jul 16 16:27:01.715675 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:01.715656 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2e58ee24-b3bc-4e3a-828c-6ac801ec7fd7-metrics-client-ca\") pod \"thanos-querier-b957bd756-6cvr2\" (UID: \"2e58ee24-b3bc-4e3a-828c-6ac801ec7fd7\") " pod="openshift-monitoring/thanos-querier-b957bd756-6cvr2" Jul 16 16:27:01.715817 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:01.715714 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2e58ee24-b3bc-4e3a-828c-6ac801ec7fd7-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-b957bd756-6cvr2\" (UID: \"2e58ee24-b3bc-4e3a-828c-6ac801ec7fd7\") " pod="openshift-monitoring/thanos-querier-b957bd756-6cvr2" Jul 16 16:27:01.715817 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:01.715748 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lmjps\" (UniqueName: \"kubernetes.io/projected/2e58ee24-b3bc-4e3a-828c-6ac801ec7fd7-kube-api-access-lmjps\") pod \"thanos-querier-b957bd756-6cvr2\" (UID: \"2e58ee24-b3bc-4e3a-828c-6ac801ec7fd7\") " pod="openshift-monitoring/thanos-querier-b957bd756-6cvr2" Jul 16 16:27:01.817275 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:01.816825 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/2e58ee24-b3bc-4e3a-828c-6ac801ec7fd7-secret-thanos-querier-tls\") pod \"thanos-querier-b957bd756-6cvr2\" (UID: \"2e58ee24-b3bc-4e3a-828c-6ac801ec7fd7\") " pod="openshift-monitoring/thanos-querier-b957bd756-6cvr2" Jul 16 16:27:01.817559 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:01.817539 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/2e58ee24-b3bc-4e3a-828c-6ac801ec7fd7-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-b957bd756-6cvr2\" (UID: \"2e58ee24-b3bc-4e3a-828c-6ac801ec7fd7\") " pod="openshift-monitoring/thanos-querier-b957bd756-6cvr2" Jul 16 16:27:01.817705 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:01.817686 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/2e58ee24-b3bc-4e3a-828c-6ac801ec7fd7-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-b957bd756-6cvr2\" (UID: \"2e58ee24-b3bc-4e3a-828c-6ac801ec7fd7\") " pod="openshift-monitoring/thanos-querier-b957bd756-6cvr2" Jul 16 16:27:01.817815 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:01.817800 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2e58ee24-b3bc-4e3a-828c-6ac801ec7fd7-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-b957bd756-6cvr2\" (UID: \"2e58ee24-b3bc-4e3a-828c-6ac801ec7fd7\") " pod="openshift-monitoring/thanos-querier-b957bd756-6cvr2" Jul 16 16:27:01.817959 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:01.817946 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2e58ee24-b3bc-4e3a-828c-6ac801ec7fd7-metrics-client-ca\") pod \"thanos-querier-b957bd756-6cvr2\" (UID: \"2e58ee24-b3bc-4e3a-828c-6ac801ec7fd7\") " pod="openshift-monitoring/thanos-querier-b957bd756-6cvr2" Jul 16 16:27:01.818761 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:01.818739 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2e58ee24-b3bc-4e3a-828c-6ac801ec7fd7-metrics-client-ca\") pod \"thanos-querier-b957bd756-6cvr2\" (UID: \"2e58ee24-b3bc-4e3a-828c-6ac801ec7fd7\") " pod="openshift-monitoring/thanos-querier-b957bd756-6cvr2" Jul 16 16:27:01.818936 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:01.818918 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2e58ee24-b3bc-4e3a-828c-6ac801ec7fd7-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-b957bd756-6cvr2\" (UID: \"2e58ee24-b3bc-4e3a-828c-6ac801ec7fd7\") " pod="openshift-monitoring/thanos-querier-b957bd756-6cvr2" Jul 16 16:27:01.819051 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:01.819038 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lmjps\" (UniqueName: \"kubernetes.io/projected/2e58ee24-b3bc-4e3a-828c-6ac801ec7fd7-kube-api-access-lmjps\") pod \"thanos-querier-b957bd756-6cvr2\" (UID: \"2e58ee24-b3bc-4e3a-828c-6ac801ec7fd7\") " pod="openshift-monitoring/thanos-querier-b957bd756-6cvr2" Jul 16 16:27:01.819231 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:01.819212 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/2e58ee24-b3bc-4e3a-828c-6ac801ec7fd7-secret-grpc-tls\") pod \"thanos-querier-b957bd756-6cvr2\" (UID: \"2e58ee24-b3bc-4e3a-828c-6ac801ec7fd7\") " pod="openshift-monitoring/thanos-querier-b957bd756-6cvr2" Jul 16 16:27:01.821031 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:01.820986 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/2e58ee24-b3bc-4e3a-828c-6ac801ec7fd7-secret-thanos-querier-tls\") pod \"thanos-querier-b957bd756-6cvr2\" (UID: \"2e58ee24-b3bc-4e3a-828c-6ac801ec7fd7\") " pod="openshift-monitoring/thanos-querier-b957bd756-6cvr2" Jul 16 16:27:01.822983 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:01.822931 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2e58ee24-b3bc-4e3a-828c-6ac801ec7fd7-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-b957bd756-6cvr2\" (UID: \"2e58ee24-b3bc-4e3a-828c-6ac801ec7fd7\") " pod="openshift-monitoring/thanos-querier-b957bd756-6cvr2" Jul 16 16:27:01.823969 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:01.823950 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/2e58ee24-b3bc-4e3a-828c-6ac801ec7fd7-secret-grpc-tls\") pod \"thanos-querier-b957bd756-6cvr2\" (UID: \"2e58ee24-b3bc-4e3a-828c-6ac801ec7fd7\") " pod="openshift-monitoring/thanos-querier-b957bd756-6cvr2" Jul 16 16:27:01.824260 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:01.824220 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/2e58ee24-b3bc-4e3a-828c-6ac801ec7fd7-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-b957bd756-6cvr2\" (UID: \"2e58ee24-b3bc-4e3a-828c-6ac801ec7fd7\") " pod="openshift-monitoring/thanos-querier-b957bd756-6cvr2" Jul 16 16:27:01.824350 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:01.824254 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/2e58ee24-b3bc-4e3a-828c-6ac801ec7fd7-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-b957bd756-6cvr2\" (UID: \"2e58ee24-b3bc-4e3a-828c-6ac801ec7fd7\") " pod="openshift-monitoring/thanos-querier-b957bd756-6cvr2" Jul 16 16:27:01.824507 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:01.824488 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2e58ee24-b3bc-4e3a-828c-6ac801ec7fd7-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-b957bd756-6cvr2\" (UID: \"2e58ee24-b3bc-4e3a-828c-6ac801ec7fd7\") " pod="openshift-monitoring/thanos-querier-b957bd756-6cvr2" Jul 16 16:27:01.838965 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:01.838825 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lmjps\" (UniqueName: \"kubernetes.io/projected/2e58ee24-b3bc-4e3a-828c-6ac801ec7fd7-kube-api-access-lmjps\") pod \"thanos-querier-b957bd756-6cvr2\" (UID: \"2e58ee24-b3bc-4e3a-828c-6ac801ec7fd7\") " pod="openshift-monitoring/thanos-querier-b957bd756-6cvr2" Jul 16 16:27:01.851457 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:01.851263 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 16 16:27:01.854947 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:27:01.854915 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c88b029_8a81_402c_90e4_77eff554183c.slice/crio-802168b911b1ed3cb6bd07aa6e649292914a88b4eeca76a7d9ae375ccfa55731 WatchSource:0}: Error finding container 802168b911b1ed3cb6bd07aa6e649292914a88b4eeca76a7d9ae375ccfa55731: Status 404 returned error can't find the container with id 802168b911b1ed3cb6bd07aa6e649292914a88b4eeca76a7d9ae375ccfa55731 Jul 16 16:27:01.906468 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:01.906350 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-b957bd756-6cvr2" Jul 16 16:27:02.054127 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:02.054083 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-b957bd756-6cvr2"] Jul 16 16:27:02.056833 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:27:02.056806 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e58ee24_b3bc_4e3a_828c_6ac801ec7fd7.slice/crio-41ff9004843fbd4d489638eb7e496565a7116c231d52c3d1630fc9b5c65c8ad6 WatchSource:0}: Error finding container 41ff9004843fbd4d489638eb7e496565a7116c231d52c3d1630fc9b5c65c8ad6: Status 404 returned error can't find the container with id 41ff9004843fbd4d489638eb7e496565a7116c231d52c3d1630fc9b5c65c8ad6 Jul 16 16:27:02.437474 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:02.437431 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-xfk8h" event={"ID":"3253b8ec-f532-448b-9a7e-6fcb71b0d3fd","Type":"ContainerStarted","Data":"a69625d0cb57cf328e588a79962b8349f168f92efc72814c679581c836203330"} Jul 16 16:27:02.439858 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:02.439823 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-785778ff74-vfxfr" event={"ID":"10bc3cb6-aa0e-4d0f-a5a9-75f494f7fffb","Type":"ContainerStarted","Data":"63998d6274bb80c5dbe737c42ee057cf7124553c52dbedfa5a791eee5cf57147"} Jul 16 16:27:02.440016 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:02.439888 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-785778ff74-vfxfr" event={"ID":"10bc3cb6-aa0e-4d0f-a5a9-75f494f7fffb","Type":"ContainerStarted","Data":"1cf1738987aa66eb1b38d8edef938cf41291777a2177a784e90a9f8a4d5b5dcc"} Jul 16 16:27:02.440016 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:02.439916 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-785778ff74-vfxfr" event={"ID":"10bc3cb6-aa0e-4d0f-a5a9-75f494f7fffb","Type":"ContainerStarted","Data":"12fac2c86c9cd47435bb534f4f9492c24029678cc906699c1d9a849b3a3895f5"} Jul 16 16:27:02.441227 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:02.441197 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-b957bd756-6cvr2" event={"ID":"2e58ee24-b3bc-4e3a-828c-6ac801ec7fd7","Type":"ContainerStarted","Data":"41ff9004843fbd4d489638eb7e496565a7116c231d52c3d1630fc9b5c65c8ad6"} Jul 16 16:27:02.442661 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:02.442632 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8c88b029-8a81-402c-90e4-77eff554183c","Type":"ContainerStarted","Data":"802168b911b1ed3cb6bd07aa6e649292914a88b4eeca76a7d9ae375ccfa55731"} Jul 16 16:27:02.447207 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:02.447180 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-76dcf44fd5-tx9p2" Jul 16 16:27:02.485730 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:02.485378 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-xfk8h" podStartSLOduration=1.942261479 podStartE2EDuration="3.485325741s" totalImagesPullingTime="1.543064262s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 16:26:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 16:27:02.460231577 +0000 UTC m=+80.964289429" watchObservedRunningTime="2026-07-16 16:27:02.485325741 +0000 UTC m=+80.989383593" Jul 16 16:27:02.486178 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:02.486145 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-785778ff74-vfxfr" podStartSLOduration=1.848415213 podStartE2EDuration="3.48613477s" totalImagesPullingTime="1.637719557s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 16:26:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 16:27:02.485503643 +0000 UTC m=+80.989561497" watchObservedRunningTime="2026-07-16 16:27:02.48613477 +0000 UTC m=+80.990192622" Jul 16 16:27:02.561140 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:02.561102 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-68b7b95d9-dnh2n"] Jul 16 16:27:03.447636 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:03.447490 2577 generic.go:356] "Generic (PLEG): container finished" podID="42c6e795-fc08-4367-8e7f-9c65cb84c814" containerID="0f3396181cdbac500bfea77590751c31826e0bb433df5384046d1ee97674ed11" exitCode=0 Jul 16 16:27:03.447636 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:03.447600 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-hvwxd" event={"ID":"42c6e795-fc08-4367-8e7f-9c65cb84c814","Type":"ContainerDied","Data":"0f3396181cdbac500bfea77590751c31826e0bb433df5384046d1ee97674ed11"} Jul 16 16:27:03.796005 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:03.795898 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-6fd5bb7967-8t82m"] Jul 16 16:27:03.798474 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:03.798452 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-6fd5bb7967-8t82m" Jul 16 16:27:03.801630 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:03.801601 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Jul 16 16:27:03.801885 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:03.801867 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Jul 16 16:27:03.801964 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:03.801873 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-8cq7659p9uq3t\"" Jul 16 16:27:03.801964 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:03.801874 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-9z4ck\"" Jul 16 16:27:03.802619 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:03.802602 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Jul 16 16:27:03.802709 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:03.802678 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Jul 16 16:27:03.816838 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:03.816810 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-6fd5bb7967-8t82m"] Jul 16 16:27:03.940227 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:03.940187 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/3ed7defb-10df-42c6-af68-8a2145ad3539-audit-log\") pod \"metrics-server-6fd5bb7967-8t82m\" (UID: \"3ed7defb-10df-42c6-af68-8a2145ad3539\") " pod="openshift-monitoring/metrics-server-6fd5bb7967-8t82m" Jul 16 16:27:03.940455 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:03.940271 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/3ed7defb-10df-42c6-af68-8a2145ad3539-secret-metrics-server-tls\") pod \"metrics-server-6fd5bb7967-8t82m\" (UID: \"3ed7defb-10df-42c6-af68-8a2145ad3539\") " pod="openshift-monitoring/metrics-server-6fd5bb7967-8t82m" Jul 16 16:27:03.940455 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:03.940306 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/3ed7defb-10df-42c6-af68-8a2145ad3539-client-ca-bundle\") pod \"metrics-server-6fd5bb7967-8t82m\" (UID: \"3ed7defb-10df-42c6-af68-8a2145ad3539\") " pod="openshift-monitoring/metrics-server-6fd5bb7967-8t82m" Jul 16 16:27:03.940455 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:03.940335 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3ed7defb-10df-42c6-af68-8a2145ad3539-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6fd5bb7967-8t82m\" (UID: \"3ed7defb-10df-42c6-af68-8a2145ad3539\") " pod="openshift-monitoring/metrics-server-6fd5bb7967-8t82m" Jul 16 16:27:03.940455 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:03.940367 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/3ed7defb-10df-42c6-af68-8a2145ad3539-secret-metrics-server-client-certs\") pod \"metrics-server-6fd5bb7967-8t82m\" (UID: \"3ed7defb-10df-42c6-af68-8a2145ad3539\") " pod="openshift-monitoring/metrics-server-6fd5bb7967-8t82m" Jul 16 16:27:03.940455 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:03.940437 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/3ed7defb-10df-42c6-af68-8a2145ad3539-metrics-server-audit-profiles\") pod \"metrics-server-6fd5bb7967-8t82m\" (UID: \"3ed7defb-10df-42c6-af68-8a2145ad3539\") " pod="openshift-monitoring/metrics-server-6fd5bb7967-8t82m" Jul 16 16:27:03.940741 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:03.940484 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w4x7h\" (UniqueName: \"kubernetes.io/projected/3ed7defb-10df-42c6-af68-8a2145ad3539-kube-api-access-w4x7h\") pod \"metrics-server-6fd5bb7967-8t82m\" (UID: \"3ed7defb-10df-42c6-af68-8a2145ad3539\") " pod="openshift-monitoring/metrics-server-6fd5bb7967-8t82m" Jul 16 16:27:04.041294 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:04.041256 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/3ed7defb-10df-42c6-af68-8a2145ad3539-secret-metrics-server-client-certs\") pod \"metrics-server-6fd5bb7967-8t82m\" (UID: \"3ed7defb-10df-42c6-af68-8a2145ad3539\") " pod="openshift-monitoring/metrics-server-6fd5bb7967-8t82m" Jul 16 16:27:04.041294 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:04.041295 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/3ed7defb-10df-42c6-af68-8a2145ad3539-metrics-server-audit-profiles\") pod \"metrics-server-6fd5bb7967-8t82m\" (UID: \"3ed7defb-10df-42c6-af68-8a2145ad3539\") " pod="openshift-monitoring/metrics-server-6fd5bb7967-8t82m" Jul 16 16:27:04.041518 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:04.041495 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w4x7h\" (UniqueName: \"kubernetes.io/projected/3ed7defb-10df-42c6-af68-8a2145ad3539-kube-api-access-w4x7h\") pod \"metrics-server-6fd5bb7967-8t82m\" (UID: \"3ed7defb-10df-42c6-af68-8a2145ad3539\") " pod="openshift-monitoring/metrics-server-6fd5bb7967-8t82m" Jul 16 16:27:04.041572 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:04.041549 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/3ed7defb-10df-42c6-af68-8a2145ad3539-audit-log\") pod \"metrics-server-6fd5bb7967-8t82m\" (UID: \"3ed7defb-10df-42c6-af68-8a2145ad3539\") " pod="openshift-monitoring/metrics-server-6fd5bb7967-8t82m" Jul 16 16:27:04.041643 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:04.041628 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/3ed7defb-10df-42c6-af68-8a2145ad3539-secret-metrics-server-tls\") pod \"metrics-server-6fd5bb7967-8t82m\" (UID: \"3ed7defb-10df-42c6-af68-8a2145ad3539\") " pod="openshift-monitoring/metrics-server-6fd5bb7967-8t82m" Jul 16 16:27:04.041703 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:04.041666 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/3ed7defb-10df-42c6-af68-8a2145ad3539-client-ca-bundle\") pod \"metrics-server-6fd5bb7967-8t82m\" (UID: \"3ed7defb-10df-42c6-af68-8a2145ad3539\") " pod="openshift-monitoring/metrics-server-6fd5bb7967-8t82m" Jul 16 16:27:04.041761 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:04.041705 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3ed7defb-10df-42c6-af68-8a2145ad3539-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6fd5bb7967-8t82m\" (UID: \"3ed7defb-10df-42c6-af68-8a2145ad3539\") " pod="openshift-monitoring/metrics-server-6fd5bb7967-8t82m" Jul 16 16:27:04.042079 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:04.042024 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/3ed7defb-10df-42c6-af68-8a2145ad3539-audit-log\") pod \"metrics-server-6fd5bb7967-8t82m\" (UID: \"3ed7defb-10df-42c6-af68-8a2145ad3539\") " pod="openshift-monitoring/metrics-server-6fd5bb7967-8t82m" Jul 16 16:27:04.042379 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:04.042333 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3ed7defb-10df-42c6-af68-8a2145ad3539-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6fd5bb7967-8t82m\" (UID: \"3ed7defb-10df-42c6-af68-8a2145ad3539\") " pod="openshift-monitoring/metrics-server-6fd5bb7967-8t82m" Jul 16 16:27:04.042494 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:04.042423 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/3ed7defb-10df-42c6-af68-8a2145ad3539-metrics-server-audit-profiles\") pod \"metrics-server-6fd5bb7967-8t82m\" (UID: \"3ed7defb-10df-42c6-af68-8a2145ad3539\") " pod="openshift-monitoring/metrics-server-6fd5bb7967-8t82m" Jul 16 16:27:04.044300 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:04.044278 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/3ed7defb-10df-42c6-af68-8a2145ad3539-secret-metrics-server-client-certs\") pod \"metrics-server-6fd5bb7967-8t82m\" (UID: \"3ed7defb-10df-42c6-af68-8a2145ad3539\") " pod="openshift-monitoring/metrics-server-6fd5bb7967-8t82m" Jul 16 16:27:04.044693 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:04.044666 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/3ed7defb-10df-42c6-af68-8a2145ad3539-secret-metrics-server-tls\") pod \"metrics-server-6fd5bb7967-8t82m\" (UID: \"3ed7defb-10df-42c6-af68-8a2145ad3539\") " pod="openshift-monitoring/metrics-server-6fd5bb7967-8t82m" Jul 16 16:27:04.044777 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:04.044740 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/3ed7defb-10df-42c6-af68-8a2145ad3539-client-ca-bundle\") pod \"metrics-server-6fd5bb7967-8t82m\" (UID: \"3ed7defb-10df-42c6-af68-8a2145ad3539\") " pod="openshift-monitoring/metrics-server-6fd5bb7967-8t82m" Jul 16 16:27:04.051378 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:04.051305 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w4x7h\" (UniqueName: \"kubernetes.io/projected/3ed7defb-10df-42c6-af68-8a2145ad3539-kube-api-access-w4x7h\") pod \"metrics-server-6fd5bb7967-8t82m\" (UID: \"3ed7defb-10df-42c6-af68-8a2145ad3539\") " pod="openshift-monitoring/metrics-server-6fd5bb7967-8t82m" Jul 16 16:27:04.108619 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:04.108579 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-6fd5bb7967-8t82m" Jul 16 16:27:04.275584 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:04.275531 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-5f68df7f58-rrhsm"] Jul 16 16:27:04.278078 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:04.278056 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-rrhsm" Jul 16 16:27:04.281113 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:04.281083 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Jul 16 16:27:04.281273 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:04.281089 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-mnp8k\"" Jul 16 16:27:04.312257 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:04.312175 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5f68df7f58-rrhsm"] Jul 16 16:27:04.446249 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:04.446213 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/342fe0d4-70b8-43b9-b052-93a1068deb59-monitoring-plugin-cert\") pod \"monitoring-plugin-5f68df7f58-rrhsm\" (UID: \"342fe0d4-70b8-43b9-b052-93a1068deb59\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-rrhsm" Jul 16 16:27:04.446249 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:04.446251 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vkl85\" (UniqueName: \"kubernetes.io/projected/342fe0d4-70b8-43b9-b052-93a1068deb59-kube-api-access-vkl85\") pod \"monitoring-plugin-5f68df7f58-rrhsm\" (UID: \"342fe0d4-70b8-43b9-b052-93a1068deb59\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-rrhsm" Jul 16 16:27:04.454755 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:04.454697 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-hvwxd" event={"ID":"42c6e795-fc08-4367-8e7f-9c65cb84c814","Type":"ContainerStarted","Data":"c4d6130e4ce0d319666bfce37b8ade5e955cd07b71dc966a64393d8018a8f664"} Jul 16 16:27:04.455195 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:04.454774 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-hvwxd" event={"ID":"42c6e795-fc08-4367-8e7f-9c65cb84c814","Type":"ContainerStarted","Data":"fa0c1461763f93d1ace9c7ac41c514e8f3348dd6ef728bfaa473ab125d9de824"} Jul 16 16:27:04.475845 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:04.475785 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-hvwxd" podStartSLOduration=3.236572069 podStartE2EDuration="5.475765759s" totalImagesPullingTime="2.169914245s" totalInitContainerRuntime="69.279445ms" isStatefulPod=true podCreationTimestamp="2026-07-16 16:26:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 16:27:04.47516207 +0000 UTC m=+82.979219922" watchObservedRunningTime="2026-07-16 16:27:04.475765759 +0000 UTC m=+82.979823613" Jul 16 16:27:04.547209 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:04.547167 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/342fe0d4-70b8-43b9-b052-93a1068deb59-monitoring-plugin-cert\") pod \"monitoring-plugin-5f68df7f58-rrhsm\" (UID: \"342fe0d4-70b8-43b9-b052-93a1068deb59\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-rrhsm" Jul 16 16:27:04.547450 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:04.547218 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vkl85\" (UniqueName: \"kubernetes.io/projected/342fe0d4-70b8-43b9-b052-93a1068deb59-kube-api-access-vkl85\") pod \"monitoring-plugin-5f68df7f58-rrhsm\" (UID: \"342fe0d4-70b8-43b9-b052-93a1068deb59\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-rrhsm" Jul 16 16:27:04.550287 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:04.550255 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/342fe0d4-70b8-43b9-b052-93a1068deb59-monitoring-plugin-cert\") pod \"monitoring-plugin-5f68df7f58-rrhsm\" (UID: \"342fe0d4-70b8-43b9-b052-93a1068deb59\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-rrhsm" Jul 16 16:27:04.556352 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:04.556322 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vkl85\" (UniqueName: \"kubernetes.io/projected/342fe0d4-70b8-43b9-b052-93a1068deb59-kube-api-access-vkl85\") pod \"monitoring-plugin-5f68df7f58-rrhsm\" (UID: \"342fe0d4-70b8-43b9-b052-93a1068deb59\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-rrhsm" Jul 16 16:27:04.590178 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:04.590144 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-rrhsm" Jul 16 16:27:04.739065 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:04.738962 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-568889b88c-4488z"] Jul 16 16:27:04.742061 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:04.742033 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-568889b88c-4488z" Jul 16 16:27:04.748619 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:04.746666 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-c8rrv\"" Jul 16 16:27:04.748619 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:04.747062 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Jul 16 16:27:04.748619 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:04.747267 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Jul 16 16:27:04.748619 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:04.747474 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Jul 16 16:27:04.748619 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:04.747796 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Jul 16 16:27:04.748619 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:04.748437 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Jul 16 16:27:04.755560 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:04.753430 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\"" Jul 16 16:27:04.764129 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:04.764026 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-568889b88c-4488z"] Jul 16 16:27:04.849315 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:04.849213 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ce591e34-5de4-45d1-ab53-e9963ffae504-serving-certs-ca-bundle\") pod \"telemeter-client-568889b88c-4488z\" (UID: \"ce591e34-5de4-45d1-ab53-e9963ffae504\") " pod="openshift-monitoring/telemeter-client-568889b88c-4488z" Jul 16 16:27:04.849315 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:04.849260 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/ce591e34-5de4-45d1-ab53-e9963ffae504-federate-client-tls\") pod \"telemeter-client-568889b88c-4488z\" (UID: \"ce591e34-5de4-45d1-ab53-e9963ffae504\") " pod="openshift-monitoring/telemeter-client-568889b88c-4488z" Jul 16 16:27:04.849599 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:04.849364 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ce591e34-5de4-45d1-ab53-e9963ffae504-metrics-client-ca\") pod \"telemeter-client-568889b88c-4488z\" (UID: \"ce591e34-5de4-45d1-ab53-e9963ffae504\") " pod="openshift-monitoring/telemeter-client-568889b88c-4488z" Jul 16 16:27:04.849599 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:04.849441 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/ce591e34-5de4-45d1-ab53-e9963ffae504-telemeter-client-tls\") pod \"telemeter-client-568889b88c-4488z\" (UID: \"ce591e34-5de4-45d1-ab53-e9963ffae504\") " pod="openshift-monitoring/telemeter-client-568889b88c-4488z" Jul 16 16:27:04.849599 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:04.849461 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ce591e34-5de4-45d1-ab53-e9963ffae504-telemeter-trusted-ca-bundle\") pod \"telemeter-client-568889b88c-4488z\" (UID: \"ce591e34-5de4-45d1-ab53-e9963ffae504\") " pod="openshift-monitoring/telemeter-client-568889b88c-4488z" Jul 16 16:27:04.849599 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:04.849545 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5w9zs\" (UniqueName: \"kubernetes.io/projected/ce591e34-5de4-45d1-ab53-e9963ffae504-kube-api-access-5w9zs\") pod \"telemeter-client-568889b88c-4488z\" (UID: \"ce591e34-5de4-45d1-ab53-e9963ffae504\") " pod="openshift-monitoring/telemeter-client-568889b88c-4488z" Jul 16 16:27:04.849599 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:04.849584 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/ce591e34-5de4-45d1-ab53-e9963ffae504-secret-telemeter-client\") pod \"telemeter-client-568889b88c-4488z\" (UID: \"ce591e34-5de4-45d1-ab53-e9963ffae504\") " pod="openshift-monitoring/telemeter-client-568889b88c-4488z" Jul 16 16:27:04.849599 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:04.849603 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ce591e34-5de4-45d1-ab53-e9963ffae504-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-568889b88c-4488z\" (UID: \"ce591e34-5de4-45d1-ab53-e9963ffae504\") " pod="openshift-monitoring/telemeter-client-568889b88c-4488z" Jul 16 16:27:04.852840 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:04.852814 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5f68df7f58-rrhsm"] Jul 16 16:27:04.860300 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:04.860277 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-6fd5bb7967-8t82m"] Jul 16 16:27:04.862901 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:27:04.862872 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ed7defb_10df_42c6_af68_8a2145ad3539.slice/crio-3275527f6fe2a377f5d2d8e9abb61e11b119e810a24ad301df84124ce9f1fa81 WatchSource:0}: Error finding container 3275527f6fe2a377f5d2d8e9abb61e11b119e810a24ad301df84124ce9f1fa81: Status 404 returned error can't find the container with id 3275527f6fe2a377f5d2d8e9abb61e11b119e810a24ad301df84124ce9f1fa81 Jul 16 16:27:04.863139 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:27:04.863120 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod342fe0d4_70b8_43b9_b052_93a1068deb59.slice/crio-7f465a5e114c5ff39b5b7141474c9b89aab70dd3c3bc357b06a50ce9311a598c WatchSource:0}: Error finding container 7f465a5e114c5ff39b5b7141474c9b89aab70dd3c3bc357b06a50ce9311a598c: Status 404 returned error can't find the container with id 7f465a5e114c5ff39b5b7141474c9b89aab70dd3c3bc357b06a50ce9311a598c Jul 16 16:27:04.950131 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:04.950062 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5w9zs\" (UniqueName: \"kubernetes.io/projected/ce591e34-5de4-45d1-ab53-e9963ffae504-kube-api-access-5w9zs\") pod \"telemeter-client-568889b88c-4488z\" (UID: \"ce591e34-5de4-45d1-ab53-e9963ffae504\") " pod="openshift-monitoring/telemeter-client-568889b88c-4488z" Jul 16 16:27:04.950131 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:04.950095 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/ce591e34-5de4-45d1-ab53-e9963ffae504-secret-telemeter-client\") pod \"telemeter-client-568889b88c-4488z\" (UID: \"ce591e34-5de4-45d1-ab53-e9963ffae504\") " pod="openshift-monitoring/telemeter-client-568889b88c-4488z" Jul 16 16:27:04.950131 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:04.950111 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ce591e34-5de4-45d1-ab53-e9963ffae504-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-568889b88c-4488z\" (UID: \"ce591e34-5de4-45d1-ab53-e9963ffae504\") " pod="openshift-monitoring/telemeter-client-568889b88c-4488z" Jul 16 16:27:04.950293 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:04.950160 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ce591e34-5de4-45d1-ab53-e9963ffae504-serving-certs-ca-bundle\") pod \"telemeter-client-568889b88c-4488z\" (UID: \"ce591e34-5de4-45d1-ab53-e9963ffae504\") " pod="openshift-monitoring/telemeter-client-568889b88c-4488z" Jul 16 16:27:04.950293 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:04.950192 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/ce591e34-5de4-45d1-ab53-e9963ffae504-federate-client-tls\") pod \"telemeter-client-568889b88c-4488z\" (UID: \"ce591e34-5de4-45d1-ab53-e9963ffae504\") " pod="openshift-monitoring/telemeter-client-568889b88c-4488z" Jul 16 16:27:04.950293 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:04.950235 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ce591e34-5de4-45d1-ab53-e9963ffae504-metrics-client-ca\") pod \"telemeter-client-568889b88c-4488z\" (UID: \"ce591e34-5de4-45d1-ab53-e9963ffae504\") " pod="openshift-monitoring/telemeter-client-568889b88c-4488z" Jul 16 16:27:04.950293 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:04.950288 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/ce591e34-5de4-45d1-ab53-e9963ffae504-telemeter-client-tls\") pod \"telemeter-client-568889b88c-4488z\" (UID: \"ce591e34-5de4-45d1-ab53-e9963ffae504\") " pod="openshift-monitoring/telemeter-client-568889b88c-4488z" Jul 16 16:27:04.950558 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:04.950314 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ce591e34-5de4-45d1-ab53-e9963ffae504-telemeter-trusted-ca-bundle\") pod \"telemeter-client-568889b88c-4488z\" (UID: \"ce591e34-5de4-45d1-ab53-e9963ffae504\") " pod="openshift-monitoring/telemeter-client-568889b88c-4488z" Jul 16 16:27:04.951360 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:04.951020 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ce591e34-5de4-45d1-ab53-e9963ffae504-serving-certs-ca-bundle\") pod \"telemeter-client-568889b88c-4488z\" (UID: \"ce591e34-5de4-45d1-ab53-e9963ffae504\") " pod="openshift-monitoring/telemeter-client-568889b88c-4488z" Jul 16 16:27:04.951360 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:04.951062 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ce591e34-5de4-45d1-ab53-e9963ffae504-metrics-client-ca\") pod \"telemeter-client-568889b88c-4488z\" (UID: \"ce591e34-5de4-45d1-ab53-e9963ffae504\") " pod="openshift-monitoring/telemeter-client-568889b88c-4488z" Jul 16 16:27:04.951360 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:04.951317 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ce591e34-5de4-45d1-ab53-e9963ffae504-telemeter-trusted-ca-bundle\") pod \"telemeter-client-568889b88c-4488z\" (UID: \"ce591e34-5de4-45d1-ab53-e9963ffae504\") " pod="openshift-monitoring/telemeter-client-568889b88c-4488z" Jul 16 16:27:04.953265 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:04.953166 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/ce591e34-5de4-45d1-ab53-e9963ffae504-secret-telemeter-client\") pod \"telemeter-client-568889b88c-4488z\" (UID: \"ce591e34-5de4-45d1-ab53-e9963ffae504\") " pod="openshift-monitoring/telemeter-client-568889b88c-4488z" Jul 16 16:27:04.953265 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:04.953188 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/ce591e34-5de4-45d1-ab53-e9963ffae504-telemeter-client-tls\") pod \"telemeter-client-568889b88c-4488z\" (UID: \"ce591e34-5de4-45d1-ab53-e9963ffae504\") " pod="openshift-monitoring/telemeter-client-568889b88c-4488z" Jul 16 16:27:04.953265 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:04.953251 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/ce591e34-5de4-45d1-ab53-e9963ffae504-federate-client-tls\") pod \"telemeter-client-568889b88c-4488z\" (UID: \"ce591e34-5de4-45d1-ab53-e9963ffae504\") " pod="openshift-monitoring/telemeter-client-568889b88c-4488z" Jul 16 16:27:04.953513 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:04.953496 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ce591e34-5de4-45d1-ab53-e9963ffae504-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-568889b88c-4488z\" (UID: \"ce591e34-5de4-45d1-ab53-e9963ffae504\") " pod="openshift-monitoring/telemeter-client-568889b88c-4488z" Jul 16 16:27:04.960701 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:04.960676 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5w9zs\" (UniqueName: \"kubernetes.io/projected/ce591e34-5de4-45d1-ab53-e9963ffae504-kube-api-access-5w9zs\") pod \"telemeter-client-568889b88c-4488z\" (UID: \"ce591e34-5de4-45d1-ab53-e9963ffae504\") " pod="openshift-monitoring/telemeter-client-568889b88c-4488z" Jul 16 16:27:05.067815 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:05.067779 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-568889b88c-4488z" Jul 16 16:27:05.211229 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:05.211199 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-568889b88c-4488z"] Jul 16 16:27:05.215770 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:27:05.215742 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce591e34_5de4_45d1_ab53_e9963ffae504.slice/crio-67e75845fa193019ad6f066f5d802f300aec2116bf027b9a378f99b6ce11ffb4 WatchSource:0}: Error finding container 67e75845fa193019ad6f066f5d802f300aec2116bf027b9a378f99b6ce11ffb4: Status 404 returned error can't find the container with id 67e75845fa193019ad6f066f5d802f300aec2116bf027b9a378f99b6ce11ffb4 Jul 16 16:27:05.463040 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:05.462944 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-568889b88c-4488z" event={"ID":"ce591e34-5de4-45d1-ab53-e9963ffae504","Type":"ContainerStarted","Data":"67e75845fa193019ad6f066f5d802f300aec2116bf027b9a378f99b6ce11ffb4"} Jul 16 16:27:05.464355 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:05.464324 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-6fd5bb7967-8t82m" event={"ID":"3ed7defb-10df-42c6-af68-8a2145ad3539","Type":"ContainerStarted","Data":"3275527f6fe2a377f5d2d8e9abb61e11b119e810a24ad301df84124ce9f1fa81"} Jul 16 16:27:05.465811 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:05.465777 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-rrhsm" event={"ID":"342fe0d4-70b8-43b9-b052-93a1068deb59","Type":"ContainerStarted","Data":"7f465a5e114c5ff39b5b7141474c9b89aab70dd3c3bc357b06a50ce9311a598c"} Jul 16 16:27:05.468191 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:05.468160 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-b957bd756-6cvr2" event={"ID":"2e58ee24-b3bc-4e3a-828c-6ac801ec7fd7","Type":"ContainerStarted","Data":"9b8b61df6e60e732e01ed0fe614603eb3bdf2031d79e5ce10068e079b8d7432e"} Jul 16 16:27:05.468313 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:05.468233 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-b957bd756-6cvr2" event={"ID":"2e58ee24-b3bc-4e3a-828c-6ac801ec7fd7","Type":"ContainerStarted","Data":"4995cf6a39cbc626633f94bfa7782c9eb6581e915d0916306fc1ec6638be74bd"} Jul 16 16:27:05.468313 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:05.468271 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-b957bd756-6cvr2" event={"ID":"2e58ee24-b3bc-4e3a-828c-6ac801ec7fd7","Type":"ContainerStarted","Data":"476d10a8b177c11dde2c4e1906fa5daf10a656d54c49fcbab0a8d01735546379"} Jul 16 16:27:05.469901 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:05.469876 2577 generic.go:356] "Generic (PLEG): container finished" podID="8c88b029-8a81-402c-90e4-77eff554183c" containerID="e5dbbb0032e1119d65117c6803f62f48e3c70241fb73f61afe14d1da40446cfe" exitCode=0 Jul 16 16:27:05.470008 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:05.469960 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8c88b029-8a81-402c-90e4-77eff554183c","Type":"ContainerDied","Data":"e5dbbb0032e1119d65117c6803f62f48e3c70241fb73f61afe14d1da40446cfe"} Jul 16 16:27:05.573433 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:05.573379 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-b8c74fcc8-njh4j"] Jul 16 16:27:05.576708 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:05.576679 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-b8c74fcc8-njh4j" Jul 16 16:27:05.587853 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:05.587824 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-b8c74fcc8-njh4j"] Jul 16 16:27:05.657633 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:05.657569 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fc506c26-ecf4-488c-a9aa-a7b52e2d04e9-console-serving-cert\") pod \"console-b8c74fcc8-njh4j\" (UID: \"fc506c26-ecf4-488c-a9aa-a7b52e2d04e9\") " pod="openshift-console/console-b8c74fcc8-njh4j" Jul 16 16:27:05.657804 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:05.657662 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fc506c26-ecf4-488c-a9aa-a7b52e2d04e9-console-oauth-config\") pod \"console-b8c74fcc8-njh4j\" (UID: \"fc506c26-ecf4-488c-a9aa-a7b52e2d04e9\") " pod="openshift-console/console-b8c74fcc8-njh4j" Jul 16 16:27:05.657804 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:05.657690 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jw8dr\" (UniqueName: \"kubernetes.io/projected/fc506c26-ecf4-488c-a9aa-a7b52e2d04e9-kube-api-access-jw8dr\") pod \"console-b8c74fcc8-njh4j\" (UID: \"fc506c26-ecf4-488c-a9aa-a7b52e2d04e9\") " pod="openshift-console/console-b8c74fcc8-njh4j" Jul 16 16:27:05.657804 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:05.657767 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fc506c26-ecf4-488c-a9aa-a7b52e2d04e9-oauth-serving-cert\") pod \"console-b8c74fcc8-njh4j\" (UID: \"fc506c26-ecf4-488c-a9aa-a7b52e2d04e9\") " pod="openshift-console/console-b8c74fcc8-njh4j" Jul 16 16:27:05.657984 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:05.657807 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fc506c26-ecf4-488c-a9aa-a7b52e2d04e9-trusted-ca-bundle\") pod \"console-b8c74fcc8-njh4j\" (UID: \"fc506c26-ecf4-488c-a9aa-a7b52e2d04e9\") " pod="openshift-console/console-b8c74fcc8-njh4j" Jul 16 16:27:05.657984 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:05.657877 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fc506c26-ecf4-488c-a9aa-a7b52e2d04e9-console-config\") pod \"console-b8c74fcc8-njh4j\" (UID: \"fc506c26-ecf4-488c-a9aa-a7b52e2d04e9\") " pod="openshift-console/console-b8c74fcc8-njh4j" Jul 16 16:27:05.657984 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:05.657918 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fc506c26-ecf4-488c-a9aa-a7b52e2d04e9-service-ca\") pod \"console-b8c74fcc8-njh4j\" (UID: \"fc506c26-ecf4-488c-a9aa-a7b52e2d04e9\") " pod="openshift-console/console-b8c74fcc8-njh4j" Jul 16 16:27:05.759308 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:05.759195 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fc506c26-ecf4-488c-a9aa-a7b52e2d04e9-console-serving-cert\") pod \"console-b8c74fcc8-njh4j\" (UID: \"fc506c26-ecf4-488c-a9aa-a7b52e2d04e9\") " pod="openshift-console/console-b8c74fcc8-njh4j" Jul 16 16:27:05.759308 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:05.759258 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fc506c26-ecf4-488c-a9aa-a7b52e2d04e9-console-oauth-config\") pod \"console-b8c74fcc8-njh4j\" (UID: \"fc506c26-ecf4-488c-a9aa-a7b52e2d04e9\") " pod="openshift-console/console-b8c74fcc8-njh4j" Jul 16 16:27:05.759308 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:05.759281 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jw8dr\" (UniqueName: \"kubernetes.io/projected/fc506c26-ecf4-488c-a9aa-a7b52e2d04e9-kube-api-access-jw8dr\") pod \"console-b8c74fcc8-njh4j\" (UID: \"fc506c26-ecf4-488c-a9aa-a7b52e2d04e9\") " pod="openshift-console/console-b8c74fcc8-njh4j" Jul 16 16:27:05.759728 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:05.759327 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fc506c26-ecf4-488c-a9aa-a7b52e2d04e9-oauth-serving-cert\") pod \"console-b8c74fcc8-njh4j\" (UID: \"fc506c26-ecf4-488c-a9aa-a7b52e2d04e9\") " pod="openshift-console/console-b8c74fcc8-njh4j" Jul 16 16:27:05.759728 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:05.759355 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fc506c26-ecf4-488c-a9aa-a7b52e2d04e9-trusted-ca-bundle\") pod \"console-b8c74fcc8-njh4j\" (UID: \"fc506c26-ecf4-488c-a9aa-a7b52e2d04e9\") " pod="openshift-console/console-b8c74fcc8-njh4j" Jul 16 16:27:05.759728 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:05.759421 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fc506c26-ecf4-488c-a9aa-a7b52e2d04e9-console-config\") pod \"console-b8c74fcc8-njh4j\" (UID: \"fc506c26-ecf4-488c-a9aa-a7b52e2d04e9\") " pod="openshift-console/console-b8c74fcc8-njh4j" Jul 16 16:27:05.759728 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:05.759456 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fc506c26-ecf4-488c-a9aa-a7b52e2d04e9-service-ca\") pod \"console-b8c74fcc8-njh4j\" (UID: \"fc506c26-ecf4-488c-a9aa-a7b52e2d04e9\") " pod="openshift-console/console-b8c74fcc8-njh4j" Jul 16 16:27:05.760318 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:05.760289 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fc506c26-ecf4-488c-a9aa-a7b52e2d04e9-service-ca\") pod \"console-b8c74fcc8-njh4j\" (UID: \"fc506c26-ecf4-488c-a9aa-a7b52e2d04e9\") " pod="openshift-console/console-b8c74fcc8-njh4j" Jul 16 16:27:05.760949 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:05.760923 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fc506c26-ecf4-488c-a9aa-a7b52e2d04e9-console-config\") pod \"console-b8c74fcc8-njh4j\" (UID: \"fc506c26-ecf4-488c-a9aa-a7b52e2d04e9\") " pod="openshift-console/console-b8c74fcc8-njh4j" Jul 16 16:27:05.761052 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:05.760932 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fc506c26-ecf4-488c-a9aa-a7b52e2d04e9-oauth-serving-cert\") pod \"console-b8c74fcc8-njh4j\" (UID: \"fc506c26-ecf4-488c-a9aa-a7b52e2d04e9\") " pod="openshift-console/console-b8c74fcc8-njh4j" Jul 16 16:27:05.761582 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:05.761556 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fc506c26-ecf4-488c-a9aa-a7b52e2d04e9-trusted-ca-bundle\") pod \"console-b8c74fcc8-njh4j\" (UID: \"fc506c26-ecf4-488c-a9aa-a7b52e2d04e9\") " pod="openshift-console/console-b8c74fcc8-njh4j" Jul 16 16:27:05.766629 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:05.766567 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fc506c26-ecf4-488c-a9aa-a7b52e2d04e9-console-serving-cert\") pod \"console-b8c74fcc8-njh4j\" (UID: \"fc506c26-ecf4-488c-a9aa-a7b52e2d04e9\") " pod="openshift-console/console-b8c74fcc8-njh4j" Jul 16 16:27:05.769197 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:05.769142 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fc506c26-ecf4-488c-a9aa-a7b52e2d04e9-console-oauth-config\") pod \"console-b8c74fcc8-njh4j\" (UID: \"fc506c26-ecf4-488c-a9aa-a7b52e2d04e9\") " pod="openshift-console/console-b8c74fcc8-njh4j" Jul 16 16:27:05.773693 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:05.773636 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jw8dr\" (UniqueName: \"kubernetes.io/projected/fc506c26-ecf4-488c-a9aa-a7b52e2d04e9-kube-api-access-jw8dr\") pod \"console-b8c74fcc8-njh4j\" (UID: \"fc506c26-ecf4-488c-a9aa-a7b52e2d04e9\") " pod="openshift-console/console-b8c74fcc8-njh4j" Jul 16 16:27:05.886569 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:05.886446 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-b8c74fcc8-njh4j" Jul 16 16:27:05.988584 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:05.988543 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 16 16:27:05.996789 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:05.996753 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 16 16:27:05.999739 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:05.999712 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Jul 16 16:27:06.000848 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:06.000821 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Jul 16 16:27:06.001002 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:06.000981 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Jul 16 16:27:06.001076 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:06.001028 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Jul 16 16:27:06.001076 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:06.001061 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-6po8hog5gfgo3\"" Jul 16 16:27:06.004701 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:06.001620 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Jul 16 16:27:06.004701 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:06.001830 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-wlv2t\"" Jul 16 16:27:06.004701 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:06.004203 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Jul 16 16:27:06.004701 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:06.004364 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Jul 16 16:27:06.004701 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:06.004447 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Jul 16 16:27:06.005018 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:06.004775 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Jul 16 16:27:06.005018 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:06.004975 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Jul 16 16:27:06.005176 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:06.005157 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Jul 16 16:27:06.005466 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:06.005447 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Jul 16 16:27:06.008288 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:06.008113 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Jul 16 16:27:06.010367 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:06.010308 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 16 16:27:06.017413 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:06.014970 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Jul 16 16:27:06.162967 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:06.162930 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/838dbd8b-f75f-4739-9052-56ecfc1cbe5e-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"838dbd8b-f75f-4739-9052-56ecfc1cbe5e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 16:27:06.162967 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:06.162980 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/838dbd8b-f75f-4739-9052-56ecfc1cbe5e-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"838dbd8b-f75f-4739-9052-56ecfc1cbe5e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 16:27:06.163206 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:06.163008 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/838dbd8b-f75f-4739-9052-56ecfc1cbe5e-config-out\") pod \"prometheus-k8s-0\" (UID: \"838dbd8b-f75f-4739-9052-56ecfc1cbe5e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 16:27:06.163206 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:06.163031 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/838dbd8b-f75f-4739-9052-56ecfc1cbe5e-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"838dbd8b-f75f-4739-9052-56ecfc1cbe5e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 16:27:06.163206 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:06.163064 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/838dbd8b-f75f-4739-9052-56ecfc1cbe5e-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"838dbd8b-f75f-4739-9052-56ecfc1cbe5e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 16:27:06.163206 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:06.163091 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/838dbd8b-f75f-4739-9052-56ecfc1cbe5e-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"838dbd8b-f75f-4739-9052-56ecfc1cbe5e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 16:27:06.163206 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:06.163111 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/838dbd8b-f75f-4739-9052-56ecfc1cbe5e-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"838dbd8b-f75f-4739-9052-56ecfc1cbe5e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 16:27:06.163463 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:06.163242 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/838dbd8b-f75f-4739-9052-56ecfc1cbe5e-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"838dbd8b-f75f-4739-9052-56ecfc1cbe5e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 16:27:06.163463 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:06.163356 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/838dbd8b-f75f-4739-9052-56ecfc1cbe5e-config\") pod \"prometheus-k8s-0\" (UID: \"838dbd8b-f75f-4739-9052-56ecfc1cbe5e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 16:27:06.163463 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:06.163405 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/838dbd8b-f75f-4739-9052-56ecfc1cbe5e-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"838dbd8b-f75f-4739-9052-56ecfc1cbe5e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 16:27:06.163463 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:06.163430 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/838dbd8b-f75f-4739-9052-56ecfc1cbe5e-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"838dbd8b-f75f-4739-9052-56ecfc1cbe5e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 16:27:06.163653 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:06.163471 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/838dbd8b-f75f-4739-9052-56ecfc1cbe5e-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"838dbd8b-f75f-4739-9052-56ecfc1cbe5e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 16:27:06.163653 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:06.163496 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/838dbd8b-f75f-4739-9052-56ecfc1cbe5e-web-config\") pod \"prometheus-k8s-0\" (UID: \"838dbd8b-f75f-4739-9052-56ecfc1cbe5e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 16:27:06.163653 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:06.163519 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/838dbd8b-f75f-4739-9052-56ecfc1cbe5e-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"838dbd8b-f75f-4739-9052-56ecfc1cbe5e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 16:27:06.163653 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:06.163556 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/838dbd8b-f75f-4739-9052-56ecfc1cbe5e-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"838dbd8b-f75f-4739-9052-56ecfc1cbe5e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 16:27:06.163653 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:06.163586 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/838dbd8b-f75f-4739-9052-56ecfc1cbe5e-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"838dbd8b-f75f-4739-9052-56ecfc1cbe5e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 16:27:06.163653 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:06.163607 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-87sp2\" (UniqueName: \"kubernetes.io/projected/838dbd8b-f75f-4739-9052-56ecfc1cbe5e-kube-api-access-87sp2\") pod \"prometheus-k8s-0\" (UID: \"838dbd8b-f75f-4739-9052-56ecfc1cbe5e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 16:27:06.163653 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:06.163634 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/838dbd8b-f75f-4739-9052-56ecfc1cbe5e-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"838dbd8b-f75f-4739-9052-56ecfc1cbe5e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 16:27:06.164586 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:06.163673 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/838dbd8b-f75f-4739-9052-56ecfc1cbe5e-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"838dbd8b-f75f-4739-9052-56ecfc1cbe5e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 16:27:06.164586 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:06.163698 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/838dbd8b-f75f-4739-9052-56ecfc1cbe5e-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"838dbd8b-f75f-4739-9052-56ecfc1cbe5e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 16:27:06.235244 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:06.235160 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-b8c74fcc8-njh4j"] Jul 16 16:27:06.265318 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:06.265214 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/838dbd8b-f75f-4739-9052-56ecfc1cbe5e-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"838dbd8b-f75f-4739-9052-56ecfc1cbe5e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 16:27:06.265318 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:06.265277 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/838dbd8b-f75f-4739-9052-56ecfc1cbe5e-config\") pod \"prometheus-k8s-0\" (UID: \"838dbd8b-f75f-4739-9052-56ecfc1cbe5e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 16:27:06.265318 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:06.265304 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/838dbd8b-f75f-4739-9052-56ecfc1cbe5e-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"838dbd8b-f75f-4739-9052-56ecfc1cbe5e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 16:27:06.265586 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:06.265331 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/838dbd8b-f75f-4739-9052-56ecfc1cbe5e-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"838dbd8b-f75f-4739-9052-56ecfc1cbe5e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 16:27:06.265586 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:06.265375 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/838dbd8b-f75f-4739-9052-56ecfc1cbe5e-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"838dbd8b-f75f-4739-9052-56ecfc1cbe5e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 16:27:06.265586 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:06.265426 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/838dbd8b-f75f-4739-9052-56ecfc1cbe5e-web-config\") pod \"prometheus-k8s-0\" (UID: \"838dbd8b-f75f-4739-9052-56ecfc1cbe5e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 16:27:06.265586 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:06.265452 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/838dbd8b-f75f-4739-9052-56ecfc1cbe5e-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"838dbd8b-f75f-4739-9052-56ecfc1cbe5e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 16:27:06.265586 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:06.265490 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/838dbd8b-f75f-4739-9052-56ecfc1cbe5e-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"838dbd8b-f75f-4739-9052-56ecfc1cbe5e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 16:27:06.265586 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:06.265523 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/838dbd8b-f75f-4739-9052-56ecfc1cbe5e-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"838dbd8b-f75f-4739-9052-56ecfc1cbe5e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 16:27:06.265586 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:06.265546 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-87sp2\" (UniqueName: \"kubernetes.io/projected/838dbd8b-f75f-4739-9052-56ecfc1cbe5e-kube-api-access-87sp2\") pod \"prometheus-k8s-0\" (UID: \"838dbd8b-f75f-4739-9052-56ecfc1cbe5e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 16:27:06.265586 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:06.265574 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/838dbd8b-f75f-4739-9052-56ecfc1cbe5e-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"838dbd8b-f75f-4739-9052-56ecfc1cbe5e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 16:27:06.265941 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:06.265610 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/838dbd8b-f75f-4739-9052-56ecfc1cbe5e-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"838dbd8b-f75f-4739-9052-56ecfc1cbe5e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 16:27:06.265941 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:06.265632 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/838dbd8b-f75f-4739-9052-56ecfc1cbe5e-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"838dbd8b-f75f-4739-9052-56ecfc1cbe5e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 16:27:06.265941 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:06.265677 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/838dbd8b-f75f-4739-9052-56ecfc1cbe5e-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"838dbd8b-f75f-4739-9052-56ecfc1cbe5e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 16:27:06.265941 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:06.265702 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/838dbd8b-f75f-4739-9052-56ecfc1cbe5e-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"838dbd8b-f75f-4739-9052-56ecfc1cbe5e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 16:27:06.265941 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:06.265730 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/838dbd8b-f75f-4739-9052-56ecfc1cbe5e-config-out\") pod \"prometheus-k8s-0\" (UID: \"838dbd8b-f75f-4739-9052-56ecfc1cbe5e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 16:27:06.265941 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:06.265752 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/838dbd8b-f75f-4739-9052-56ecfc1cbe5e-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"838dbd8b-f75f-4739-9052-56ecfc1cbe5e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 16:27:06.265941 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:06.265793 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/838dbd8b-f75f-4739-9052-56ecfc1cbe5e-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"838dbd8b-f75f-4739-9052-56ecfc1cbe5e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 16:27:06.265941 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:06.265828 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/838dbd8b-f75f-4739-9052-56ecfc1cbe5e-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"838dbd8b-f75f-4739-9052-56ecfc1cbe5e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 16:27:06.265941 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:06.265853 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/838dbd8b-f75f-4739-9052-56ecfc1cbe5e-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"838dbd8b-f75f-4739-9052-56ecfc1cbe5e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 16:27:06.267152 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:06.266741 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/838dbd8b-f75f-4739-9052-56ecfc1cbe5e-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"838dbd8b-f75f-4739-9052-56ecfc1cbe5e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 16:27:06.267152 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:06.266996 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/838dbd8b-f75f-4739-9052-56ecfc1cbe5e-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"838dbd8b-f75f-4739-9052-56ecfc1cbe5e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 16:27:06.268564 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:06.267684 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/838dbd8b-f75f-4739-9052-56ecfc1cbe5e-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"838dbd8b-f75f-4739-9052-56ecfc1cbe5e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 16:27:06.268564 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:06.268116 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/838dbd8b-f75f-4739-9052-56ecfc1cbe5e-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"838dbd8b-f75f-4739-9052-56ecfc1cbe5e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 16:27:06.269129 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:06.269063 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/838dbd8b-f75f-4739-9052-56ecfc1cbe5e-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"838dbd8b-f75f-4739-9052-56ecfc1cbe5e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 16:27:06.269336 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:06.269233 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/838dbd8b-f75f-4739-9052-56ecfc1cbe5e-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"838dbd8b-f75f-4739-9052-56ecfc1cbe5e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 16:27:06.270062 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:06.269753 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/838dbd8b-f75f-4739-9052-56ecfc1cbe5e-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"838dbd8b-f75f-4739-9052-56ecfc1cbe5e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 16:27:06.270062 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:06.269963 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/838dbd8b-f75f-4739-9052-56ecfc1cbe5e-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"838dbd8b-f75f-4739-9052-56ecfc1cbe5e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 16:27:06.271996 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:06.271956 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/838dbd8b-f75f-4739-9052-56ecfc1cbe5e-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"838dbd8b-f75f-4739-9052-56ecfc1cbe5e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 16:27:06.273559 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:06.273512 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/838dbd8b-f75f-4739-9052-56ecfc1cbe5e-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"838dbd8b-f75f-4739-9052-56ecfc1cbe5e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 16:27:06.274149 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:06.274097 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/838dbd8b-f75f-4739-9052-56ecfc1cbe5e-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"838dbd8b-f75f-4739-9052-56ecfc1cbe5e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 16:27:06.274234 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:06.274220 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/838dbd8b-f75f-4739-9052-56ecfc1cbe5e-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"838dbd8b-f75f-4739-9052-56ecfc1cbe5e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 16:27:06.274610 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:06.274569 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/838dbd8b-f75f-4739-9052-56ecfc1cbe5e-config-out\") pod \"prometheus-k8s-0\" (UID: \"838dbd8b-f75f-4739-9052-56ecfc1cbe5e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 16:27:06.275119 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:06.275079 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/838dbd8b-f75f-4739-9052-56ecfc1cbe5e-config\") pod \"prometheus-k8s-0\" (UID: \"838dbd8b-f75f-4739-9052-56ecfc1cbe5e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 16:27:06.275473 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:06.275410 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/838dbd8b-f75f-4739-9052-56ecfc1cbe5e-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"838dbd8b-f75f-4739-9052-56ecfc1cbe5e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 16:27:06.275654 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:06.275632 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/838dbd8b-f75f-4739-9052-56ecfc1cbe5e-web-config\") pod \"prometheus-k8s-0\" (UID: \"838dbd8b-f75f-4739-9052-56ecfc1cbe5e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 16:27:06.278146 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:06.278102 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/838dbd8b-f75f-4739-9052-56ecfc1cbe5e-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"838dbd8b-f75f-4739-9052-56ecfc1cbe5e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 16:27:06.280898 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:06.280855 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-87sp2\" (UniqueName: \"kubernetes.io/projected/838dbd8b-f75f-4739-9052-56ecfc1cbe5e-kube-api-access-87sp2\") pod \"prometheus-k8s-0\" (UID: \"838dbd8b-f75f-4739-9052-56ecfc1cbe5e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 16:27:06.281580 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:06.281557 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/838dbd8b-f75f-4739-9052-56ecfc1cbe5e-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"838dbd8b-f75f-4739-9052-56ecfc1cbe5e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 16:27:06.285852 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:06.285808 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/838dbd8b-f75f-4739-9052-56ecfc1cbe5e-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"838dbd8b-f75f-4739-9052-56ecfc1cbe5e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 16:27:06.316949 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:06.316665 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 16 16:27:06.659786 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:27:06.659740 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc506c26_ecf4_488c_a9aa_a7b52e2d04e9.slice/crio-6dbc23d3ea12cc4802f2b16bdc1bc0fd3a0e791fed0353707b0aede6bb6e891f WatchSource:0}: Error finding container 6dbc23d3ea12cc4802f2b16bdc1bc0fd3a0e791fed0353707b0aede6bb6e891f: Status 404 returned error can't find the container with id 6dbc23d3ea12cc4802f2b16bdc1bc0fd3a0e791fed0353707b0aede6bb6e891f Jul 16 16:27:07.484771 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:07.484695 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-b8c74fcc8-njh4j" event={"ID":"fc506c26-ecf4-488c-a9aa-a7b52e2d04e9","Type":"ContainerStarted","Data":"6dbc23d3ea12cc4802f2b16bdc1bc0fd3a0e791fed0353707b0aede6bb6e891f"} Jul 16 16:27:07.626346 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:07.626241 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 16 16:27:07.635282 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:27:07.635208 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod838dbd8b_f75f_4739_9052_56ecfc1cbe5e.slice/crio-423babd017c7e74512f6f09da13fb910ec4f0c2f01999bea54442c2c7fe811b8 WatchSource:0}: Error finding container 423babd017c7e74512f6f09da13fb910ec4f0c2f01999bea54442c2c7fe811b8: Status 404 returned error can't find the container with id 423babd017c7e74512f6f09da13fb910ec4f0c2f01999bea54442c2c7fe811b8 Jul 16 16:27:08.492224 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:08.492143 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-6fd5bb7967-8t82m" event={"ID":"3ed7defb-10df-42c6-af68-8a2145ad3539","Type":"ContainerStarted","Data":"c83a2fcfbb42f6e6a3e658c984cfc4406b2d593c2ca2b3a1f0acd056da3e5c2f"} Jul 16 16:27:08.494220 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:08.494183 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-rrhsm" event={"ID":"342fe0d4-70b8-43b9-b052-93a1068deb59","Type":"ContainerStarted","Data":"1e960e36b40226adeb0d4e594962ee545e3e6d404a2a33f07dc4490e62da6d19"} Jul 16 16:27:08.494432 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:08.494407 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-rrhsm" Jul 16 16:27:08.496842 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:08.496786 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-b8c74fcc8-njh4j" event={"ID":"fc506c26-ecf4-488c-a9aa-a7b52e2d04e9","Type":"ContainerStarted","Data":"6c8f71a8a42ef95dc2f04e381eb7f3e985c20e8d529bea0353bc78e5988c911b"} Jul 16 16:27:08.499440 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:08.499357 2577 generic.go:356] "Generic (PLEG): container finished" podID="838dbd8b-f75f-4739-9052-56ecfc1cbe5e" containerID="faaef2d8530abd87f24486d50bedc594fc104551f55539bef6850873be59b76d" exitCode=0 Jul 16 16:27:08.499440 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:08.499426 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"838dbd8b-f75f-4739-9052-56ecfc1cbe5e","Type":"ContainerDied","Data":"faaef2d8530abd87f24486d50bedc594fc104551f55539bef6850873be59b76d"} Jul 16 16:27:08.499591 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:08.499502 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"838dbd8b-f75f-4739-9052-56ecfc1cbe5e","Type":"ContainerStarted","Data":"423babd017c7e74512f6f09da13fb910ec4f0c2f01999bea54442c2c7fe811b8"} Jul 16 16:27:08.500927 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:08.500874 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-rrhsm" Jul 16 16:27:08.504109 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:08.504077 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-b957bd756-6cvr2" event={"ID":"2e58ee24-b3bc-4e3a-828c-6ac801ec7fd7","Type":"ContainerStarted","Data":"866c0dbc0d339ea0d16272a85428cd4dc6d44cff686cb407d00c50b8a3118e72"} Jul 16 16:27:08.504214 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:08.504144 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-b957bd756-6cvr2" event={"ID":"2e58ee24-b3bc-4e3a-828c-6ac801ec7fd7","Type":"ContainerStarted","Data":"1d5f5f9d59b36b9451726331066f6cce59311f33ea3fc0412cccf74b0619f082"} Jul 16 16:27:08.504214 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:08.504189 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-b957bd756-6cvr2" event={"ID":"2e58ee24-b3bc-4e3a-828c-6ac801ec7fd7","Type":"ContainerStarted","Data":"456a22c2dcfd53df608329e2276b5e3b1cd2a6f33cb79e683b62687a7173c5cb"} Jul 16 16:27:08.504313 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:08.504235 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-b957bd756-6cvr2" Jul 16 16:27:08.508628 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:08.508605 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8c88b029-8a81-402c-90e4-77eff554183c","Type":"ContainerStarted","Data":"648b94bad83a73c838176c43a95050f789ffa285843d356634a1b63cf4e29526"} Jul 16 16:27:08.508707 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:08.508670 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8c88b029-8a81-402c-90e4-77eff554183c","Type":"ContainerStarted","Data":"48b61e12bb28e6ab048714994999a92f8f9abb73b6b67ffc0e85372cda417156"} Jul 16 16:27:08.508759 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:08.508712 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8c88b029-8a81-402c-90e4-77eff554183c","Type":"ContainerStarted","Data":"135f81d3a5aa643313bffe0e166eb7a6e53d0f06405a6ad43f12cfa76c3751d7"} Jul 16 16:27:08.508759 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:08.508751 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8c88b029-8a81-402c-90e4-77eff554183c","Type":"ContainerStarted","Data":"cb11d353e80c9f772f3efe23e583a8c4766a06571190187af1b9e8bd0156a325"} Jul 16 16:27:08.508842 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:08.508780 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8c88b029-8a81-402c-90e4-77eff554183c","Type":"ContainerStarted","Data":"3a63a0710d4c374958232c6e8f55a492cfbf7e6734eaeb6437bd985e806708e9"} Jul 16 16:27:08.508842 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:08.508818 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8c88b029-8a81-402c-90e4-77eff554183c","Type":"ContainerStarted","Data":"b0a314b361a43e35379709e849c91bf7ab25abc8a82c3359de1743b90c66733c"} Jul 16 16:27:08.518950 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:08.518900 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-6fd5bb7967-8t82m" podStartSLOduration=2.966117304 podStartE2EDuration="5.518883646s" totalImagesPullingTime="2.552766342s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 16:27:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 16:27:08.517567792 +0000 UTC m=+87.021625645" watchObservedRunningTime="2026-07-16 16:27:08.518883646 +0000 UTC m=+87.022941475" Jul 16 16:27:08.579206 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:08.578985 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-b8c74fcc8-njh4j" podStartSLOduration=3.578962823 podStartE2EDuration="3.578962823s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 16:27:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 16:27:08.578469582 +0000 UTC m=+87.082527435" watchObservedRunningTime="2026-07-16 16:27:08.578962823 +0000 UTC m=+87.083020675" Jul 16 16:27:08.580279 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:08.580221 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-rrhsm" podStartSLOduration=2.027567151 podStartE2EDuration="4.580207292s" totalImagesPullingTime="2.552640141s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 16:27:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 16:27:08.549974839 +0000 UTC m=+87.054032696" watchObservedRunningTime="2026-07-16 16:27:08.580207292 +0000 UTC m=+87.084265144" Jul 16 16:27:08.658336 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:08.658263 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.873222254 podStartE2EDuration="8.658240971s" totalImagesPullingTime="4.752568121s" totalInitContainerRuntime="32.450596ms" isStatefulPod=false podCreationTimestamp="2026-07-16 16:27:00 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 16:27:08.655289188 +0000 UTC m=+87.159347038" watchObservedRunningTime="2026-07-16 16:27:08.658240971 +0000 UTC m=+87.162298822" Jul 16 16:27:08.698708 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:08.698645 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-b957bd756-6cvr2" podStartSLOduration=2.638931096 podStartE2EDuration="7.698621189s" totalImagesPullingTime="5.059690093s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 16:27:01 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 16:27:08.697268265 +0000 UTC m=+87.201326117" watchObservedRunningTime="2026-07-16 16:27:08.698621189 +0000 UTC m=+87.202679041" Jul 16 16:27:09.522340 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:09.522247 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-568889b88c-4488z" event={"ID":"ce591e34-5de4-45d1-ab53-e9963ffae504","Type":"ContainerStarted","Data":"d1ca856692be6c3131c32435b569f228fd7d2542fd0f46b72fceb6604383a291"} Jul 16 16:27:09.522796 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:09.522360 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-568889b88c-4488z" event={"ID":"ce591e34-5de4-45d1-ab53-e9963ffae504","Type":"ContainerStarted","Data":"68c33be3bc8f40cbf96d13eadf0f2489f1b2326b20425474675683564d0183d9"} Jul 16 16:27:09.522796 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:09.522399 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-568889b88c-4488z" event={"ID":"ce591e34-5de4-45d1-ab53-e9963ffae504","Type":"ContainerStarted","Data":"8a8cc72af9e1c58be55ca5358134dda6ed2f1d091fea6fd4c69938449388b734"} Jul 16 16:27:09.529974 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:09.529948 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-b957bd756-6cvr2" Jul 16 16:27:09.548547 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:09.548487 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-568889b88c-4488z" podStartSLOduration=1.5145203170000001 podStartE2EDuration="5.548472259s" totalImagesPullingTime="4.033951942s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 16:27:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 16:27:09.547405694 +0000 UTC m=+88.051463545" watchObservedRunningTime="2026-07-16 16:27:09.548472259 +0000 UTC m=+88.052530109" Jul 16 16:27:10.338252 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:10.338211 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-b8c74fcc8-njh4j"] Jul 16 16:27:12.544221 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:12.544184 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"838dbd8b-f75f-4739-9052-56ecfc1cbe5e","Type":"ContainerStarted","Data":"e10bc69952a585fa6762e61776e483eb7e3787c7a132d9b79885bb8e184aa075"} Jul 16 16:27:12.544613 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:12.544259 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"838dbd8b-f75f-4739-9052-56ecfc1cbe5e","Type":"ContainerStarted","Data":"b0d9a29e2cc50e98b6e03987e9b3f8ff9e0f139132c4775d09d82ebb6b698c76"} Jul 16 16:27:12.544613 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:12.544284 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"838dbd8b-f75f-4739-9052-56ecfc1cbe5e","Type":"ContainerStarted","Data":"80c8454b5d7d8a17756684b674a9b52e5849acc67ca8687b85ebdde0ca36bfb1"} Jul 16 16:27:12.544613 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:12.544308 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"838dbd8b-f75f-4739-9052-56ecfc1cbe5e","Type":"ContainerStarted","Data":"d4067bba4ae22b4d5efa2280ce5e385f2e648cb8699bb1e1af03041332411a3d"} Jul 16 16:27:13.551656 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:13.551618 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"838dbd8b-f75f-4739-9052-56ecfc1cbe5e","Type":"ContainerStarted","Data":"0ab36244b62bf265c6948c8203782b9f09f2869261c835c0a0e1a3399bea3b31"} Jul 16 16:27:13.552045 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:13.551689 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"838dbd8b-f75f-4739-9052-56ecfc1cbe5e","Type":"ContainerStarted","Data":"9cee351c24a45cb1a0973f3189985e363b010c49c1509fbf5bf6c1b2fcd930a8"} Jul 16 16:27:13.593193 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:13.593136 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=4.873506081 podStartE2EDuration="8.593119153s" totalImagesPullingTime="3.690043459s" totalInitContainerRuntime="29.569613ms" isStatefulPod=false podCreationTimestamp="2026-07-16 16:27:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 16:27:13.591662033 +0000 UTC m=+92.095719898" watchObservedRunningTime="2026-07-16 16:27:13.593119153 +0000 UTC m=+92.097177003" Jul 16 16:27:15.887492 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:15.887452 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-b8c74fcc8-njh4j" Jul 16 16:27:16.317190 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:16.317108 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 16 16:27:21.430212 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:21.430177 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-76dcf44fd5-tx9p2"] Jul 16 16:27:24.109102 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:24.109063 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-6fd5bb7967-8t82m" Jul 16 16:27:24.109499 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:24.109126 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-6fd5bb7967-8t82m" Jul 16 16:27:27.584663 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:27.584594 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-68b7b95d9-dnh2n" podUID="cabb304d-fabd-4944-87f1-b7deccca8616" containerName="console" containerID="cri-o://704d5aada534d8e3c194bff401788a7021278ef9f8f99309cd4085a0ac99eeb1" gracePeriod=15 Jul 16 16:27:28.371644 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:28.371613 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-68b7b95d9-dnh2n" Jul 16 16:27:28.415072 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:28.415031 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-ntktc" Jul 16 16:27:28.483480 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:28.483399 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/cabb304d-fabd-4944-87f1-b7deccca8616-console-serving-cert\") pod \"cabb304d-fabd-4944-87f1-b7deccca8616\" (UID: \"cabb304d-fabd-4944-87f1-b7deccca8616\") " Jul 16 16:27:28.483480 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:28.483443 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/cabb304d-fabd-4944-87f1-b7deccca8616-console-config\") pod \"cabb304d-fabd-4944-87f1-b7deccca8616\" (UID: \"cabb304d-fabd-4944-87f1-b7deccca8616\") " Jul 16 16:27:28.483480 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:28.483461 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/cabb304d-fabd-4944-87f1-b7deccca8616-console-oauth-config\") pod \"cabb304d-fabd-4944-87f1-b7deccca8616\" (UID: \"cabb304d-fabd-4944-87f1-b7deccca8616\") " Jul 16 16:27:28.483752 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:28.483487 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/cabb304d-fabd-4944-87f1-b7deccca8616-oauth-serving-cert\") pod \"cabb304d-fabd-4944-87f1-b7deccca8616\" (UID: \"cabb304d-fabd-4944-87f1-b7deccca8616\") " Jul 16 16:27:28.483752 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:28.483554 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h5spp\" (UniqueName: \"kubernetes.io/projected/cabb304d-fabd-4944-87f1-b7deccca8616-kube-api-access-h5spp\") pod \"cabb304d-fabd-4944-87f1-b7deccca8616\" (UID: \"cabb304d-fabd-4944-87f1-b7deccca8616\") " Jul 16 16:27:28.483752 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:28.483601 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/cabb304d-fabd-4944-87f1-b7deccca8616-service-ca\") pod \"cabb304d-fabd-4944-87f1-b7deccca8616\" (UID: \"cabb304d-fabd-4944-87f1-b7deccca8616\") " Jul 16 16:27:28.483910 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:28.483866 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cabb304d-fabd-4944-87f1-b7deccca8616-console-config" (OuterVolumeSpecName: "console-config") pod "cabb304d-fabd-4944-87f1-b7deccca8616" (UID: "cabb304d-fabd-4944-87f1-b7deccca8616"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 16:27:28.483910 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:28.483885 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cabb304d-fabd-4944-87f1-b7deccca8616-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "cabb304d-fabd-4944-87f1-b7deccca8616" (UID: "cabb304d-fabd-4944-87f1-b7deccca8616"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 16:27:28.484315 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:28.484287 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cabb304d-fabd-4944-87f1-b7deccca8616-service-ca" (OuterVolumeSpecName: "service-ca") pod "cabb304d-fabd-4944-87f1-b7deccca8616" (UID: "cabb304d-fabd-4944-87f1-b7deccca8616"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 16:27:28.486045 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:28.486012 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cabb304d-fabd-4944-87f1-b7deccca8616-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "cabb304d-fabd-4944-87f1-b7deccca8616" (UID: "cabb304d-fabd-4944-87f1-b7deccca8616"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 16:27:28.486172 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:28.486137 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cabb304d-fabd-4944-87f1-b7deccca8616-kube-api-access-h5spp" (OuterVolumeSpecName: "kube-api-access-h5spp") pod "cabb304d-fabd-4944-87f1-b7deccca8616" (UID: "cabb304d-fabd-4944-87f1-b7deccca8616"). InnerVolumeSpecName "kube-api-access-h5spp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 16:27:28.486172 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:28.486148 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cabb304d-fabd-4944-87f1-b7deccca8616-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "cabb304d-fabd-4944-87f1-b7deccca8616" (UID: "cabb304d-fabd-4944-87f1-b7deccca8616"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 16:27:28.585169 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:28.585135 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h5spp\" (UniqueName: \"kubernetes.io/projected/cabb304d-fabd-4944-87f1-b7deccca8616-kube-api-access-h5spp\") on node \"ip-10-0-142-101.ec2.internal\" DevicePath \"\"" Jul 16 16:27:28.585169 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:28.585164 2577 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/cabb304d-fabd-4944-87f1-b7deccca8616-service-ca\") on node \"ip-10-0-142-101.ec2.internal\" DevicePath \"\"" Jul 16 16:27:28.585169 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:28.585173 2577 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/cabb304d-fabd-4944-87f1-b7deccca8616-console-serving-cert\") on node \"ip-10-0-142-101.ec2.internal\" DevicePath \"\"" Jul 16 16:27:28.585608 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:28.585182 2577 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/cabb304d-fabd-4944-87f1-b7deccca8616-console-config\") on node \"ip-10-0-142-101.ec2.internal\" DevicePath \"\"" Jul 16 16:27:28.585608 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:28.585190 2577 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/cabb304d-fabd-4944-87f1-b7deccca8616-console-oauth-config\") on node \"ip-10-0-142-101.ec2.internal\" DevicePath \"\"" Jul 16 16:27:28.585608 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:28.585198 2577 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/cabb304d-fabd-4944-87f1-b7deccca8616-oauth-serving-cert\") on node \"ip-10-0-142-101.ec2.internal\" DevicePath \"\"" Jul 16 16:27:28.620013 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:28.619981 2577 generic.go:356] "Generic (PLEG): container finished" podID="cabb304d-fabd-4944-87f1-b7deccca8616" containerID="704d5aada534d8e3c194bff401788a7021278ef9f8f99309cd4085a0ac99eeb1" exitCode=0 Jul 16 16:27:28.620178 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:28.620043 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-68b7b95d9-dnh2n" Jul 16 16:27:28.620178 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:28.620072 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-68b7b95d9-dnh2n" event={"ID":"cabb304d-fabd-4944-87f1-b7deccca8616","Type":"ContainerDied","Data":"704d5aada534d8e3c194bff401788a7021278ef9f8f99309cd4085a0ac99eeb1"} Jul 16 16:27:28.620178 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:28.620139 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-68b7b95d9-dnh2n" event={"ID":"cabb304d-fabd-4944-87f1-b7deccca8616","Type":"ContainerDied","Data":"c4305d6867f91dbcb47fc20684165eb81209439fb08f2f170bfe192b7c809deb"} Jul 16 16:27:28.620178 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:28.620167 2577 scope.go:117] "RemoveContainer" containerID="704d5aada534d8e3c194bff401788a7021278ef9f8f99309cd4085a0ac99eeb1" Jul 16 16:27:28.629827 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:28.629812 2577 scope.go:117] "RemoveContainer" containerID="704d5aada534d8e3c194bff401788a7021278ef9f8f99309cd4085a0ac99eeb1" Jul 16 16:27:28.630079 ip-10-0-142-101 kubenswrapper[2577]: E0716 16:27:28.630060 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"704d5aada534d8e3c194bff401788a7021278ef9f8f99309cd4085a0ac99eeb1\": container with ID starting with 704d5aada534d8e3c194bff401788a7021278ef9f8f99309cd4085a0ac99eeb1 not found: ID does not exist" containerID="704d5aada534d8e3c194bff401788a7021278ef9f8f99309cd4085a0ac99eeb1" Jul 16 16:27:28.630145 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:28.630088 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"704d5aada534d8e3c194bff401788a7021278ef9f8f99309cd4085a0ac99eeb1"} err="failed to get container status \"704d5aada534d8e3c194bff401788a7021278ef9f8f99309cd4085a0ac99eeb1\": rpc error: code = NotFound desc = could not find container \"704d5aada534d8e3c194bff401788a7021278ef9f8f99309cd4085a0ac99eeb1\": container with ID starting with 704d5aada534d8e3c194bff401788a7021278ef9f8f99309cd4085a0ac99eeb1 not found: ID does not exist" Jul 16 16:27:28.656685 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:28.656653 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-68b7b95d9-dnh2n"] Jul 16 16:27:28.660934 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:28.660912 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-68b7b95d9-dnh2n"] Jul 16 16:27:30.016806 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:30.016773 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cabb304d-fabd-4944-87f1-b7deccca8616" path="/var/lib/kubelet/pods/cabb304d-fabd-4944-87f1-b7deccca8616/volumes" Jul 16 16:27:35.557768 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:35.557707 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-b8c74fcc8-njh4j" podUID="fc506c26-ecf4-488c-a9aa-a7b52e2d04e9" containerName="console" containerID="cri-o://6c8f71a8a42ef95dc2f04e381eb7f3e985c20e8d529bea0353bc78e5988c911b" gracePeriod=15 Jul 16 16:27:35.846287 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:35.846259 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-b8c74fcc8-njh4j" Jul 16 16:27:35.954398 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:35.954347 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fc506c26-ecf4-488c-a9aa-a7b52e2d04e9-console-config\") pod \"fc506c26-ecf4-488c-a9aa-a7b52e2d04e9\" (UID: \"fc506c26-ecf4-488c-a9aa-a7b52e2d04e9\") " Jul 16 16:27:35.954583 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:35.954455 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fc506c26-ecf4-488c-a9aa-a7b52e2d04e9-oauth-serving-cert\") pod \"fc506c26-ecf4-488c-a9aa-a7b52e2d04e9\" (UID: \"fc506c26-ecf4-488c-a9aa-a7b52e2d04e9\") " Jul 16 16:27:35.954583 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:35.954487 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fc506c26-ecf4-488c-a9aa-a7b52e2d04e9-service-ca\") pod \"fc506c26-ecf4-488c-a9aa-a7b52e2d04e9\" (UID: \"fc506c26-ecf4-488c-a9aa-a7b52e2d04e9\") " Jul 16 16:27:35.954583 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:35.954544 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fc506c26-ecf4-488c-a9aa-a7b52e2d04e9-console-oauth-config\") pod \"fc506c26-ecf4-488c-a9aa-a7b52e2d04e9\" (UID: \"fc506c26-ecf4-488c-a9aa-a7b52e2d04e9\") " Jul 16 16:27:35.954583 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:35.954580 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fc506c26-ecf4-488c-a9aa-a7b52e2d04e9-console-serving-cert\") pod \"fc506c26-ecf4-488c-a9aa-a7b52e2d04e9\" (UID: \"fc506c26-ecf4-488c-a9aa-a7b52e2d04e9\") " Jul 16 16:27:35.954807 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:35.954606 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jw8dr\" (UniqueName: \"kubernetes.io/projected/fc506c26-ecf4-488c-a9aa-a7b52e2d04e9-kube-api-access-jw8dr\") pod \"fc506c26-ecf4-488c-a9aa-a7b52e2d04e9\" (UID: \"fc506c26-ecf4-488c-a9aa-a7b52e2d04e9\") " Jul 16 16:27:35.954807 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:35.954656 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fc506c26-ecf4-488c-a9aa-a7b52e2d04e9-trusted-ca-bundle\") pod \"fc506c26-ecf4-488c-a9aa-a7b52e2d04e9\" (UID: \"fc506c26-ecf4-488c-a9aa-a7b52e2d04e9\") " Jul 16 16:27:35.954929 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:35.954862 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc506c26-ecf4-488c-a9aa-a7b52e2d04e9-console-config" (OuterVolumeSpecName: "console-config") pod "fc506c26-ecf4-488c-a9aa-a7b52e2d04e9" (UID: "fc506c26-ecf4-488c-a9aa-a7b52e2d04e9"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 16:27:35.954986 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:35.954948 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc506c26-ecf4-488c-a9aa-a7b52e2d04e9-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "fc506c26-ecf4-488c-a9aa-a7b52e2d04e9" (UID: "fc506c26-ecf4-488c-a9aa-a7b52e2d04e9"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 16:27:35.955150 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:35.955120 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc506c26-ecf4-488c-a9aa-a7b52e2d04e9-service-ca" (OuterVolumeSpecName: "service-ca") pod "fc506c26-ecf4-488c-a9aa-a7b52e2d04e9" (UID: "fc506c26-ecf4-488c-a9aa-a7b52e2d04e9"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 16:27:35.955292 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:35.955173 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc506c26-ecf4-488c-a9aa-a7b52e2d04e9-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "fc506c26-ecf4-488c-a9aa-a7b52e2d04e9" (UID: "fc506c26-ecf4-488c-a9aa-a7b52e2d04e9"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 16:27:35.956931 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:35.956894 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fc506c26-ecf4-488c-a9aa-a7b52e2d04e9-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "fc506c26-ecf4-488c-a9aa-a7b52e2d04e9" (UID: "fc506c26-ecf4-488c-a9aa-a7b52e2d04e9"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 16:27:35.957336 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:35.957307 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fc506c26-ecf4-488c-a9aa-a7b52e2d04e9-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "fc506c26-ecf4-488c-a9aa-a7b52e2d04e9" (UID: "fc506c26-ecf4-488c-a9aa-a7b52e2d04e9"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 16:27:35.957410 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:35.957318 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc506c26-ecf4-488c-a9aa-a7b52e2d04e9-kube-api-access-jw8dr" (OuterVolumeSpecName: "kube-api-access-jw8dr") pod "fc506c26-ecf4-488c-a9aa-a7b52e2d04e9" (UID: "fc506c26-ecf4-488c-a9aa-a7b52e2d04e9"). InnerVolumeSpecName "kube-api-access-jw8dr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 16:27:36.055560 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:36.055525 2577 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fc506c26-ecf4-488c-a9aa-a7b52e2d04e9-trusted-ca-bundle\") on node \"ip-10-0-142-101.ec2.internal\" DevicePath \"\"" Jul 16 16:27:36.055560 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:36.055559 2577 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fc506c26-ecf4-488c-a9aa-a7b52e2d04e9-console-config\") on node \"ip-10-0-142-101.ec2.internal\" DevicePath \"\"" Jul 16 16:27:36.055729 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:36.055573 2577 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fc506c26-ecf4-488c-a9aa-a7b52e2d04e9-oauth-serving-cert\") on node \"ip-10-0-142-101.ec2.internal\" DevicePath \"\"" Jul 16 16:27:36.055729 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:36.055582 2577 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fc506c26-ecf4-488c-a9aa-a7b52e2d04e9-service-ca\") on node \"ip-10-0-142-101.ec2.internal\" DevicePath \"\"" Jul 16 16:27:36.055729 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:36.055591 2577 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fc506c26-ecf4-488c-a9aa-a7b52e2d04e9-console-oauth-config\") on node \"ip-10-0-142-101.ec2.internal\" DevicePath \"\"" Jul 16 16:27:36.055729 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:36.055599 2577 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fc506c26-ecf4-488c-a9aa-a7b52e2d04e9-console-serving-cert\") on node \"ip-10-0-142-101.ec2.internal\" DevicePath \"\"" Jul 16 16:27:36.055729 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:36.055607 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jw8dr\" (UniqueName: \"kubernetes.io/projected/fc506c26-ecf4-488c-a9aa-a7b52e2d04e9-kube-api-access-jw8dr\") on node \"ip-10-0-142-101.ec2.internal\" DevicePath \"\"" Jul 16 16:27:36.660282 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:36.660243 2577 generic.go:356] "Generic (PLEG): container finished" podID="fc506c26-ecf4-488c-a9aa-a7b52e2d04e9" containerID="6c8f71a8a42ef95dc2f04e381eb7f3e985c20e8d529bea0353bc78e5988c911b" exitCode=0 Jul 16 16:27:36.660787 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:36.660291 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-b8c74fcc8-njh4j" Jul 16 16:27:36.660787 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:36.660332 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-b8c74fcc8-njh4j" event={"ID":"fc506c26-ecf4-488c-a9aa-a7b52e2d04e9","Type":"ContainerDied","Data":"6c8f71a8a42ef95dc2f04e381eb7f3e985c20e8d529bea0353bc78e5988c911b"} Jul 16 16:27:36.660787 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:36.660418 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-b8c74fcc8-njh4j" event={"ID":"fc506c26-ecf4-488c-a9aa-a7b52e2d04e9","Type":"ContainerDied","Data":"6dbc23d3ea12cc4802f2b16bdc1bc0fd3a0e791fed0353707b0aede6bb6e891f"} Jul 16 16:27:36.660787 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:36.660454 2577 scope.go:117] "RemoveContainer" containerID="6c8f71a8a42ef95dc2f04e381eb7f3e985c20e8d529bea0353bc78e5988c911b" Jul 16 16:27:36.668670 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:36.668654 2577 scope.go:117] "RemoveContainer" containerID="6c8f71a8a42ef95dc2f04e381eb7f3e985c20e8d529bea0353bc78e5988c911b" Jul 16 16:27:36.668927 ip-10-0-142-101 kubenswrapper[2577]: E0716 16:27:36.668909 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c8f71a8a42ef95dc2f04e381eb7f3e985c20e8d529bea0353bc78e5988c911b\": container with ID starting with 6c8f71a8a42ef95dc2f04e381eb7f3e985c20e8d529bea0353bc78e5988c911b not found: ID does not exist" containerID="6c8f71a8a42ef95dc2f04e381eb7f3e985c20e8d529bea0353bc78e5988c911b" Jul 16 16:27:36.668980 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:36.668938 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c8f71a8a42ef95dc2f04e381eb7f3e985c20e8d529bea0353bc78e5988c911b"} err="failed to get container status \"6c8f71a8a42ef95dc2f04e381eb7f3e985c20e8d529bea0353bc78e5988c911b\": rpc error: code = NotFound desc = could not find container \"6c8f71a8a42ef95dc2f04e381eb7f3e985c20e8d529bea0353bc78e5988c911b\": container with ID starting with 6c8f71a8a42ef95dc2f04e381eb7f3e985c20e8d529bea0353bc78e5988c911b not found: ID does not exist" Jul 16 16:27:36.678350 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:36.678328 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-b8c74fcc8-njh4j"] Jul 16 16:27:36.682398 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:36.682368 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-b8c74fcc8-njh4j"] Jul 16 16:27:38.020667 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:38.018327 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fc506c26-ecf4-488c-a9aa-a7b52e2d04e9" path="/var/lib/kubelet/pods/fc506c26-ecf4-488c-a9aa-a7b52e2d04e9/volumes" Jul 16 16:27:44.114693 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:44.114660 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-6fd5bb7967-8t82m" Jul 16 16:27:44.118492 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:44.118465 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-6fd5bb7967-8t82m" Jul 16 16:27:46.451116 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:46.451071 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-76dcf44fd5-tx9p2" podUID="744cb445-c1cf-423d-9f10-7f00df75a709" containerName="console" containerID="cri-o://bbfeeaa3bedd1cc9ca63fa6e08dd5157028e0aa1eddb4fd8c35a4e6564af8cde" gracePeriod=15 Jul 16 16:27:46.702288 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:46.702222 2577 generic.go:356] "Generic (PLEG): container finished" podID="744cb445-c1cf-423d-9f10-7f00df75a709" containerID="bbfeeaa3bedd1cc9ca63fa6e08dd5157028e0aa1eddb4fd8c35a4e6564af8cde" exitCode=0 Jul 16 16:27:46.702417 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:46.702292 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-76dcf44fd5-tx9p2" event={"ID":"744cb445-c1cf-423d-9f10-7f00df75a709","Type":"ContainerDied","Data":"bbfeeaa3bedd1cc9ca63fa6e08dd5157028e0aa1eddb4fd8c35a4e6564af8cde"} Jul 16 16:27:46.702417 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:46.702342 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-76dcf44fd5-tx9p2" event={"ID":"744cb445-c1cf-423d-9f10-7f00df75a709","Type":"ContainerDied","Data":"07f74b231b30f0acd150d2e4126367cc29ebabeee1554d9e3d92d8033981f773"} Jul 16 16:27:46.702417 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:46.702357 2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="07f74b231b30f0acd150d2e4126367cc29ebabeee1554d9e3d92d8033981f773" Jul 16 16:27:46.708112 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:46.708092 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-76dcf44fd5-tx9p2" Jul 16 16:27:46.853632 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:46.853598 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/744cb445-c1cf-423d-9f10-7f00df75a709-console-oauth-config\") pod \"744cb445-c1cf-423d-9f10-7f00df75a709\" (UID: \"744cb445-c1cf-423d-9f10-7f00df75a709\") " Jul 16 16:27:46.853632 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:46.853633 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/744cb445-c1cf-423d-9f10-7f00df75a709-service-ca\") pod \"744cb445-c1cf-423d-9f10-7f00df75a709\" (UID: \"744cb445-c1cf-423d-9f10-7f00df75a709\") " Jul 16 16:27:46.853888 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:46.853668 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/744cb445-c1cf-423d-9f10-7f00df75a709-console-serving-cert\") pod \"744cb445-c1cf-423d-9f10-7f00df75a709\" (UID: \"744cb445-c1cf-423d-9f10-7f00df75a709\") " Jul 16 16:27:46.853888 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:46.853794 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4p9nn\" (UniqueName: \"kubernetes.io/projected/744cb445-c1cf-423d-9f10-7f00df75a709-kube-api-access-4p9nn\") pod \"744cb445-c1cf-423d-9f10-7f00df75a709\" (UID: \"744cb445-c1cf-423d-9f10-7f00df75a709\") " Jul 16 16:27:46.853888 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:46.853843 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/744cb445-c1cf-423d-9f10-7f00df75a709-console-config\") pod \"744cb445-c1cf-423d-9f10-7f00df75a709\" (UID: \"744cb445-c1cf-423d-9f10-7f00df75a709\") " Jul 16 16:27:46.853888 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:46.853881 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/744cb445-c1cf-423d-9f10-7f00df75a709-oauth-serving-cert\") pod \"744cb445-c1cf-423d-9f10-7f00df75a709\" (UID: \"744cb445-c1cf-423d-9f10-7f00df75a709\") " Jul 16 16:27:46.854077 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:46.853914 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/744cb445-c1cf-423d-9f10-7f00df75a709-trusted-ca-bundle\") pod \"744cb445-c1cf-423d-9f10-7f00df75a709\" (UID: \"744cb445-c1cf-423d-9f10-7f00df75a709\") " Jul 16 16:27:46.854077 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:46.854006 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/744cb445-c1cf-423d-9f10-7f00df75a709-service-ca" (OuterVolumeSpecName: "service-ca") pod "744cb445-c1cf-423d-9f10-7f00df75a709" (UID: "744cb445-c1cf-423d-9f10-7f00df75a709"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 16:27:46.854245 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:46.854224 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/744cb445-c1cf-423d-9f10-7f00df75a709-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "744cb445-c1cf-423d-9f10-7f00df75a709" (UID: "744cb445-c1cf-423d-9f10-7f00df75a709"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 16:27:46.854320 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:46.854273 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/744cb445-c1cf-423d-9f10-7f00df75a709-console-config" (OuterVolumeSpecName: "console-config") pod "744cb445-c1cf-423d-9f10-7f00df75a709" (UID: "744cb445-c1cf-423d-9f10-7f00df75a709"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 16:27:46.854410 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:46.854320 2577 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/744cb445-c1cf-423d-9f10-7f00df75a709-service-ca\") on node \"ip-10-0-142-101.ec2.internal\" DevicePath \"\"" Jul 16 16:27:46.854410 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:46.854340 2577 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/744cb445-c1cf-423d-9f10-7f00df75a709-oauth-serving-cert\") on node \"ip-10-0-142-101.ec2.internal\" DevicePath \"\"" Jul 16 16:27:46.854410 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:46.854366 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/744cb445-c1cf-423d-9f10-7f00df75a709-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "744cb445-c1cf-423d-9f10-7f00df75a709" (UID: "744cb445-c1cf-423d-9f10-7f00df75a709"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 16:27:46.855836 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:46.855817 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/744cb445-c1cf-423d-9f10-7f00df75a709-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "744cb445-c1cf-423d-9f10-7f00df75a709" (UID: "744cb445-c1cf-423d-9f10-7f00df75a709"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 16:27:46.856267 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:46.856248 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/744cb445-c1cf-423d-9f10-7f00df75a709-kube-api-access-4p9nn" (OuterVolumeSpecName: "kube-api-access-4p9nn") pod "744cb445-c1cf-423d-9f10-7f00df75a709" (UID: "744cb445-c1cf-423d-9f10-7f00df75a709"). InnerVolumeSpecName "kube-api-access-4p9nn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 16:27:46.856314 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:46.856269 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/744cb445-c1cf-423d-9f10-7f00df75a709-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "744cb445-c1cf-423d-9f10-7f00df75a709" (UID: "744cb445-c1cf-423d-9f10-7f00df75a709"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 16:27:46.955681 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:46.955592 2577 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/744cb445-c1cf-423d-9f10-7f00df75a709-console-oauth-config\") on node \"ip-10-0-142-101.ec2.internal\" DevicePath \"\"" Jul 16 16:27:46.955681 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:46.955625 2577 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/744cb445-c1cf-423d-9f10-7f00df75a709-console-serving-cert\") on node \"ip-10-0-142-101.ec2.internal\" DevicePath \"\"" Jul 16 16:27:46.955681 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:46.955635 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4p9nn\" (UniqueName: \"kubernetes.io/projected/744cb445-c1cf-423d-9f10-7f00df75a709-kube-api-access-4p9nn\") on node \"ip-10-0-142-101.ec2.internal\" DevicePath \"\"" Jul 16 16:27:46.955681 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:46.955644 2577 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/744cb445-c1cf-423d-9f10-7f00df75a709-console-config\") on node \"ip-10-0-142-101.ec2.internal\" DevicePath \"\"" Jul 16 16:27:46.955681 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:46.955665 2577 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/744cb445-c1cf-423d-9f10-7f00df75a709-trusted-ca-bundle\") on node \"ip-10-0-142-101.ec2.internal\" DevicePath \"\"" Jul 16 16:27:47.706643 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:47.706607 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-76dcf44fd5-tx9p2" Jul 16 16:27:47.732455 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:47.732427 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-76dcf44fd5-tx9p2"] Jul 16 16:27:47.736494 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:47.736468 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-76dcf44fd5-tx9p2"] Jul 16 16:27:48.017091 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:27:48.017016 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="744cb445-c1cf-423d-9f10-7f00df75a709" path="/var/lib/kubelet/pods/744cb445-c1cf-423d-9f10-7f00df75a709/volumes" Jul 16 16:28:06.317864 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:28:06.317821 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Jul 16 16:28:06.338041 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:28:06.338015 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Jul 16 16:28:06.800099 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:28:06.800064 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 16 16:29:47.529130 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:29:47.529094 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-42m2m"] Jul 16 16:29:47.529599 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:29:47.529451 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fc506c26-ecf4-488c-a9aa-a7b52e2d04e9" containerName="console" Jul 16 16:29:47.529599 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:29:47.529464 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="fc506c26-ecf4-488c-a9aa-a7b52e2d04e9" containerName="console" Jul 16 16:29:47.529599 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:29:47.529478 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cabb304d-fabd-4944-87f1-b7deccca8616" containerName="console" Jul 16 16:29:47.529599 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:29:47.529484 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="cabb304d-fabd-4944-87f1-b7deccca8616" containerName="console" Jul 16 16:29:47.529599 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:29:47.529498 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="744cb445-c1cf-423d-9f10-7f00df75a709" containerName="console" Jul 16 16:29:47.529599 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:29:47.529505 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="744cb445-c1cf-423d-9f10-7f00df75a709" containerName="console" Jul 16 16:29:47.529599 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:29:47.529560 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="cabb304d-fabd-4944-87f1-b7deccca8616" containerName="console" Jul 16 16:29:47.529599 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:29:47.529572 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="fc506c26-ecf4-488c-a9aa-a7b52e2d04e9" containerName="console" Jul 16 16:29:47.529599 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:29:47.529578 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="744cb445-c1cf-423d-9f10-7f00df75a709" containerName="console" Jul 16 16:29:47.532617 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:29:47.532600 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-42m2m" Jul 16 16:29:47.537483 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:29:47.537462 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Jul 16 16:29:47.539768 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:29:47.539745 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Jul 16 16:29:47.540655 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:29:47.540638 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-7bwsf\"" Jul 16 16:29:47.561289 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:29:47.561254 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-42m2m"] Jul 16 16:29:47.599534 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:29:47.599493 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pcbq9\" (UniqueName: \"kubernetes.io/projected/479e924c-b5f9-4a40-992d-41336dc361a5-kube-api-access-pcbq9\") pod \"cert-manager-webhook-6486ffd997-42m2m\" (UID: \"479e924c-b5f9-4a40-992d-41336dc361a5\") " pod="cert-manager/cert-manager-webhook-6486ffd997-42m2m" Jul 16 16:29:47.599724 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:29:47.599598 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/479e924c-b5f9-4a40-992d-41336dc361a5-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-42m2m\" (UID: \"479e924c-b5f9-4a40-992d-41336dc361a5\") " pod="cert-manager/cert-manager-webhook-6486ffd997-42m2m" Jul 16 16:29:47.700365 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:29:47.700333 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pcbq9\" (UniqueName: \"kubernetes.io/projected/479e924c-b5f9-4a40-992d-41336dc361a5-kube-api-access-pcbq9\") pod \"cert-manager-webhook-6486ffd997-42m2m\" (UID: \"479e924c-b5f9-4a40-992d-41336dc361a5\") " pod="cert-manager/cert-manager-webhook-6486ffd997-42m2m" Jul 16 16:29:47.700565 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:29:47.700402 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/479e924c-b5f9-4a40-992d-41336dc361a5-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-42m2m\" (UID: \"479e924c-b5f9-4a40-992d-41336dc361a5\") " pod="cert-manager/cert-manager-webhook-6486ffd997-42m2m" Jul 16 16:29:47.714507 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:29:47.714480 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pcbq9\" (UniqueName: \"kubernetes.io/projected/479e924c-b5f9-4a40-992d-41336dc361a5-kube-api-access-pcbq9\") pod \"cert-manager-webhook-6486ffd997-42m2m\" (UID: \"479e924c-b5f9-4a40-992d-41336dc361a5\") " pod="cert-manager/cert-manager-webhook-6486ffd997-42m2m" Jul 16 16:29:47.718025 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:29:47.717997 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/479e924c-b5f9-4a40-992d-41336dc361a5-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-42m2m\" (UID: \"479e924c-b5f9-4a40-992d-41336dc361a5\") " pod="cert-manager/cert-manager-webhook-6486ffd997-42m2m" Jul 16 16:29:47.850648 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:29:47.850603 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-42m2m" Jul 16 16:29:47.988959 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:29:47.988902 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-42m2m"] Jul 16 16:29:47.991302 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:29:47.991271 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod479e924c_b5f9_4a40_992d_41336dc361a5.slice/crio-dd16fcd408eef9b7183998a8be97b8e14f4086d0756c352fed2444d36ba71e72 WatchSource:0}: Error finding container dd16fcd408eef9b7183998a8be97b8e14f4086d0756c352fed2444d36ba71e72: Status 404 returned error can't find the container with id dd16fcd408eef9b7183998a8be97b8e14f4086d0756c352fed2444d36ba71e72 Jul 16 16:29:48.187203 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:29:48.187120 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-42m2m" event={"ID":"479e924c-b5f9-4a40-992d-41336dc361a5","Type":"ContainerStarted","Data":"dd16fcd408eef9b7183998a8be97b8e14f4086d0756c352fed2444d36ba71e72"} Jul 16 16:29:50.214615 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:29:50.214504 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-rm2lb"] Jul 16 16:29:50.218284 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:29:50.218260 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-rm2lb" Jul 16 16:29:50.221140 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:29:50.220915 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-6znfm\"" Jul 16 16:29:50.227525 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:29:50.227492 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-rm2lb"] Jul 16 16:29:50.325331 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:29:50.325283 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hmjsr\" (UniqueName: \"kubernetes.io/projected/b7d8a728-98dd-4e14-81eb-62d7d7e7021f-kube-api-access-hmjsr\") pod \"cert-manager-cainjector-559cdd8c87-rm2lb\" (UID: \"b7d8a728-98dd-4e14-81eb-62d7d7e7021f\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-rm2lb" Jul 16 16:29:50.325531 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:29:50.325400 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b7d8a728-98dd-4e14-81eb-62d7d7e7021f-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-rm2lb\" (UID: \"b7d8a728-98dd-4e14-81eb-62d7d7e7021f\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-rm2lb" Jul 16 16:29:50.426635 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:29:50.426589 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hmjsr\" (UniqueName: \"kubernetes.io/projected/b7d8a728-98dd-4e14-81eb-62d7d7e7021f-kube-api-access-hmjsr\") pod \"cert-manager-cainjector-559cdd8c87-rm2lb\" (UID: \"b7d8a728-98dd-4e14-81eb-62d7d7e7021f\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-rm2lb" Jul 16 16:29:50.426846 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:29:50.426666 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b7d8a728-98dd-4e14-81eb-62d7d7e7021f-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-rm2lb\" (UID: \"b7d8a728-98dd-4e14-81eb-62d7d7e7021f\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-rm2lb" Jul 16 16:29:50.436588 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:29:50.436525 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b7d8a728-98dd-4e14-81eb-62d7d7e7021f-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-rm2lb\" (UID: \"b7d8a728-98dd-4e14-81eb-62d7d7e7021f\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-rm2lb" Jul 16 16:29:50.436759 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:29:50.436726 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hmjsr\" (UniqueName: \"kubernetes.io/projected/b7d8a728-98dd-4e14-81eb-62d7d7e7021f-kube-api-access-hmjsr\") pod \"cert-manager-cainjector-559cdd8c87-rm2lb\" (UID: \"b7d8a728-98dd-4e14-81eb-62d7d7e7021f\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-rm2lb" Jul 16 16:29:50.534336 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:29:50.534244 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-rm2lb" Jul 16 16:29:50.701049 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:29:50.701022 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-rm2lb"] Jul 16 16:29:52.327421 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:29:52.327363 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb7d8a728_98dd_4e14_81eb_62d7d7e7021f.slice/crio-4a297369136b4fc6272cddf1ea60c3e8f1f95cf48064d7d96cc1f1ead6344448 WatchSource:0}: Error finding container 4a297369136b4fc6272cddf1ea60c3e8f1f95cf48064d7d96cc1f1ead6344448: Status 404 returned error can't find the container with id 4a297369136b4fc6272cddf1ea60c3e8f1f95cf48064d7d96cc1f1ead6344448 Jul 16 16:29:53.212222 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:29:53.212170 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-42m2m" event={"ID":"479e924c-b5f9-4a40-992d-41336dc361a5","Type":"ContainerStarted","Data":"2b08eeb5ef310eca3073b313397c51f7167a5ba35c9f048af3b63436b1bbab86"} Jul 16 16:29:53.212469 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:29:53.212363 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-42m2m" Jul 16 16:29:53.213649 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:29:53.213630 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-rm2lb" event={"ID":"b7d8a728-98dd-4e14-81eb-62d7d7e7021f","Type":"ContainerStarted","Data":"d3cd2720e792e4b1f96a73f0438b5a846670f4969e7cd63badd42f3aa1e3832a"} Jul 16 16:29:53.213712 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:29:53.213658 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-rm2lb" event={"ID":"b7d8a728-98dd-4e14-81eb-62d7d7e7021f","Type":"ContainerStarted","Data":"4a297369136b4fc6272cddf1ea60c3e8f1f95cf48064d7d96cc1f1ead6344448"} Jul 16 16:29:53.261436 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:29:53.261358 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-42m2m" podStartSLOduration=1.863973606 podStartE2EDuration="6.261341024s" totalImagesPullingTime="4.397367418s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 16:29:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 16:29:53.260496577 +0000 UTC m=+251.764554430" watchObservedRunningTime="2026-07-16 16:29:53.261341024 +0000 UTC m=+251.765398875" Jul 16 16:29:59.219825 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:29:59.219796 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-42m2m" Jul 16 16:29:59.266767 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:29:59.266710 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-rm2lb" podStartSLOduration=8.918522637 podStartE2EDuration="9.266691882s" totalImagesPullingTime="348.169245ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 16:29:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 16:29:53.291659291 +0000 UTC m=+251.795717154" watchObservedRunningTime="2026-07-16 16:29:59.266691882 +0000 UTC m=+257.770749732" Jul 16 16:30:20.568906 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:30:20.568822 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-846bd4c774-wlzp8"] Jul 16 16:30:20.573453 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:30:20.573430 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-846bd4c774-wlzp8" Jul 16 16:30:20.577423 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:30:20.577375 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 16 16:30:20.577573 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:30:20.577408 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 16 16:30:20.577573 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:30:20.577473 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-7fcqd\"" Jul 16 16:30:20.577667 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:30:20.577586 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 16 16:30:20.578089 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:30:20.578076 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 16 16:30:20.603802 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:30:20.603764 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/2dd9c4bf-dd03-4c43-8f5f-b5fa4a178b6d-manifests\") pod \"opendatahub-operator-controller-manager-846bd4c774-wlzp8\" (UID: \"2dd9c4bf-dd03-4c43-8f5f-b5fa4a178b6d\") " pod="opendatahub/opendatahub-operator-controller-manager-846bd4c774-wlzp8" Jul 16 16:30:20.603990 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:30:20.603870 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2dd9c4bf-dd03-4c43-8f5f-b5fa4a178b6d-webhook-cert\") pod \"opendatahub-operator-controller-manager-846bd4c774-wlzp8\" (UID: \"2dd9c4bf-dd03-4c43-8f5f-b5fa4a178b6d\") " pod="opendatahub/opendatahub-operator-controller-manager-846bd4c774-wlzp8" Jul 16 16:30:20.603990 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:30:20.603902 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2dd9c4bf-dd03-4c43-8f5f-b5fa4a178b6d-tmp\") pod \"opendatahub-operator-controller-manager-846bd4c774-wlzp8\" (UID: \"2dd9c4bf-dd03-4c43-8f5f-b5fa4a178b6d\") " pod="opendatahub/opendatahub-operator-controller-manager-846bd4c774-wlzp8" Jul 16 16:30:20.603990 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:30:20.603927 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/2dd9c4bf-dd03-4c43-8f5f-b5fa4a178b6d-apiservice-cert\") pod \"opendatahub-operator-controller-manager-846bd4c774-wlzp8\" (UID: \"2dd9c4bf-dd03-4c43-8f5f-b5fa4a178b6d\") " pod="opendatahub/opendatahub-operator-controller-manager-846bd4c774-wlzp8" Jul 16 16:30:20.604153 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:30:20.604035 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c465s\" (UniqueName: \"kubernetes.io/projected/2dd9c4bf-dd03-4c43-8f5f-b5fa4a178b6d-kube-api-access-c465s\") pod \"opendatahub-operator-controller-manager-846bd4c774-wlzp8\" (UID: \"2dd9c4bf-dd03-4c43-8f5f-b5fa4a178b6d\") " pod="opendatahub/opendatahub-operator-controller-manager-846bd4c774-wlzp8" Jul 16 16:30:20.609313 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:30:20.609284 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-846bd4c774-wlzp8"] Jul 16 16:30:20.704750 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:30:20.704705 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c465s\" (UniqueName: \"kubernetes.io/projected/2dd9c4bf-dd03-4c43-8f5f-b5fa4a178b6d-kube-api-access-c465s\") pod \"opendatahub-operator-controller-manager-846bd4c774-wlzp8\" (UID: \"2dd9c4bf-dd03-4c43-8f5f-b5fa4a178b6d\") " pod="opendatahub/opendatahub-operator-controller-manager-846bd4c774-wlzp8" Jul 16 16:30:20.704958 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:30:20.704763 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/2dd9c4bf-dd03-4c43-8f5f-b5fa4a178b6d-manifests\") pod \"opendatahub-operator-controller-manager-846bd4c774-wlzp8\" (UID: \"2dd9c4bf-dd03-4c43-8f5f-b5fa4a178b6d\") " pod="opendatahub/opendatahub-operator-controller-manager-846bd4c774-wlzp8" Jul 16 16:30:20.704958 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:30:20.704809 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2dd9c4bf-dd03-4c43-8f5f-b5fa4a178b6d-webhook-cert\") pod \"opendatahub-operator-controller-manager-846bd4c774-wlzp8\" (UID: \"2dd9c4bf-dd03-4c43-8f5f-b5fa4a178b6d\") " pod="opendatahub/opendatahub-operator-controller-manager-846bd4c774-wlzp8" Jul 16 16:30:20.704958 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:30:20.704827 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2dd9c4bf-dd03-4c43-8f5f-b5fa4a178b6d-tmp\") pod \"opendatahub-operator-controller-manager-846bd4c774-wlzp8\" (UID: \"2dd9c4bf-dd03-4c43-8f5f-b5fa4a178b6d\") " pod="opendatahub/opendatahub-operator-controller-manager-846bd4c774-wlzp8" Jul 16 16:30:20.704958 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:30:20.704850 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/2dd9c4bf-dd03-4c43-8f5f-b5fa4a178b6d-apiservice-cert\") pod \"opendatahub-operator-controller-manager-846bd4c774-wlzp8\" (UID: \"2dd9c4bf-dd03-4c43-8f5f-b5fa4a178b6d\") " pod="opendatahub/opendatahub-operator-controller-manager-846bd4c774-wlzp8" Jul 16 16:30:20.705242 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:30:20.705214 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/2dd9c4bf-dd03-4c43-8f5f-b5fa4a178b6d-manifests\") pod \"opendatahub-operator-controller-manager-846bd4c774-wlzp8\" (UID: \"2dd9c4bf-dd03-4c43-8f5f-b5fa4a178b6d\") " pod="opendatahub/opendatahub-operator-controller-manager-846bd4c774-wlzp8" Jul 16 16:30:20.707314 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:30:20.707279 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2dd9c4bf-dd03-4c43-8f5f-b5fa4a178b6d-tmp\") pod \"opendatahub-operator-controller-manager-846bd4c774-wlzp8\" (UID: \"2dd9c4bf-dd03-4c43-8f5f-b5fa4a178b6d\") " pod="opendatahub/opendatahub-operator-controller-manager-846bd4c774-wlzp8" Jul 16 16:30:20.707460 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:30:20.707424 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/2dd9c4bf-dd03-4c43-8f5f-b5fa4a178b6d-apiservice-cert\") pod \"opendatahub-operator-controller-manager-846bd4c774-wlzp8\" (UID: \"2dd9c4bf-dd03-4c43-8f5f-b5fa4a178b6d\") " pod="opendatahub/opendatahub-operator-controller-manager-846bd4c774-wlzp8" Jul 16 16:30:20.707617 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:30:20.707595 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2dd9c4bf-dd03-4c43-8f5f-b5fa4a178b6d-webhook-cert\") pod \"opendatahub-operator-controller-manager-846bd4c774-wlzp8\" (UID: \"2dd9c4bf-dd03-4c43-8f5f-b5fa4a178b6d\") " pod="opendatahub/opendatahub-operator-controller-manager-846bd4c774-wlzp8" Jul 16 16:30:20.714645 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:30:20.714613 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c465s\" (UniqueName: \"kubernetes.io/projected/2dd9c4bf-dd03-4c43-8f5f-b5fa4a178b6d-kube-api-access-c465s\") pod \"opendatahub-operator-controller-manager-846bd4c774-wlzp8\" (UID: \"2dd9c4bf-dd03-4c43-8f5f-b5fa4a178b6d\") " pod="opendatahub/opendatahub-operator-controller-manager-846bd4c774-wlzp8" Jul 16 16:30:20.884486 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:30:20.884396 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-846bd4c774-wlzp8" Jul 16 16:30:21.059076 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:30:21.059048 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-846bd4c774-wlzp8"] Jul 16 16:30:21.061429 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:30:21.061378 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2dd9c4bf_dd03_4c43_8f5f_b5fa4a178b6d.slice/crio-b924f990e79250c2a6c19d1695b664984f65886a2ac52b1b7e98c652117353cd WatchSource:0}: Error finding container b924f990e79250c2a6c19d1695b664984f65886a2ac52b1b7e98c652117353cd: Status 404 returned error can't find the container with id b924f990e79250c2a6c19d1695b664984f65886a2ac52b1b7e98c652117353cd Jul 16 16:30:21.327579 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:30:21.327541 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-846bd4c774-wlzp8" event={"ID":"2dd9c4bf-dd03-4c43-8f5f-b5fa4a178b6d","Type":"ContainerStarted","Data":"b924f990e79250c2a6c19d1695b664984f65886a2ac52b1b7e98c652117353cd"} Jul 16 16:30:24.345379 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:30:24.345339 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-846bd4c774-wlzp8" event={"ID":"2dd9c4bf-dd03-4c43-8f5f-b5fa4a178b6d","Type":"ContainerStarted","Data":"364df40543c9eb964415e0bf99522c84b60fb898c76991962eeeadb3f02ac5a8"} Jul 16 16:30:26.356099 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:30:26.356064 2577 generic.go:356] "Generic (PLEG): container finished" podID="2dd9c4bf-dd03-4c43-8f5f-b5fa4a178b6d" containerID="364df40543c9eb964415e0bf99522c84b60fb898c76991962eeeadb3f02ac5a8" exitCode=0 Jul 16 16:30:26.356529 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:30:26.356113 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-846bd4c774-wlzp8" event={"ID":"2dd9c4bf-dd03-4c43-8f5f-b5fa4a178b6d","Type":"ContainerDied","Data":"364df40543c9eb964415e0bf99522c84b60fb898c76991962eeeadb3f02ac5a8"} Jul 16 16:30:31.380321 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:30:31.380281 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-846bd4c774-wlzp8" event={"ID":"2dd9c4bf-dd03-4c43-8f5f-b5fa4a178b6d","Type":"ContainerStarted","Data":"2e3f18314cca19b8fdafafa8e1e8d01680796c96b354feaff4137a615a9a17a0"} Jul 16 16:30:31.380706 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:30:31.380522 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-846bd4c774-wlzp8" Jul 16 16:30:31.418297 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:30:31.418243 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-846bd4c774-wlzp8" podStartSLOduration=3.160097587 podStartE2EDuration="11.418225953s" totalImagesPullingTime="7.188964166s" totalInitContainerRuntime="1.0691642s" isStatefulPod=false podCreationTimestamp="2026-07-16 16:30:20 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 16:30:31.415967632 +0000 UTC m=+289.920025473" watchObservedRunningTime="2026-07-16 16:30:31.418225953 +0000 UTC m=+289.922283803" Jul 16 16:30:41.898054 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:30:41.898019 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9bjst_b20b154f-1a4c-4b25-86db-aebe2d4ffc08/ovn-acl-logging/0.log" Jul 16 16:30:41.898589 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:30:41.898570 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9bjst_b20b154f-1a4c-4b25-86db-aebe2d4ffc08/ovn-acl-logging/0.log" Jul 16 16:30:41.901467 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:30:41.901370 2577 kubelet.go:1648] "Image garbage collection succeeded" Jul 16 16:30:42.387824 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:30:42.387795 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-846bd4c774-wlzp8" Jul 16 16:30:50.288497 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:30:50.288457 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-6697dd89c8-whsxg"] Jul 16 16:30:50.291919 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:30:50.291900 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-6697dd89c8-whsxg" Jul 16 16:30:50.294948 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:30:50.294926 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Jul 16 16:30:50.295063 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:30:50.294926 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 16 16:30:50.295656 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:30:50.295639 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Jul 16 16:30:50.295958 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:30:50.295939 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-cmt5n\"" Jul 16 16:30:50.296046 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:30:50.295974 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 16 16:30:50.296046 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:30:50.295944 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Jul 16 16:30:50.309104 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:30:50.309076 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-6697dd89c8-whsxg"] Jul 16 16:30:50.378230 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:30:50.378192 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bf9f4e29-a305-490b-b161-ae5f9890dd38-cert\") pod \"lws-controller-manager-6697dd89c8-whsxg\" (UID: \"bf9f4e29-a305-490b-b161-ae5f9890dd38\") " pod="openshift-lws-operator/lws-controller-manager-6697dd89c8-whsxg" Jul 16 16:30:50.378230 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:30:50.378231 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/bf9f4e29-a305-490b-b161-ae5f9890dd38-metrics-cert\") pod \"lws-controller-manager-6697dd89c8-whsxg\" (UID: \"bf9f4e29-a305-490b-b161-ae5f9890dd38\") " pod="openshift-lws-operator/lws-controller-manager-6697dd89c8-whsxg" Jul 16 16:30:50.378570 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:30:50.378315 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/bf9f4e29-a305-490b-b161-ae5f9890dd38-manager-config\") pod \"lws-controller-manager-6697dd89c8-whsxg\" (UID: \"bf9f4e29-a305-490b-b161-ae5f9890dd38\") " pod="openshift-lws-operator/lws-controller-manager-6697dd89c8-whsxg" Jul 16 16:30:50.378570 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:30:50.378487 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pgp54\" (UniqueName: \"kubernetes.io/projected/bf9f4e29-a305-490b-b161-ae5f9890dd38-kube-api-access-pgp54\") pod \"lws-controller-manager-6697dd89c8-whsxg\" (UID: \"bf9f4e29-a305-490b-b161-ae5f9890dd38\") " pod="openshift-lws-operator/lws-controller-manager-6697dd89c8-whsxg" Jul 16 16:30:50.479137 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:30:50.479098 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pgp54\" (UniqueName: \"kubernetes.io/projected/bf9f4e29-a305-490b-b161-ae5f9890dd38-kube-api-access-pgp54\") pod \"lws-controller-manager-6697dd89c8-whsxg\" (UID: \"bf9f4e29-a305-490b-b161-ae5f9890dd38\") " pod="openshift-lws-operator/lws-controller-manager-6697dd89c8-whsxg" Jul 16 16:30:50.479335 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:30:50.479153 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bf9f4e29-a305-490b-b161-ae5f9890dd38-cert\") pod \"lws-controller-manager-6697dd89c8-whsxg\" (UID: \"bf9f4e29-a305-490b-b161-ae5f9890dd38\") " pod="openshift-lws-operator/lws-controller-manager-6697dd89c8-whsxg" Jul 16 16:30:50.479335 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:30:50.479183 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/bf9f4e29-a305-490b-b161-ae5f9890dd38-metrics-cert\") pod \"lws-controller-manager-6697dd89c8-whsxg\" (UID: \"bf9f4e29-a305-490b-b161-ae5f9890dd38\") " pod="openshift-lws-operator/lws-controller-manager-6697dd89c8-whsxg" Jul 16 16:30:50.479335 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:30:50.479238 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/bf9f4e29-a305-490b-b161-ae5f9890dd38-manager-config\") pod \"lws-controller-manager-6697dd89c8-whsxg\" (UID: \"bf9f4e29-a305-490b-b161-ae5f9890dd38\") " pod="openshift-lws-operator/lws-controller-manager-6697dd89c8-whsxg" Jul 16 16:30:50.479910 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:30:50.479884 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/bf9f4e29-a305-490b-b161-ae5f9890dd38-manager-config\") pod \"lws-controller-manager-6697dd89c8-whsxg\" (UID: \"bf9f4e29-a305-490b-b161-ae5f9890dd38\") " pod="openshift-lws-operator/lws-controller-manager-6697dd89c8-whsxg" Jul 16 16:30:50.481683 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:30:50.481659 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/bf9f4e29-a305-490b-b161-ae5f9890dd38-metrics-cert\") pod \"lws-controller-manager-6697dd89c8-whsxg\" (UID: \"bf9f4e29-a305-490b-b161-ae5f9890dd38\") " pod="openshift-lws-operator/lws-controller-manager-6697dd89c8-whsxg" Jul 16 16:30:50.481788 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:30:50.481703 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bf9f4e29-a305-490b-b161-ae5f9890dd38-cert\") pod \"lws-controller-manager-6697dd89c8-whsxg\" (UID: \"bf9f4e29-a305-490b-b161-ae5f9890dd38\") " pod="openshift-lws-operator/lws-controller-manager-6697dd89c8-whsxg" Jul 16 16:30:50.488157 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:30:50.488135 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pgp54\" (UniqueName: \"kubernetes.io/projected/bf9f4e29-a305-490b-b161-ae5f9890dd38-kube-api-access-pgp54\") pod \"lws-controller-manager-6697dd89c8-whsxg\" (UID: \"bf9f4e29-a305-490b-b161-ae5f9890dd38\") " pod="openshift-lws-operator/lws-controller-manager-6697dd89c8-whsxg" Jul 16 16:30:50.602323 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:30:50.602275 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-6697dd89c8-whsxg" Jul 16 16:30:50.738815 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:30:50.738781 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-6697dd89c8-whsxg"] Jul 16 16:30:50.745492 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:30:50.745465 2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 16 16:30:51.475680 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:30:51.475637 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-6697dd89c8-whsxg" event={"ID":"bf9f4e29-a305-490b-b161-ae5f9890dd38","Type":"ContainerStarted","Data":"c9c33bd4f5d38c359a9428e957c9b3582819173b9e3d4f1cddf0585bf27c50fb"} Jul 16 16:30:53.487861 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:30:53.487822 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-6697dd89c8-whsxg" event={"ID":"bf9f4e29-a305-490b-b161-ae5f9890dd38","Type":"ContainerStarted","Data":"44517c4531893d16d51974567a0c723f4a65de10652061376b2135fce17291a7"} Jul 16 16:30:53.488282 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:30:53.487909 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-6697dd89c8-whsxg" Jul 16 16:30:53.515041 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:30:53.514982 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-6697dd89c8-whsxg" podStartSLOduration=1.354177418 podStartE2EDuration="3.514962851s" totalImagesPullingTime="2.160785433s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 16:30:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 16:30:53.513138453 +0000 UTC m=+312.017196306" watchObservedRunningTime="2026-07-16 16:30:53.514962851 +0000 UTC m=+312.019020704" Jul 16 16:30:55.488021 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:30:55.487988 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4hr8c9"] Jul 16 16:30:55.491670 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:30:55.491650 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4hr8c9" Jul 16 16:30:55.497908 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:30:55.497872 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-bg88q\"" Jul 16 16:30:55.498069 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:30:55.497922 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 16 16:30:55.498069 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:30:55.498031 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 16 16:30:55.498428 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:30:55.498409 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 16 16:30:55.568030 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:30:55.567997 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4hr8c9"] Jul 16 16:30:55.628845 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:30:55.628806 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1be39f1d-1d72-4e13-b292-8d7cc8389fc1-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4hr8c9\" (UID: \"1be39f1d-1d72-4e13-b292-8d7cc8389fc1\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4hr8c9" Jul 16 16:30:55.628845 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:30:55.628847 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1be39f1d-1d72-4e13-b292-8d7cc8389fc1-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4hr8c9\" (UID: \"1be39f1d-1d72-4e13-b292-8d7cc8389fc1\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4hr8c9" Jul 16 16:30:55.629089 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:30:55.628867 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1be39f1d-1d72-4e13-b292-8d7cc8389fc1-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4hr8c9\" (UID: \"1be39f1d-1d72-4e13-b292-8d7cc8389fc1\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4hr8c9" Jul 16 16:30:55.629089 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:30:55.628942 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pdzcz\" (UniqueName: \"kubernetes.io/projected/1be39f1d-1d72-4e13-b292-8d7cc8389fc1-kube-api-access-pdzcz\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4hr8c9\" (UID: \"1be39f1d-1d72-4e13-b292-8d7cc8389fc1\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4hr8c9" Jul 16 16:30:55.629089 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:30:55.628991 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1be39f1d-1d72-4e13-b292-8d7cc8389fc1-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4hr8c9\" (UID: \"1be39f1d-1d72-4e13-b292-8d7cc8389fc1\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4hr8c9" Jul 16 16:30:55.629089 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:30:55.629023 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1be39f1d-1d72-4e13-b292-8d7cc8389fc1-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4hr8c9\" (UID: \"1be39f1d-1d72-4e13-b292-8d7cc8389fc1\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4hr8c9" Jul 16 16:30:55.629089 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:30:55.629048 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1be39f1d-1d72-4e13-b292-8d7cc8389fc1-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4hr8c9\" (UID: \"1be39f1d-1d72-4e13-b292-8d7cc8389fc1\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4hr8c9" Jul 16 16:30:55.629274 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:30:55.629123 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1be39f1d-1d72-4e13-b292-8d7cc8389fc1-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4hr8c9\" (UID: \"1be39f1d-1d72-4e13-b292-8d7cc8389fc1\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4hr8c9" Jul 16 16:30:55.629274 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:30:55.629145 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1be39f1d-1d72-4e13-b292-8d7cc8389fc1-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4hr8c9\" (UID: \"1be39f1d-1d72-4e13-b292-8d7cc8389fc1\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4hr8c9" Jul 16 16:30:55.729722 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:30:55.729681 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1be39f1d-1d72-4e13-b292-8d7cc8389fc1-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4hr8c9\" (UID: \"1be39f1d-1d72-4e13-b292-8d7cc8389fc1\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4hr8c9" Jul 16 16:30:55.729916 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:30:55.729731 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1be39f1d-1d72-4e13-b292-8d7cc8389fc1-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4hr8c9\" (UID: \"1be39f1d-1d72-4e13-b292-8d7cc8389fc1\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4hr8c9" Jul 16 16:30:55.729916 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:30:55.729769 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1be39f1d-1d72-4e13-b292-8d7cc8389fc1-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4hr8c9\" (UID: \"1be39f1d-1d72-4e13-b292-8d7cc8389fc1\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4hr8c9" Jul 16 16:30:55.729916 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:30:55.729795 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1be39f1d-1d72-4e13-b292-8d7cc8389fc1-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4hr8c9\" (UID: \"1be39f1d-1d72-4e13-b292-8d7cc8389fc1\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4hr8c9" Jul 16 16:30:55.730073 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:30:55.729977 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1be39f1d-1d72-4e13-b292-8d7cc8389fc1-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4hr8c9\" (UID: \"1be39f1d-1d72-4e13-b292-8d7cc8389fc1\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4hr8c9" Jul 16 16:30:55.730073 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:30:55.730029 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1be39f1d-1d72-4e13-b292-8d7cc8389fc1-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4hr8c9\" (UID: \"1be39f1d-1d72-4e13-b292-8d7cc8389fc1\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4hr8c9" Jul 16 16:30:55.730073 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:30:55.730049 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1be39f1d-1d72-4e13-b292-8d7cc8389fc1-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4hr8c9\" (UID: \"1be39f1d-1d72-4e13-b292-8d7cc8389fc1\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4hr8c9" Jul 16 16:30:55.730258 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:30:55.730092 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pdzcz\" (UniqueName: \"kubernetes.io/projected/1be39f1d-1d72-4e13-b292-8d7cc8389fc1-kube-api-access-pdzcz\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4hr8c9\" (UID: \"1be39f1d-1d72-4e13-b292-8d7cc8389fc1\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4hr8c9" Jul 16 16:30:55.730258 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:30:55.730159 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1be39f1d-1d72-4e13-b292-8d7cc8389fc1-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4hr8c9\" (UID: \"1be39f1d-1d72-4e13-b292-8d7cc8389fc1\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4hr8c9" Jul 16 16:30:55.730558 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:30:55.730534 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1be39f1d-1d72-4e13-b292-8d7cc8389fc1-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4hr8c9\" (UID: \"1be39f1d-1d72-4e13-b292-8d7cc8389fc1\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4hr8c9" Jul 16 16:30:55.730758 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:30:55.730730 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1be39f1d-1d72-4e13-b292-8d7cc8389fc1-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4hr8c9\" (UID: \"1be39f1d-1d72-4e13-b292-8d7cc8389fc1\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4hr8c9" Jul 16 16:30:55.730859 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:30:55.730797 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1be39f1d-1d72-4e13-b292-8d7cc8389fc1-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4hr8c9\" (UID: \"1be39f1d-1d72-4e13-b292-8d7cc8389fc1\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4hr8c9" Jul 16 16:30:55.730859 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:30:55.730755 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1be39f1d-1d72-4e13-b292-8d7cc8389fc1-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4hr8c9\" (UID: \"1be39f1d-1d72-4e13-b292-8d7cc8389fc1\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4hr8c9" Jul 16 16:30:55.730859 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:30:55.730735 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1be39f1d-1d72-4e13-b292-8d7cc8389fc1-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4hr8c9\" (UID: \"1be39f1d-1d72-4e13-b292-8d7cc8389fc1\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4hr8c9" Jul 16 16:30:55.732415 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:30:55.732373 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1be39f1d-1d72-4e13-b292-8d7cc8389fc1-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4hr8c9\" (UID: \"1be39f1d-1d72-4e13-b292-8d7cc8389fc1\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4hr8c9" Jul 16 16:30:55.732470 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:30:55.732379 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1be39f1d-1d72-4e13-b292-8d7cc8389fc1-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4hr8c9\" (UID: \"1be39f1d-1d72-4e13-b292-8d7cc8389fc1\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4hr8c9" Jul 16 16:30:55.746513 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:30:55.746444 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1be39f1d-1d72-4e13-b292-8d7cc8389fc1-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4hr8c9\" (UID: \"1be39f1d-1d72-4e13-b292-8d7cc8389fc1\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4hr8c9" Jul 16 16:30:55.765744 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:30:55.765709 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pdzcz\" (UniqueName: \"kubernetes.io/projected/1be39f1d-1d72-4e13-b292-8d7cc8389fc1-kube-api-access-pdzcz\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4hr8c9\" (UID: \"1be39f1d-1d72-4e13-b292-8d7cc8389fc1\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4hr8c9" Jul 16 16:30:55.803648 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:30:55.803602 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4hr8c9" Jul 16 16:30:56.037205 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:30:56.037130 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-j5pgj"] Jul 16 16:30:56.041799 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:30:56.041776 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-j5pgj" Jul 16 16:30:56.048638 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:30:56.048595 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Jul 16 16:30:56.050571 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:30:56.050546 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Jul 16 16:30:56.050865 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:30:56.050768 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-wsbgv\"" Jul 16 16:30:56.066425 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:30:56.066379 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Jul 16 16:30:56.107431 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:30:56.107355 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1be39f1d_1d72_4e13_b292_8d7cc8389fc1.slice/crio-598e935a732dcb0af44c5bc82ee0b1df867aafec6b5d7717379a3add8a8b1852 WatchSource:0}: Error finding container 598e935a732dcb0af44c5bc82ee0b1df867aafec6b5d7717379a3add8a8b1852: Status 404 returned error can't find the container with id 598e935a732dcb0af44c5bc82ee0b1df867aafec6b5d7717379a3add8a8b1852 Jul 16 16:30:56.109114 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:30:56.109085 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4hr8c9"] Jul 16 16:30:56.111632 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:30:56.111607 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-j5pgj"] Jul 16 16:30:56.134405 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:30:56.134355 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/f161514a-da71-4244-88f3-1a42a7c4a87d-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-j5pgj\" (UID: \"f161514a-da71-4244-88f3-1a42a7c4a87d\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-j5pgj" Jul 16 16:30:56.134584 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:30:56.134475 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zk2kr\" (UniqueName: \"kubernetes.io/projected/f161514a-da71-4244-88f3-1a42a7c4a87d-kube-api-access-zk2kr\") pod \"istiod-openshift-gateway-94bb8fbfd-j5pgj\" (UID: \"f161514a-da71-4244-88f3-1a42a7c4a87d\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-j5pgj" Jul 16 16:30:56.134584 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:30:56.134566 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/f161514a-da71-4244-88f3-1a42a7c4a87d-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-j5pgj\" (UID: \"f161514a-da71-4244-88f3-1a42a7c4a87d\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-j5pgj" Jul 16 16:30:56.134682 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:30:56.134661 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/f161514a-da71-4244-88f3-1a42a7c4a87d-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-j5pgj\" (UID: \"f161514a-da71-4244-88f3-1a42a7c4a87d\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-j5pgj" Jul 16 16:30:56.134733 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:30:56.134718 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f161514a-da71-4244-88f3-1a42a7c4a87d-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-j5pgj\" (UID: \"f161514a-da71-4244-88f3-1a42a7c4a87d\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-j5pgj" Jul 16 16:30:56.134772 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:30:56.134736 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/f161514a-da71-4244-88f3-1a42a7c4a87d-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-j5pgj\" (UID: \"f161514a-da71-4244-88f3-1a42a7c4a87d\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-j5pgj" Jul 16 16:30:56.134772 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:30:56.134758 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f161514a-da71-4244-88f3-1a42a7c4a87d-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-j5pgj\" (UID: \"f161514a-da71-4244-88f3-1a42a7c4a87d\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-j5pgj" Jul 16 16:30:56.235556 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:30:56.235521 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/f161514a-da71-4244-88f3-1a42a7c4a87d-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-j5pgj\" (UID: \"f161514a-da71-4244-88f3-1a42a7c4a87d\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-j5pgj" Jul 16 16:30:56.235761 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:30:56.235573 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zk2kr\" (UniqueName: \"kubernetes.io/projected/f161514a-da71-4244-88f3-1a42a7c4a87d-kube-api-access-zk2kr\") pod \"istiod-openshift-gateway-94bb8fbfd-j5pgj\" (UID: \"f161514a-da71-4244-88f3-1a42a7c4a87d\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-j5pgj" Jul 16 16:30:56.235761 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:30:56.235715 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/f161514a-da71-4244-88f3-1a42a7c4a87d-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-j5pgj\" (UID: \"f161514a-da71-4244-88f3-1a42a7c4a87d\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-j5pgj" Jul 16 16:30:56.235899 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:30:56.235818 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/f161514a-da71-4244-88f3-1a42a7c4a87d-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-j5pgj\" (UID: \"f161514a-da71-4244-88f3-1a42a7c4a87d\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-j5pgj" Jul 16 16:30:56.235899 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:30:56.235855 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f161514a-da71-4244-88f3-1a42a7c4a87d-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-j5pgj\" (UID: \"f161514a-da71-4244-88f3-1a42a7c4a87d\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-j5pgj" Jul 16 16:30:56.235899 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:30:56.235885 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/f161514a-da71-4244-88f3-1a42a7c4a87d-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-j5pgj\" (UID: \"f161514a-da71-4244-88f3-1a42a7c4a87d\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-j5pgj" Jul 16 16:30:56.236060 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:30:56.235920 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f161514a-da71-4244-88f3-1a42a7c4a87d-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-j5pgj\" (UID: \"f161514a-da71-4244-88f3-1a42a7c4a87d\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-j5pgj" Jul 16 16:30:56.236713 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:30:56.236682 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/f161514a-da71-4244-88f3-1a42a7c4a87d-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-j5pgj\" (UID: \"f161514a-da71-4244-88f3-1a42a7c4a87d\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-j5pgj" Jul 16 16:30:56.238227 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:30:56.238195 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/f161514a-da71-4244-88f3-1a42a7c4a87d-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-j5pgj\" (UID: \"f161514a-da71-4244-88f3-1a42a7c4a87d\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-j5pgj" Jul 16 16:30:56.238320 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:30:56.238202 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/f161514a-da71-4244-88f3-1a42a7c4a87d-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-j5pgj\" (UID: \"f161514a-da71-4244-88f3-1a42a7c4a87d\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-j5pgj" Jul 16 16:30:56.238320 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:30:56.238296 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/f161514a-da71-4244-88f3-1a42a7c4a87d-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-j5pgj\" (UID: \"f161514a-da71-4244-88f3-1a42a7c4a87d\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-j5pgj" Jul 16 16:30:56.238526 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:30:56.238509 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f161514a-da71-4244-88f3-1a42a7c4a87d-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-j5pgj\" (UID: \"f161514a-da71-4244-88f3-1a42a7c4a87d\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-j5pgj" Jul 16 16:30:56.250214 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:30:56.250188 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f161514a-da71-4244-88f3-1a42a7c4a87d-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-j5pgj\" (UID: \"f161514a-da71-4244-88f3-1a42a7c4a87d\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-j5pgj" Jul 16 16:30:56.250483 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:30:56.250455 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zk2kr\" (UniqueName: \"kubernetes.io/projected/f161514a-da71-4244-88f3-1a42a7c4a87d-kube-api-access-zk2kr\") pod \"istiod-openshift-gateway-94bb8fbfd-j5pgj\" (UID: \"f161514a-da71-4244-88f3-1a42a7c4a87d\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-j5pgj" Jul 16 16:30:56.352722 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:30:56.352689 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-j5pgj" Jul 16 16:30:56.499441 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:30:56.499403 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-j5pgj"] Jul 16 16:30:56.504144 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:30:56.504113 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf161514a_da71_4244_88f3_1a42a7c4a87d.slice/crio-f970e4e885841a2cf24269b009f21942c20b569263e19e005b396791db6b51ad WatchSource:0}: Error finding container f970e4e885841a2cf24269b009f21942c20b569263e19e005b396791db6b51ad: Status 404 returned error can't find the container with id f970e4e885841a2cf24269b009f21942c20b569263e19e005b396791db6b51ad Jul 16 16:30:56.504293 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:30:56.504265 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4hr8c9" event={"ID":"1be39f1d-1d72-4e13-b292-8d7cc8389fc1","Type":"ContainerStarted","Data":"598e935a732dcb0af44c5bc82ee0b1df867aafec6b5d7717379a3add8a8b1852"} Jul 16 16:30:57.511684 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:30:57.511641 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-j5pgj" event={"ID":"f161514a-da71-4244-88f3-1a42a7c4a87d","Type":"ContainerStarted","Data":"f970e4e885841a2cf24269b009f21942c20b569263e19e005b396791db6b51ad"} Jul 16 16:30:59.034040 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:30:59.034001 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 16 16:30:59.034333 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:30:59.034076 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 16 16:30:59.525785 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:30:59.525744 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-j5pgj" event={"ID":"f161514a-da71-4244-88f3-1a42a7c4a87d","Type":"ContainerStarted","Data":"d07677d6847fe31ae6bd30f357ecb10a03602354098b99c7d2229ae43e27c8f0"} Jul 16 16:30:59.525998 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:30:59.525826 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-j5pgj" Jul 16 16:30:59.602181 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:30:59.602120 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-j5pgj" podStartSLOduration=2.074759812 podStartE2EDuration="4.602097885s" totalImagesPullingTime="2.527338073s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 16:30:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 16:30:59.594180826 +0000 UTC m=+318.098238678" watchObservedRunningTime="2026-07-16 16:30:59.602097885 +0000 UTC m=+318.106155732" Jul 16 16:31:00.532064 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:31:00.532035 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-j5pgj" Jul 16 16:31:03.910153 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:31:03.910115 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 16 16:31:03.910506 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:31:03.910190 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 16 16:31:03.910506 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:31:03.910220 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 16 16:31:04.495578 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:31:04.495528 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-6697dd89c8-whsxg" Jul 16 16:31:04.557326 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:31:04.557288 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4hr8c9" event={"ID":"1be39f1d-1d72-4e13-b292-8d7cc8389fc1","Type":"ContainerStarted","Data":"3b039a1437263eeb03d7ac8e38048ad2f664fb5ea540225f8cc1ff319a61c78b"} Jul 16 16:31:04.596771 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:31:04.596701 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4hr8c9" podStartSLOduration=1.7977574760000001 podStartE2EDuration="9.596676065s" totalImagesPullingTime="7.798918589s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 16:30:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 16:31:04.591126789 +0000 UTC m=+323.095184662" watchObservedRunningTime="2026-07-16 16:31:04.596676065 +0000 UTC m=+323.100733917" Jul 16 16:31:04.804321 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:31:04.804235 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4hr8c9" Jul 16 16:31:04.809013 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:31:04.808990 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4hr8c9" Jul 16 16:31:05.562269 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:31:05.562239 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4hr8c9" Jul 16 16:31:05.563350 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:31:05.563331 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4hr8c9" Jul 16 16:32:20.701157 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:32:20.701123 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-67f5cb746f-bn5m2"] Jul 16 16:32:20.704567 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:32:20.704548 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-67f5cb746f-bn5m2" Jul 16 16:32:20.708840 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:32:20.708807 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 16 16:32:20.708981 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:32:20.708862 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 16 16:32:20.708981 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:32:20.708871 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 16 16:32:20.709129 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:32:20.709078 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-jx67x\"" Jul 16 16:32:20.712501 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:32:20.712480 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 16 16:32:20.719897 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:32:20.719871 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 16 16:32:20.720036 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:32:20.719920 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 16 16:32:20.720036 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:32:20.719920 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 16 16:32:20.728593 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:32:20.728568 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 16 16:32:20.747465 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:32:20.747433 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-67f5cb746f-bn5m2"] Jul 16 16:32:20.813588 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:32:20.813550 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5439ffd6-38a7-4d37-a229-38d02ee0e167-service-ca\") pod \"console-67f5cb746f-bn5m2\" (UID: \"5439ffd6-38a7-4d37-a229-38d02ee0e167\") " pod="openshift-console/console-67f5cb746f-bn5m2" Jul 16 16:32:20.813759 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:32:20.813655 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5439ffd6-38a7-4d37-a229-38d02ee0e167-oauth-serving-cert\") pod \"console-67f5cb746f-bn5m2\" (UID: \"5439ffd6-38a7-4d37-a229-38d02ee0e167\") " pod="openshift-console/console-67f5cb746f-bn5m2" Jul 16 16:32:20.813759 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:32:20.813692 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5439ffd6-38a7-4d37-a229-38d02ee0e167-console-oauth-config\") pod \"console-67f5cb746f-bn5m2\" (UID: \"5439ffd6-38a7-4d37-a229-38d02ee0e167\") " pod="openshift-console/console-67f5cb746f-bn5m2" Jul 16 16:32:20.813759 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:32:20.813714 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5439ffd6-38a7-4d37-a229-38d02ee0e167-console-config\") pod \"console-67f5cb746f-bn5m2\" (UID: \"5439ffd6-38a7-4d37-a229-38d02ee0e167\") " pod="openshift-console/console-67f5cb746f-bn5m2" Jul 16 16:32:20.813866 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:32:20.813771 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5439ffd6-38a7-4d37-a229-38d02ee0e167-trusted-ca-bundle\") pod \"console-67f5cb746f-bn5m2\" (UID: \"5439ffd6-38a7-4d37-a229-38d02ee0e167\") " pod="openshift-console/console-67f5cb746f-bn5m2" Jul 16 16:32:20.813866 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:32:20.813832 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5439ffd6-38a7-4d37-a229-38d02ee0e167-console-serving-cert\") pod \"console-67f5cb746f-bn5m2\" (UID: \"5439ffd6-38a7-4d37-a229-38d02ee0e167\") " pod="openshift-console/console-67f5cb746f-bn5m2" Jul 16 16:32:20.813931 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:32:20.813864 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gflxc\" (UniqueName: \"kubernetes.io/projected/5439ffd6-38a7-4d37-a229-38d02ee0e167-kube-api-access-gflxc\") pod \"console-67f5cb746f-bn5m2\" (UID: \"5439ffd6-38a7-4d37-a229-38d02ee0e167\") " pod="openshift-console/console-67f5cb746f-bn5m2" Jul 16 16:32:20.915162 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:32:20.915125 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5439ffd6-38a7-4d37-a229-38d02ee0e167-service-ca\") pod \"console-67f5cb746f-bn5m2\" (UID: \"5439ffd6-38a7-4d37-a229-38d02ee0e167\") " pod="openshift-console/console-67f5cb746f-bn5m2" Jul 16 16:32:20.915358 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:32:20.915182 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5439ffd6-38a7-4d37-a229-38d02ee0e167-oauth-serving-cert\") pod \"console-67f5cb746f-bn5m2\" (UID: \"5439ffd6-38a7-4d37-a229-38d02ee0e167\") " pod="openshift-console/console-67f5cb746f-bn5m2" Jul 16 16:32:20.915358 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:32:20.915204 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5439ffd6-38a7-4d37-a229-38d02ee0e167-console-oauth-config\") pod \"console-67f5cb746f-bn5m2\" (UID: \"5439ffd6-38a7-4d37-a229-38d02ee0e167\") " pod="openshift-console/console-67f5cb746f-bn5m2" Jul 16 16:32:20.915358 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:32:20.915219 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5439ffd6-38a7-4d37-a229-38d02ee0e167-console-config\") pod \"console-67f5cb746f-bn5m2\" (UID: \"5439ffd6-38a7-4d37-a229-38d02ee0e167\") " pod="openshift-console/console-67f5cb746f-bn5m2" Jul 16 16:32:20.915358 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:32:20.915253 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5439ffd6-38a7-4d37-a229-38d02ee0e167-trusted-ca-bundle\") pod \"console-67f5cb746f-bn5m2\" (UID: \"5439ffd6-38a7-4d37-a229-38d02ee0e167\") " pod="openshift-console/console-67f5cb746f-bn5m2" Jul 16 16:32:20.915358 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:32:20.915284 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5439ffd6-38a7-4d37-a229-38d02ee0e167-console-serving-cert\") pod \"console-67f5cb746f-bn5m2\" (UID: \"5439ffd6-38a7-4d37-a229-38d02ee0e167\") " pod="openshift-console/console-67f5cb746f-bn5m2" Jul 16 16:32:20.915358 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:32:20.915312 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gflxc\" (UniqueName: \"kubernetes.io/projected/5439ffd6-38a7-4d37-a229-38d02ee0e167-kube-api-access-gflxc\") pod \"console-67f5cb746f-bn5m2\" (UID: \"5439ffd6-38a7-4d37-a229-38d02ee0e167\") " pod="openshift-console/console-67f5cb746f-bn5m2" Jul 16 16:32:20.916078 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:32:20.916053 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5439ffd6-38a7-4d37-a229-38d02ee0e167-console-config\") pod \"console-67f5cb746f-bn5m2\" (UID: \"5439ffd6-38a7-4d37-a229-38d02ee0e167\") " pod="openshift-console/console-67f5cb746f-bn5m2" Jul 16 16:32:20.916133 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:32:20.916089 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5439ffd6-38a7-4d37-a229-38d02ee0e167-oauth-serving-cert\") pod \"console-67f5cb746f-bn5m2\" (UID: \"5439ffd6-38a7-4d37-a229-38d02ee0e167\") " pod="openshift-console/console-67f5cb746f-bn5m2" Jul 16 16:32:20.916133 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:32:20.916087 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5439ffd6-38a7-4d37-a229-38d02ee0e167-service-ca\") pod \"console-67f5cb746f-bn5m2\" (UID: \"5439ffd6-38a7-4d37-a229-38d02ee0e167\") " pod="openshift-console/console-67f5cb746f-bn5m2" Jul 16 16:32:20.916196 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:32:20.916179 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5439ffd6-38a7-4d37-a229-38d02ee0e167-trusted-ca-bundle\") pod \"console-67f5cb746f-bn5m2\" (UID: \"5439ffd6-38a7-4d37-a229-38d02ee0e167\") " pod="openshift-console/console-67f5cb746f-bn5m2" Jul 16 16:32:20.917853 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:32:20.917824 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5439ffd6-38a7-4d37-a229-38d02ee0e167-console-oauth-config\") pod \"console-67f5cb746f-bn5m2\" (UID: \"5439ffd6-38a7-4d37-a229-38d02ee0e167\") " pod="openshift-console/console-67f5cb746f-bn5m2" Jul 16 16:32:20.917989 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:32:20.917899 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5439ffd6-38a7-4d37-a229-38d02ee0e167-console-serving-cert\") pod \"console-67f5cb746f-bn5m2\" (UID: \"5439ffd6-38a7-4d37-a229-38d02ee0e167\") " pod="openshift-console/console-67f5cb746f-bn5m2" Jul 16 16:32:20.945007 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:32:20.944971 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gflxc\" (UniqueName: \"kubernetes.io/projected/5439ffd6-38a7-4d37-a229-38d02ee0e167-kube-api-access-gflxc\") pod \"console-67f5cb746f-bn5m2\" (UID: \"5439ffd6-38a7-4d37-a229-38d02ee0e167\") " pod="openshift-console/console-67f5cb746f-bn5m2" Jul 16 16:32:21.014734 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:32:21.014638 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-67f5cb746f-bn5m2" Jul 16 16:32:21.262433 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:32:21.262396 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-67f5cb746f-bn5m2"] Jul 16 16:32:21.265795 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:32:21.265739 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5439ffd6_38a7_4d37_a229_38d02ee0e167.slice/crio-38aba8e9fde216d1318425c4c68ed598849ebfdc81fc39c022abffc099ef20ea WatchSource:0}: Error finding container 38aba8e9fde216d1318425c4c68ed598849ebfdc81fc39c022abffc099ef20ea: Status 404 returned error can't find the container with id 38aba8e9fde216d1318425c4c68ed598849ebfdc81fc39c022abffc099ef20ea Jul 16 16:32:21.917476 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:32:21.917429 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-67f5cb746f-bn5m2" event={"ID":"5439ffd6-38a7-4d37-a229-38d02ee0e167","Type":"ContainerStarted","Data":"af284aefa4f403537aa3a61f550aa974024be8c06cdadd38cead8ff577ad3ae7"} Jul 16 16:32:21.917931 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:32:21.917485 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-67f5cb746f-bn5m2" event={"ID":"5439ffd6-38a7-4d37-a229-38d02ee0e167","Type":"ContainerStarted","Data":"38aba8e9fde216d1318425c4c68ed598849ebfdc81fc39c022abffc099ef20ea"} Jul 16 16:32:21.949520 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:32:21.949465 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-67f5cb746f-bn5m2" podStartSLOduration=1.9494454719999998 podStartE2EDuration="1.949445472s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 16:32:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 16:32:21.948776037 +0000 UTC m=+400.452833890" watchObservedRunningTime="2026-07-16 16:32:21.949445472 +0000 UTC m=+400.453503324" Jul 16 16:32:31.015846 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:32:31.015753 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-67f5cb746f-bn5m2" Jul 16 16:32:31.015846 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:32:31.015812 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-67f5cb746f-bn5m2" Jul 16 16:32:31.020528 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:32:31.020502 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-67f5cb746f-bn5m2" Jul 16 16:32:31.969142 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:32:31.969104 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-67f5cb746f-bn5m2" Jul 16 16:32:52.731133 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:32:52.731095 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-n4jzp"] Jul 16 16:32:52.734748 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:32:52.734730 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-n4jzp" Jul 16 16:32:52.738023 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:32:52.737990 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 16 16:32:52.738158 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:32:52.738036 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 16 16:32:52.738316 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:32:52.738301 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-r7nns\"" Jul 16 16:32:52.769097 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:32:52.769066 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-n4jzp"] Jul 16 16:32:52.827734 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:32:52.822350 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hpsz2\" (UniqueName: \"kubernetes.io/projected/3a820af3-e614-4661-88c6-9bc6828356f0-kube-api-access-hpsz2\") pod \"kuadrant-operator-controller-manager-86878c769f-n4jzp\" (UID: \"3a820af3-e614-4661-88c6-9bc6828356f0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-n4jzp" Jul 16 16:32:52.827734 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:32:52.822505 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3a820af3-e614-4661-88c6-9bc6828356f0-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-n4jzp\" (UID: \"3a820af3-e614-4661-88c6-9bc6828356f0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-n4jzp" Jul 16 16:32:52.923610 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:32:52.923564 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3a820af3-e614-4661-88c6-9bc6828356f0-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-n4jzp\" (UID: \"3a820af3-e614-4661-88c6-9bc6828356f0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-n4jzp" Jul 16 16:32:52.923818 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:32:52.923667 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hpsz2\" (UniqueName: \"kubernetes.io/projected/3a820af3-e614-4661-88c6-9bc6828356f0-kube-api-access-hpsz2\") pod \"kuadrant-operator-controller-manager-86878c769f-n4jzp\" (UID: \"3a820af3-e614-4661-88c6-9bc6828356f0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-n4jzp" Jul 16 16:32:52.924048 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:32:52.924020 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3a820af3-e614-4661-88c6-9bc6828356f0-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-n4jzp\" (UID: \"3a820af3-e614-4661-88c6-9bc6828356f0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-n4jzp" Jul 16 16:32:52.937048 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:32:52.937011 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hpsz2\" (UniqueName: \"kubernetes.io/projected/3a820af3-e614-4661-88c6-9bc6828356f0-kube-api-access-hpsz2\") pod \"kuadrant-operator-controller-manager-86878c769f-n4jzp\" (UID: \"3a820af3-e614-4661-88c6-9bc6828356f0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-n4jzp" Jul 16 16:32:53.045349 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:32:53.045265 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-n4jzp" Jul 16 16:32:53.210549 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:32:53.210507 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a820af3_e614_4661_88c6_9bc6828356f0.slice/crio-6465b00f272a0587161f595ad872ae8188c92aa06b7e6e5370b5d275c8948acb WatchSource:0}: Error finding container 6465b00f272a0587161f595ad872ae8188c92aa06b7e6e5370b5d275c8948acb: Status 404 returned error can't find the container with id 6465b00f272a0587161f595ad872ae8188c92aa06b7e6e5370b5d275c8948acb Jul 16 16:32:53.214040 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:32:53.214013 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-n4jzp"] Jul 16 16:32:54.073627 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:32:54.073593 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-n4jzp" event={"ID":"3a820af3-e614-4661-88c6-9bc6828356f0","Type":"ContainerStarted","Data":"6465b00f272a0587161f595ad872ae8188c92aa06b7e6e5370b5d275c8948acb"} Jul 16 16:33:01.113654 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:33:01.113621 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-n4jzp" event={"ID":"3a820af3-e614-4661-88c6-9bc6828356f0","Type":"ContainerStarted","Data":"1a8c708d4067d0733ee88f8784f739293c3a2b1fe10df522f07f35200c66ecec"} Jul 16 16:33:01.114034 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:33:01.113698 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-n4jzp" Jul 16 16:33:01.153164 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:33:01.153112 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-n4jzp" podStartSLOduration=1.763253459 podStartE2EDuration="9.153092719s" totalImagesPullingTime="7.38983926s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 16:32:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 16:33:01.151221862 +0000 UTC m=+439.655279712" watchObservedRunningTime="2026-07-16 16:33:01.153092719 +0000 UTC m=+439.657150627" Jul 16 16:33:12.122027 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:33:12.121936 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-n4jzp" Jul 16 16:33:41.921478 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:33:41.921443 2577 scope.go:117] "RemoveContainer" containerID="bbfeeaa3bedd1cc9ca63fa6e08dd5157028e0aa1eddb4fd8c35a4e6564af8cde" Jul 16 16:34:48.757932 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:34:48.757891 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-74584bb9f8-ht4px"] Jul 16 16:34:48.760835 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:34:48.760814 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-74584bb9f8-ht4px" Jul 16 16:34:48.763855 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:34:48.763834 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Jul 16 16:34:48.763966 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:34:48.763837 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-l77qm\"" Jul 16 16:34:48.772427 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:34:48.772368 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-74584bb9f8-ht4px"] Jul 16 16:34:48.836723 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:34:48.836677 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/998f1366-d52e-447f-8465-1dfc4b4e6c2e-plugins-config-volume\") pod \"payload-pre-processing-74584bb9f8-ht4px\" (UID: \"998f1366-d52e-447f-8465-1dfc4b4e6c2e\") " pod="openshift-ingress/payload-pre-processing-74584bb9f8-ht4px" Jul 16 16:34:48.836908 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:34:48.836824 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m7psf\" (UniqueName: \"kubernetes.io/projected/998f1366-d52e-447f-8465-1dfc4b4e6c2e-kube-api-access-m7psf\") pod \"payload-pre-processing-74584bb9f8-ht4px\" (UID: \"998f1366-d52e-447f-8465-1dfc4b4e6c2e\") " pod="openshift-ingress/payload-pre-processing-74584bb9f8-ht4px" Jul 16 16:34:48.938085 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:34:48.938049 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/998f1366-d52e-447f-8465-1dfc4b4e6c2e-plugins-config-volume\") pod \"payload-pre-processing-74584bb9f8-ht4px\" (UID: \"998f1366-d52e-447f-8465-1dfc4b4e6c2e\") " pod="openshift-ingress/payload-pre-processing-74584bb9f8-ht4px" Jul 16 16:34:48.938296 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:34:48.938139 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7psf\" (UniqueName: \"kubernetes.io/projected/998f1366-d52e-447f-8465-1dfc4b4e6c2e-kube-api-access-m7psf\") pod \"payload-pre-processing-74584bb9f8-ht4px\" (UID: \"998f1366-d52e-447f-8465-1dfc4b4e6c2e\") " pod="openshift-ingress/payload-pre-processing-74584bb9f8-ht4px" Jul 16 16:34:48.938903 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:34:48.938875 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/998f1366-d52e-447f-8465-1dfc4b4e6c2e-plugins-config-volume\") pod \"payload-pre-processing-74584bb9f8-ht4px\" (UID: \"998f1366-d52e-447f-8465-1dfc4b4e6c2e\") " pod="openshift-ingress/payload-pre-processing-74584bb9f8-ht4px" Jul 16 16:34:48.946610 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:34:48.946580 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m7psf\" (UniqueName: \"kubernetes.io/projected/998f1366-d52e-447f-8465-1dfc4b4e6c2e-kube-api-access-m7psf\") pod \"payload-pre-processing-74584bb9f8-ht4px\" (UID: \"998f1366-d52e-447f-8465-1dfc4b4e6c2e\") " pod="openshift-ingress/payload-pre-processing-74584bb9f8-ht4px" Jul 16 16:34:49.072706 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:34:49.072669 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-74584bb9f8-ht4px" Jul 16 16:34:49.210514 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:34:49.210476 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-74584bb9f8-ht4px"] Jul 16 16:34:49.213765 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:34:49.213728 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod998f1366_d52e_447f_8465_1dfc4b4e6c2e.slice/crio-4564da4d3ea8c9cb99f42049beed1d9b0226e65cf46e600f9308aad53128f0ef WatchSource:0}: Error finding container 4564da4d3ea8c9cb99f42049beed1d9b0226e65cf46e600f9308aad53128f0ef: Status 404 returned error can't find the container with id 4564da4d3ea8c9cb99f42049beed1d9b0226e65cf46e600f9308aad53128f0ef Jul 16 16:34:49.632954 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:34:49.632918 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-74584bb9f8-ht4px" event={"ID":"998f1366-d52e-447f-8465-1dfc4b4e6c2e","Type":"ContainerStarted","Data":"4564da4d3ea8c9cb99f42049beed1d9b0226e65cf46e600f9308aad53128f0ef"} Jul 16 16:34:53.654454 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:34:53.654419 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-74584bb9f8-ht4px" event={"ID":"998f1366-d52e-447f-8465-1dfc4b4e6c2e","Type":"ContainerStarted","Data":"90c4e723586a3130e21e200daefd44e60c773ec0ccc47ef6cfd41ed3e3be9c9e"} Jul 16 16:34:53.654945 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:34:53.654550 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-74584bb9f8-ht4px" Jul 16 16:34:53.673154 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:34:53.673093 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-74584bb9f8-ht4px" podStartSLOduration=2.263445969 podStartE2EDuration="5.673077901s" totalImagesPullingTime="3.409631932s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 16:34:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 16:34:53.671504965 +0000 UTC m=+552.175562820" watchObservedRunningTime="2026-07-16 16:34:53.673077901 +0000 UTC m=+552.177135752" Jul 16 16:35:04.661588 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:35:04.661549 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-74584bb9f8-ht4px" Jul 16 16:35:27.025850 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:35:27.025815 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-n76ht"] Jul 16 16:35:27.028525 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:35:27.028496 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-n76ht" Jul 16 16:35:27.031274 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:35:27.031252 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 16 16:35:27.032368 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:35:27.032341 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\"" Jul 16 16:35:27.032489 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:35:27.032431 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-p8r4s\"" Jul 16 16:35:27.032489 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:35:27.032476 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 16 16:35:27.041250 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:35:27.041228 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-n76ht"] Jul 16 16:35:27.102582 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:35:27.102535 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/416f1afc-880f-4e22-83da-bf4d8ad0e5d5-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-n76ht\" (UID: \"416f1afc-880f-4e22-83da-bf4d8ad0e5d5\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-n76ht" Jul 16 16:35:27.102582 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:35:27.102576 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bkk2w\" (UniqueName: \"kubernetes.io/projected/416f1afc-880f-4e22-83da-bf4d8ad0e5d5-kube-api-access-bkk2w\") pod \"e2e-trlp-test-simulated-kserve-598466548d-n76ht\" (UID: \"416f1afc-880f-4e22-83da-bf4d8ad0e5d5\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-n76ht" Jul 16 16:35:27.102822 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:35:27.102611 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/416f1afc-880f-4e22-83da-bf4d8ad0e5d5-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-n76ht\" (UID: \"416f1afc-880f-4e22-83da-bf4d8ad0e5d5\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-n76ht" Jul 16 16:35:27.102822 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:35:27.102675 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/416f1afc-880f-4e22-83da-bf4d8ad0e5d5-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-n76ht\" (UID: \"416f1afc-880f-4e22-83da-bf4d8ad0e5d5\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-n76ht" Jul 16 16:35:27.102822 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:35:27.102708 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/416f1afc-880f-4e22-83da-bf4d8ad0e5d5-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-n76ht\" (UID: \"416f1afc-880f-4e22-83da-bf4d8ad0e5d5\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-n76ht" Jul 16 16:35:27.102822 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:35:27.102740 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/416f1afc-880f-4e22-83da-bf4d8ad0e5d5-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-n76ht\" (UID: \"416f1afc-880f-4e22-83da-bf4d8ad0e5d5\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-n76ht" Jul 16 16:35:27.203590 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:35:27.203547 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/416f1afc-880f-4e22-83da-bf4d8ad0e5d5-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-n76ht\" (UID: \"416f1afc-880f-4e22-83da-bf4d8ad0e5d5\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-n76ht" Jul 16 16:35:27.203590 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:35:27.203593 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bkk2w\" (UniqueName: \"kubernetes.io/projected/416f1afc-880f-4e22-83da-bf4d8ad0e5d5-kube-api-access-bkk2w\") pod \"e2e-trlp-test-simulated-kserve-598466548d-n76ht\" (UID: \"416f1afc-880f-4e22-83da-bf4d8ad0e5d5\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-n76ht" Jul 16 16:35:27.203858 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:35:27.203708 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/416f1afc-880f-4e22-83da-bf4d8ad0e5d5-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-n76ht\" (UID: \"416f1afc-880f-4e22-83da-bf4d8ad0e5d5\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-n76ht" Jul 16 16:35:27.203858 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:35:27.203745 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/416f1afc-880f-4e22-83da-bf4d8ad0e5d5-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-n76ht\" (UID: \"416f1afc-880f-4e22-83da-bf4d8ad0e5d5\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-n76ht" Jul 16 16:35:27.203858 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:35:27.203768 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/416f1afc-880f-4e22-83da-bf4d8ad0e5d5-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-n76ht\" (UID: \"416f1afc-880f-4e22-83da-bf4d8ad0e5d5\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-n76ht" Jul 16 16:35:27.203858 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:35:27.203811 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/416f1afc-880f-4e22-83da-bf4d8ad0e5d5-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-n76ht\" (UID: \"416f1afc-880f-4e22-83da-bf4d8ad0e5d5\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-n76ht" Jul 16 16:35:27.204058 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:35:27.204039 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/416f1afc-880f-4e22-83da-bf4d8ad0e5d5-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-n76ht\" (UID: \"416f1afc-880f-4e22-83da-bf4d8ad0e5d5\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-n76ht" Jul 16 16:35:27.204109 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:35:27.204070 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/416f1afc-880f-4e22-83da-bf4d8ad0e5d5-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-n76ht\" (UID: \"416f1afc-880f-4e22-83da-bf4d8ad0e5d5\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-n76ht" Jul 16 16:35:27.204170 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:35:27.204145 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/416f1afc-880f-4e22-83da-bf4d8ad0e5d5-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-n76ht\" (UID: \"416f1afc-880f-4e22-83da-bf4d8ad0e5d5\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-n76ht" Jul 16 16:35:27.206040 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:35:27.206013 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/416f1afc-880f-4e22-83da-bf4d8ad0e5d5-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-n76ht\" (UID: \"416f1afc-880f-4e22-83da-bf4d8ad0e5d5\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-n76ht" Jul 16 16:35:27.206174 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:35:27.206158 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/416f1afc-880f-4e22-83da-bf4d8ad0e5d5-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-n76ht\" (UID: \"416f1afc-880f-4e22-83da-bf4d8ad0e5d5\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-n76ht" Jul 16 16:35:27.220274 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:35:27.220251 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bkk2w\" (UniqueName: \"kubernetes.io/projected/416f1afc-880f-4e22-83da-bf4d8ad0e5d5-kube-api-access-bkk2w\") pod \"e2e-trlp-test-simulated-kserve-598466548d-n76ht\" (UID: \"416f1afc-880f-4e22-83da-bf4d8ad0e5d5\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-n76ht" Jul 16 16:35:27.339358 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:35:27.339328 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-n76ht" Jul 16 16:35:27.479991 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:35:27.479966 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-n76ht"] Jul 16 16:35:27.482751 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:35:27.482725 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod416f1afc_880f_4e22_83da_bf4d8ad0e5d5.slice/crio-1195bd467fc58286594bd9ad9df0fce6d4308f1955e6eb5d7847202aa5b523d4 WatchSource:0}: Error finding container 1195bd467fc58286594bd9ad9df0fce6d4308f1955e6eb5d7847202aa5b523d4: Status 404 returned error can't find the container with id 1195bd467fc58286594bd9ad9df0fce6d4308f1955e6eb5d7847202aa5b523d4 Jul 16 16:35:27.822481 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:35:27.822435 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-n76ht" event={"ID":"416f1afc-880f-4e22-83da-bf4d8ad0e5d5","Type":"ContainerStarted","Data":"1195bd467fc58286594bd9ad9df0fce6d4308f1955e6eb5d7847202aa5b523d4"} Jul 16 16:35:28.829002 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:35:28.828969 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-n76ht" event={"ID":"416f1afc-880f-4e22-83da-bf4d8ad0e5d5","Type":"ContainerStarted","Data":"89db4b5438534ea74280e06093e93df657e786336a95c85cf92cdf9fbc51bb5e"} Jul 16 16:35:28.852217 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:35:28.852156 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-n76ht" podStartSLOduration=0.828960728 podStartE2EDuration="1.852137232s" totalImagesPullingTime="1.023176504s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 16:35:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 16:35:28.850460782 +0000 UTC m=+587.354518635" watchObservedRunningTime="2026-07-16 16:35:28.852137232 +0000 UTC m=+587.356195083" Jul 16 16:35:37.339644 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:35:37.339606 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-n76ht" Jul 16 16:35:37.340040 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:35:37.339668 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-n76ht" Jul 16 16:35:37.352394 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:35:37.352363 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-n76ht" Jul 16 16:35:37.891631 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:35:37.891602 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-n76ht" Jul 16 16:35:41.929152 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:35:41.929122 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9bjst_b20b154f-1a4c-4b25-86db-aebe2d4ffc08/ovn-acl-logging/0.log" Jul 16 16:35:41.929672 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:35:41.929656 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9bjst_b20b154f-1a4c-4b25-86db-aebe2d4ffc08/ovn-acl-logging/0.log" Jul 16 16:40:41.959720 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:40:41.959638 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9bjst_b20b154f-1a4c-4b25-86db-aebe2d4ffc08/ovn-acl-logging/0.log" Jul 16 16:40:41.962487 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:40:41.962465 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9bjst_b20b154f-1a4c-4b25-86db-aebe2d4ffc08/ovn-acl-logging/0.log" Jul 16 16:42:45.255635 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:42:45.255554 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-n4jzp"] Jul 16 16:42:45.257072 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:42:45.255781 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-n4jzp" podUID="3a820af3-e614-4661-88c6-9bc6828356f0" containerName="manager" containerID="cri-o://1a8c708d4067d0733ee88f8784f739293c3a2b1fe10df522f07f35200c66ecec" gracePeriod=10 Jul 16 16:42:47.619531 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:42:47.619505 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-n4jzp" Jul 16 16:42:47.707054 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:42:47.706972 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hpsz2\" (UniqueName: \"kubernetes.io/projected/3a820af3-e614-4661-88c6-9bc6828356f0-kube-api-access-hpsz2\") pod \"3a820af3-e614-4661-88c6-9bc6828356f0\" (UID: \"3a820af3-e614-4661-88c6-9bc6828356f0\") " Jul 16 16:42:47.707185 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:42:47.707057 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3a820af3-e614-4661-88c6-9bc6828356f0-extensions-socket-volume\") pod \"3a820af3-e614-4661-88c6-9bc6828356f0\" (UID: \"3a820af3-e614-4661-88c6-9bc6828356f0\") " Jul 16 16:42:47.707471 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:42:47.707440 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a820af3-e614-4661-88c6-9bc6828356f0-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "3a820af3-e614-4661-88c6-9bc6828356f0" (UID: "3a820af3-e614-4661-88c6-9bc6828356f0"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 16:42:47.709105 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:42:47.709081 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a820af3-e614-4661-88c6-9bc6828356f0-kube-api-access-hpsz2" (OuterVolumeSpecName: "kube-api-access-hpsz2") pod "3a820af3-e614-4661-88c6-9bc6828356f0" (UID: "3a820af3-e614-4661-88c6-9bc6828356f0"). InnerVolumeSpecName "kube-api-access-hpsz2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 16:42:47.808805 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:42:47.808767 2577 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3a820af3-e614-4661-88c6-9bc6828356f0-extensions-socket-volume\") on node \"ip-10-0-142-101.ec2.internal\" DevicePath \"\"" Jul 16 16:42:47.808805 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:42:47.808797 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hpsz2\" (UniqueName: \"kubernetes.io/projected/3a820af3-e614-4661-88c6-9bc6828356f0-kube-api-access-hpsz2\") on node \"ip-10-0-142-101.ec2.internal\" DevicePath \"\"" Jul 16 16:42:47.967794 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:42:47.967712 2577 generic.go:356] "Generic (PLEG): container finished" podID="3a820af3-e614-4661-88c6-9bc6828356f0" containerID="1a8c708d4067d0733ee88f8784f739293c3a2b1fe10df522f07f35200c66ecec" exitCode=0 Jul 16 16:42:47.967794 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:42:47.967752 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-n4jzp" Jul 16 16:42:47.967971 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:42:47.967791 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-n4jzp" event={"ID":"3a820af3-e614-4661-88c6-9bc6828356f0","Type":"ContainerDied","Data":"1a8c708d4067d0733ee88f8784f739293c3a2b1fe10df522f07f35200c66ecec"} Jul 16 16:42:47.967971 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:42:47.967845 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-n4jzp" event={"ID":"3a820af3-e614-4661-88c6-9bc6828356f0","Type":"ContainerDied","Data":"6465b00f272a0587161f595ad872ae8188c92aa06b7e6e5370b5d275c8948acb"} Jul 16 16:42:47.967971 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:42:47.967867 2577 scope.go:117] "RemoveContainer" containerID="1a8c708d4067d0733ee88f8784f739293c3a2b1fe10df522f07f35200c66ecec" Jul 16 16:42:47.978897 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:42:47.978882 2577 scope.go:117] "RemoveContainer" containerID="1a8c708d4067d0733ee88f8784f739293c3a2b1fe10df522f07f35200c66ecec" Jul 16 16:42:47.979155 ip-10-0-142-101 kubenswrapper[2577]: E0716 16:42:47.979134 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a8c708d4067d0733ee88f8784f739293c3a2b1fe10df522f07f35200c66ecec\": container with ID starting with 1a8c708d4067d0733ee88f8784f739293c3a2b1fe10df522f07f35200c66ecec not found: ID does not exist" containerID="1a8c708d4067d0733ee88f8784f739293c3a2b1fe10df522f07f35200c66ecec" Jul 16 16:42:47.979199 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:42:47.979164 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a8c708d4067d0733ee88f8784f739293c3a2b1fe10df522f07f35200c66ecec"} err="failed to get container status \"1a8c708d4067d0733ee88f8784f739293c3a2b1fe10df522f07f35200c66ecec\": rpc error: code = NotFound desc = could not find container \"1a8c708d4067d0733ee88f8784f739293c3a2b1fe10df522f07f35200c66ecec\": container with ID starting with 1a8c708d4067d0733ee88f8784f739293c3a2b1fe10df522f07f35200c66ecec not found: ID does not exist" Jul 16 16:42:47.993346 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:42:47.993324 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-n4jzp"] Jul 16 16:42:47.996269 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:42:47.996246 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-n4jzp"] Jul 16 16:42:48.017560 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:42:48.017532 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3a820af3-e614-4661-88c6-9bc6828356f0" path="/var/lib/kubelet/pods/3a820af3-e614-4661-88c6-9bc6828356f0/volumes" Jul 16 16:43:46.489872 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:43:46.489827 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8jf9w"] Jul 16 16:43:46.490431 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:43:46.490410 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3a820af3-e614-4661-88c6-9bc6828356f0" containerName="manager" Jul 16 16:43:46.490513 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:43:46.490434 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="3a820af3-e614-4661-88c6-9bc6828356f0" containerName="manager" Jul 16 16:43:46.490573 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:43:46.490559 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="3a820af3-e614-4661-88c6-9bc6828356f0" containerName="manager" Jul 16 16:43:46.492763 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:43:46.492742 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8jf9w" Jul 16 16:43:46.495306 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:43:46.495280 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 16 16:43:46.495684 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:43:46.495666 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-r7nns\"" Jul 16 16:43:46.496017 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:43:46.495995 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 16 16:43:46.505121 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:43:46.505095 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8jf9w"] Jul 16 16:43:46.561120 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:43:46.561079 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/30c25bef-1662-4e8f-8389-f075eda51313-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-8jf9w\" (UID: \"30c25bef-1662-4e8f-8389-f075eda51313\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8jf9w" Jul 16 16:43:46.561302 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:43:46.561169 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dh4np\" (UniqueName: \"kubernetes.io/projected/30c25bef-1662-4e8f-8389-f075eda51313-kube-api-access-dh4np\") pod \"kuadrant-operator-controller-manager-86878c769f-8jf9w\" (UID: \"30c25bef-1662-4e8f-8389-f075eda51313\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8jf9w" Jul 16 16:43:46.662310 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:43:46.662257 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dh4np\" (UniqueName: \"kubernetes.io/projected/30c25bef-1662-4e8f-8389-f075eda51313-kube-api-access-dh4np\") pod \"kuadrant-operator-controller-manager-86878c769f-8jf9w\" (UID: \"30c25bef-1662-4e8f-8389-f075eda51313\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8jf9w" Jul 16 16:43:46.662527 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:43:46.662367 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/30c25bef-1662-4e8f-8389-f075eda51313-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-8jf9w\" (UID: \"30c25bef-1662-4e8f-8389-f075eda51313\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8jf9w" Jul 16 16:43:46.662743 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:43:46.662715 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/30c25bef-1662-4e8f-8389-f075eda51313-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-8jf9w\" (UID: \"30c25bef-1662-4e8f-8389-f075eda51313\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8jf9w" Jul 16 16:43:46.670941 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:43:46.670910 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dh4np\" (UniqueName: \"kubernetes.io/projected/30c25bef-1662-4e8f-8389-f075eda51313-kube-api-access-dh4np\") pod \"kuadrant-operator-controller-manager-86878c769f-8jf9w\" (UID: \"30c25bef-1662-4e8f-8389-f075eda51313\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8jf9w" Jul 16 16:43:46.804053 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:43:46.803971 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8jf9w" Jul 16 16:43:46.943721 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:43:46.943653 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8jf9w"] Jul 16 16:43:46.948996 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:43:46.948965 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30c25bef_1662_4e8f_8389_f075eda51313.slice/crio-da43f71022d8c87d6ee4e2731b7c351da2ae828f185bee62868f5af580834002 WatchSource:0}: Error finding container da43f71022d8c87d6ee4e2731b7c351da2ae828f185bee62868f5af580834002: Status 404 returned error can't find the container with id da43f71022d8c87d6ee4e2731b7c351da2ae828f185bee62868f5af580834002 Jul 16 16:43:47.252341 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:43:47.252295 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8jf9w" event={"ID":"30c25bef-1662-4e8f-8389-f075eda51313","Type":"ContainerStarted","Data":"74bd0b35b9995af6091a66398d4fbf8a70d31bddfb5a02aa409b53b64fb0b1f3"} Jul 16 16:43:47.252341 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:43:47.252349 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8jf9w" event={"ID":"30c25bef-1662-4e8f-8389-f075eda51313","Type":"ContainerStarted","Data":"da43f71022d8c87d6ee4e2731b7c351da2ae828f185bee62868f5af580834002"} Jul 16 16:43:47.252582 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:43:47.252447 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8jf9w" Jul 16 16:43:47.275488 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:43:47.275433 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8jf9w" podStartSLOduration=1.275414515 podStartE2EDuration="1.275414515s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 16:43:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 16:43:47.273441522 +0000 UTC m=+1085.777499374" watchObservedRunningTime="2026-07-16 16:43:47.275414515 +0000 UTC m=+1085.779472365" Jul 16 16:43:58.260285 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:43:58.260249 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8jf9w" Jul 16 16:45:00.202457 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:45:00.202420 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29737005-t72lz"] Jul 16 16:45:00.206241 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:45:00.206223 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29737005-t72lz" Jul 16 16:45:00.210139 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:45:00.210112 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dssfh\"" Jul 16 16:45:00.215910 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:45:00.215887 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29737005-t72lz" Jul 16 16:45:00.223678 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:45:00.223649 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29737005-t72lz"] Jul 16 16:45:00.360310 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:45:00.360285 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29737005-t72lz"] Jul 16 16:45:00.363073 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:45:00.363043 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e211b39_1283_45ca_8b0d_6bce4939e466.slice/crio-4db021ed905e1345a36f3c1899e99bba109f72071ac2e192f2a98a32a3e33de7 WatchSource:0}: Error finding container 4db021ed905e1345a36f3c1899e99bba109f72071ac2e192f2a98a32a3e33de7: Status 404 returned error can't find the container with id 4db021ed905e1345a36f3c1899e99bba109f72071ac2e192f2a98a32a3e33de7 Jul 16 16:45:00.365038 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:45:00.365023 2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 16 16:45:00.605860 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:45:00.605821 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29737005-t72lz" event={"ID":"7e211b39-1283-45ca-8b0d-6bce4939e466","Type":"ContainerStarted","Data":"4db021ed905e1345a36f3c1899e99bba109f72071ac2e192f2a98a32a3e33de7"} Jul 16 16:45:03.622210 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:45:03.622116 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29737005-t72lz" event={"ID":"7e211b39-1283-45ca-8b0d-6bce4939e466","Type":"ContainerStarted","Data":"40a76db35c0bdb9424a4dfb593524d57db1b81d4b23afc235f6d3afd0c54a438"} Jul 16 16:45:03.640467 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:45:03.640405 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29737005-t72lz" podStartSLOduration=0.951543863 podStartE2EDuration="3.640366629s" totalImagesPullingTime="2.688822766s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 16:45:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 16:45:03.637262797 +0000 UTC m=+1162.141320674" watchObservedRunningTime="2026-07-16 16:45:03.640366629 +0000 UTC m=+1162.144424482" Jul 16 16:45:04.627931 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:45:04.627892 2577 generic.go:356] "Generic (PLEG): container finished" podID="7e211b39-1283-45ca-8b0d-6bce4939e466" containerID="40a76db35c0bdb9424a4dfb593524d57db1b81d4b23afc235f6d3afd0c54a438" exitCode=0 Jul 16 16:45:04.628308 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:45:04.627980 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29737005-t72lz" event={"ID":"7e211b39-1283-45ca-8b0d-6bce4939e466","Type":"ContainerDied","Data":"40a76db35c0bdb9424a4dfb593524d57db1b81d4b23afc235f6d3afd0c54a438"} Jul 16 16:45:05.768563 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:45:05.768531 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29737005-t72lz" Jul 16 16:45:06.638905 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:45:06.638868 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29737005-t72lz" event={"ID":"7e211b39-1283-45ca-8b0d-6bce4939e466","Type":"ContainerDied","Data":"4db021ed905e1345a36f3c1899e99bba109f72071ac2e192f2a98a32a3e33de7"} Jul 16 16:45:06.639076 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:45:06.638929 2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4db021ed905e1345a36f3c1899e99bba109f72071ac2e192f2a98a32a3e33de7" Jul 16 16:45:06.639076 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:45:06.638888 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29737005-t72lz" Jul 16 16:45:41.991572 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:45:41.991540 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9bjst_b20b154f-1a4c-4b25-86db-aebe2d4ffc08/ovn-acl-logging/0.log" Jul 16 16:45:41.995047 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:45:41.995027 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9bjst_b20b154f-1a4c-4b25-86db-aebe2d4ffc08/ovn-acl-logging/0.log" Jul 16 16:50:19.084924 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:50:19.084898 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-7a95ef74-openshift-default-c6b94f6c5-fndx2"] Jul 16 16:50:19.085303 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:50:19.085290 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7e211b39-1283-45ca-8b0d-6bce4939e466" containerName="cleanup" Jul 16 16:50:19.085303 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:50:19.085304 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="7e211b39-1283-45ca-8b0d-6bce4939e466" containerName="cleanup" Jul 16 16:50:19.085380 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:50:19.085371 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="7e211b39-1283-45ca-8b0d-6bce4939e466" containerName="cleanup" Jul 16 16:50:19.088549 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:50:19.088530 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-7a95ef74-openshift-default-c6b94f6c5-fndx2" Jul 16 16:50:19.091578 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:50:19.091558 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-7a95ef74-openshift-default-dockercfg-c6jrw\"" Jul 16 16:50:19.119345 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:50:19.119315 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-7a95ef74-openshift-default-c6b94f6c5-fndx2"] Jul 16 16:50:19.209572 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:50:19.209534 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12adb8c6-7644-4a72-8e70-7ef5b5306805-workload-certs\") pod \"e2e-ait-7a95ef74-openshift-default-c6b94f6c5-fndx2\" (UID: \"12adb8c6-7644-4a72-8e70-7ef5b5306805\") " pod="openshift-ingress/e2e-ait-7a95ef74-openshift-default-c6b94f6c5-fndx2" Jul 16 16:50:19.209572 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:50:19.209571 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12adb8c6-7644-4a72-8e70-7ef5b5306805-istio-podinfo\") pod \"e2e-ait-7a95ef74-openshift-default-c6b94f6c5-fndx2\" (UID: \"12adb8c6-7644-4a72-8e70-7ef5b5306805\") " pod="openshift-ingress/e2e-ait-7a95ef74-openshift-default-c6b94f6c5-fndx2" Jul 16 16:50:19.209782 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:50:19.209660 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12adb8c6-7644-4a72-8e70-7ef5b5306805-credential-socket\") pod \"e2e-ait-7a95ef74-openshift-default-c6b94f6c5-fndx2\" (UID: \"12adb8c6-7644-4a72-8e70-7ef5b5306805\") " pod="openshift-ingress/e2e-ait-7a95ef74-openshift-default-c6b94f6c5-fndx2" Jul 16 16:50:19.209782 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:50:19.209701 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12adb8c6-7644-4a72-8e70-7ef5b5306805-istio-data\") pod \"e2e-ait-7a95ef74-openshift-default-c6b94f6c5-fndx2\" (UID: \"12adb8c6-7644-4a72-8e70-7ef5b5306805\") " pod="openshift-ingress/e2e-ait-7a95ef74-openshift-default-c6b94f6c5-fndx2" Jul 16 16:50:19.209782 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:50:19.209739 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12adb8c6-7644-4a72-8e70-7ef5b5306805-workload-socket\") pod \"e2e-ait-7a95ef74-openshift-default-c6b94f6c5-fndx2\" (UID: \"12adb8c6-7644-4a72-8e70-7ef5b5306805\") " pod="openshift-ingress/e2e-ait-7a95ef74-openshift-default-c6b94f6c5-fndx2" Jul 16 16:50:19.209882 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:50:19.209784 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12adb8c6-7644-4a72-8e70-7ef5b5306805-istio-envoy\") pod \"e2e-ait-7a95ef74-openshift-default-c6b94f6c5-fndx2\" (UID: \"12adb8c6-7644-4a72-8e70-7ef5b5306805\") " pod="openshift-ingress/e2e-ait-7a95ef74-openshift-default-c6b94f6c5-fndx2" Jul 16 16:50:19.209882 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:50:19.209833 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12adb8c6-7644-4a72-8e70-7ef5b5306805-istiod-ca-cert\") pod \"e2e-ait-7a95ef74-openshift-default-c6b94f6c5-fndx2\" (UID: \"12adb8c6-7644-4a72-8e70-7ef5b5306805\") " pod="openshift-ingress/e2e-ait-7a95ef74-openshift-default-c6b94f6c5-fndx2" Jul 16 16:50:19.209882 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:50:19.209850 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-phfzm\" (UniqueName: \"kubernetes.io/projected/12adb8c6-7644-4a72-8e70-7ef5b5306805-kube-api-access-phfzm\") pod \"e2e-ait-7a95ef74-openshift-default-c6b94f6c5-fndx2\" (UID: \"12adb8c6-7644-4a72-8e70-7ef5b5306805\") " pod="openshift-ingress/e2e-ait-7a95ef74-openshift-default-c6b94f6c5-fndx2" Jul 16 16:50:19.209974 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:50:19.209879 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/12adb8c6-7644-4a72-8e70-7ef5b5306805-istio-token\") pod \"e2e-ait-7a95ef74-openshift-default-c6b94f6c5-fndx2\" (UID: \"12adb8c6-7644-4a72-8e70-7ef5b5306805\") " pod="openshift-ingress/e2e-ait-7a95ef74-openshift-default-c6b94f6c5-fndx2" Jul 16 16:50:19.310845 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:50:19.310802 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12adb8c6-7644-4a72-8e70-7ef5b5306805-istiod-ca-cert\") pod \"e2e-ait-7a95ef74-openshift-default-c6b94f6c5-fndx2\" (UID: \"12adb8c6-7644-4a72-8e70-7ef5b5306805\") " pod="openshift-ingress/e2e-ait-7a95ef74-openshift-default-c6b94f6c5-fndx2" Jul 16 16:50:19.310845 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:50:19.310847 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-phfzm\" (UniqueName: \"kubernetes.io/projected/12adb8c6-7644-4a72-8e70-7ef5b5306805-kube-api-access-phfzm\") pod \"e2e-ait-7a95ef74-openshift-default-c6b94f6c5-fndx2\" (UID: \"12adb8c6-7644-4a72-8e70-7ef5b5306805\") " pod="openshift-ingress/e2e-ait-7a95ef74-openshift-default-c6b94f6c5-fndx2" Jul 16 16:50:19.311104 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:50:19.310878 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/12adb8c6-7644-4a72-8e70-7ef5b5306805-istio-token\") pod \"e2e-ait-7a95ef74-openshift-default-c6b94f6c5-fndx2\" (UID: \"12adb8c6-7644-4a72-8e70-7ef5b5306805\") " pod="openshift-ingress/e2e-ait-7a95ef74-openshift-default-c6b94f6c5-fndx2" Jul 16 16:50:19.311104 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:50:19.310901 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12adb8c6-7644-4a72-8e70-7ef5b5306805-workload-certs\") pod \"e2e-ait-7a95ef74-openshift-default-c6b94f6c5-fndx2\" (UID: \"12adb8c6-7644-4a72-8e70-7ef5b5306805\") " pod="openshift-ingress/e2e-ait-7a95ef74-openshift-default-c6b94f6c5-fndx2" Jul 16 16:50:19.311104 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:50:19.310916 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12adb8c6-7644-4a72-8e70-7ef5b5306805-istio-podinfo\") pod \"e2e-ait-7a95ef74-openshift-default-c6b94f6c5-fndx2\" (UID: \"12adb8c6-7644-4a72-8e70-7ef5b5306805\") " pod="openshift-ingress/e2e-ait-7a95ef74-openshift-default-c6b94f6c5-fndx2" Jul 16 16:50:19.311104 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:50:19.310979 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12adb8c6-7644-4a72-8e70-7ef5b5306805-credential-socket\") pod \"e2e-ait-7a95ef74-openshift-default-c6b94f6c5-fndx2\" (UID: \"12adb8c6-7644-4a72-8e70-7ef5b5306805\") " pod="openshift-ingress/e2e-ait-7a95ef74-openshift-default-c6b94f6c5-fndx2" Jul 16 16:50:19.311104 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:50:19.311016 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12adb8c6-7644-4a72-8e70-7ef5b5306805-istio-data\") pod \"e2e-ait-7a95ef74-openshift-default-c6b94f6c5-fndx2\" (UID: \"12adb8c6-7644-4a72-8e70-7ef5b5306805\") " pod="openshift-ingress/e2e-ait-7a95ef74-openshift-default-c6b94f6c5-fndx2" Jul 16 16:50:19.311104 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:50:19.311057 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12adb8c6-7644-4a72-8e70-7ef5b5306805-workload-socket\") pod \"e2e-ait-7a95ef74-openshift-default-c6b94f6c5-fndx2\" (UID: \"12adb8c6-7644-4a72-8e70-7ef5b5306805\") " pod="openshift-ingress/e2e-ait-7a95ef74-openshift-default-c6b94f6c5-fndx2" Jul 16 16:50:19.311104 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:50:19.311082 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12adb8c6-7644-4a72-8e70-7ef5b5306805-istio-envoy\") pod \"e2e-ait-7a95ef74-openshift-default-c6b94f6c5-fndx2\" (UID: \"12adb8c6-7644-4a72-8e70-7ef5b5306805\") " pod="openshift-ingress/e2e-ait-7a95ef74-openshift-default-c6b94f6c5-fndx2" Jul 16 16:50:19.311489 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:50:19.311376 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12adb8c6-7644-4a72-8e70-7ef5b5306805-credential-socket\") pod \"e2e-ait-7a95ef74-openshift-default-c6b94f6c5-fndx2\" (UID: \"12adb8c6-7644-4a72-8e70-7ef5b5306805\") " pod="openshift-ingress/e2e-ait-7a95ef74-openshift-default-c6b94f6c5-fndx2" Jul 16 16:50:19.311489 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:50:19.311471 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12adb8c6-7644-4a72-8e70-7ef5b5306805-workload-certs\") pod \"e2e-ait-7a95ef74-openshift-default-c6b94f6c5-fndx2\" (UID: \"12adb8c6-7644-4a72-8e70-7ef5b5306805\") " pod="openshift-ingress/e2e-ait-7a95ef74-openshift-default-c6b94f6c5-fndx2" Jul 16 16:50:19.311646 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:50:19.311623 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12adb8c6-7644-4a72-8e70-7ef5b5306805-istio-data\") pod \"e2e-ait-7a95ef74-openshift-default-c6b94f6c5-fndx2\" (UID: \"12adb8c6-7644-4a72-8e70-7ef5b5306805\") " pod="openshift-ingress/e2e-ait-7a95ef74-openshift-default-c6b94f6c5-fndx2" Jul 16 16:50:19.311700 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:50:19.311668 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12adb8c6-7644-4a72-8e70-7ef5b5306805-workload-socket\") pod \"e2e-ait-7a95ef74-openshift-default-c6b94f6c5-fndx2\" (UID: \"12adb8c6-7644-4a72-8e70-7ef5b5306805\") " pod="openshift-ingress/e2e-ait-7a95ef74-openshift-default-c6b94f6c5-fndx2" Jul 16 16:50:19.311700 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:50:19.311689 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12adb8c6-7644-4a72-8e70-7ef5b5306805-istiod-ca-cert\") pod \"e2e-ait-7a95ef74-openshift-default-c6b94f6c5-fndx2\" (UID: \"12adb8c6-7644-4a72-8e70-7ef5b5306805\") " pod="openshift-ingress/e2e-ait-7a95ef74-openshift-default-c6b94f6c5-fndx2" Jul 16 16:50:19.313283 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:50:19.313264 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12adb8c6-7644-4a72-8e70-7ef5b5306805-istio-envoy\") pod \"e2e-ait-7a95ef74-openshift-default-c6b94f6c5-fndx2\" (UID: \"12adb8c6-7644-4a72-8e70-7ef5b5306805\") " pod="openshift-ingress/e2e-ait-7a95ef74-openshift-default-c6b94f6c5-fndx2" Jul 16 16:50:19.313559 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:50:19.313540 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12adb8c6-7644-4a72-8e70-7ef5b5306805-istio-podinfo\") pod \"e2e-ait-7a95ef74-openshift-default-c6b94f6c5-fndx2\" (UID: \"12adb8c6-7644-4a72-8e70-7ef5b5306805\") " pod="openshift-ingress/e2e-ait-7a95ef74-openshift-default-c6b94f6c5-fndx2" Jul 16 16:50:19.319664 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:50:19.319637 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/12adb8c6-7644-4a72-8e70-7ef5b5306805-istio-token\") pod \"e2e-ait-7a95ef74-openshift-default-c6b94f6c5-fndx2\" (UID: \"12adb8c6-7644-4a72-8e70-7ef5b5306805\") " pod="openshift-ingress/e2e-ait-7a95ef74-openshift-default-c6b94f6c5-fndx2" Jul 16 16:50:19.319792 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:50:19.319771 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-phfzm\" (UniqueName: \"kubernetes.io/projected/12adb8c6-7644-4a72-8e70-7ef5b5306805-kube-api-access-phfzm\") pod \"e2e-ait-7a95ef74-openshift-default-c6b94f6c5-fndx2\" (UID: \"12adb8c6-7644-4a72-8e70-7ef5b5306805\") " pod="openshift-ingress/e2e-ait-7a95ef74-openshift-default-c6b94f6c5-fndx2" Jul 16 16:50:19.399894 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:50:19.399808 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-7a95ef74-openshift-default-c6b94f6c5-fndx2" Jul 16 16:50:19.548342 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:50:19.548305 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-7a95ef74-openshift-default-c6b94f6c5-fndx2"] Jul 16 16:50:19.555660 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:50:19.555616 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12adb8c6_7644_4a72_8e70_7ef5b5306805.slice/crio-96818e98db4f2e2a21ca1aa2a3b85f36c2e4d35ebc44cd46842959930d6071ac WatchSource:0}: Error finding container 96818e98db4f2e2a21ca1aa2a3b85f36c2e4d35ebc44cd46842959930d6071ac: Status 404 returned error can't find the container with id 96818e98db4f2e2a21ca1aa2a3b85f36c2e4d35ebc44cd46842959930d6071ac Jul 16 16:50:19.557705 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:50:19.557667 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 16 16:50:19.557814 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:50:19.557746 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 16 16:50:19.557814 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:50:19.557791 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 16 16:50:20.215691 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:50:20.215655 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-7a95ef74-openshift-default-c6b94f6c5-fndx2" event={"ID":"12adb8c6-7644-4a72-8e70-7ef5b5306805","Type":"ContainerStarted","Data":"8a84d0b7399c07b81c6ff3b36aab00677d68cec3be9324957d0cdb0114356376"} Jul 16 16:50:20.216048 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:50:20.215714 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-7a95ef74-openshift-default-c6b94f6c5-fndx2" event={"ID":"12adb8c6-7644-4a72-8e70-7ef5b5306805","Type":"ContainerStarted","Data":"96818e98db4f2e2a21ca1aa2a3b85f36c2e4d35ebc44cd46842959930d6071ac"} Jul 16 16:50:20.238802 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:50:20.238745 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-7a95ef74-openshift-default-c6b94f6c5-fndx2" podStartSLOduration=1.2387274640000001 podStartE2EDuration="1.238727464s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 16:50:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 16:50:20.236912319 +0000 UTC m=+1478.740970173" watchObservedRunningTime="2026-07-16 16:50:20.238727464 +0000 UTC m=+1478.742785315" Jul 16 16:50:20.400345 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:50:20.400310 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-7a95ef74-openshift-default-c6b94f6c5-fndx2" Jul 16 16:50:20.405446 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:50:20.405417 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-7a95ef74-openshift-default-c6b94f6c5-fndx2" Jul 16 16:50:21.222125 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:50:21.222091 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-7a95ef74-openshift-default-c6b94f6c5-fndx2" Jul 16 16:50:21.223031 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:50:21.223014 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-7a95ef74-openshift-default-c6b94f6c5-fndx2" Jul 16 16:50:25.368250 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:50:25.368211 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7a95ef74-tx5b9"] Jul 16 16:50:25.372355 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:50:25.372330 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7a95ef74-tx5b9" Jul 16 16:50:25.377371 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:50:25.377353 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 16 16:50:25.377956 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:50:25.377936 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dssfh\"" Jul 16 16:50:25.383260 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:50:25.383232 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7a95ef74-tx5b9"] Jul 16 16:50:25.480778 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:50:25.480741 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/511780d3-effc-4b1b-a560-82d2ea21e1b7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-7a95ef74-tx5b9\" (UID: \"511780d3-effc-4b1b-a560-82d2ea21e1b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7a95ef74-tx5b9" Jul 16 16:50:25.581802 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:50:25.581765 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/511780d3-effc-4b1b-a560-82d2ea21e1b7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-7a95ef74-tx5b9\" (UID: \"511780d3-effc-4b1b-a560-82d2ea21e1b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7a95ef74-tx5b9" Jul 16 16:50:25.582464 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:50:25.582442 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/511780d3-effc-4b1b-a560-82d2ea21e1b7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-7a95ef74-tx5b9\" (UID: \"511780d3-effc-4b1b-a560-82d2ea21e1b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7a95ef74-tx5b9" Jul 16 16:50:25.683968 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:50:25.683877 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7a95ef74-tx5b9" Jul 16 16:50:25.823327 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:50:25.823302 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7a95ef74-tx5b9"] Jul 16 16:50:25.825614 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:50:25.825577 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod511780d3_effc_4b1b_a560_82d2ea21e1b7.slice/crio-93606e27a838c184e3b38dbf548aa6f07f5ccd114d31678ef2a81becbd8cb1b8 WatchSource:0}: Error finding container 93606e27a838c184e3b38dbf548aa6f07f5ccd114d31678ef2a81becbd8cb1b8: Status 404 returned error can't find the container with id 93606e27a838c184e3b38dbf548aa6f07f5ccd114d31678ef2a81becbd8cb1b8 Jul 16 16:50:26.247741 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:50:26.247644 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7a95ef74-tx5b9" event={"ID":"511780d3-effc-4b1b-a560-82d2ea21e1b7","Type":"ContainerStarted","Data":"71a36d5d57e790789f66fb6347dffe1ef1c5790caded85efd8e72b38fc6cfc83"} Jul 16 16:50:26.247741 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:50:26.247718 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7a95ef74-tx5b9" event={"ID":"511780d3-effc-4b1b-a560-82d2ea21e1b7","Type":"ContainerStarted","Data":"93606e27a838c184e3b38dbf548aa6f07f5ccd114d31678ef2a81becbd8cb1b8"} Jul 16 16:50:26.265114 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:50:26.265060 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7a95ef74-tx5b9" podStartSLOduration=1.26503913 podStartE2EDuration="1.26503913s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 16:50:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 16:50:26.264188423 +0000 UTC m=+1484.768246276" watchObservedRunningTime="2026-07-16 16:50:26.26503913 +0000 UTC m=+1484.769096980" Jul 16 16:50:27.254394 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:50:27.254353 2577 generic.go:356] "Generic (PLEG): container finished" podID="511780d3-effc-4b1b-a560-82d2ea21e1b7" containerID="71a36d5d57e790789f66fb6347dffe1ef1c5790caded85efd8e72b38fc6cfc83" exitCode=7 Jul 16 16:50:27.254794 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:50:27.254426 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7a95ef74-tx5b9" event={"ID":"511780d3-effc-4b1b-a560-82d2ea21e1b7","Type":"ContainerDied","Data":"71a36d5d57e790789f66fb6347dffe1ef1c5790caded85efd8e72b38fc6cfc83"} Jul 16 16:50:27.254794 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:50:27.254670 2577 scope.go:117] "RemoveContainer" containerID="71a36d5d57e790789f66fb6347dffe1ef1c5790caded85efd8e72b38fc6cfc83" Jul 16 16:50:28.261709 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:50:28.261627 2577 generic.go:356] "Generic (PLEG): container finished" podID="511780d3-effc-4b1b-a560-82d2ea21e1b7" containerID="1aab42c30e27a5a59d8c329ca55b1954e79baeb6853b19ac6403182ee7b5bea7" exitCode=7 Jul 16 16:50:28.261709 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:50:28.261667 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7a95ef74-tx5b9" event={"ID":"511780d3-effc-4b1b-a560-82d2ea21e1b7","Type":"ContainerDied","Data":"1aab42c30e27a5a59d8c329ca55b1954e79baeb6853b19ac6403182ee7b5bea7"} Jul 16 16:50:28.262248 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:50:28.261729 2577 scope.go:117] "RemoveContainer" containerID="71a36d5d57e790789f66fb6347dffe1ef1c5790caded85efd8e72b38fc6cfc83" Jul 16 16:50:28.262248 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:50:28.261947 2577 scope.go:117] "RemoveContainer" containerID="1aab42c30e27a5a59d8c329ca55b1954e79baeb6853b19ac6403182ee7b5bea7" Jul 16 16:50:28.262248 ip-10-0-142-101 kubenswrapper[2577]: E0716 16:50:28.262212 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-7a95ef74-tx5b9_odh-ai-gateway-infra(511780d3-effc-4b1b-a560-82d2ea21e1b7)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7a95ef74-tx5b9" podUID="511780d3-effc-4b1b-a560-82d2ea21e1b7" Jul 16 16:50:29.268434 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:50:29.268408 2577 scope.go:117] "RemoveContainer" containerID="1aab42c30e27a5a59d8c329ca55b1954e79baeb6853b19ac6403182ee7b5bea7" Jul 16 16:50:29.268797 ip-10-0-142-101 kubenswrapper[2577]: E0716 16:50:29.268618 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-7a95ef74-tx5b9_odh-ai-gateway-infra(511780d3-effc-4b1b-a560-82d2ea21e1b7)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7a95ef74-tx5b9" podUID="511780d3-effc-4b1b-a560-82d2ea21e1b7" Jul 16 16:50:37.263580 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:50:37.263534 2577 patch_prober.go:28] interesting pod/e2e-ait-7a95ef74-openshift-default-c6b94f6c5-fndx2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 16 16:50:37.263959 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:50:37.263608 2577 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-7a95ef74-openshift-default-c6b94f6c5-fndx2" podUID="12adb8c6-7644-4a72-8e70-7ef5b5306805" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 16 16:50:38.316012 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:50:38.315903 2577 generic.go:356] "Generic (PLEG): container finished" podID="12adb8c6-7644-4a72-8e70-7ef5b5306805" containerID="8a84d0b7399c07b81c6ff3b36aab00677d68cec3be9324957d0cdb0114356376" exitCode=0 Jul 16 16:50:38.316012 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:50:38.315984 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-7a95ef74-openshift-default-c6b94f6c5-fndx2" event={"ID":"12adb8c6-7644-4a72-8e70-7ef5b5306805","Type":"ContainerDied","Data":"8a84d0b7399c07b81c6ff3b36aab00677d68cec3be9324957d0cdb0114356376"} Jul 16 16:50:38.316521 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:50:38.316503 2577 scope.go:117] "RemoveContainer" containerID="8a84d0b7399c07b81c6ff3b36aab00677d68cec3be9324957d0cdb0114356376" Jul 16 16:50:38.317640 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:50:38.317612 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 16 16:50:38.317723 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:50:38.317684 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 16 16:50:38.317723 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:50:38.317716 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 16 16:50:39.325916 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:50:39.325881 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-7a95ef74-openshift-default-c6b94f6c5-fndx2" event={"ID":"12adb8c6-7644-4a72-8e70-7ef5b5306805","Type":"ContainerStarted","Data":"e94d5b7fb5483e7f53c7d31b21a90a1aeef1c8f99adb597d29847660f52381cb"} Jul 16 16:50:39.400186 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:50:39.400145 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-7a95ef74-openshift-default-c6b94f6c5-fndx2" Jul 16 16:50:39.400481 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:50:39.400461 2577 patch_prober.go:28] interesting pod/e2e-ait-7a95ef74-openshift-default-c6b94f6c5-fndx2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" start-of-body= Jul 16 16:50:39.400546 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:50:39.400505 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-7a95ef74-openshift-default-c6b94f6c5-fndx2" podUID="12adb8c6-7644-4a72-8e70-7ef5b5306805" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" Jul 16 16:50:40.400218 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:50:40.400182 2577 patch_prober.go:28] interesting pod/e2e-ait-7a95ef74-openshift-default-c6b94f6c5-fndx2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" start-of-body= Jul 16 16:50:40.400662 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:50:40.400240 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-7a95ef74-openshift-default-c6b94f6c5-fndx2" podUID="12adb8c6-7644-4a72-8e70-7ef5b5306805" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" Jul 16 16:50:41.401086 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:50:41.401035 2577 patch_prober.go:28] interesting pod/e2e-ait-7a95ef74-openshift-default-c6b94f6c5-fndx2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" start-of-body= Jul 16 16:50:41.401536 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:50:41.401116 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-7a95ef74-openshift-default-c6b94f6c5-fndx2" podUID="12adb8c6-7644-4a72-8e70-7ef5b5306805" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" Jul 16 16:50:42.024684 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:50:42.024657 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9bjst_b20b154f-1a4c-4b25-86db-aebe2d4ffc08/ovn-acl-logging/0.log" Jul 16 16:50:42.028917 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:50:42.028896 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9bjst_b20b154f-1a4c-4b25-86db-aebe2d4ffc08/ovn-acl-logging/0.log" Jul 16 16:50:42.400595 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:50:42.400563 2577 patch_prober.go:28] interesting pod/e2e-ait-7a95ef74-openshift-default-c6b94f6c5-fndx2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" start-of-body= Jul 16 16:50:42.400777 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:50:42.400630 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-7a95ef74-openshift-default-c6b94f6c5-fndx2" podUID="12adb8c6-7644-4a72-8e70-7ef5b5306805" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" Jul 16 16:50:43.404052 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:50:43.404022 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-7a95ef74-openshift-default-c6b94f6c5-fndx2" Jul 16 16:50:43.404464 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:50:43.404291 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-7a95ef74-openshift-default-c6b94f6c5-fndx2" Jul 16 16:50:43.404856 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:50:43.404840 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-7a95ef74-openshift-default-c6b94f6c5-fndx2" Jul 16 16:50:44.012039 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:50:44.012009 2577 scope.go:117] "RemoveContainer" containerID="1aab42c30e27a5a59d8c329ca55b1954e79baeb6853b19ac6403182ee7b5bea7" Jul 16 16:50:44.376267 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:50:44.376232 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7a95ef74-tx5b9" event={"ID":"511780d3-effc-4b1b-a560-82d2ea21e1b7","Type":"ContainerStarted","Data":"04ae557e51a785edbd5752b4c5f9b10d915ee3370337f2c48cf523c236bc48de"} Jul 16 16:50:45.382460 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:50:45.382426 2577 generic.go:356] "Generic (PLEG): container finished" podID="511780d3-effc-4b1b-a560-82d2ea21e1b7" containerID="04ae557e51a785edbd5752b4c5f9b10d915ee3370337f2c48cf523c236bc48de" exitCode=0 Jul 16 16:50:45.382887 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:50:45.382505 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7a95ef74-tx5b9" event={"ID":"511780d3-effc-4b1b-a560-82d2ea21e1b7","Type":"ContainerDied","Data":"04ae557e51a785edbd5752b4c5f9b10d915ee3370337f2c48cf523c236bc48de"} Jul 16 16:50:45.382887 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:50:45.382569 2577 scope.go:117] "RemoveContainer" containerID="1aab42c30e27a5a59d8c329ca55b1954e79baeb6853b19ac6403182ee7b5bea7" Jul 16 16:50:45.406926 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:50:45.406887 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7a95ef74-tx5b9"] Jul 16 16:50:46.552825 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:50:46.552795 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7a95ef74-tx5b9" Jul 16 16:50:46.608331 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:50:46.608297 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/511780d3-effc-4b1b-a560-82d2ea21e1b7-maas-api-service-ca\") pod \"511780d3-effc-4b1b-a560-82d2ea21e1b7\" (UID: \"511780d3-effc-4b1b-a560-82d2ea21e1b7\") " Jul 16 16:50:46.608667 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:50:46.608642 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/511780d3-effc-4b1b-a560-82d2ea21e1b7-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "511780d3-effc-4b1b-a560-82d2ea21e1b7" (UID: "511780d3-effc-4b1b-a560-82d2ea21e1b7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 16:50:46.709629 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:50:46.709524 2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/511780d3-effc-4b1b-a560-82d2ea21e1b7-maas-api-service-ca\") on node \"ip-10-0-142-101.ec2.internal\" DevicePath \"\"" Jul 16 16:50:47.396986 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:50:47.396952 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7a95ef74-tx5b9" event={"ID":"511780d3-effc-4b1b-a560-82d2ea21e1b7","Type":"ContainerDied","Data":"93606e27a838c184e3b38dbf548aa6f07f5ccd114d31678ef2a81becbd8cb1b8"} Jul 16 16:50:47.397167 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:50:47.396999 2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="93606e27a838c184e3b38dbf548aa6f07f5ccd114d31678ef2a81becbd8cb1b8" Jul 16 16:50:47.397167 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:50:47.397003 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7a95ef74-tx5b9" Jul 16 16:50:47.426995 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:50:47.426959 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7a95ef74-tx5b9"] Jul 16 16:50:47.431966 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:50:47.431936 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7a95ef74-tx5b9"] Jul 16 16:50:48.018508 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:50:48.018474 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="511780d3-effc-4b1b-a560-82d2ea21e1b7" path="/var/lib/kubelet/pods/511780d3-effc-4b1b-a560-82d2ea21e1b7/volumes" Jul 16 16:51:25.403953 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:51:25.403870 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7a95ef74-7xgr9"] Jul 16 16:51:25.404461 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:51:25.404271 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="511780d3-effc-4b1b-a560-82d2ea21e1b7" containerName="revoke-keys" Jul 16 16:51:25.404461 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:51:25.404281 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="511780d3-effc-4b1b-a560-82d2ea21e1b7" containerName="revoke-keys" Jul 16 16:51:25.404461 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:51:25.404294 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="511780d3-effc-4b1b-a560-82d2ea21e1b7" containerName="revoke-keys" Jul 16 16:51:25.404461 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:51:25.404299 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="511780d3-effc-4b1b-a560-82d2ea21e1b7" containerName="revoke-keys" Jul 16 16:51:25.404461 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:51:25.404307 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="511780d3-effc-4b1b-a560-82d2ea21e1b7" containerName="revoke-keys" Jul 16 16:51:25.404461 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:51:25.404313 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="511780d3-effc-4b1b-a560-82d2ea21e1b7" containerName="revoke-keys" Jul 16 16:51:25.404461 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:51:25.404377 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="511780d3-effc-4b1b-a560-82d2ea21e1b7" containerName="revoke-keys" Jul 16 16:51:25.404461 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:51:25.404417 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="511780d3-effc-4b1b-a560-82d2ea21e1b7" containerName="revoke-keys" Jul 16 16:51:25.407530 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:51:25.407512 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7a95ef74-7xgr9" Jul 16 16:51:25.410220 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:51:25.410195 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 16 16:51:25.410327 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:51:25.410194 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dssfh\"" Jul 16 16:51:25.416132 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:51:25.416109 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7a95ef74-7xgr9"] Jul 16 16:51:25.503420 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:51:25.503368 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/86d20cca-30d2-4ed9-bed9-659c339429ea-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-7a95ef74-7xgr9\" (UID: \"86d20cca-30d2-4ed9-bed9-659c339429ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7a95ef74-7xgr9" Jul 16 16:51:25.604692 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:51:25.604654 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/86d20cca-30d2-4ed9-bed9-659c339429ea-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-7a95ef74-7xgr9\" (UID: \"86d20cca-30d2-4ed9-bed9-659c339429ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7a95ef74-7xgr9" Jul 16 16:51:25.605270 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:51:25.605248 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/86d20cca-30d2-4ed9-bed9-659c339429ea-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-7a95ef74-7xgr9\" (UID: \"86d20cca-30d2-4ed9-bed9-659c339429ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7a95ef74-7xgr9" Jul 16 16:51:25.719750 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:51:25.719666 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7a95ef74-7xgr9" Jul 16 16:51:25.888163 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:51:25.888056 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7a95ef74-7xgr9"] Jul 16 16:51:25.890732 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:51:25.890701 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86d20cca_30d2_4ed9_bed9_659c339429ea.slice/crio-5ac14108d0a21d8a28cc19d90b755009895f2a883a34ed979ff4107aa3159b15 WatchSource:0}: Error finding container 5ac14108d0a21d8a28cc19d90b755009895f2a883a34ed979ff4107aa3159b15: Status 404 returned error can't find the container with id 5ac14108d0a21d8a28cc19d90b755009895f2a883a34ed979ff4107aa3159b15 Jul 16 16:51:26.598511 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:51:26.598477 2577 generic.go:356] "Generic (PLEG): container finished" podID="86d20cca-30d2-4ed9-bed9-659c339429ea" containerID="7bb38ed8743bbd43f6210f83f91a646f29daa58662cd57e6a18d082aaa5be71c" exitCode=0 Jul 16 16:51:26.598906 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:51:26.598520 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7a95ef74-7xgr9" event={"ID":"86d20cca-30d2-4ed9-bed9-659c339429ea","Type":"ContainerDied","Data":"7bb38ed8743bbd43f6210f83f91a646f29daa58662cd57e6a18d082aaa5be71c"} Jul 16 16:51:26.598906 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:51:26.598576 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7a95ef74-7xgr9" event={"ID":"86d20cca-30d2-4ed9-bed9-659c339429ea","Type":"ContainerStarted","Data":"5ac14108d0a21d8a28cc19d90b755009895f2a883a34ed979ff4107aa3159b15"} Jul 16 16:51:27.739187 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:51:27.739150 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7a95ef74-7xgr9" Jul 16 16:51:27.829111 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:51:27.829069 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/86d20cca-30d2-4ed9-bed9-659c339429ea-maas-api-service-ca\") pod \"86d20cca-30d2-4ed9-bed9-659c339429ea\" (UID: \"86d20cca-30d2-4ed9-bed9-659c339429ea\") " Jul 16 16:51:27.829481 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:51:27.829455 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86d20cca-30d2-4ed9-bed9-659c339429ea-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "86d20cca-30d2-4ed9-bed9-659c339429ea" (UID: "86d20cca-30d2-4ed9-bed9-659c339429ea"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 16:51:27.930853 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:51:27.930757 2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/86d20cca-30d2-4ed9-bed9-659c339429ea-maas-api-service-ca\") on node \"ip-10-0-142-101.ec2.internal\" DevicePath \"\"" Jul 16 16:51:28.610088 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:51:28.610057 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7a95ef74-7xgr9" Jul 16 16:51:28.610269 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:51:28.610055 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7a95ef74-7xgr9" event={"ID":"86d20cca-30d2-4ed9-bed9-659c339429ea","Type":"ContainerDied","Data":"5ac14108d0a21d8a28cc19d90b755009895f2a883a34ed979ff4107aa3159b15"} Jul 16 16:51:28.610269 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:51:28.610189 2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5ac14108d0a21d8a28cc19d90b755009895f2a883a34ed979ff4107aa3159b15" Jul 16 16:51:50.772627 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:51:50.772583 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-7a95ef74-openshift-default-c6b94f6c5-fndx2"] Jul 16 16:51:50.773109 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:51:50.772880 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-7a95ef74-openshift-default-c6b94f6c5-fndx2" podUID="12adb8c6-7644-4a72-8e70-7ef5b5306805" containerName="istio-proxy" containerID="cri-o://e94d5b7fb5483e7f53c7d31b21a90a1aeef1c8f99adb597d29847660f52381cb" gracePeriod=30 Jul 16 16:51:51.414719 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:51:51.414682 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-c61ec96a-gw-openshift-default-769498d775-8cpnt"] Jul 16 16:51:51.415162 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:51:51.415145 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="86d20cca-30d2-4ed9-bed9-659c339429ea" containerName="revoke-keys" Jul 16 16:51:51.415243 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:51:51.415165 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="86d20cca-30d2-4ed9-bed9-659c339429ea" containerName="revoke-keys" Jul 16 16:51:51.415313 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:51:51.415301 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="86d20cca-30d2-4ed9-bed9-659c339429ea" containerName="revoke-keys" Jul 16 16:51:51.415367 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:51:51.415317 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="511780d3-effc-4b1b-a560-82d2ea21e1b7" containerName="revoke-keys" Jul 16 16:51:51.418576 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:51:51.418551 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-c61ec96a-gw-openshift-default-769498d775-8cpnt" Jul 16 16:51:51.432088 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:51:51.432063 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-c61ec96a-gw-openshift-default-dockercfg-47f5j\"" Jul 16 16:51:51.443813 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:51:51.443778 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-c61ec96a-gw-openshift-default-769498d775-8cpnt"] Jul 16 16:51:51.568779 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:51:51.568740 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3da487a-0665-40b7-ab25-329cba6f1b2c-credential-socket\") pod \"e2e-migrate-c61ec96a-gw-openshift-default-769498d775-8cpnt\" (UID: \"e3da487a-0665-40b7-ab25-329cba6f1b2c\") " pod="openshift-ingress/e2e-migrate-c61ec96a-gw-openshift-default-769498d775-8cpnt" Jul 16 16:51:51.568957 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:51:51.568793 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3da487a-0665-40b7-ab25-329cba6f1b2c-istiod-ca-cert\") pod \"e2e-migrate-c61ec96a-gw-openshift-default-769498d775-8cpnt\" (UID: \"e3da487a-0665-40b7-ab25-329cba6f1b2c\") " pod="openshift-ingress/e2e-migrate-c61ec96a-gw-openshift-default-769498d775-8cpnt" Jul 16 16:51:51.568957 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:51:51.568830 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3da487a-0665-40b7-ab25-329cba6f1b2c-istio-token\") pod \"e2e-migrate-c61ec96a-gw-openshift-default-769498d775-8cpnt\" (UID: \"e3da487a-0665-40b7-ab25-329cba6f1b2c\") " pod="openshift-ingress/e2e-migrate-c61ec96a-gw-openshift-default-769498d775-8cpnt" Jul 16 16:51:51.568957 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:51:51.568852 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3da487a-0665-40b7-ab25-329cba6f1b2c-istio-envoy\") pod \"e2e-migrate-c61ec96a-gw-openshift-default-769498d775-8cpnt\" (UID: \"e3da487a-0665-40b7-ab25-329cba6f1b2c\") " pod="openshift-ingress/e2e-migrate-c61ec96a-gw-openshift-default-769498d775-8cpnt" Jul 16 16:51:51.568957 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:51:51.568875 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3da487a-0665-40b7-ab25-329cba6f1b2c-istio-data\") pod \"e2e-migrate-c61ec96a-gw-openshift-default-769498d775-8cpnt\" (UID: \"e3da487a-0665-40b7-ab25-329cba6f1b2c\") " pod="openshift-ingress/e2e-migrate-c61ec96a-gw-openshift-default-769498d775-8cpnt" Jul 16 16:51:51.568957 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:51:51.568941 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3da487a-0665-40b7-ab25-329cba6f1b2c-workload-socket\") pod \"e2e-migrate-c61ec96a-gw-openshift-default-769498d775-8cpnt\" (UID: \"e3da487a-0665-40b7-ab25-329cba6f1b2c\") " pod="openshift-ingress/e2e-migrate-c61ec96a-gw-openshift-default-769498d775-8cpnt" Jul 16 16:51:51.569119 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:51:51.568982 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3da487a-0665-40b7-ab25-329cba6f1b2c-workload-certs\") pod \"e2e-migrate-c61ec96a-gw-openshift-default-769498d775-8cpnt\" (UID: \"e3da487a-0665-40b7-ab25-329cba6f1b2c\") " pod="openshift-ingress/e2e-migrate-c61ec96a-gw-openshift-default-769498d775-8cpnt" Jul 16 16:51:51.569119 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:51:51.569000 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3da487a-0665-40b7-ab25-329cba6f1b2c-istio-podinfo\") pod \"e2e-migrate-c61ec96a-gw-openshift-default-769498d775-8cpnt\" (UID: \"e3da487a-0665-40b7-ab25-329cba6f1b2c\") " pod="openshift-ingress/e2e-migrate-c61ec96a-gw-openshift-default-769498d775-8cpnt" Jul 16 16:51:51.569119 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:51:51.569037 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nx8cg\" (UniqueName: \"kubernetes.io/projected/e3da487a-0665-40b7-ab25-329cba6f1b2c-kube-api-access-nx8cg\") pod \"e2e-migrate-c61ec96a-gw-openshift-default-769498d775-8cpnt\" (UID: \"e3da487a-0665-40b7-ab25-329cba6f1b2c\") " pod="openshift-ingress/e2e-migrate-c61ec96a-gw-openshift-default-769498d775-8cpnt" Jul 16 16:51:51.669896 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:51:51.669788 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3da487a-0665-40b7-ab25-329cba6f1b2c-credential-socket\") pod \"e2e-migrate-c61ec96a-gw-openshift-default-769498d775-8cpnt\" (UID: \"e3da487a-0665-40b7-ab25-329cba6f1b2c\") " pod="openshift-ingress/e2e-migrate-c61ec96a-gw-openshift-default-769498d775-8cpnt" Jul 16 16:51:51.670075 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:51:51.669945 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3da487a-0665-40b7-ab25-329cba6f1b2c-istiod-ca-cert\") pod \"e2e-migrate-c61ec96a-gw-openshift-default-769498d775-8cpnt\" (UID: \"e3da487a-0665-40b7-ab25-329cba6f1b2c\") " pod="openshift-ingress/e2e-migrate-c61ec96a-gw-openshift-default-769498d775-8cpnt" Jul 16 16:51:51.670075 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:51:51.669997 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3da487a-0665-40b7-ab25-329cba6f1b2c-istio-token\") pod \"e2e-migrate-c61ec96a-gw-openshift-default-769498d775-8cpnt\" (UID: \"e3da487a-0665-40b7-ab25-329cba6f1b2c\") " pod="openshift-ingress/e2e-migrate-c61ec96a-gw-openshift-default-769498d775-8cpnt" Jul 16 16:51:51.670075 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:51:51.670018 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3da487a-0665-40b7-ab25-329cba6f1b2c-istio-envoy\") pod \"e2e-migrate-c61ec96a-gw-openshift-default-769498d775-8cpnt\" (UID: \"e3da487a-0665-40b7-ab25-329cba6f1b2c\") " pod="openshift-ingress/e2e-migrate-c61ec96a-gw-openshift-default-769498d775-8cpnt" Jul 16 16:51:51.670075 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:51:51.670052 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3da487a-0665-40b7-ab25-329cba6f1b2c-istio-data\") pod \"e2e-migrate-c61ec96a-gw-openshift-default-769498d775-8cpnt\" (UID: \"e3da487a-0665-40b7-ab25-329cba6f1b2c\") " pod="openshift-ingress/e2e-migrate-c61ec96a-gw-openshift-default-769498d775-8cpnt" Jul 16 16:51:51.670299 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:51:51.670105 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3da487a-0665-40b7-ab25-329cba6f1b2c-workload-socket\") pod \"e2e-migrate-c61ec96a-gw-openshift-default-769498d775-8cpnt\" (UID: \"e3da487a-0665-40b7-ab25-329cba6f1b2c\") " pod="openshift-ingress/e2e-migrate-c61ec96a-gw-openshift-default-769498d775-8cpnt" Jul 16 16:51:51.670299 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:51:51.670185 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3da487a-0665-40b7-ab25-329cba6f1b2c-workload-certs\") pod \"e2e-migrate-c61ec96a-gw-openshift-default-769498d775-8cpnt\" (UID: \"e3da487a-0665-40b7-ab25-329cba6f1b2c\") " pod="openshift-ingress/e2e-migrate-c61ec96a-gw-openshift-default-769498d775-8cpnt" Jul 16 16:51:51.670299 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:51:51.670215 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3da487a-0665-40b7-ab25-329cba6f1b2c-istio-podinfo\") pod \"e2e-migrate-c61ec96a-gw-openshift-default-769498d775-8cpnt\" (UID: \"e3da487a-0665-40b7-ab25-329cba6f1b2c\") " pod="openshift-ingress/e2e-migrate-c61ec96a-gw-openshift-default-769498d775-8cpnt" Jul 16 16:51:51.670299 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:51:51.670270 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nx8cg\" (UniqueName: \"kubernetes.io/projected/e3da487a-0665-40b7-ab25-329cba6f1b2c-kube-api-access-nx8cg\") pod \"e2e-migrate-c61ec96a-gw-openshift-default-769498d775-8cpnt\" (UID: \"e3da487a-0665-40b7-ab25-329cba6f1b2c\") " pod="openshift-ingress/e2e-migrate-c61ec96a-gw-openshift-default-769498d775-8cpnt" Jul 16 16:51:51.670299 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:51:51.670279 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3da487a-0665-40b7-ab25-329cba6f1b2c-credential-socket\") pod \"e2e-migrate-c61ec96a-gw-openshift-default-769498d775-8cpnt\" (UID: \"e3da487a-0665-40b7-ab25-329cba6f1b2c\") " pod="openshift-ingress/e2e-migrate-c61ec96a-gw-openshift-default-769498d775-8cpnt" Jul 16 16:51:51.670565 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:51:51.670453 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3da487a-0665-40b7-ab25-329cba6f1b2c-istio-data\") pod \"e2e-migrate-c61ec96a-gw-openshift-default-769498d775-8cpnt\" (UID: \"e3da487a-0665-40b7-ab25-329cba6f1b2c\") " pod="openshift-ingress/e2e-migrate-c61ec96a-gw-openshift-default-769498d775-8cpnt" Jul 16 16:51:51.670653 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:51:51.670630 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3da487a-0665-40b7-ab25-329cba6f1b2c-workload-socket\") pod \"e2e-migrate-c61ec96a-gw-openshift-default-769498d775-8cpnt\" (UID: \"e3da487a-0665-40b7-ab25-329cba6f1b2c\") " pod="openshift-ingress/e2e-migrate-c61ec96a-gw-openshift-default-769498d775-8cpnt" Jul 16 16:51:51.670719 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:51:51.670662 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3da487a-0665-40b7-ab25-329cba6f1b2c-workload-certs\") pod \"e2e-migrate-c61ec96a-gw-openshift-default-769498d775-8cpnt\" (UID: \"e3da487a-0665-40b7-ab25-329cba6f1b2c\") " pod="openshift-ingress/e2e-migrate-c61ec96a-gw-openshift-default-769498d775-8cpnt" Jul 16 16:51:51.670768 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:51:51.670727 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3da487a-0665-40b7-ab25-329cba6f1b2c-istiod-ca-cert\") pod \"e2e-migrate-c61ec96a-gw-openshift-default-769498d775-8cpnt\" (UID: \"e3da487a-0665-40b7-ab25-329cba6f1b2c\") " pod="openshift-ingress/e2e-migrate-c61ec96a-gw-openshift-default-769498d775-8cpnt" Jul 16 16:51:51.672370 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:51:51.672340 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3da487a-0665-40b7-ab25-329cba6f1b2c-istio-envoy\") pod \"e2e-migrate-c61ec96a-gw-openshift-default-769498d775-8cpnt\" (UID: \"e3da487a-0665-40b7-ab25-329cba6f1b2c\") " pod="openshift-ingress/e2e-migrate-c61ec96a-gw-openshift-default-769498d775-8cpnt" Jul 16 16:51:51.672795 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:51:51.672777 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3da487a-0665-40b7-ab25-329cba6f1b2c-istio-podinfo\") pod \"e2e-migrate-c61ec96a-gw-openshift-default-769498d775-8cpnt\" (UID: \"e3da487a-0665-40b7-ab25-329cba6f1b2c\") " pod="openshift-ingress/e2e-migrate-c61ec96a-gw-openshift-default-769498d775-8cpnt" Jul 16 16:51:51.679419 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:51:51.679359 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3da487a-0665-40b7-ab25-329cba6f1b2c-istio-token\") pod \"e2e-migrate-c61ec96a-gw-openshift-default-769498d775-8cpnt\" (UID: \"e3da487a-0665-40b7-ab25-329cba6f1b2c\") " pod="openshift-ingress/e2e-migrate-c61ec96a-gw-openshift-default-769498d775-8cpnt" Jul 16 16:51:51.679504 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:51:51.679490 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nx8cg\" (UniqueName: \"kubernetes.io/projected/e3da487a-0665-40b7-ab25-329cba6f1b2c-kube-api-access-nx8cg\") pod \"e2e-migrate-c61ec96a-gw-openshift-default-769498d775-8cpnt\" (UID: \"e3da487a-0665-40b7-ab25-329cba6f1b2c\") " pod="openshift-ingress/e2e-migrate-c61ec96a-gw-openshift-default-769498d775-8cpnt" Jul 16 16:51:51.729347 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:51:51.729314 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-c61ec96a-gw-openshift-default-769498d775-8cpnt" Jul 16 16:51:51.915326 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:51:51.915299 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-c61ec96a-gw-openshift-default-769498d775-8cpnt"] Jul 16 16:51:51.916649 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:51:51.916628 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3da487a_0665_40b7_ab25_329cba6f1b2c.slice/crio-48b43aed09bdd3c163d4eaccf9d27e90cf44de32aaa09e57d1bd605060764fc0 WatchSource:0}: Error finding container 48b43aed09bdd3c163d4eaccf9d27e90cf44de32aaa09e57d1bd605060764fc0: Status 404 returned error can't find the container with id 48b43aed09bdd3c163d4eaccf9d27e90cf44de32aaa09e57d1bd605060764fc0 Jul 16 16:51:51.918969 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:51:51.918933 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 16 16:51:51.919042 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:51:51.919020 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 16 16:51:51.919082 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:51:51.919064 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 16 16:51:52.732865 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:51:52.732826 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-c61ec96a-gw-openshift-default-769498d775-8cpnt" event={"ID":"e3da487a-0665-40b7-ab25-329cba6f1b2c","Type":"ContainerStarted","Data":"fbbcfb44f0be91cad0b80ca4fdd7b0fd0a366132faa7d6e25bf7bb522949f3b3"} Jul 16 16:51:52.733055 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:51:52.732893 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-c61ec96a-gw-openshift-default-769498d775-8cpnt" event={"ID":"e3da487a-0665-40b7-ab25-329cba6f1b2c","Type":"ContainerStarted","Data":"48b43aed09bdd3c163d4eaccf9d27e90cf44de32aaa09e57d1bd605060764fc0"} Jul 16 16:51:52.756673 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:51:52.756612 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-c61ec96a-gw-openshift-default-769498d775-8cpnt" podStartSLOduration=1.756595823 podStartE2EDuration="1.756595823s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 16:51:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 16:51:52.75464168 +0000 UTC m=+1571.258699544" watchObservedRunningTime="2026-07-16 16:51:52.756595823 +0000 UTC m=+1571.260653674" Jul 16 16:51:53.729774 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:51:53.729737 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-c61ec96a-gw-openshift-default-769498d775-8cpnt" Jul 16 16:51:53.734603 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:51:53.734578 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-c61ec96a-gw-openshift-default-769498d775-8cpnt" Jul 16 16:51:53.738050 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:51:53.738017 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-c61ec96a-gw-openshift-default-769498d775-8cpnt" Jul 16 16:51:53.739156 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:51:53.739131 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-c61ec96a-gw-openshift-default-769498d775-8cpnt" Jul 16 16:51:56.030015 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:51:56.029988 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-7a95ef74-openshift-default-c6b94f6c5-fndx2" Jul 16 16:51:56.112149 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:51:56.112113 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-phfzm\" (UniqueName: \"kubernetes.io/projected/12adb8c6-7644-4a72-8e70-7ef5b5306805-kube-api-access-phfzm\") pod \"12adb8c6-7644-4a72-8e70-7ef5b5306805\" (UID: \"12adb8c6-7644-4a72-8e70-7ef5b5306805\") " Jul 16 16:51:56.112336 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:51:56.112176 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12adb8c6-7644-4a72-8e70-7ef5b5306805-workload-certs\") pod \"12adb8c6-7644-4a72-8e70-7ef5b5306805\" (UID: \"12adb8c6-7644-4a72-8e70-7ef5b5306805\") " Jul 16 16:51:56.112336 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:51:56.112265 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/12adb8c6-7644-4a72-8e70-7ef5b5306805-istio-token\") pod \"12adb8c6-7644-4a72-8e70-7ef5b5306805\" (UID: \"12adb8c6-7644-4a72-8e70-7ef5b5306805\") " Jul 16 16:51:56.112336 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:51:56.112313 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12adb8c6-7644-4a72-8e70-7ef5b5306805-credential-socket\") pod \"12adb8c6-7644-4a72-8e70-7ef5b5306805\" (UID: \"12adb8c6-7644-4a72-8e70-7ef5b5306805\") " Jul 16 16:51:56.112539 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:51:56.112362 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12adb8c6-7644-4a72-8e70-7ef5b5306805-workload-socket\") pod \"12adb8c6-7644-4a72-8e70-7ef5b5306805\" (UID: \"12adb8c6-7644-4a72-8e70-7ef5b5306805\") " Jul 16 16:51:56.112539 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:51:56.112425 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12adb8c6-7644-4a72-8e70-7ef5b5306805-istio-envoy\") pod \"12adb8c6-7644-4a72-8e70-7ef5b5306805\" (UID: \"12adb8c6-7644-4a72-8e70-7ef5b5306805\") " Jul 16 16:51:56.112539 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:51:56.112466 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12adb8c6-7644-4a72-8e70-7ef5b5306805-istio-podinfo\") pod \"12adb8c6-7644-4a72-8e70-7ef5b5306805\" (UID: \"12adb8c6-7644-4a72-8e70-7ef5b5306805\") " Jul 16 16:51:56.112539 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:51:56.112482 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12adb8c6-7644-4a72-8e70-7ef5b5306805-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "12adb8c6-7644-4a72-8e70-7ef5b5306805" (UID: "12adb8c6-7644-4a72-8e70-7ef5b5306805"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 16:51:56.112539 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:51:56.112511 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12adb8c6-7644-4a72-8e70-7ef5b5306805-istiod-ca-cert\") pod \"12adb8c6-7644-4a72-8e70-7ef5b5306805\" (UID: \"12adb8c6-7644-4a72-8e70-7ef5b5306805\") " Jul 16 16:51:56.112799 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:51:56.112568 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12adb8c6-7644-4a72-8e70-7ef5b5306805-istio-data\") pod \"12adb8c6-7644-4a72-8e70-7ef5b5306805\" (UID: \"12adb8c6-7644-4a72-8e70-7ef5b5306805\") " Jul 16 16:51:56.112799 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:51:56.112610 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12adb8c6-7644-4a72-8e70-7ef5b5306805-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "12adb8c6-7644-4a72-8e70-7ef5b5306805" (UID: "12adb8c6-7644-4a72-8e70-7ef5b5306805"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 16:51:56.112799 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:51:56.112629 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12adb8c6-7644-4a72-8e70-7ef5b5306805-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "12adb8c6-7644-4a72-8e70-7ef5b5306805" (UID: "12adb8c6-7644-4a72-8e70-7ef5b5306805"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 16:51:56.112957 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:51:56.112908 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/12adb8c6-7644-4a72-8e70-7ef5b5306805-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "12adb8c6-7644-4a72-8e70-7ef5b5306805" (UID: "12adb8c6-7644-4a72-8e70-7ef5b5306805"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 16:51:56.113026 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:51:56.113009 2577 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12adb8c6-7644-4a72-8e70-7ef5b5306805-workload-certs\") on node \"ip-10-0-142-101.ec2.internal\" DevicePath \"\"" Jul 16 16:51:56.113082 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:51:56.113031 2577 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12adb8c6-7644-4a72-8e70-7ef5b5306805-credential-socket\") on node \"ip-10-0-142-101.ec2.internal\" DevicePath \"\"" Jul 16 16:51:56.113082 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:51:56.113048 2577 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12adb8c6-7644-4a72-8e70-7ef5b5306805-workload-socket\") on node \"ip-10-0-142-101.ec2.internal\" DevicePath \"\"" Jul 16 16:51:56.113082 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:51:56.113065 2577 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12adb8c6-7644-4a72-8e70-7ef5b5306805-istiod-ca-cert\") on node \"ip-10-0-142-101.ec2.internal\" DevicePath \"\"" Jul 16 16:51:56.113888 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:51:56.113864 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12adb8c6-7644-4a72-8e70-7ef5b5306805-istio-data" (OuterVolumeSpecName: "istio-data") pod "12adb8c6-7644-4a72-8e70-7ef5b5306805" (UID: "12adb8c6-7644-4a72-8e70-7ef5b5306805"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 16:51:56.114917 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:51:56.114771 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12adb8c6-7644-4a72-8e70-7ef5b5306805-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "12adb8c6-7644-4a72-8e70-7ef5b5306805" (UID: "12adb8c6-7644-4a72-8e70-7ef5b5306805"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 16:51:56.114917 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:51:56.114853 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12adb8c6-7644-4a72-8e70-7ef5b5306805-istio-token" (OuterVolumeSpecName: "istio-token") pod "12adb8c6-7644-4a72-8e70-7ef5b5306805" (UID: "12adb8c6-7644-4a72-8e70-7ef5b5306805"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 16:51:56.114917 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:51:56.114880 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/12adb8c6-7644-4a72-8e70-7ef5b5306805-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "12adb8c6-7644-4a72-8e70-7ef5b5306805" (UID: "12adb8c6-7644-4a72-8e70-7ef5b5306805"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 16 16:51:56.114917 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:51:56.114880 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12adb8c6-7644-4a72-8e70-7ef5b5306805-kube-api-access-phfzm" (OuterVolumeSpecName: "kube-api-access-phfzm") pod "12adb8c6-7644-4a72-8e70-7ef5b5306805" (UID: "12adb8c6-7644-4a72-8e70-7ef5b5306805"). InnerVolumeSpecName "kube-api-access-phfzm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 16:51:56.214031 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:51:56.213988 2577 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12adb8c6-7644-4a72-8e70-7ef5b5306805-istio-envoy\") on node \"ip-10-0-142-101.ec2.internal\" DevicePath \"\"" Jul 16 16:51:56.214031 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:51:56.214023 2577 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12adb8c6-7644-4a72-8e70-7ef5b5306805-istio-podinfo\") on node \"ip-10-0-142-101.ec2.internal\" DevicePath \"\"" Jul 16 16:51:56.214031 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:51:56.214032 2577 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12adb8c6-7644-4a72-8e70-7ef5b5306805-istio-data\") on node \"ip-10-0-142-101.ec2.internal\" DevicePath \"\"" Jul 16 16:51:56.214031 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:51:56.214041 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-phfzm\" (UniqueName: \"kubernetes.io/projected/12adb8c6-7644-4a72-8e70-7ef5b5306805-kube-api-access-phfzm\") on node \"ip-10-0-142-101.ec2.internal\" DevicePath \"\"" Jul 16 16:51:56.214302 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:51:56.214050 2577 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/12adb8c6-7644-4a72-8e70-7ef5b5306805-istio-token\") on node \"ip-10-0-142-101.ec2.internal\" DevicePath \"\"" Jul 16 16:51:56.755226 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:51:56.755188 2577 generic.go:356] "Generic (PLEG): container finished" podID="12adb8c6-7644-4a72-8e70-7ef5b5306805" containerID="e94d5b7fb5483e7f53c7d31b21a90a1aeef1c8f99adb597d29847660f52381cb" exitCode=0 Jul 16 16:51:56.755511 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:51:56.755243 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-7a95ef74-openshift-default-c6b94f6c5-fndx2" Jul 16 16:51:56.755511 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:51:56.755275 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-7a95ef74-openshift-default-c6b94f6c5-fndx2" event={"ID":"12adb8c6-7644-4a72-8e70-7ef5b5306805","Type":"ContainerDied","Data":"e94d5b7fb5483e7f53c7d31b21a90a1aeef1c8f99adb597d29847660f52381cb"} Jul 16 16:51:56.755511 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:51:56.755334 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-7a95ef74-openshift-default-c6b94f6c5-fndx2" event={"ID":"12adb8c6-7644-4a72-8e70-7ef5b5306805","Type":"ContainerDied","Data":"96818e98db4f2e2a21ca1aa2a3b85f36c2e4d35ebc44cd46842959930d6071ac"} Jul 16 16:51:56.755511 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:51:56.755357 2577 scope.go:117] "RemoveContainer" containerID="e94d5b7fb5483e7f53c7d31b21a90a1aeef1c8f99adb597d29847660f52381cb" Jul 16 16:51:56.764708 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:51:56.764689 2577 scope.go:117] "RemoveContainer" containerID="8a84d0b7399c07b81c6ff3b36aab00677d68cec3be9324957d0cdb0114356376" Jul 16 16:51:56.773079 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:51:56.773060 2577 scope.go:117] "RemoveContainer" containerID="e94d5b7fb5483e7f53c7d31b21a90a1aeef1c8f99adb597d29847660f52381cb" Jul 16 16:51:56.773354 ip-10-0-142-101 kubenswrapper[2577]: E0716 16:51:56.773331 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e94d5b7fb5483e7f53c7d31b21a90a1aeef1c8f99adb597d29847660f52381cb\": container with ID starting with e94d5b7fb5483e7f53c7d31b21a90a1aeef1c8f99adb597d29847660f52381cb not found: ID does not exist" containerID="e94d5b7fb5483e7f53c7d31b21a90a1aeef1c8f99adb597d29847660f52381cb" Jul 16 16:51:56.773424 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:51:56.773363 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e94d5b7fb5483e7f53c7d31b21a90a1aeef1c8f99adb597d29847660f52381cb"} err="failed to get container status \"e94d5b7fb5483e7f53c7d31b21a90a1aeef1c8f99adb597d29847660f52381cb\": rpc error: code = NotFound desc = could not find container \"e94d5b7fb5483e7f53c7d31b21a90a1aeef1c8f99adb597d29847660f52381cb\": container with ID starting with e94d5b7fb5483e7f53c7d31b21a90a1aeef1c8f99adb597d29847660f52381cb not found: ID does not exist" Jul 16 16:51:56.773424 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:51:56.773416 2577 scope.go:117] "RemoveContainer" containerID="8a84d0b7399c07b81c6ff3b36aab00677d68cec3be9324957d0cdb0114356376" Jul 16 16:51:56.773670 ip-10-0-142-101 kubenswrapper[2577]: E0716 16:51:56.773654 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a84d0b7399c07b81c6ff3b36aab00677d68cec3be9324957d0cdb0114356376\": container with ID starting with 8a84d0b7399c07b81c6ff3b36aab00677d68cec3be9324957d0cdb0114356376 not found: ID does not exist" containerID="8a84d0b7399c07b81c6ff3b36aab00677d68cec3be9324957d0cdb0114356376" Jul 16 16:51:56.773712 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:51:56.773677 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a84d0b7399c07b81c6ff3b36aab00677d68cec3be9324957d0cdb0114356376"} err="failed to get container status \"8a84d0b7399c07b81c6ff3b36aab00677d68cec3be9324957d0cdb0114356376\": rpc error: code = NotFound desc = could not find container \"8a84d0b7399c07b81c6ff3b36aab00677d68cec3be9324957d0cdb0114356376\": container with ID starting with 8a84d0b7399c07b81c6ff3b36aab00677d68cec3be9324957d0cdb0114356376 not found: ID does not exist" Jul 16 16:51:56.778291 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:51:56.778268 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-7a95ef74-openshift-default-c6b94f6c5-fndx2"] Jul 16 16:51:56.784131 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:51:56.784104 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-7a95ef74-openshift-default-c6b94f6c5-fndx2"] Jul 16 16:51:57.162438 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:51:57.162400 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c61ec96a-kp984"] Jul 16 16:51:57.162838 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:51:57.162824 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="12adb8c6-7644-4a72-8e70-7ef5b5306805" containerName="istio-proxy" Jul 16 16:51:57.162838 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:51:57.162839 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="12adb8c6-7644-4a72-8e70-7ef5b5306805" containerName="istio-proxy" Jul 16 16:51:57.162913 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:51:57.162860 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="12adb8c6-7644-4a72-8e70-7ef5b5306805" containerName="istio-proxy" Jul 16 16:51:57.162913 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:51:57.162875 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="12adb8c6-7644-4a72-8e70-7ef5b5306805" containerName="istio-proxy" Jul 16 16:51:57.163020 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:51:57.162937 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="12adb8c6-7644-4a72-8e70-7ef5b5306805" containerName="istio-proxy" Jul 16 16:51:57.163020 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:51:57.162947 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="12adb8c6-7644-4a72-8e70-7ef5b5306805" containerName="istio-proxy" Jul 16 16:51:57.167810 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:51:57.167787 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c61ec96a-kp984" Jul 16 16:51:57.170256 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:51:57.170233 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dssfh\"" Jul 16 16:51:57.170359 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:51:57.170314 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 16 16:51:57.173663 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:51:57.173642 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c61ec96a-kp984"] Jul 16 16:51:57.325988 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:51:57.325946 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c86f886-9e85-4e26-b83f-eab8135b4a9a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-c61ec96a-kp984\" (UID: \"2c86f886-9e85-4e26-b83f-eab8135b4a9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c61ec96a-kp984" Jul 16 16:51:57.427330 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:51:57.427221 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c86f886-9e85-4e26-b83f-eab8135b4a9a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-c61ec96a-kp984\" (UID: \"2c86f886-9e85-4e26-b83f-eab8135b4a9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c61ec96a-kp984" Jul 16 16:51:57.427904 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:51:57.427879 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c86f886-9e85-4e26-b83f-eab8135b4a9a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-c61ec96a-kp984\" (UID: \"2c86f886-9e85-4e26-b83f-eab8135b4a9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c61ec96a-kp984" Jul 16 16:51:57.480199 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:51:57.480164 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c61ec96a-kp984" Jul 16 16:51:57.626787 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:51:57.626746 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c61ec96a-kp984"] Jul 16 16:51:57.630827 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:51:57.630796 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c86f886_9e85_4e26_b83f_eab8135b4a9a.slice/crio-787a543c048d7e6e07262c8e522ae88dce8f87808ba5be4cdb595140da1a39a5 WatchSource:0}: Error finding container 787a543c048d7e6e07262c8e522ae88dce8f87808ba5be4cdb595140da1a39a5: Status 404 returned error can't find the container with id 787a543c048d7e6e07262c8e522ae88dce8f87808ba5be4cdb595140da1a39a5 Jul 16 16:51:57.763192 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:51:57.763162 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c61ec96a-kp984" event={"ID":"2c86f886-9e85-4e26-b83f-eab8135b4a9a","Type":"ContainerStarted","Data":"787a543c048d7e6e07262c8e522ae88dce8f87808ba5be4cdb595140da1a39a5"} Jul 16 16:51:58.017846 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:51:58.017769 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="12adb8c6-7644-4a72-8e70-7ef5b5306805" path="/var/lib/kubelet/pods/12adb8c6-7644-4a72-8e70-7ef5b5306805/volumes" Jul 16 16:51:58.769617 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:51:58.769573 2577 generic.go:356] "Generic (PLEG): container finished" podID="2c86f886-9e85-4e26-b83f-eab8135b4a9a" containerID="4202ed9e00dca4298b783cc2d8b550b90060a6cb10c7f7d4611d4efe0c59915f" exitCode=7 Jul 16 16:51:58.770103 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:51:58.769662 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c61ec96a-kp984" event={"ID":"2c86f886-9e85-4e26-b83f-eab8135b4a9a","Type":"ContainerDied","Data":"4202ed9e00dca4298b783cc2d8b550b90060a6cb10c7f7d4611d4efe0c59915f"} Jul 16 16:51:58.770103 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:51:58.769863 2577 scope.go:117] "RemoveContainer" containerID="4202ed9e00dca4298b783cc2d8b550b90060a6cb10c7f7d4611d4efe0c59915f" Jul 16 16:51:59.779183 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:51:59.779089 2577 generic.go:356] "Generic (PLEG): container finished" podID="2c86f886-9e85-4e26-b83f-eab8135b4a9a" containerID="04deb3deaf54c7bd7faba2781bfce34bbd3896444d8af58452e7c9252daeb1ac" exitCode=7 Jul 16 16:51:59.779774 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:51:59.779149 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c61ec96a-kp984" event={"ID":"2c86f886-9e85-4e26-b83f-eab8135b4a9a","Type":"ContainerDied","Data":"04deb3deaf54c7bd7faba2781bfce34bbd3896444d8af58452e7c9252daeb1ac"} Jul 16 16:51:59.779846 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:51:59.779507 2577 scope.go:117] "RemoveContainer" containerID="04deb3deaf54c7bd7faba2781bfce34bbd3896444d8af58452e7c9252daeb1ac" Jul 16 16:51:59.780034 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:51:59.780017 2577 scope.go:117] "RemoveContainer" containerID="4202ed9e00dca4298b783cc2d8b550b90060a6cb10c7f7d4611d4efe0c59915f" Jul 16 16:51:59.780227 ip-10-0-142-101 kubenswrapper[2577]: E0716 16:51:59.780084 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-c61ec96a-kp984_odh-ai-gateway-infra(2c86f886-9e85-4e26-b83f-eab8135b4a9a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c61ec96a-kp984" podUID="2c86f886-9e85-4e26-b83f-eab8135b4a9a" Jul 16 16:52:00.786934 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:52:00.786907 2577 scope.go:117] "RemoveContainer" containerID="04deb3deaf54c7bd7faba2781bfce34bbd3896444d8af58452e7c9252daeb1ac" Jul 16 16:52:00.787294 ip-10-0-142-101 kubenswrapper[2577]: E0716 16:52:00.787102 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-c61ec96a-kp984_odh-ai-gateway-infra(2c86f886-9e85-4e26-b83f-eab8135b4a9a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c61ec96a-kp984" podUID="2c86f886-9e85-4e26-b83f-eab8135b4a9a" Jul 16 16:52:07.826922 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:52:07.826887 2577 generic.go:356] "Generic (PLEG): container finished" podID="e3da487a-0665-40b7-ab25-329cba6f1b2c" containerID="fbbcfb44f0be91cad0b80ca4fdd7b0fd0a366132faa7d6e25bf7bb522949f3b3" exitCode=0 Jul 16 16:52:07.827336 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:52:07.826962 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-c61ec96a-gw-openshift-default-769498d775-8cpnt" event={"ID":"e3da487a-0665-40b7-ab25-329cba6f1b2c","Type":"ContainerDied","Data":"fbbcfb44f0be91cad0b80ca4fdd7b0fd0a366132faa7d6e25bf7bb522949f3b3"} Jul 16 16:52:07.827436 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:52:07.827418 2577 scope.go:117] "RemoveContainer" containerID="fbbcfb44f0be91cad0b80ca4fdd7b0fd0a366132faa7d6e25bf7bb522949f3b3" Jul 16 16:52:07.828480 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:52:07.828453 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 16 16:52:07.828552 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:52:07.828535 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 16 16:52:07.828595 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:52:07.828582 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 16 16:52:08.737966 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:52:08.737922 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-c61ec96a-gw-openshift-default-769498d775-8cpnt" Jul 16 16:52:08.839409 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:52:08.835427 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-c61ec96a-gw-openshift-default-769498d775-8cpnt" event={"ID":"e3da487a-0665-40b7-ab25-329cba6f1b2c","Type":"ContainerStarted","Data":"1268fe8b637ce28d0e84d172aac82d3b8bd0171f45416311d7e9be93a7722b3c"} Jul 16 16:52:09.730270 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:52:09.730223 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-c61ec96a-gw-openshift-default-769498d775-8cpnt" Jul 16 16:52:09.731509 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:52:09.731485 2577 patch_prober.go:28] interesting pod/e2e-migrate-c61ec96a-gw-openshift-default-769498d775-8cpnt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body= Jul 16 16:52:09.731613 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:52:09.731543 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-c61ec96a-gw-openshift-default-769498d775-8cpnt" podUID="e3da487a-0665-40b7-ab25-329cba6f1b2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" Jul 16 16:52:10.730501 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:52:10.730459 2577 patch_prober.go:28] interesting pod/e2e-migrate-c61ec96a-gw-openshift-default-769498d775-8cpnt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body= Jul 16 16:52:10.730905 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:52:10.730532 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-c61ec96a-gw-openshift-default-769498d775-8cpnt" podUID="e3da487a-0665-40b7-ab25-329cba6f1b2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" Jul 16 16:52:11.729840 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:52:11.729805 2577 patch_prober.go:28] interesting pod/e2e-migrate-c61ec96a-gw-openshift-default-769498d775-8cpnt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body= Jul 16 16:52:11.730019 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:52:11.729868 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-c61ec96a-gw-openshift-default-769498d775-8cpnt" podUID="e3da487a-0665-40b7-ab25-329cba6f1b2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" Jul 16 16:52:12.730062 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:52:12.730020 2577 patch_prober.go:28] interesting pod/e2e-migrate-c61ec96a-gw-openshift-default-769498d775-8cpnt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body= Jul 16 16:52:12.730502 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:52:12.730091 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-c61ec96a-gw-openshift-default-769498d775-8cpnt" podUID="e3da487a-0665-40b7-ab25-329cba6f1b2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" Jul 16 16:52:13.734111 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:52:13.734082 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-c61ec96a-gw-openshift-default-769498d775-8cpnt" Jul 16 16:52:13.734551 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:52:13.734366 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-c61ec96a-gw-openshift-default-769498d775-8cpnt" Jul 16 16:52:13.735068 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:52:13.735052 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-c61ec96a-gw-openshift-default-769498d775-8cpnt" Jul 16 16:52:14.012749 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:52:14.012667 2577 scope.go:117] "RemoveContainer" containerID="04deb3deaf54c7bd7faba2781bfce34bbd3896444d8af58452e7c9252daeb1ac" Jul 16 16:52:14.876902 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:52:14.876870 2577 generic.go:356] "Generic (PLEG): container finished" podID="2c86f886-9e85-4e26-b83f-eab8135b4a9a" containerID="1c1bef34ca10986cdb489bd597e99e382b01e19cff2e2e2d4852b2cf2f2c82e7" exitCode=0 Jul 16 16:52:14.877302 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:52:14.876946 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c61ec96a-kp984" event={"ID":"2c86f886-9e85-4e26-b83f-eab8135b4a9a","Type":"ContainerDied","Data":"1c1bef34ca10986cdb489bd597e99e382b01e19cff2e2e2d4852b2cf2f2c82e7"} Jul 16 16:52:14.877302 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:52:14.877010 2577 scope.go:117] "RemoveContainer" containerID="04deb3deaf54c7bd7faba2781bfce34bbd3896444d8af58452e7c9252daeb1ac" Jul 16 16:52:15.924515 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:52:15.924480 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c61ec96a-kp984"] Jul 16 16:52:16.023740 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:52:16.023715 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c61ec96a-kp984" Jul 16 16:52:16.119045 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:52:16.119010 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c86f886-9e85-4e26-b83f-eab8135b4a9a-maas-api-service-ca\") pod \"2c86f886-9e85-4e26-b83f-eab8135b4a9a\" (UID: \"2c86f886-9e85-4e26-b83f-eab8135b4a9a\") " Jul 16 16:52:16.119400 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:52:16.119361 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c86f886-9e85-4e26-b83f-eab8135b4a9a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "2c86f886-9e85-4e26-b83f-eab8135b4a9a" (UID: "2c86f886-9e85-4e26-b83f-eab8135b4a9a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 16:52:16.220676 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:52:16.220590 2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c86f886-9e85-4e26-b83f-eab8135b4a9a-maas-api-service-ca\") on node \"ip-10-0-142-101.ec2.internal\" DevicePath \"\"" Jul 16 16:52:16.889442 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:52:16.889413 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c61ec96a-kp984" Jul 16 16:52:16.889442 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:52:16.889434 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c61ec96a-kp984" event={"ID":"2c86f886-9e85-4e26-b83f-eab8135b4a9a","Type":"ContainerDied","Data":"787a543c048d7e6e07262c8e522ae88dce8f87808ba5be4cdb595140da1a39a5"} Jul 16 16:52:16.889688 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:52:16.889520 2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="787a543c048d7e6e07262c8e522ae88dce8f87808ba5be4cdb595140da1a39a5" Jul 16 16:52:16.918490 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:52:16.918457 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c61ec96a-kp984"] Jul 16 16:52:16.924599 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:52:16.924568 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c61ec96a-kp984"] Jul 16 16:52:18.018104 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:52:18.018063 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2c86f886-9e85-4e26-b83f-eab8135b4a9a" path="/var/lib/kubelet/pods/2c86f886-9e85-4e26-b83f-eab8135b4a9a/volumes" Jul 16 16:52:47.235833 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:52:47.235799 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c61ec96a-mswhf"] Jul 16 16:52:47.236245 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:52:47.236223 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2c86f886-9e85-4e26-b83f-eab8135b4a9a" containerName="revoke-keys" Jul 16 16:52:47.236245 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:52:47.236235 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="2c86f886-9e85-4e26-b83f-eab8135b4a9a" containerName="revoke-keys" Jul 16 16:52:47.236316 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:52:47.236248 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2c86f886-9e85-4e26-b83f-eab8135b4a9a" containerName="revoke-keys" Jul 16 16:52:47.236316 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:52:47.236254 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="2c86f886-9e85-4e26-b83f-eab8135b4a9a" containerName="revoke-keys" Jul 16 16:52:47.236380 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:52:47.236335 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="2c86f886-9e85-4e26-b83f-eab8135b4a9a" containerName="revoke-keys" Jul 16 16:52:47.236380 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:52:47.236343 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="2c86f886-9e85-4e26-b83f-eab8135b4a9a" containerName="revoke-keys" Jul 16 16:52:47.236380 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:52:47.236350 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="2c86f886-9e85-4e26-b83f-eab8135b4a9a" containerName="revoke-keys" Jul 16 16:52:47.239359 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:52:47.239341 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c61ec96a-mswhf" Jul 16 16:52:47.243019 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:52:47.242988 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 16 16:52:47.243155 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:52:47.243057 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dssfh\"" Jul 16 16:52:47.268018 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:52:47.263262 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c61ec96a-mswhf"] Jul 16 16:52:47.330803 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:52:47.330757 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4df3ac0-fff6-41e5-a4fd-82d06f73226f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-c61ec96a-mswhf\" (UID: \"c4df3ac0-fff6-41e5-a4fd-82d06f73226f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c61ec96a-mswhf" Jul 16 16:52:47.431569 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:52:47.431526 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4df3ac0-fff6-41e5-a4fd-82d06f73226f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-c61ec96a-mswhf\" (UID: \"c4df3ac0-fff6-41e5-a4fd-82d06f73226f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c61ec96a-mswhf" Jul 16 16:52:47.432202 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:52:47.432181 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4df3ac0-fff6-41e5-a4fd-82d06f73226f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-c61ec96a-mswhf\" (UID: \"c4df3ac0-fff6-41e5-a4fd-82d06f73226f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c61ec96a-mswhf" Jul 16 16:52:47.549748 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:52:47.549649 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c61ec96a-mswhf" Jul 16 16:52:47.897993 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:52:47.897968 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c61ec96a-mswhf"] Jul 16 16:52:47.900271 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:52:47.900241 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4df3ac0_fff6_41e5_a4fd_82d06f73226f.slice/crio-fe67eeded6e82b7addccac9ed63e3d3772daaec7e40a5fae4c647e437d8be6ef WatchSource:0}: Error finding container fe67eeded6e82b7addccac9ed63e3d3772daaec7e40a5fae4c647e437d8be6ef: Status 404 returned error can't find the container with id fe67eeded6e82b7addccac9ed63e3d3772daaec7e40a5fae4c647e437d8be6ef Jul 16 16:52:48.057869 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:52:48.057832 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c61ec96a-mswhf" event={"ID":"c4df3ac0-fff6-41e5-a4fd-82d06f73226f","Type":"ContainerStarted","Data":"14261209d8288a1a1c58258f48a40ba7b746315e8606d54ff0df905af09b949b"} Jul 16 16:52:48.058035 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:52:48.057892 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c61ec96a-mswhf" event={"ID":"c4df3ac0-fff6-41e5-a4fd-82d06f73226f","Type":"ContainerStarted","Data":"fe67eeded6e82b7addccac9ed63e3d3772daaec7e40a5fae4c647e437d8be6ef"} Jul 16 16:52:48.077904 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:52:48.077845 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c61ec96a-mswhf" podStartSLOduration=1.077826842 podStartE2EDuration="1.077826842s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 16:52:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 16:52:48.076303924 +0000 UTC m=+1626.580361776" watchObservedRunningTime="2026-07-16 16:52:48.077826842 +0000 UTC m=+1626.581884692" Jul 16 16:52:49.064241 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:52:49.064207 2577 generic.go:356] "Generic (PLEG): container finished" podID="c4df3ac0-fff6-41e5-a4fd-82d06f73226f" containerID="14261209d8288a1a1c58258f48a40ba7b746315e8606d54ff0df905af09b949b" exitCode=0 Jul 16 16:52:49.064643 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:52:49.064295 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c61ec96a-mswhf" event={"ID":"c4df3ac0-fff6-41e5-a4fd-82d06f73226f","Type":"ContainerDied","Data":"14261209d8288a1a1c58258f48a40ba7b746315e8606d54ff0df905af09b949b"} Jul 16 16:52:50.209114 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:52:50.209083 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c61ec96a-mswhf" Jul 16 16:52:50.262558 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:52:50.262525 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4df3ac0-fff6-41e5-a4fd-82d06f73226f-maas-api-service-ca\") pod \"c4df3ac0-fff6-41e5-a4fd-82d06f73226f\" (UID: \"c4df3ac0-fff6-41e5-a4fd-82d06f73226f\") " Jul 16 16:52:50.262889 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:52:50.262866 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4df3ac0-fff6-41e5-a4fd-82d06f73226f-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "c4df3ac0-fff6-41e5-a4fd-82d06f73226f" (UID: "c4df3ac0-fff6-41e5-a4fd-82d06f73226f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 16:52:50.363457 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:52:50.363354 2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4df3ac0-fff6-41e5-a4fd-82d06f73226f-maas-api-service-ca\") on node \"ip-10-0-142-101.ec2.internal\" DevicePath \"\"" Jul 16 16:52:51.076402 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:52:51.076363 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c61ec96a-mswhf" Jul 16 16:52:51.076599 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:52:51.076365 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c61ec96a-mswhf" event={"ID":"c4df3ac0-fff6-41e5-a4fd-82d06f73226f","Type":"ContainerDied","Data":"fe67eeded6e82b7addccac9ed63e3d3772daaec7e40a5fae4c647e437d8be6ef"} Jul 16 16:52:51.076599 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:52:51.076507 2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fe67eeded6e82b7addccac9ed63e3d3772daaec7e40a5fae4c647e437d8be6ef" Jul 16 16:52:57.292601 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:52:57.292570 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-c61ec96a-gw-openshift-default-769498d775-8cpnt"] Jul 16 16:52:57.293018 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:52:57.292785 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-c61ec96a-gw-openshift-default-769498d775-8cpnt" podUID="e3da487a-0665-40b7-ab25-329cba6f1b2c" containerName="istio-proxy" containerID="cri-o://1268fe8b637ce28d0e84d172aac82d3b8bd0171f45416311d7e9be93a7722b3c" gracePeriod=30 Jul 16 16:52:58.737076 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:52:58.737037 2577 patch_prober.go:28] interesting pod/e2e-migrate-c61ec96a-gw-openshift-default-769498d775-8cpnt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 16 16:52:58.737499 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:52:58.737099 2577 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-migrate-c61ec96a-gw-openshift-default-769498d775-8cpnt" podUID="e3da487a-0665-40b7-ab25-329cba6f1b2c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 16 16:53:02.557030 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:53:02.556966 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-c61ec96a-gw-openshift-default-769498d775-8cpnt" Jul 16 16:53:02.689041 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:53:02.689000 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3da487a-0665-40b7-ab25-329cba6f1b2c-istiod-ca-cert\") pod \"e3da487a-0665-40b7-ab25-329cba6f1b2c\" (UID: \"e3da487a-0665-40b7-ab25-329cba6f1b2c\") " Jul 16 16:53:02.689208 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:53:02.689051 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3da487a-0665-40b7-ab25-329cba6f1b2c-credential-socket\") pod \"e3da487a-0665-40b7-ab25-329cba6f1b2c\" (UID: \"e3da487a-0665-40b7-ab25-329cba6f1b2c\") " Jul 16 16:53:02.689208 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:53:02.689080 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3da487a-0665-40b7-ab25-329cba6f1b2c-istio-envoy\") pod \"e3da487a-0665-40b7-ab25-329cba6f1b2c\" (UID: \"e3da487a-0665-40b7-ab25-329cba6f1b2c\") " Jul 16 16:53:02.689208 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:53:02.689112 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nx8cg\" (UniqueName: \"kubernetes.io/projected/e3da487a-0665-40b7-ab25-329cba6f1b2c-kube-api-access-nx8cg\") pod \"e3da487a-0665-40b7-ab25-329cba6f1b2c\" (UID: \"e3da487a-0665-40b7-ab25-329cba6f1b2c\") " Jul 16 16:53:02.689208 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:53:02.689153 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3da487a-0665-40b7-ab25-329cba6f1b2c-workload-socket\") pod \"e3da487a-0665-40b7-ab25-329cba6f1b2c\" (UID: \"e3da487a-0665-40b7-ab25-329cba6f1b2c\") " Jul 16 16:53:02.689465 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:53:02.689208 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3da487a-0665-40b7-ab25-329cba6f1b2c-istio-podinfo\") pod \"e3da487a-0665-40b7-ab25-329cba6f1b2c\" (UID: \"e3da487a-0665-40b7-ab25-329cba6f1b2c\") " Jul 16 16:53:02.689465 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:53:02.689273 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3da487a-0665-40b7-ab25-329cba6f1b2c-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "e3da487a-0665-40b7-ab25-329cba6f1b2c" (UID: "e3da487a-0665-40b7-ab25-329cba6f1b2c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 16:53:02.689465 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:53:02.689343 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3da487a-0665-40b7-ab25-329cba6f1b2c-workload-certs\") pod \"e3da487a-0665-40b7-ab25-329cba6f1b2c\" (UID: \"e3da487a-0665-40b7-ab25-329cba6f1b2c\") " Jul 16 16:53:02.689465 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:53:02.689370 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3da487a-0665-40b7-ab25-329cba6f1b2c-istio-token\") pod \"e3da487a-0665-40b7-ab25-329cba6f1b2c\" (UID: \"e3da487a-0665-40b7-ab25-329cba6f1b2c\") " Jul 16 16:53:02.689465 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:53:02.689425 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3da487a-0665-40b7-ab25-329cba6f1b2c-istio-data\") pod \"e3da487a-0665-40b7-ab25-329cba6f1b2c\" (UID: \"e3da487a-0665-40b7-ab25-329cba6f1b2c\") " Jul 16 16:53:02.689726 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:53:02.689506 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3da487a-0665-40b7-ab25-329cba6f1b2c-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "e3da487a-0665-40b7-ab25-329cba6f1b2c" (UID: "e3da487a-0665-40b7-ab25-329cba6f1b2c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 16:53:02.689726 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:53:02.689514 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3da487a-0665-40b7-ab25-329cba6f1b2c-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "e3da487a-0665-40b7-ab25-329cba6f1b2c" (UID: "e3da487a-0665-40b7-ab25-329cba6f1b2c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 16:53:02.689726 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:53:02.689673 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3da487a-0665-40b7-ab25-329cba6f1b2c-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "e3da487a-0665-40b7-ab25-329cba6f1b2c" (UID: "e3da487a-0665-40b7-ab25-329cba6f1b2c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 16:53:02.689878 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:53:02.689839 2577 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3da487a-0665-40b7-ab25-329cba6f1b2c-workload-certs\") on node \"ip-10-0-142-101.ec2.internal\" DevicePath \"\"" Jul 16 16:53:02.689878 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:53:02.689860 2577 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3da487a-0665-40b7-ab25-329cba6f1b2c-istiod-ca-cert\") on node \"ip-10-0-142-101.ec2.internal\" DevicePath \"\"" Jul 16 16:53:02.689878 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:53:02.689873 2577 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3da487a-0665-40b7-ab25-329cba6f1b2c-credential-socket\") on node \"ip-10-0-142-101.ec2.internal\" DevicePath \"\"" Jul 16 16:53:02.690029 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:53:02.689888 2577 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3da487a-0665-40b7-ab25-329cba6f1b2c-workload-socket\") on node \"ip-10-0-142-101.ec2.internal\" DevicePath \"\"" Jul 16 16:53:02.690264 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:53:02.690239 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3da487a-0665-40b7-ab25-329cba6f1b2c-istio-data" (OuterVolumeSpecName: "istio-data") pod "e3da487a-0665-40b7-ab25-329cba6f1b2c" (UID: "e3da487a-0665-40b7-ab25-329cba6f1b2c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 16:53:02.691487 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:53:02.691456 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3da487a-0665-40b7-ab25-329cba6f1b2c-istio-token" (OuterVolumeSpecName: "istio-token") pod "e3da487a-0665-40b7-ab25-329cba6f1b2c" (UID: "e3da487a-0665-40b7-ab25-329cba6f1b2c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 16:53:02.691697 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:53:02.691679 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e3da487a-0665-40b7-ab25-329cba6f1b2c-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "e3da487a-0665-40b7-ab25-329cba6f1b2c" (UID: "e3da487a-0665-40b7-ab25-329cba6f1b2c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 16 16:53:02.691842 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:53:02.691823 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3da487a-0665-40b7-ab25-329cba6f1b2c-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "e3da487a-0665-40b7-ab25-329cba6f1b2c" (UID: "e3da487a-0665-40b7-ab25-329cba6f1b2c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 16:53:02.691908 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:53:02.691826 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3da487a-0665-40b7-ab25-329cba6f1b2c-kube-api-access-nx8cg" (OuterVolumeSpecName: "kube-api-access-nx8cg") pod "e3da487a-0665-40b7-ab25-329cba6f1b2c" (UID: "e3da487a-0665-40b7-ab25-329cba6f1b2c"). InnerVolumeSpecName "kube-api-access-nx8cg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 16:53:02.791143 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:53:02.791101 2577 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3da487a-0665-40b7-ab25-329cba6f1b2c-istio-token\") on node \"ip-10-0-142-101.ec2.internal\" DevicePath \"\"" Jul 16 16:53:02.791143 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:53:02.791134 2577 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3da487a-0665-40b7-ab25-329cba6f1b2c-istio-data\") on node \"ip-10-0-142-101.ec2.internal\" DevicePath \"\"" Jul 16 16:53:02.791143 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:53:02.791142 2577 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3da487a-0665-40b7-ab25-329cba6f1b2c-istio-envoy\") on node \"ip-10-0-142-101.ec2.internal\" DevicePath \"\"" Jul 16 16:53:02.791143 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:53:02.791150 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nx8cg\" (UniqueName: \"kubernetes.io/projected/e3da487a-0665-40b7-ab25-329cba6f1b2c-kube-api-access-nx8cg\") on node \"ip-10-0-142-101.ec2.internal\" DevicePath \"\"" Jul 16 16:53:02.791445 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:53:02.791159 2577 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3da487a-0665-40b7-ab25-329cba6f1b2c-istio-podinfo\") on node \"ip-10-0-142-101.ec2.internal\" DevicePath \"\"" Jul 16 16:53:03.137073 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:53:03.137037 2577 generic.go:356] "Generic (PLEG): container finished" podID="e3da487a-0665-40b7-ab25-329cba6f1b2c" containerID="1268fe8b637ce28d0e84d172aac82d3b8bd0171f45416311d7e9be93a7722b3c" exitCode=0 Jul 16 16:53:03.137242 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:53:03.137096 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-c61ec96a-gw-openshift-default-769498d775-8cpnt" Jul 16 16:53:03.137242 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:53:03.137122 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-c61ec96a-gw-openshift-default-769498d775-8cpnt" event={"ID":"e3da487a-0665-40b7-ab25-329cba6f1b2c","Type":"ContainerDied","Data":"1268fe8b637ce28d0e84d172aac82d3b8bd0171f45416311d7e9be93a7722b3c"} Jul 16 16:53:03.137242 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:53:03.137171 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-c61ec96a-gw-openshift-default-769498d775-8cpnt" event={"ID":"e3da487a-0665-40b7-ab25-329cba6f1b2c","Type":"ContainerDied","Data":"48b43aed09bdd3c163d4eaccf9d27e90cf44de32aaa09e57d1bd605060764fc0"} Jul 16 16:53:03.137242 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:53:03.137185 2577 scope.go:117] "RemoveContainer" containerID="1268fe8b637ce28d0e84d172aac82d3b8bd0171f45416311d7e9be93a7722b3c" Jul 16 16:53:03.147715 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:53:03.147693 2577 scope.go:117] "RemoveContainer" containerID="fbbcfb44f0be91cad0b80ca4fdd7b0fd0a366132faa7d6e25bf7bb522949f3b3" Jul 16 16:53:03.157027 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:53:03.157006 2577 scope.go:117] "RemoveContainer" containerID="1268fe8b637ce28d0e84d172aac82d3b8bd0171f45416311d7e9be93a7722b3c" Jul 16 16:53:03.157331 ip-10-0-142-101 kubenswrapper[2577]: E0716 16:53:03.157312 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1268fe8b637ce28d0e84d172aac82d3b8bd0171f45416311d7e9be93a7722b3c\": container with ID starting with 1268fe8b637ce28d0e84d172aac82d3b8bd0171f45416311d7e9be93a7722b3c not found: ID does not exist" containerID="1268fe8b637ce28d0e84d172aac82d3b8bd0171f45416311d7e9be93a7722b3c" Jul 16 16:53:03.157375 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:53:03.157340 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1268fe8b637ce28d0e84d172aac82d3b8bd0171f45416311d7e9be93a7722b3c"} err="failed to get container status \"1268fe8b637ce28d0e84d172aac82d3b8bd0171f45416311d7e9be93a7722b3c\": rpc error: code = NotFound desc = could not find container \"1268fe8b637ce28d0e84d172aac82d3b8bd0171f45416311d7e9be93a7722b3c\": container with ID starting with 1268fe8b637ce28d0e84d172aac82d3b8bd0171f45416311d7e9be93a7722b3c not found: ID does not exist" Jul 16 16:53:03.157375 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:53:03.157357 2577 scope.go:117] "RemoveContainer" containerID="fbbcfb44f0be91cad0b80ca4fdd7b0fd0a366132faa7d6e25bf7bb522949f3b3" Jul 16 16:53:03.157657 ip-10-0-142-101 kubenswrapper[2577]: E0716 16:53:03.157636 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fbbcfb44f0be91cad0b80ca4fdd7b0fd0a366132faa7d6e25bf7bb522949f3b3\": container with ID starting with fbbcfb44f0be91cad0b80ca4fdd7b0fd0a366132faa7d6e25bf7bb522949f3b3 not found: ID does not exist" containerID="fbbcfb44f0be91cad0b80ca4fdd7b0fd0a366132faa7d6e25bf7bb522949f3b3" Jul 16 16:53:03.157716 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:53:03.157662 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fbbcfb44f0be91cad0b80ca4fdd7b0fd0a366132faa7d6e25bf7bb522949f3b3"} err="failed to get container status \"fbbcfb44f0be91cad0b80ca4fdd7b0fd0a366132faa7d6e25bf7bb522949f3b3\": rpc error: code = NotFound desc = could not find container \"fbbcfb44f0be91cad0b80ca4fdd7b0fd0a366132faa7d6e25bf7bb522949f3b3\": container with ID starting with fbbcfb44f0be91cad0b80ca4fdd7b0fd0a366132faa7d6e25bf7bb522949f3b3 not found: ID does not exist" Jul 16 16:53:03.198282 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:53:03.198247 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-c61ec96a-gw-openshift-default-769498d775-8cpnt"] Jul 16 16:53:03.207648 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:53:03.207613 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-c61ec96a-gw-openshift-default-769498d775-8cpnt"] Jul 16 16:53:04.017724 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:53:04.017685 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e3da487a-0665-40b7-ab25-329cba6f1b2c" path="/var/lib/kubelet/pods/e3da487a-0665-40b7-ab25-329cba6f1b2c/volumes" Jul 16 16:53:04.158258 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:53:04.158231 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-fb78c5b0-openshift-default-56f54df885-8ss9l"] Jul 16 16:53:04.158708 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:53:04.158690 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c4df3ac0-fff6-41e5-a4fd-82d06f73226f" containerName="revoke-keys" Jul 16 16:53:04.158708 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:53:04.158708 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="c4df3ac0-fff6-41e5-a4fd-82d06f73226f" containerName="revoke-keys" Jul 16 16:53:04.158879 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:53:04.158725 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e3da487a-0665-40b7-ab25-329cba6f1b2c" containerName="istio-proxy" Jul 16 16:53:04.158879 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:53:04.158732 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="e3da487a-0665-40b7-ab25-329cba6f1b2c" containerName="istio-proxy" Jul 16 16:53:04.158879 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:53:04.158741 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2c86f886-9e85-4e26-b83f-eab8135b4a9a" containerName="revoke-keys" Jul 16 16:53:04.158879 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:53:04.158747 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="2c86f886-9e85-4e26-b83f-eab8135b4a9a" containerName="revoke-keys" Jul 16 16:53:04.158879 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:53:04.158821 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="e3da487a-0665-40b7-ab25-329cba6f1b2c" containerName="istio-proxy" Jul 16 16:53:04.158879 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:53:04.158830 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="c4df3ac0-fff6-41e5-a4fd-82d06f73226f" containerName="revoke-keys" Jul 16 16:53:04.162293 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:53:04.162271 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-fb78c5b0-openshift-default-56f54df885-8ss9l" Jul 16 16:53:04.171955 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:53:04.171934 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-fb78c5b0-openshift-default-dockercfg-mhrfm\"" Jul 16 16:53:04.183930 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:53:04.183902 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-fb78c5b0-openshift-default-56f54df885-8ss9l"] Jul 16 16:53:04.308135 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:53:04.308096 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ad76b56-3e04-49ff-9376-e8353e009e35-workload-socket\") pod \"e2e-ait-fb78c5b0-openshift-default-56f54df885-8ss9l\" (UID: \"0ad76b56-3e04-49ff-9376-e8353e009e35\") " pod="openshift-ingress/e2e-ait-fb78c5b0-openshift-default-56f54df885-8ss9l" Jul 16 16:53:04.308135 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:53:04.308136 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ad76b56-3e04-49ff-9376-e8353e009e35-istio-token\") pod \"e2e-ait-fb78c5b0-openshift-default-56f54df885-8ss9l\" (UID: \"0ad76b56-3e04-49ff-9376-e8353e009e35\") " pod="openshift-ingress/e2e-ait-fb78c5b0-openshift-default-56f54df885-8ss9l" Jul 16 16:53:04.308356 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:53:04.308196 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ad76b56-3e04-49ff-9376-e8353e009e35-credential-socket\") pod \"e2e-ait-fb78c5b0-openshift-default-56f54df885-8ss9l\" (UID: \"0ad76b56-3e04-49ff-9376-e8353e009e35\") " pod="openshift-ingress/e2e-ait-fb78c5b0-openshift-default-56f54df885-8ss9l" Jul 16 16:53:04.308356 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:53:04.308215 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zz44s\" (UniqueName: \"kubernetes.io/projected/0ad76b56-3e04-49ff-9376-e8353e009e35-kube-api-access-zz44s\") pod \"e2e-ait-fb78c5b0-openshift-default-56f54df885-8ss9l\" (UID: \"0ad76b56-3e04-49ff-9376-e8353e009e35\") " pod="openshift-ingress/e2e-ait-fb78c5b0-openshift-default-56f54df885-8ss9l" Jul 16 16:53:04.308356 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:53:04.308264 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ad76b56-3e04-49ff-9376-e8353e009e35-istiod-ca-cert\") pod \"e2e-ait-fb78c5b0-openshift-default-56f54df885-8ss9l\" (UID: \"0ad76b56-3e04-49ff-9376-e8353e009e35\") " pod="openshift-ingress/e2e-ait-fb78c5b0-openshift-default-56f54df885-8ss9l" Jul 16 16:53:04.308356 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:53:04.308287 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ad76b56-3e04-49ff-9376-e8353e009e35-istio-data\") pod \"e2e-ait-fb78c5b0-openshift-default-56f54df885-8ss9l\" (UID: \"0ad76b56-3e04-49ff-9376-e8353e009e35\") " pod="openshift-ingress/e2e-ait-fb78c5b0-openshift-default-56f54df885-8ss9l" Jul 16 16:53:04.308525 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:53:04.308495 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ad76b56-3e04-49ff-9376-e8353e009e35-istio-envoy\") pod \"e2e-ait-fb78c5b0-openshift-default-56f54df885-8ss9l\" (UID: \"0ad76b56-3e04-49ff-9376-e8353e009e35\") " pod="openshift-ingress/e2e-ait-fb78c5b0-openshift-default-56f54df885-8ss9l" Jul 16 16:53:04.308569 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:53:04.308529 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ad76b56-3e04-49ff-9376-e8353e009e35-istio-podinfo\") pod \"e2e-ait-fb78c5b0-openshift-default-56f54df885-8ss9l\" (UID: \"0ad76b56-3e04-49ff-9376-e8353e009e35\") " pod="openshift-ingress/e2e-ait-fb78c5b0-openshift-default-56f54df885-8ss9l" Jul 16 16:53:04.308569 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:53:04.308549 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ad76b56-3e04-49ff-9376-e8353e009e35-workload-certs\") pod \"e2e-ait-fb78c5b0-openshift-default-56f54df885-8ss9l\" (UID: \"0ad76b56-3e04-49ff-9376-e8353e009e35\") " pod="openshift-ingress/e2e-ait-fb78c5b0-openshift-default-56f54df885-8ss9l" Jul 16 16:53:04.410118 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:53:04.410010 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ad76b56-3e04-49ff-9376-e8353e009e35-istio-envoy\") pod \"e2e-ait-fb78c5b0-openshift-default-56f54df885-8ss9l\" (UID: \"0ad76b56-3e04-49ff-9376-e8353e009e35\") " pod="openshift-ingress/e2e-ait-fb78c5b0-openshift-default-56f54df885-8ss9l" Jul 16 16:53:04.410118 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:53:04.410059 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ad76b56-3e04-49ff-9376-e8353e009e35-istio-podinfo\") pod \"e2e-ait-fb78c5b0-openshift-default-56f54df885-8ss9l\" (UID: \"0ad76b56-3e04-49ff-9376-e8353e009e35\") " pod="openshift-ingress/e2e-ait-fb78c5b0-openshift-default-56f54df885-8ss9l" Jul 16 16:53:04.410118 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:53:04.410079 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ad76b56-3e04-49ff-9376-e8353e009e35-workload-certs\") pod \"e2e-ait-fb78c5b0-openshift-default-56f54df885-8ss9l\" (UID: \"0ad76b56-3e04-49ff-9376-e8353e009e35\") " pod="openshift-ingress/e2e-ait-fb78c5b0-openshift-default-56f54df885-8ss9l" Jul 16 16:53:04.410118 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:53:04.410102 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ad76b56-3e04-49ff-9376-e8353e009e35-workload-socket\") pod \"e2e-ait-fb78c5b0-openshift-default-56f54df885-8ss9l\" (UID: \"0ad76b56-3e04-49ff-9376-e8353e009e35\") " pod="openshift-ingress/e2e-ait-fb78c5b0-openshift-default-56f54df885-8ss9l" Jul 16 16:53:04.410118 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:53:04.410119 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ad76b56-3e04-49ff-9376-e8353e009e35-istio-token\") pod \"e2e-ait-fb78c5b0-openshift-default-56f54df885-8ss9l\" (UID: \"0ad76b56-3e04-49ff-9376-e8353e009e35\") " pod="openshift-ingress/e2e-ait-fb78c5b0-openshift-default-56f54df885-8ss9l" Jul 16 16:53:04.410851 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:53:04.410148 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ad76b56-3e04-49ff-9376-e8353e009e35-credential-socket\") pod \"e2e-ait-fb78c5b0-openshift-default-56f54df885-8ss9l\" (UID: \"0ad76b56-3e04-49ff-9376-e8353e009e35\") " pod="openshift-ingress/e2e-ait-fb78c5b0-openshift-default-56f54df885-8ss9l" Jul 16 16:53:04.410851 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:53:04.410164 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zz44s\" (UniqueName: \"kubernetes.io/projected/0ad76b56-3e04-49ff-9376-e8353e009e35-kube-api-access-zz44s\") pod \"e2e-ait-fb78c5b0-openshift-default-56f54df885-8ss9l\" (UID: \"0ad76b56-3e04-49ff-9376-e8353e009e35\") " pod="openshift-ingress/e2e-ait-fb78c5b0-openshift-default-56f54df885-8ss9l" Jul 16 16:53:04.410851 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:53:04.410195 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ad76b56-3e04-49ff-9376-e8353e009e35-istiod-ca-cert\") pod \"e2e-ait-fb78c5b0-openshift-default-56f54df885-8ss9l\" (UID: \"0ad76b56-3e04-49ff-9376-e8353e009e35\") " pod="openshift-ingress/e2e-ait-fb78c5b0-openshift-default-56f54df885-8ss9l" Jul 16 16:53:04.410851 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:53:04.410219 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ad76b56-3e04-49ff-9376-e8353e009e35-istio-data\") pod \"e2e-ait-fb78c5b0-openshift-default-56f54df885-8ss9l\" (UID: \"0ad76b56-3e04-49ff-9376-e8353e009e35\") " pod="openshift-ingress/e2e-ait-fb78c5b0-openshift-default-56f54df885-8ss9l" Jul 16 16:53:04.410851 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:53:04.410591 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ad76b56-3e04-49ff-9376-e8353e009e35-workload-socket\") pod \"e2e-ait-fb78c5b0-openshift-default-56f54df885-8ss9l\" (UID: \"0ad76b56-3e04-49ff-9376-e8353e009e35\") " pod="openshift-ingress/e2e-ait-fb78c5b0-openshift-default-56f54df885-8ss9l" Jul 16 16:53:04.410851 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:53:04.410668 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ad76b56-3e04-49ff-9376-e8353e009e35-credential-socket\") pod \"e2e-ait-fb78c5b0-openshift-default-56f54df885-8ss9l\" (UID: \"0ad76b56-3e04-49ff-9376-e8353e009e35\") " pod="openshift-ingress/e2e-ait-fb78c5b0-openshift-default-56f54df885-8ss9l" Jul 16 16:53:04.410851 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:53:04.410690 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ad76b56-3e04-49ff-9376-e8353e009e35-workload-certs\") pod \"e2e-ait-fb78c5b0-openshift-default-56f54df885-8ss9l\" (UID: \"0ad76b56-3e04-49ff-9376-e8353e009e35\") " pod="openshift-ingress/e2e-ait-fb78c5b0-openshift-default-56f54df885-8ss9l" Jul 16 16:53:04.410851 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:53:04.410728 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ad76b56-3e04-49ff-9376-e8353e009e35-istio-data\") pod \"e2e-ait-fb78c5b0-openshift-default-56f54df885-8ss9l\" (UID: \"0ad76b56-3e04-49ff-9376-e8353e009e35\") " pod="openshift-ingress/e2e-ait-fb78c5b0-openshift-default-56f54df885-8ss9l" Jul 16 16:53:04.411676 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:53:04.411022 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ad76b56-3e04-49ff-9376-e8353e009e35-istiod-ca-cert\") pod \"e2e-ait-fb78c5b0-openshift-default-56f54df885-8ss9l\" (UID: \"0ad76b56-3e04-49ff-9376-e8353e009e35\") " pod="openshift-ingress/e2e-ait-fb78c5b0-openshift-default-56f54df885-8ss9l" Jul 16 16:53:04.412635 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:53:04.412615 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ad76b56-3e04-49ff-9376-e8353e009e35-istio-envoy\") pod \"e2e-ait-fb78c5b0-openshift-default-56f54df885-8ss9l\" (UID: \"0ad76b56-3e04-49ff-9376-e8353e009e35\") " pod="openshift-ingress/e2e-ait-fb78c5b0-openshift-default-56f54df885-8ss9l" Jul 16 16:53:04.412749 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:53:04.412731 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ad76b56-3e04-49ff-9376-e8353e009e35-istio-podinfo\") pod \"e2e-ait-fb78c5b0-openshift-default-56f54df885-8ss9l\" (UID: \"0ad76b56-3e04-49ff-9376-e8353e009e35\") " pod="openshift-ingress/e2e-ait-fb78c5b0-openshift-default-56f54df885-8ss9l" Jul 16 16:53:04.423957 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:53:04.423930 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ad76b56-3e04-49ff-9376-e8353e009e35-istio-token\") pod \"e2e-ait-fb78c5b0-openshift-default-56f54df885-8ss9l\" (UID: \"0ad76b56-3e04-49ff-9376-e8353e009e35\") " pod="openshift-ingress/e2e-ait-fb78c5b0-openshift-default-56f54df885-8ss9l" Jul 16 16:53:04.425758 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:53:04.425730 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zz44s\" (UniqueName: \"kubernetes.io/projected/0ad76b56-3e04-49ff-9376-e8353e009e35-kube-api-access-zz44s\") pod \"e2e-ait-fb78c5b0-openshift-default-56f54df885-8ss9l\" (UID: \"0ad76b56-3e04-49ff-9376-e8353e009e35\") " pod="openshift-ingress/e2e-ait-fb78c5b0-openshift-default-56f54df885-8ss9l" Jul 16 16:53:04.478857 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:53:04.478814 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-fb78c5b0-openshift-default-56f54df885-8ss9l" Jul 16 16:53:04.622723 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:53:04.622693 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-fb78c5b0-openshift-default-56f54df885-8ss9l"] Jul 16 16:53:04.624733 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:53:04.624704 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ad76b56_3e04_49ff_9376_e8353e009e35.slice/crio-0f5ef76bad0fa7ae7f41aa4c6509f663cbeb38477c9890e3bf465fc9dba090bf WatchSource:0}: Error finding container 0f5ef76bad0fa7ae7f41aa4c6509f663cbeb38477c9890e3bf465fc9dba090bf: Status 404 returned error can't find the container with id 0f5ef76bad0fa7ae7f41aa4c6509f663cbeb38477c9890e3bf465fc9dba090bf Jul 16 16:53:04.626850 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:53:04.626818 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 16 16:53:04.626924 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:53:04.626880 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 16 16:53:04.626924 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:53:04.626908 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 16 16:53:05.151839 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:53:05.151792 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-fb78c5b0-openshift-default-56f54df885-8ss9l" event={"ID":"0ad76b56-3e04-49ff-9376-e8353e009e35","Type":"ContainerStarted","Data":"ded25a27348f6d62ef91fbd6325a49641f00b32ee0e00a1ababe2029f0aac0ce"} Jul 16 16:53:05.152305 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:53:05.151861 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-fb78c5b0-openshift-default-56f54df885-8ss9l" event={"ID":"0ad76b56-3e04-49ff-9376-e8353e009e35","Type":"ContainerStarted","Data":"0f5ef76bad0fa7ae7f41aa4c6509f663cbeb38477c9890e3bf465fc9dba090bf"} Jul 16 16:53:05.180816 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:53:05.180766 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-fb78c5b0-openshift-default-56f54df885-8ss9l" podStartSLOduration=1.180745993 podStartE2EDuration="1.180745993s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 16:53:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 16:53:05.178894945 +0000 UTC m=+1643.682952797" watchObservedRunningTime="2026-07-16 16:53:05.180745993 +0000 UTC m=+1643.684803843" Jul 16 16:53:05.480045 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:53:05.479957 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-fb78c5b0-openshift-default-56f54df885-8ss9l" Jul 16 16:53:05.484730 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:53:05.484703 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-fb78c5b0-openshift-default-56f54df885-8ss9l" Jul 16 16:53:06.157188 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:53:06.157156 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-fb78c5b0-openshift-default-56f54df885-8ss9l" Jul 16 16:53:06.158111 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:53:06.158093 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-fb78c5b0-openshift-default-56f54df885-8ss9l" Jul 16 16:53:10.466541 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:53:10.466505 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fb78c5b0-gwhcq"] Jul 16 16:53:10.467407 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:53:10.467367 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e3da487a-0665-40b7-ab25-329cba6f1b2c" containerName="istio-proxy" Jul 16 16:53:10.467645 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:53:10.467615 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="e3da487a-0665-40b7-ab25-329cba6f1b2c" containerName="istio-proxy" Jul 16 16:53:10.467931 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:53:10.467915 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="e3da487a-0665-40b7-ab25-329cba6f1b2c" containerName="istio-proxy" Jul 16 16:53:10.471070 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:53:10.471044 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fb78c5b0-gwhcq" Jul 16 16:53:10.474206 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:53:10.474181 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dssfh\"" Jul 16 16:53:10.475444 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:53:10.475195 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 16 16:53:10.475444 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:53:10.475349 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fb78c5b0-gwhcq"] Jul 16 16:53:10.573426 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:53:10.573358 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/08a0ffe4-dc1b-418a-81c4-c7d2fdf1611c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-fb78c5b0-gwhcq\" (UID: \"08a0ffe4-dc1b-418a-81c4-c7d2fdf1611c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fb78c5b0-gwhcq" Jul 16 16:53:10.674486 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:53:10.674447 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/08a0ffe4-dc1b-418a-81c4-c7d2fdf1611c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-fb78c5b0-gwhcq\" (UID: \"08a0ffe4-dc1b-418a-81c4-c7d2fdf1611c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fb78c5b0-gwhcq" Jul 16 16:53:10.675067 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:53:10.675047 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/08a0ffe4-dc1b-418a-81c4-c7d2fdf1611c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-fb78c5b0-gwhcq\" (UID: \"08a0ffe4-dc1b-418a-81c4-c7d2fdf1611c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fb78c5b0-gwhcq" Jul 16 16:53:10.785448 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:53:10.785339 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fb78c5b0-gwhcq" Jul 16 16:53:11.127333 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:53:11.127286 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fb78c5b0-gwhcq"] Jul 16 16:53:11.130109 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:53:11.130078 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08a0ffe4_dc1b_418a_81c4_c7d2fdf1611c.slice/crio-23de163f15e8d1719265acf99d22a2886be95519da0c7f71c0f957b2c0c78d9a WatchSource:0}: Error finding container 23de163f15e8d1719265acf99d22a2886be95519da0c7f71c0f957b2c0c78d9a: Status 404 returned error can't find the container with id 23de163f15e8d1719265acf99d22a2886be95519da0c7f71c0f957b2c0c78d9a Jul 16 16:53:11.184179 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:53:11.184141 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fb78c5b0-gwhcq" event={"ID":"08a0ffe4-dc1b-418a-81c4-c7d2fdf1611c","Type":"ContainerStarted","Data":"23de163f15e8d1719265acf99d22a2886be95519da0c7f71c0f957b2c0c78d9a"} Jul 16 16:53:12.190817 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:53:12.190767 2577 generic.go:356] "Generic (PLEG): container finished" podID="08a0ffe4-dc1b-418a-81c4-c7d2fdf1611c" containerID="e2800ed60961ed7446637d0b4b6cfdce9dee6b84efa54757fc0111056ed45846" exitCode=7 Jul 16 16:53:12.191227 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:53:12.190815 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fb78c5b0-gwhcq" event={"ID":"08a0ffe4-dc1b-418a-81c4-c7d2fdf1611c","Type":"ContainerDied","Data":"e2800ed60961ed7446637d0b4b6cfdce9dee6b84efa54757fc0111056ed45846"} Jul 16 16:53:12.191227 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:53:12.191017 2577 scope.go:117] "RemoveContainer" containerID="e2800ed60961ed7446637d0b4b6cfdce9dee6b84efa54757fc0111056ed45846" Jul 16 16:53:13.199008 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:53:13.198977 2577 generic.go:356] "Generic (PLEG): container finished" podID="08a0ffe4-dc1b-418a-81c4-c7d2fdf1611c" containerID="4337e6be8a28c2f2169e7baef82e3b90b267f9ba9a25452ec1b3a19b0be63681" exitCode=7 Jul 16 16:53:13.199506 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:53:13.199056 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fb78c5b0-gwhcq" event={"ID":"08a0ffe4-dc1b-418a-81c4-c7d2fdf1611c","Type":"ContainerDied","Data":"4337e6be8a28c2f2169e7baef82e3b90b267f9ba9a25452ec1b3a19b0be63681"} Jul 16 16:53:13.199506 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:53:13.199132 2577 scope.go:117] "RemoveContainer" containerID="e2800ed60961ed7446637d0b4b6cfdce9dee6b84efa54757fc0111056ed45846" Jul 16 16:53:13.199506 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:53:13.199281 2577 scope.go:117] "RemoveContainer" containerID="4337e6be8a28c2f2169e7baef82e3b90b267f9ba9a25452ec1b3a19b0be63681" Jul 16 16:53:13.199636 ip-10-0-142-101 kubenswrapper[2577]: E0716 16:53:13.199570 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-fb78c5b0-gwhcq_odh-ai-gateway-infra(08a0ffe4-dc1b-418a-81c4-c7d2fdf1611c)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fb78c5b0-gwhcq" podUID="08a0ffe4-dc1b-418a-81c4-c7d2fdf1611c" Jul 16 16:53:14.205479 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:53:14.205452 2577 scope.go:117] "RemoveContainer" containerID="4337e6be8a28c2f2169e7baef82e3b90b267f9ba9a25452ec1b3a19b0be63681" Jul 16 16:53:14.205855 ip-10-0-142-101 kubenswrapper[2577]: E0716 16:53:14.205672 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-fb78c5b0-gwhcq_odh-ai-gateway-infra(08a0ffe4-dc1b-418a-81c4-c7d2fdf1611c)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fb78c5b0-gwhcq" podUID="08a0ffe4-dc1b-418a-81c4-c7d2fdf1611c" Jul 16 16:53:21.158627 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:53:21.158592 2577 patch_prober.go:28] interesting pod/e2e-ait-fb78c5b0-openshift-default-56f54df885-8ss9l container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Jul 16 16:53:21.159072 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:53:21.158661 2577 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-fb78c5b0-openshift-default-56f54df885-8ss9l" podUID="0ad76b56-3e04-49ff-9376-e8353e009e35" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Jul 16 16:53:21.247728 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:53:21.247689 2577 generic.go:356] "Generic (PLEG): container finished" podID="0ad76b56-3e04-49ff-9376-e8353e009e35" containerID="ded25a27348f6d62ef91fbd6325a49641f00b32ee0e00a1ababe2029f0aac0ce" exitCode=0 Jul 16 16:53:21.247923 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:53:21.247764 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-fb78c5b0-openshift-default-56f54df885-8ss9l" event={"ID":"0ad76b56-3e04-49ff-9376-e8353e009e35","Type":"ContainerDied","Data":"ded25a27348f6d62ef91fbd6325a49641f00b32ee0e00a1ababe2029f0aac0ce"} Jul 16 16:53:21.248284 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:53:21.248265 2577 scope.go:117] "RemoveContainer" containerID="ded25a27348f6d62ef91fbd6325a49641f00b32ee0e00a1ababe2029f0aac0ce" Jul 16 16:53:21.249319 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:53:21.249291 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 16 16:53:21.249441 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:53:21.249354 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 16 16:53:21.249441 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:53:21.249411 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 16 16:53:22.259321 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:53:22.259283 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-fb78c5b0-openshift-default-56f54df885-8ss9l" event={"ID":"0ad76b56-3e04-49ff-9376-e8353e009e35","Type":"ContainerStarted","Data":"258dccaf84d0c94f2d8e1ae37c8df3f301674d570422c27b61d32f376dd8951a"} Jul 16 16:53:22.480055 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:53:22.480010 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-fb78c5b0-openshift-default-56f54df885-8ss9l" Jul 16 16:53:22.480264 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:53:22.480240 2577 patch_prober.go:28] interesting pod/e2e-ait-fb78c5b0-openshift-default-56f54df885-8ss9l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Jul 16 16:53:22.480315 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:53:22.480295 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-fb78c5b0-openshift-default-56f54df885-8ss9l" podUID="0ad76b56-3e04-49ff-9376-e8353e009e35" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Jul 16 16:53:23.480270 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:53:23.480228 2577 patch_prober.go:28] interesting pod/e2e-ait-fb78c5b0-openshift-default-56f54df885-8ss9l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Jul 16 16:53:23.480732 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:53:23.480290 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-fb78c5b0-openshift-default-56f54df885-8ss9l" podUID="0ad76b56-3e04-49ff-9376-e8353e009e35" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Jul 16 16:53:24.479907 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:53:24.479861 2577 patch_prober.go:28] interesting pod/e2e-ait-fb78c5b0-openshift-default-56f54df885-8ss9l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Jul 16 16:53:24.480098 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:53:24.479927 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-fb78c5b0-openshift-default-56f54df885-8ss9l" podUID="0ad76b56-3e04-49ff-9376-e8353e009e35" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Jul 16 16:53:25.479183 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:53:25.479149 2577 patch_prober.go:28] interesting pod/e2e-ait-fb78c5b0-openshift-default-56f54df885-8ss9l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Jul 16 16:53:25.479610 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:53:25.479211 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-fb78c5b0-openshift-default-56f54df885-8ss9l" podUID="0ad76b56-3e04-49ff-9376-e8353e009e35" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Jul 16 16:53:26.482977 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:53:26.482948 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-fb78c5b0-openshift-default-56f54df885-8ss9l" Jul 16 16:53:26.483475 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:53:26.483228 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-fb78c5b0-openshift-default-56f54df885-8ss9l" Jul 16 16:53:26.483862 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:53:26.483844 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-fb78c5b0-openshift-default-56f54df885-8ss9l" Jul 16 16:53:29.012396 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:53:29.012347 2577 scope.go:117] "RemoveContainer" containerID="4337e6be8a28c2f2169e7baef82e3b90b267f9ba9a25452ec1b3a19b0be63681" Jul 16 16:53:30.300724 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:53:30.300691 2577 generic.go:356] "Generic (PLEG): container finished" podID="08a0ffe4-dc1b-418a-81c4-c7d2fdf1611c" containerID="797a0cdb57d9f3d1d06e24bef5eb8b44ec3dbca8ff7b4f56a10db88a271ea4e3" exitCode=0 Jul 16 16:53:30.301139 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:53:30.300767 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fb78c5b0-gwhcq" event={"ID":"08a0ffe4-dc1b-418a-81c4-c7d2fdf1611c","Type":"ContainerDied","Data":"797a0cdb57d9f3d1d06e24bef5eb8b44ec3dbca8ff7b4f56a10db88a271ea4e3"} Jul 16 16:53:30.301139 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:53:30.300833 2577 scope.go:117] "RemoveContainer" containerID="4337e6be8a28c2f2169e7baef82e3b90b267f9ba9a25452ec1b3a19b0be63681" Jul 16 16:53:31.334815 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:53:31.334776 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fb78c5b0-gwhcq"] Jul 16 16:53:31.447886 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:53:31.447856 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fb78c5b0-gwhcq" Jul 16 16:53:31.602784 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:53:31.602744 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/08a0ffe4-dc1b-418a-81c4-c7d2fdf1611c-maas-api-service-ca\") pod \"08a0ffe4-dc1b-418a-81c4-c7d2fdf1611c\" (UID: \"08a0ffe4-dc1b-418a-81c4-c7d2fdf1611c\") " Jul 16 16:53:31.603035 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:53:31.603013 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08a0ffe4-dc1b-418a-81c4-c7d2fdf1611c-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "08a0ffe4-dc1b-418a-81c4-c7d2fdf1611c" (UID: "08a0ffe4-dc1b-418a-81c4-c7d2fdf1611c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 16:53:31.603134 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:53:31.603121 2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/08a0ffe4-dc1b-418a-81c4-c7d2fdf1611c-maas-api-service-ca\") on node \"ip-10-0-142-101.ec2.internal\" DevicePath \"\"" Jul 16 16:53:32.314284 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:53:32.314202 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fb78c5b0-gwhcq" Jul 16 16:53:32.314284 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:53:32.314230 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fb78c5b0-gwhcq" event={"ID":"08a0ffe4-dc1b-418a-81c4-c7d2fdf1611c","Type":"ContainerDied","Data":"23de163f15e8d1719265acf99d22a2886be95519da0c7f71c0f957b2c0c78d9a"} Jul 16 16:53:32.314284 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:53:32.314275 2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="23de163f15e8d1719265acf99d22a2886be95519da0c7f71c0f957b2c0c78d9a" Jul 16 16:53:32.344581 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:53:32.344548 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fb78c5b0-gwhcq"] Jul 16 16:53:32.350361 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:53:32.350337 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fb78c5b0-gwhcq"] Jul 16 16:53:34.017512 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:53:34.017477 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="08a0ffe4-dc1b-418a-81c4-c7d2fdf1611c" path="/var/lib/kubelet/pods/08a0ffe4-dc1b-418a-81c4-c7d2fdf1611c/volumes" Jul 16 16:54:10.519482 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:54:10.519401 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fb78c5b0-zv8kw"] Jul 16 16:54:10.519949 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:54:10.519837 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="08a0ffe4-dc1b-418a-81c4-c7d2fdf1611c" containerName="revoke-keys" Jul 16 16:54:10.519949 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:54:10.519851 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="08a0ffe4-dc1b-418a-81c4-c7d2fdf1611c" containerName="revoke-keys" Jul 16 16:54:10.519949 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:54:10.519874 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="08a0ffe4-dc1b-418a-81c4-c7d2fdf1611c" containerName="revoke-keys" Jul 16 16:54:10.519949 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:54:10.519879 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="08a0ffe4-dc1b-418a-81c4-c7d2fdf1611c" containerName="revoke-keys" Jul 16 16:54:10.519949 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:54:10.519889 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="08a0ffe4-dc1b-418a-81c4-c7d2fdf1611c" containerName="revoke-keys" Jul 16 16:54:10.519949 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:54:10.519895 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="08a0ffe4-dc1b-418a-81c4-c7d2fdf1611c" containerName="revoke-keys" Jul 16 16:54:10.520140 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:54:10.519964 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="08a0ffe4-dc1b-418a-81c4-c7d2fdf1611c" containerName="revoke-keys" Jul 16 16:54:10.520140 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:54:10.519974 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="08a0ffe4-dc1b-418a-81c4-c7d2fdf1611c" containerName="revoke-keys" Jul 16 16:54:10.520140 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:54:10.519980 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="08a0ffe4-dc1b-418a-81c4-c7d2fdf1611c" containerName="revoke-keys" Jul 16 16:54:10.522081 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:54:10.522063 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fb78c5b0-zv8kw" Jul 16 16:54:10.525213 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:54:10.525190 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dssfh\"" Jul 16 16:54:10.525336 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:54:10.525192 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 16 16:54:10.531478 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:54:10.531453 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fb78c5b0-zv8kw"] Jul 16 16:54:10.681364 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:54:10.681322 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c116afa5-da9b-4c3d-9022-7826683596c8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-fb78c5b0-zv8kw\" (UID: \"c116afa5-da9b-4c3d-9022-7826683596c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fb78c5b0-zv8kw" Jul 16 16:54:10.782543 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:54:10.782449 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c116afa5-da9b-4c3d-9022-7826683596c8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-fb78c5b0-zv8kw\" (UID: \"c116afa5-da9b-4c3d-9022-7826683596c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fb78c5b0-zv8kw" Jul 16 16:54:10.783090 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:54:10.783068 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c116afa5-da9b-4c3d-9022-7826683596c8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-fb78c5b0-zv8kw\" (UID: \"c116afa5-da9b-4c3d-9022-7826683596c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fb78c5b0-zv8kw" Jul 16 16:54:10.834190 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:54:10.834149 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fb78c5b0-zv8kw" Jul 16 16:54:10.971794 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:54:10.971763 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fb78c5b0-zv8kw"] Jul 16 16:54:10.974085 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:54:10.974060 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc116afa5_da9b_4c3d_9022_7826683596c8.slice/crio-1060bebca0ee2981b188cf5fde13d9d778e0474d61b2c642f88e404baebeb42f WatchSource:0}: Error finding container 1060bebca0ee2981b188cf5fde13d9d778e0474d61b2c642f88e404baebeb42f: Status 404 returned error can't find the container with id 1060bebca0ee2981b188cf5fde13d9d778e0474d61b2c642f88e404baebeb42f Jul 16 16:54:11.520920 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:54:11.520817 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fb78c5b0-zv8kw" event={"ID":"c116afa5-da9b-4c3d-9022-7826683596c8","Type":"ContainerStarted","Data":"efaa8eee654f00a827caeb643f246bef79f527b1b778db8c4c26c5cfd66b18db"} Jul 16 16:54:11.520920 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:54:11.520876 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fb78c5b0-zv8kw" event={"ID":"c116afa5-da9b-4c3d-9022-7826683596c8","Type":"ContainerStarted","Data":"1060bebca0ee2981b188cf5fde13d9d778e0474d61b2c642f88e404baebeb42f"} Jul 16 16:54:11.538764 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:54:11.538706 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fb78c5b0-zv8kw" podStartSLOduration=1.5386897400000001 podStartE2EDuration="1.53868974s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 16:54:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 16:54:11.53573953 +0000 UTC m=+1710.039797381" watchObservedRunningTime="2026-07-16 16:54:11.53868974 +0000 UTC m=+1710.042747591" Jul 16 16:54:12.528240 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:54:12.528195 2577 generic.go:356] "Generic (PLEG): container finished" podID="c116afa5-da9b-4c3d-9022-7826683596c8" containerID="efaa8eee654f00a827caeb643f246bef79f527b1b778db8c4c26c5cfd66b18db" exitCode=0 Jul 16 16:54:12.528733 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:54:12.528261 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fb78c5b0-zv8kw" event={"ID":"c116afa5-da9b-4c3d-9022-7826683596c8","Type":"ContainerDied","Data":"efaa8eee654f00a827caeb643f246bef79f527b1b778db8c4c26c5cfd66b18db"} Jul 16 16:54:13.666521 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:54:13.666494 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fb78c5b0-zv8kw" Jul 16 16:54:13.810952 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:54:13.810861 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c116afa5-da9b-4c3d-9022-7826683596c8-maas-api-service-ca\") pod \"c116afa5-da9b-4c3d-9022-7826683596c8\" (UID: \"c116afa5-da9b-4c3d-9022-7826683596c8\") " Jul 16 16:54:13.811247 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:54:13.811224 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c116afa5-da9b-4c3d-9022-7826683596c8-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "c116afa5-da9b-4c3d-9022-7826683596c8" (UID: "c116afa5-da9b-4c3d-9022-7826683596c8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 16:54:13.911794 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:54:13.911746 2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c116afa5-da9b-4c3d-9022-7826683596c8-maas-api-service-ca\") on node \"ip-10-0-142-101.ec2.internal\" DevicePath \"\"" Jul 16 16:54:14.538708 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:54:14.538677 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fb78c5b0-zv8kw" Jul 16 16:54:14.538708 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:54:14.538694 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fb78c5b0-zv8kw" event={"ID":"c116afa5-da9b-4c3d-9022-7826683596c8","Type":"ContainerDied","Data":"1060bebca0ee2981b188cf5fde13d9d778e0474d61b2c642f88e404baebeb42f"} Jul 16 16:54:14.538933 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:54:14.538736 2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1060bebca0ee2981b188cf5fde13d9d778e0474d61b2c642f88e404baebeb42f" Jul 16 16:54:36.723013 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:54:36.722975 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-fb78c5b0-openshift-default-56f54df885-8ss9l"] Jul 16 16:54:36.723555 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:54:36.723207 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-fb78c5b0-openshift-default-56f54df885-8ss9l" podUID="0ad76b56-3e04-49ff-9376-e8353e009e35" containerName="istio-proxy" containerID="cri-o://258dccaf84d0c94f2d8e1ae37c8df3f301674d570422c27b61d32f376dd8951a" gracePeriod=30 Jul 16 16:54:37.027628 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:54:37.027551 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-ac435f2b-openshift-default-77c4b545cf-j8rq8"] Jul 16 16:54:37.028070 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:54:37.028046 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c116afa5-da9b-4c3d-9022-7826683596c8" containerName="revoke-keys" Jul 16 16:54:37.028250 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:54:37.028233 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="c116afa5-da9b-4c3d-9022-7826683596c8" containerName="revoke-keys" Jul 16 16:54:37.028688 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:54:37.028498 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="c116afa5-da9b-4c3d-9022-7826683596c8" containerName="revoke-keys" Jul 16 16:54:37.031356 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:54:37.031336 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-ac435f2b-openshift-default-77c4b545cf-j8rq8" Jul 16 16:54:37.034079 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:54:37.034059 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-ac435f2b-openshift-default-dockercfg-fdwld\"" Jul 16 16:54:37.046666 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:54:37.046636 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-ac435f2b-openshift-default-77c4b545cf-j8rq8"] Jul 16 16:54:37.139371 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:54:37.139330 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/66594dbb-e95f-4449-a8bc-54ef2f289968-istio-token\") pod \"e2e-derive-ac435f2b-openshift-default-77c4b545cf-j8rq8\" (UID: \"66594dbb-e95f-4449-a8bc-54ef2f289968\") " pod="openshift-ingress/e2e-derive-ac435f2b-openshift-default-77c4b545cf-j8rq8" Jul 16 16:54:37.139858 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:54:37.139749 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66594dbb-e95f-4449-a8bc-54ef2f289968-istio-envoy\") pod \"e2e-derive-ac435f2b-openshift-default-77c4b545cf-j8rq8\" (UID: \"66594dbb-e95f-4449-a8bc-54ef2f289968\") " pod="openshift-ingress/e2e-derive-ac435f2b-openshift-default-77c4b545cf-j8rq8" Jul 16 16:54:37.140092 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:54:37.140075 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66594dbb-e95f-4449-a8bc-54ef2f289968-istio-podinfo\") pod \"e2e-derive-ac435f2b-openshift-default-77c4b545cf-j8rq8\" (UID: \"66594dbb-e95f-4449-a8bc-54ef2f289968\") " pod="openshift-ingress/e2e-derive-ac435f2b-openshift-default-77c4b545cf-j8rq8" Jul 16 16:54:37.140420 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:54:37.140401 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66594dbb-e95f-4449-a8bc-54ef2f289968-credential-socket\") pod \"e2e-derive-ac435f2b-openshift-default-77c4b545cf-j8rq8\" (UID: \"66594dbb-e95f-4449-a8bc-54ef2f289968\") " pod="openshift-ingress/e2e-derive-ac435f2b-openshift-default-77c4b545cf-j8rq8" Jul 16 16:54:37.140633 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:54:37.140617 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66594dbb-e95f-4449-a8bc-54ef2f289968-istio-data\") pod \"e2e-derive-ac435f2b-openshift-default-77c4b545cf-j8rq8\" (UID: \"66594dbb-e95f-4449-a8bc-54ef2f289968\") " pod="openshift-ingress/e2e-derive-ac435f2b-openshift-default-77c4b545cf-j8rq8" Jul 16 16:54:37.140812 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:54:37.140795 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66594dbb-e95f-4449-a8bc-54ef2f289968-workload-socket\") pod \"e2e-derive-ac435f2b-openshift-default-77c4b545cf-j8rq8\" (UID: \"66594dbb-e95f-4449-a8bc-54ef2f289968\") " pod="openshift-ingress/e2e-derive-ac435f2b-openshift-default-77c4b545cf-j8rq8" Jul 16 16:54:37.140966 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:54:37.140953 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zgzg2\" (UniqueName: \"kubernetes.io/projected/66594dbb-e95f-4449-a8bc-54ef2f289968-kube-api-access-zgzg2\") pod \"e2e-derive-ac435f2b-openshift-default-77c4b545cf-j8rq8\" (UID: \"66594dbb-e95f-4449-a8bc-54ef2f289968\") " pod="openshift-ingress/e2e-derive-ac435f2b-openshift-default-77c4b545cf-j8rq8" Jul 16 16:54:37.141115 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:54:37.141103 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66594dbb-e95f-4449-a8bc-54ef2f289968-istiod-ca-cert\") pod \"e2e-derive-ac435f2b-openshift-default-77c4b545cf-j8rq8\" (UID: \"66594dbb-e95f-4449-a8bc-54ef2f289968\") " pod="openshift-ingress/e2e-derive-ac435f2b-openshift-default-77c4b545cf-j8rq8" Jul 16 16:54:37.141244 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:54:37.141230 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66594dbb-e95f-4449-a8bc-54ef2f289968-workload-certs\") pod \"e2e-derive-ac435f2b-openshift-default-77c4b545cf-j8rq8\" (UID: \"66594dbb-e95f-4449-a8bc-54ef2f289968\") " pod="openshift-ingress/e2e-derive-ac435f2b-openshift-default-77c4b545cf-j8rq8" Jul 16 16:54:37.242932 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:54:37.242888 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zgzg2\" (UniqueName: \"kubernetes.io/projected/66594dbb-e95f-4449-a8bc-54ef2f289968-kube-api-access-zgzg2\") pod \"e2e-derive-ac435f2b-openshift-default-77c4b545cf-j8rq8\" (UID: \"66594dbb-e95f-4449-a8bc-54ef2f289968\") " pod="openshift-ingress/e2e-derive-ac435f2b-openshift-default-77c4b545cf-j8rq8" Jul 16 16:54:37.243148 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:54:37.242968 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66594dbb-e95f-4449-a8bc-54ef2f289968-istiod-ca-cert\") pod \"e2e-derive-ac435f2b-openshift-default-77c4b545cf-j8rq8\" (UID: \"66594dbb-e95f-4449-a8bc-54ef2f289968\") " pod="openshift-ingress/e2e-derive-ac435f2b-openshift-default-77c4b545cf-j8rq8" Jul 16 16:54:37.243148 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:54:37.243001 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66594dbb-e95f-4449-a8bc-54ef2f289968-workload-certs\") pod \"e2e-derive-ac435f2b-openshift-default-77c4b545cf-j8rq8\" (UID: \"66594dbb-e95f-4449-a8bc-54ef2f289968\") " pod="openshift-ingress/e2e-derive-ac435f2b-openshift-default-77c4b545cf-j8rq8" Jul 16 16:54:37.243148 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:54:37.243034 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/66594dbb-e95f-4449-a8bc-54ef2f289968-istio-token\") pod \"e2e-derive-ac435f2b-openshift-default-77c4b545cf-j8rq8\" (UID: \"66594dbb-e95f-4449-a8bc-54ef2f289968\") " pod="openshift-ingress/e2e-derive-ac435f2b-openshift-default-77c4b545cf-j8rq8" Jul 16 16:54:37.243148 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:54:37.243070 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66594dbb-e95f-4449-a8bc-54ef2f289968-istio-envoy\") pod \"e2e-derive-ac435f2b-openshift-default-77c4b545cf-j8rq8\" (UID: \"66594dbb-e95f-4449-a8bc-54ef2f289968\") " pod="openshift-ingress/e2e-derive-ac435f2b-openshift-default-77c4b545cf-j8rq8" Jul 16 16:54:37.243148 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:54:37.243096 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66594dbb-e95f-4449-a8bc-54ef2f289968-istio-podinfo\") pod \"e2e-derive-ac435f2b-openshift-default-77c4b545cf-j8rq8\" (UID: \"66594dbb-e95f-4449-a8bc-54ef2f289968\") " pod="openshift-ingress/e2e-derive-ac435f2b-openshift-default-77c4b545cf-j8rq8" Jul 16 16:54:37.243469 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:54:37.243155 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66594dbb-e95f-4449-a8bc-54ef2f289968-credential-socket\") pod \"e2e-derive-ac435f2b-openshift-default-77c4b545cf-j8rq8\" (UID: \"66594dbb-e95f-4449-a8bc-54ef2f289968\") " pod="openshift-ingress/e2e-derive-ac435f2b-openshift-default-77c4b545cf-j8rq8" Jul 16 16:54:37.243469 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:54:37.243189 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66594dbb-e95f-4449-a8bc-54ef2f289968-istio-data\") pod \"e2e-derive-ac435f2b-openshift-default-77c4b545cf-j8rq8\" (UID: \"66594dbb-e95f-4449-a8bc-54ef2f289968\") " pod="openshift-ingress/e2e-derive-ac435f2b-openshift-default-77c4b545cf-j8rq8" Jul 16 16:54:37.243469 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:54:37.243241 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66594dbb-e95f-4449-a8bc-54ef2f289968-workload-socket\") pod \"e2e-derive-ac435f2b-openshift-default-77c4b545cf-j8rq8\" (UID: \"66594dbb-e95f-4449-a8bc-54ef2f289968\") " pod="openshift-ingress/e2e-derive-ac435f2b-openshift-default-77c4b545cf-j8rq8" Jul 16 16:54:37.243626 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:54:37.243539 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66594dbb-e95f-4449-a8bc-54ef2f289968-workload-certs\") pod \"e2e-derive-ac435f2b-openshift-default-77c4b545cf-j8rq8\" (UID: \"66594dbb-e95f-4449-a8bc-54ef2f289968\") " pod="openshift-ingress/e2e-derive-ac435f2b-openshift-default-77c4b545cf-j8rq8" Jul 16 16:54:37.243626 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:54:37.243600 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66594dbb-e95f-4449-a8bc-54ef2f289968-workload-socket\") pod \"e2e-derive-ac435f2b-openshift-default-77c4b545cf-j8rq8\" (UID: \"66594dbb-e95f-4449-a8bc-54ef2f289968\") " pod="openshift-ingress/e2e-derive-ac435f2b-openshift-default-77c4b545cf-j8rq8" Jul 16 16:54:37.243729 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:54:37.243675 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66594dbb-e95f-4449-a8bc-54ef2f289968-credential-socket\") pod \"e2e-derive-ac435f2b-openshift-default-77c4b545cf-j8rq8\" (UID: \"66594dbb-e95f-4449-a8bc-54ef2f289968\") " pod="openshift-ingress/e2e-derive-ac435f2b-openshift-default-77c4b545cf-j8rq8" Jul 16 16:54:37.243786 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:54:37.243758 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66594dbb-e95f-4449-a8bc-54ef2f289968-istio-data\") pod \"e2e-derive-ac435f2b-openshift-default-77c4b545cf-j8rq8\" (UID: \"66594dbb-e95f-4449-a8bc-54ef2f289968\") " pod="openshift-ingress/e2e-derive-ac435f2b-openshift-default-77c4b545cf-j8rq8" Jul 16 16:54:37.244179 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:54:37.244151 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66594dbb-e95f-4449-a8bc-54ef2f289968-istiod-ca-cert\") pod \"e2e-derive-ac435f2b-openshift-default-77c4b545cf-j8rq8\" (UID: \"66594dbb-e95f-4449-a8bc-54ef2f289968\") " pod="openshift-ingress/e2e-derive-ac435f2b-openshift-default-77c4b545cf-j8rq8" Jul 16 16:54:37.245589 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:54:37.245568 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66594dbb-e95f-4449-a8bc-54ef2f289968-istio-envoy\") pod \"e2e-derive-ac435f2b-openshift-default-77c4b545cf-j8rq8\" (UID: \"66594dbb-e95f-4449-a8bc-54ef2f289968\") " pod="openshift-ingress/e2e-derive-ac435f2b-openshift-default-77c4b545cf-j8rq8" Jul 16 16:54:37.245682 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:54:37.245658 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66594dbb-e95f-4449-a8bc-54ef2f289968-istio-podinfo\") pod \"e2e-derive-ac435f2b-openshift-default-77c4b545cf-j8rq8\" (UID: \"66594dbb-e95f-4449-a8bc-54ef2f289968\") " pod="openshift-ingress/e2e-derive-ac435f2b-openshift-default-77c4b545cf-j8rq8" Jul 16 16:54:37.251237 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:54:37.251208 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/66594dbb-e95f-4449-a8bc-54ef2f289968-istio-token\") pod \"e2e-derive-ac435f2b-openshift-default-77c4b545cf-j8rq8\" (UID: \"66594dbb-e95f-4449-a8bc-54ef2f289968\") " pod="openshift-ingress/e2e-derive-ac435f2b-openshift-default-77c4b545cf-j8rq8" Jul 16 16:54:37.251365 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:54:37.251286 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zgzg2\" (UniqueName: \"kubernetes.io/projected/66594dbb-e95f-4449-a8bc-54ef2f289968-kube-api-access-zgzg2\") pod \"e2e-derive-ac435f2b-openshift-default-77c4b545cf-j8rq8\" (UID: \"66594dbb-e95f-4449-a8bc-54ef2f289968\") " pod="openshift-ingress/e2e-derive-ac435f2b-openshift-default-77c4b545cf-j8rq8" Jul 16 16:54:37.343506 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:54:37.343467 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-ac435f2b-openshift-default-77c4b545cf-j8rq8" Jul 16 16:54:37.484698 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:54:37.484667 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-ac435f2b-openshift-default-77c4b545cf-j8rq8"] Jul 16 16:54:37.486304 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:54:37.486274 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66594dbb_e95f_4449_a8bc_54ef2f289968.slice/crio-3dd1890113dd3aaa045586a4279259500ff6e3149b244043348360f821ea2c69 WatchSource:0}: Error finding container 3dd1890113dd3aaa045586a4279259500ff6e3149b244043348360f821ea2c69: Status 404 returned error can't find the container with id 3dd1890113dd3aaa045586a4279259500ff6e3149b244043348360f821ea2c69 Jul 16 16:54:37.488852 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:54:37.488814 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 16 16:54:37.488964 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:54:37.488907 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 16 16:54:37.488964 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:54:37.488955 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 16 16:54:37.660450 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:54:37.660325 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-ac435f2b-openshift-default-77c4b545cf-j8rq8" event={"ID":"66594dbb-e95f-4449-a8bc-54ef2f289968","Type":"ContainerStarted","Data":"abcb4136e83cc3468c8a2652aee9687fd1375884d346e3ceab65fe16e88c09ed"} Jul 16 16:54:37.660450 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:54:37.660421 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-ac435f2b-openshift-default-77c4b545cf-j8rq8" event={"ID":"66594dbb-e95f-4449-a8bc-54ef2f289968","Type":"ContainerStarted","Data":"3dd1890113dd3aaa045586a4279259500ff6e3149b244043348360f821ea2c69"} Jul 16 16:54:37.681625 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:54:37.681566 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-ac435f2b-openshift-default-77c4b545cf-j8rq8" podStartSLOduration=0.681547247 podStartE2EDuration="681.547247ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 16:54:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 16:54:37.679315236 +0000 UTC m=+1736.183373116" watchObservedRunningTime="2026-07-16 16:54:37.681547247 +0000 UTC m=+1736.185605097" Jul 16 16:54:38.344517 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:54:38.344473 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-ac435f2b-openshift-default-77c4b545cf-j8rq8" Jul 16 16:54:38.349563 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:54:38.349524 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-ac435f2b-openshift-default-77c4b545cf-j8rq8" Jul 16 16:54:38.665588 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:54:38.665495 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-ac435f2b-openshift-default-77c4b545cf-j8rq8" Jul 16 16:54:38.666595 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:54:38.666565 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-ac435f2b-openshift-default-77c4b545cf-j8rq8" Jul 16 16:54:41.485503 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:54:41.485465 2577 patch_prober.go:28] interesting pod/e2e-ait-fb78c5b0-openshift-default-56f54df885-8ss9l container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 16 16:54:41.485919 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:54:41.485525 2577 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-fb78c5b0-openshift-default-56f54df885-8ss9l" podUID="0ad76b56-3e04-49ff-9376-e8353e009e35" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 16 16:54:41.999244 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:54:41.999172 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-fb78c5b0-openshift-default-56f54df885-8ss9l" Jul 16 16:54:42.095032 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:54:42.094999 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zz44s\" (UniqueName: \"kubernetes.io/projected/0ad76b56-3e04-49ff-9376-e8353e009e35-kube-api-access-zz44s\") pod \"0ad76b56-3e04-49ff-9376-e8353e009e35\" (UID: \"0ad76b56-3e04-49ff-9376-e8353e009e35\") " Jul 16 16:54:42.095210 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:54:42.095051 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ad76b56-3e04-49ff-9376-e8353e009e35-istiod-ca-cert\") pod \"0ad76b56-3e04-49ff-9376-e8353e009e35\" (UID: \"0ad76b56-3e04-49ff-9376-e8353e009e35\") " Jul 16 16:54:42.095210 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:54:42.095078 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ad76b56-3e04-49ff-9376-e8353e009e35-workload-certs\") pod \"0ad76b56-3e04-49ff-9376-e8353e009e35\" (UID: \"0ad76b56-3e04-49ff-9376-e8353e009e35\") " Jul 16 16:54:42.095210 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:54:42.095144 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ad76b56-3e04-49ff-9376-e8353e009e35-credential-socket\") pod \"0ad76b56-3e04-49ff-9376-e8353e009e35\" (UID: \"0ad76b56-3e04-49ff-9376-e8353e009e35\") " Jul 16 16:54:42.095210 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:54:42.095185 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ad76b56-3e04-49ff-9376-e8353e009e35-istio-envoy\") pod \"0ad76b56-3e04-49ff-9376-e8353e009e35\" (UID: \"0ad76b56-3e04-49ff-9376-e8353e009e35\") " Jul 16 16:54:42.095491 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:54:42.095219 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ad76b56-3e04-49ff-9376-e8353e009e35-workload-socket\") pod \"0ad76b56-3e04-49ff-9376-e8353e009e35\" (UID: \"0ad76b56-3e04-49ff-9376-e8353e009e35\") " Jul 16 16:54:42.095491 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:54:42.095266 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ad76b56-3e04-49ff-9376-e8353e009e35-istio-token\") pod \"0ad76b56-3e04-49ff-9376-e8353e009e35\" (UID: \"0ad76b56-3e04-49ff-9376-e8353e009e35\") " Jul 16 16:54:42.095491 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:54:42.095356 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ad76b56-3e04-49ff-9376-e8353e009e35-istio-data\") pod \"0ad76b56-3e04-49ff-9376-e8353e009e35\" (UID: \"0ad76b56-3e04-49ff-9376-e8353e009e35\") " Jul 16 16:54:42.095491 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:54:42.095462 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ad76b56-3e04-49ff-9376-e8353e009e35-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "0ad76b56-3e04-49ff-9376-e8353e009e35" (UID: "0ad76b56-3e04-49ff-9376-e8353e009e35"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 16:54:42.095727 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:54:42.095483 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ad76b56-3e04-49ff-9376-e8353e009e35-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "0ad76b56-3e04-49ff-9376-e8353e009e35" (UID: "0ad76b56-3e04-49ff-9376-e8353e009e35"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 16:54:42.095727 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:54:42.095508 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ad76b56-3e04-49ff-9376-e8353e009e35-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "0ad76b56-3e04-49ff-9376-e8353e009e35" (UID: "0ad76b56-3e04-49ff-9376-e8353e009e35"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 16:54:42.095727 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:54:42.095528 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ad76b56-3e04-49ff-9376-e8353e009e35-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "0ad76b56-3e04-49ff-9376-e8353e009e35" (UID: "0ad76b56-3e04-49ff-9376-e8353e009e35"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 16:54:42.095935 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:54:42.095748 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ad76b56-3e04-49ff-9376-e8353e009e35-istio-podinfo\") pod \"0ad76b56-3e04-49ff-9376-e8353e009e35\" (UID: \"0ad76b56-3e04-49ff-9376-e8353e009e35\") " Jul 16 16:54:42.096147 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:54:42.096123 2577 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ad76b56-3e04-49ff-9376-e8353e009e35-istiod-ca-cert\") on node \"ip-10-0-142-101.ec2.internal\" DevicePath \"\"" Jul 16 16:54:42.096231 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:54:42.096169 2577 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ad76b56-3e04-49ff-9376-e8353e009e35-workload-certs\") on node \"ip-10-0-142-101.ec2.internal\" DevicePath \"\"" Jul 16 16:54:42.096231 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:54:42.096184 2577 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ad76b56-3e04-49ff-9376-e8353e009e35-credential-socket\") on node \"ip-10-0-142-101.ec2.internal\" DevicePath \"\"" Jul 16 16:54:42.096231 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:54:42.096198 2577 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ad76b56-3e04-49ff-9376-e8353e009e35-workload-socket\") on node \"ip-10-0-142-101.ec2.internal\" DevicePath \"\"" Jul 16 16:54:42.096416 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:54:42.096344 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ad76b56-3e04-49ff-9376-e8353e009e35-istio-data" (OuterVolumeSpecName: "istio-data") pod "0ad76b56-3e04-49ff-9376-e8353e009e35" (UID: "0ad76b56-3e04-49ff-9376-e8353e009e35"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 16:54:42.097364 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:54:42.097336 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ad76b56-3e04-49ff-9376-e8353e009e35-kube-api-access-zz44s" (OuterVolumeSpecName: "kube-api-access-zz44s") pod "0ad76b56-3e04-49ff-9376-e8353e009e35" (UID: "0ad76b56-3e04-49ff-9376-e8353e009e35"). InnerVolumeSpecName "kube-api-access-zz44s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 16:54:42.097645 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:54:42.097618 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0ad76b56-3e04-49ff-9376-e8353e009e35-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "0ad76b56-3e04-49ff-9376-e8353e009e35" (UID: "0ad76b56-3e04-49ff-9376-e8353e009e35"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 16 16:54:42.097757 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:54:42.097736 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ad76b56-3e04-49ff-9376-e8353e009e35-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "0ad76b56-3e04-49ff-9376-e8353e009e35" (UID: "0ad76b56-3e04-49ff-9376-e8353e009e35"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 16:54:42.098367 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:54:42.098341 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ad76b56-3e04-49ff-9376-e8353e009e35-istio-token" (OuterVolumeSpecName: "istio-token") pod "0ad76b56-3e04-49ff-9376-e8353e009e35" (UID: "0ad76b56-3e04-49ff-9376-e8353e009e35"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 16:54:42.197627 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:54:42.197588 2577 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ad76b56-3e04-49ff-9376-e8353e009e35-istio-token\") on node \"ip-10-0-142-101.ec2.internal\" DevicePath \"\"" Jul 16 16:54:42.197627 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:54:42.197620 2577 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ad76b56-3e04-49ff-9376-e8353e009e35-istio-data\") on node \"ip-10-0-142-101.ec2.internal\" DevicePath \"\"" Jul 16 16:54:42.197627 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:54:42.197629 2577 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ad76b56-3e04-49ff-9376-e8353e009e35-istio-podinfo\") on node \"ip-10-0-142-101.ec2.internal\" DevicePath \"\"" Jul 16 16:54:42.197627 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:54:42.197638 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zz44s\" (UniqueName: \"kubernetes.io/projected/0ad76b56-3e04-49ff-9376-e8353e009e35-kube-api-access-zz44s\") on node \"ip-10-0-142-101.ec2.internal\" DevicePath \"\"" Jul 16 16:54:42.197905 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:54:42.197648 2577 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ad76b56-3e04-49ff-9376-e8353e009e35-istio-envoy\") on node \"ip-10-0-142-101.ec2.internal\" DevicePath \"\"" Jul 16 16:54:42.633729 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:54:42.633689 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ac435f2b-8b2j6"] Jul 16 16:54:42.634121 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:54:42.634113 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0ad76b56-3e04-49ff-9376-e8353e009e35" containerName="istio-proxy" Jul 16 16:54:42.634163 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:54:42.634126 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="0ad76b56-3e04-49ff-9376-e8353e009e35" containerName="istio-proxy" Jul 16 16:54:42.634163 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:54:42.634141 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0ad76b56-3e04-49ff-9376-e8353e009e35" containerName="istio-proxy" Jul 16 16:54:42.634163 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:54:42.634147 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="0ad76b56-3e04-49ff-9376-e8353e009e35" containerName="istio-proxy" Jul 16 16:54:42.634252 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:54:42.634237 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="0ad76b56-3e04-49ff-9376-e8353e009e35" containerName="istio-proxy" Jul 16 16:54:42.637911 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:54:42.637892 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ac435f2b-8b2j6" Jul 16 16:54:42.640528 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:54:42.640500 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 16 16:54:42.641288 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:54:42.641234 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dssfh\"" Jul 16 16:54:42.643587 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:54:42.643555 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ac435f2b-8b2j6"] Jul 16 16:54:42.687264 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:54:42.687223 2577 generic.go:356] "Generic (PLEG): container finished" podID="0ad76b56-3e04-49ff-9376-e8353e009e35" containerID="258dccaf84d0c94f2d8e1ae37c8df3f301674d570422c27b61d32f376dd8951a" exitCode=0 Jul 16 16:54:42.687499 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:54:42.687278 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-fb78c5b0-openshift-default-56f54df885-8ss9l" Jul 16 16:54:42.687499 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:54:42.687318 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-fb78c5b0-openshift-default-56f54df885-8ss9l" event={"ID":"0ad76b56-3e04-49ff-9376-e8353e009e35","Type":"ContainerDied","Data":"258dccaf84d0c94f2d8e1ae37c8df3f301674d570422c27b61d32f376dd8951a"} Jul 16 16:54:42.687499 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:54:42.687416 2577 scope.go:117] "RemoveContainer" containerID="258dccaf84d0c94f2d8e1ae37c8df3f301674d570422c27b61d32f376dd8951a" Jul 16 16:54:42.687499 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:54:42.687414 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-fb78c5b0-openshift-default-56f54df885-8ss9l" event={"ID":"0ad76b56-3e04-49ff-9376-e8353e009e35","Type":"ContainerDied","Data":"0f5ef76bad0fa7ae7f41aa4c6509f663cbeb38477c9890e3bf465fc9dba090bf"} Jul 16 16:54:42.696860 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:54:42.696841 2577 scope.go:117] "RemoveContainer" containerID="ded25a27348f6d62ef91fbd6325a49641f00b32ee0e00a1ababe2029f0aac0ce" Jul 16 16:54:42.702366 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:54:42.702343 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/737d554c-df19-4b9e-87e1-8c5b5195d1d6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-ac435f2b-8b2j6\" (UID: \"737d554c-df19-4b9e-87e1-8c5b5195d1d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ac435f2b-8b2j6" Jul 16 16:54:42.704877 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:54:42.704859 2577 scope.go:117] "RemoveContainer" containerID="258dccaf84d0c94f2d8e1ae37c8df3f301674d570422c27b61d32f376dd8951a" Jul 16 16:54:42.705174 ip-10-0-142-101 kubenswrapper[2577]: E0716 16:54:42.705153 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"258dccaf84d0c94f2d8e1ae37c8df3f301674d570422c27b61d32f376dd8951a\": container with ID starting with 258dccaf84d0c94f2d8e1ae37c8df3f301674d570422c27b61d32f376dd8951a not found: ID does not exist" containerID="258dccaf84d0c94f2d8e1ae37c8df3f301674d570422c27b61d32f376dd8951a" Jul 16 16:54:42.705218 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:54:42.705184 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"258dccaf84d0c94f2d8e1ae37c8df3f301674d570422c27b61d32f376dd8951a"} err="failed to get container status \"258dccaf84d0c94f2d8e1ae37c8df3f301674d570422c27b61d32f376dd8951a\": rpc error: code = NotFound desc = could not find container \"258dccaf84d0c94f2d8e1ae37c8df3f301674d570422c27b61d32f376dd8951a\": container with ID starting with 258dccaf84d0c94f2d8e1ae37c8df3f301674d570422c27b61d32f376dd8951a not found: ID does not exist" Jul 16 16:54:42.705218 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:54:42.705201 2577 scope.go:117] "RemoveContainer" containerID="ded25a27348f6d62ef91fbd6325a49641f00b32ee0e00a1ababe2029f0aac0ce" Jul 16 16:54:42.705454 ip-10-0-142-101 kubenswrapper[2577]: E0716 16:54:42.705435 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ded25a27348f6d62ef91fbd6325a49641f00b32ee0e00a1ababe2029f0aac0ce\": container with ID starting with ded25a27348f6d62ef91fbd6325a49641f00b32ee0e00a1ababe2029f0aac0ce not found: ID does not exist" containerID="ded25a27348f6d62ef91fbd6325a49641f00b32ee0e00a1ababe2029f0aac0ce" Jul 16 16:54:42.705522 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:54:42.705464 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ded25a27348f6d62ef91fbd6325a49641f00b32ee0e00a1ababe2029f0aac0ce"} err="failed to get container status \"ded25a27348f6d62ef91fbd6325a49641f00b32ee0e00a1ababe2029f0aac0ce\": rpc error: code = NotFound desc = could not find container \"ded25a27348f6d62ef91fbd6325a49641f00b32ee0e00a1ababe2029f0aac0ce\": container with ID starting with ded25a27348f6d62ef91fbd6325a49641f00b32ee0e00a1ababe2029f0aac0ce not found: ID does not exist" Jul 16 16:54:42.709864 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:54:42.709838 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-fb78c5b0-openshift-default-56f54df885-8ss9l"] Jul 16 16:54:42.714577 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:54:42.714552 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-fb78c5b0-openshift-default-56f54df885-8ss9l"] Jul 16 16:54:42.803466 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:54:42.803422 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/737d554c-df19-4b9e-87e1-8c5b5195d1d6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-ac435f2b-8b2j6\" (UID: \"737d554c-df19-4b9e-87e1-8c5b5195d1d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ac435f2b-8b2j6" Jul 16 16:54:42.804034 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:54:42.804014 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/737d554c-df19-4b9e-87e1-8c5b5195d1d6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-ac435f2b-8b2j6\" (UID: \"737d554c-df19-4b9e-87e1-8c5b5195d1d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ac435f2b-8b2j6" Jul 16 16:54:42.952026 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:54:42.951939 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ac435f2b-8b2j6" Jul 16 16:54:43.092378 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:54:43.092350 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ac435f2b-8b2j6"] Jul 16 16:54:43.093965 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:54:43.093939 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod737d554c_df19_4b9e_87e1_8c5b5195d1d6.slice/crio-5a7e800bfb4a87d73c3317076d4d29fd8c80d8515d1a3c5427eff98d28f13514 WatchSource:0}: Error finding container 5a7e800bfb4a87d73c3317076d4d29fd8c80d8515d1a3c5427eff98d28f13514: Status 404 returned error can't find the container with id 5a7e800bfb4a87d73c3317076d4d29fd8c80d8515d1a3c5427eff98d28f13514 Jul 16 16:54:43.701372 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:54:43.701334 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ac435f2b-8b2j6" event={"ID":"737d554c-df19-4b9e-87e1-8c5b5195d1d6","Type":"ContainerStarted","Data":"fdbc6c2cce48150cd293e7ee00abd40585efc83b37df468d7f6e5704e6202f57"} Jul 16 16:54:43.701777 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:54:43.701440 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ac435f2b-8b2j6" event={"ID":"737d554c-df19-4b9e-87e1-8c5b5195d1d6","Type":"ContainerStarted","Data":"5a7e800bfb4a87d73c3317076d4d29fd8c80d8515d1a3c5427eff98d28f13514"} Jul 16 16:54:43.718764 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:54:43.718710 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ac435f2b-8b2j6" podStartSLOduration=1.718692858 podStartE2EDuration="1.718692858s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 16:54:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 16:54:43.717781788 +0000 UTC m=+1742.221839641" watchObservedRunningTime="2026-07-16 16:54:43.718692858 +0000 UTC m=+1742.222750709" Jul 16 16:54:44.017804 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:54:44.017717 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0ad76b56-3e04-49ff-9376-e8353e009e35" path="/var/lib/kubelet/pods/0ad76b56-3e04-49ff-9376-e8353e009e35/volumes" Jul 16 16:54:44.707847 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:54:44.707813 2577 generic.go:356] "Generic (PLEG): container finished" podID="737d554c-df19-4b9e-87e1-8c5b5195d1d6" containerID="fdbc6c2cce48150cd293e7ee00abd40585efc83b37df468d7f6e5704e6202f57" exitCode=7 Jul 16 16:54:44.708276 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:54:44.707908 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ac435f2b-8b2j6" event={"ID":"737d554c-df19-4b9e-87e1-8c5b5195d1d6","Type":"ContainerDied","Data":"fdbc6c2cce48150cd293e7ee00abd40585efc83b37df468d7f6e5704e6202f57"} Jul 16 16:54:44.708276 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:54:44.708085 2577 scope.go:117] "RemoveContainer" containerID="fdbc6c2cce48150cd293e7ee00abd40585efc83b37df468d7f6e5704e6202f57" Jul 16 16:54:45.718974 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:54:45.718879 2577 generic.go:356] "Generic (PLEG): container finished" podID="737d554c-df19-4b9e-87e1-8c5b5195d1d6" containerID="a362cd4d2ff4cb22134f0d7f106e78dff776bfb5b75b03184bb060cb15368e0e" exitCode=7 Jul 16 16:54:45.719450 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:54:45.718968 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ac435f2b-8b2j6" event={"ID":"737d554c-df19-4b9e-87e1-8c5b5195d1d6","Type":"ContainerDied","Data":"a362cd4d2ff4cb22134f0d7f106e78dff776bfb5b75b03184bb060cb15368e0e"} Jul 16 16:54:45.719450 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:54:45.719034 2577 scope.go:117] "RemoveContainer" containerID="fdbc6c2cce48150cd293e7ee00abd40585efc83b37df468d7f6e5704e6202f57" Jul 16 16:54:45.719450 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:54:45.719263 2577 scope.go:117] "RemoveContainer" containerID="a362cd4d2ff4cb22134f0d7f106e78dff776bfb5b75b03184bb060cb15368e0e" Jul 16 16:54:45.719628 ip-10-0-142-101 kubenswrapper[2577]: E0716 16:54:45.719577 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-ac435f2b-8b2j6_odh-ai-gateway-infra(737d554c-df19-4b9e-87e1-8c5b5195d1d6)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ac435f2b-8b2j6" podUID="737d554c-df19-4b9e-87e1-8c5b5195d1d6" Jul 16 16:54:46.725407 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:54:46.725344 2577 scope.go:117] "RemoveContainer" containerID="a362cd4d2ff4cb22134f0d7f106e78dff776bfb5b75b03184bb060cb15368e0e" Jul 16 16:54:46.725796 ip-10-0-142-101 kubenswrapper[2577]: E0716 16:54:46.725625 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-ac435f2b-8b2j6_odh-ai-gateway-infra(737d554c-df19-4b9e-87e1-8c5b5195d1d6)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ac435f2b-8b2j6" podUID="737d554c-df19-4b9e-87e1-8c5b5195d1d6" Jul 16 16:54:53.820000 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:54:53.819877 2577 patch_prober.go:28] interesting pod/e2e-derive-ac435f2b-openshift-default-77c4b545cf-j8rq8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": read tcp 10.134.0.2:56402->10.134.0.43:15021: read: connection reset by peer" start-of-body= Jul 16 16:54:53.820000 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:54:53.819971 2577 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-derive-ac435f2b-openshift-default-77c4b545cf-j8rq8" podUID="66594dbb-e95f-4449-a8bc-54ef2f289968" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": read tcp 10.134.0.2:56402->10.134.0.43:15021: read: connection reset by peer" Jul 16 16:54:54.777327 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:54:54.777292 2577 generic.go:356] "Generic (PLEG): container finished" podID="66594dbb-e95f-4449-a8bc-54ef2f289968" containerID="abcb4136e83cc3468c8a2652aee9687fd1375884d346e3ceab65fe16e88c09ed" exitCode=0 Jul 16 16:54:54.777522 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:54:54.777337 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-ac435f2b-openshift-default-77c4b545cf-j8rq8" event={"ID":"66594dbb-e95f-4449-a8bc-54ef2f289968","Type":"ContainerDied","Data":"abcb4136e83cc3468c8a2652aee9687fd1375884d346e3ceab65fe16e88c09ed"} Jul 16 16:54:54.777785 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:54:54.777770 2577 scope.go:117] "RemoveContainer" containerID="abcb4136e83cc3468c8a2652aee9687fd1375884d346e3ceab65fe16e88c09ed" Jul 16 16:54:54.778930 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:54:54.778897 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 16 16:54:54.779009 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:54:54.778994 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 16 16:54:54.779051 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:54:54.779038 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 16 16:54:55.784282 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:54:55.784247 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-ac435f2b-openshift-default-77c4b545cf-j8rq8" event={"ID":"66594dbb-e95f-4449-a8bc-54ef2f289968","Type":"ContainerStarted","Data":"5e7ec071d606b5c145d9adc3e7f9633fcb3fda1491da3c37559e2f3bb3a26f6b"} Jul 16 16:54:56.344219 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:54:56.343815 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-ac435f2b-openshift-default-77c4b545cf-j8rq8" Jul 16 16:54:56.344219 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:54:56.344094 2577 patch_prober.go:28] interesting pod/e2e-derive-ac435f2b-openshift-default-77c4b545cf-j8rq8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body= Jul 16 16:54:56.344219 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:54:56.344164 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-ac435f2b-openshift-default-77c4b545cf-j8rq8" podUID="66594dbb-e95f-4449-a8bc-54ef2f289968" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" Jul 16 16:54:57.344871 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:54:57.344357 2577 patch_prober.go:28] interesting pod/e2e-derive-ac435f2b-openshift-default-77c4b545cf-j8rq8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body= Jul 16 16:54:57.344871 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:54:57.344457 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-ac435f2b-openshift-default-77c4b545cf-j8rq8" podUID="66594dbb-e95f-4449-a8bc-54ef2f289968" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" Jul 16 16:54:58.344374 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:54:58.344327 2577 patch_prober.go:28] interesting pod/e2e-derive-ac435f2b-openshift-default-77c4b545cf-j8rq8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body= Jul 16 16:54:58.344718 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:54:58.344679 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-ac435f2b-openshift-default-77c4b545cf-j8rq8" podUID="66594dbb-e95f-4449-a8bc-54ef2f289968" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" Jul 16 16:54:59.344139 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:54:59.344093 2577 patch_prober.go:28] interesting pod/e2e-derive-ac435f2b-openshift-default-77c4b545cf-j8rq8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body= Jul 16 16:54:59.344646 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:54:59.344175 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-ac435f2b-openshift-default-77c4b545cf-j8rq8" podUID="66594dbb-e95f-4449-a8bc-54ef2f289968" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" Jul 16 16:55:00.012525 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:55:00.012493 2577 scope.go:117] "RemoveContainer" containerID="a362cd4d2ff4cb22134f0d7f106e78dff776bfb5b75b03184bb060cb15368e0e" Jul 16 16:55:00.348007 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:55:00.347976 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-ac435f2b-openshift-default-77c4b545cf-j8rq8" Jul 16 16:55:00.348369 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:55:00.348246 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-ac435f2b-openshift-default-77c4b545cf-j8rq8" Jul 16 16:55:00.349001 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:55:00.348982 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-ac435f2b-openshift-default-77c4b545cf-j8rq8" Jul 16 16:55:00.812815 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:55:00.812728 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ac435f2b-8b2j6" event={"ID":"737d554c-df19-4b9e-87e1-8c5b5195d1d6","Type":"ContainerStarted","Data":"2a75387222194b18cda407593cea741153e28f49814b64821b0608e70bff2cbd"} Jul 16 16:55:01.819285 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:55:01.819253 2577 generic.go:356] "Generic (PLEG): container finished" podID="737d554c-df19-4b9e-87e1-8c5b5195d1d6" containerID="2a75387222194b18cda407593cea741153e28f49814b64821b0608e70bff2cbd" exitCode=0 Jul 16 16:55:01.819693 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:55:01.819325 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ac435f2b-8b2j6" event={"ID":"737d554c-df19-4b9e-87e1-8c5b5195d1d6","Type":"ContainerDied","Data":"2a75387222194b18cda407593cea741153e28f49814b64821b0608e70bff2cbd"} Jul 16 16:55:01.819693 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:55:01.819399 2577 scope.go:117] "RemoveContainer" containerID="a362cd4d2ff4cb22134f0d7f106e78dff776bfb5b75b03184bb060cb15368e0e" Jul 16 16:55:01.847101 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:55:01.847068 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ac435f2b-8b2j6"] Jul 16 16:55:02.964439 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:55:02.964408 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ac435f2b-8b2j6" Jul 16 16:55:03.003955 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:55:03.003913 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/737d554c-df19-4b9e-87e1-8c5b5195d1d6-maas-api-service-ca\") pod \"737d554c-df19-4b9e-87e1-8c5b5195d1d6\" (UID: \"737d554c-df19-4b9e-87e1-8c5b5195d1d6\") " Jul 16 16:55:03.004297 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:55:03.004275 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/737d554c-df19-4b9e-87e1-8c5b5195d1d6-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "737d554c-df19-4b9e-87e1-8c5b5195d1d6" (UID: "737d554c-df19-4b9e-87e1-8c5b5195d1d6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 16:55:03.105224 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:55:03.105132 2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/737d554c-df19-4b9e-87e1-8c5b5195d1d6-maas-api-service-ca\") on node \"ip-10-0-142-101.ec2.internal\" DevicePath \"\"" Jul 16 16:55:03.832842 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:55:03.832810 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ac435f2b-8b2j6" Jul 16 16:55:03.833026 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:55:03.832863 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ac435f2b-8b2j6" event={"ID":"737d554c-df19-4b9e-87e1-8c5b5195d1d6","Type":"ContainerDied","Data":"5a7e800bfb4a87d73c3317076d4d29fd8c80d8515d1a3c5427eff98d28f13514"} Jul 16 16:55:03.833026 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:55:03.832917 2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5a7e800bfb4a87d73c3317076d4d29fd8c80d8515d1a3c5427eff98d28f13514" Jul 16 16:55:03.861784 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:55:03.861743 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ac435f2b-8b2j6"] Jul 16 16:55:03.869117 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:55:03.869087 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ac435f2b-8b2j6"] Jul 16 16:55:04.018111 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:55:04.018078 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="737d554c-df19-4b9e-87e1-8c5b5195d1d6" path="/var/lib/kubelet/pods/737d554c-df19-4b9e-87e1-8c5b5195d1d6/volumes" Jul 16 16:55:42.061328 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:55:42.061297 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9bjst_b20b154f-1a4c-4b25-86db-aebe2d4ffc08/ovn-acl-logging/0.log" Jul 16 16:55:42.066328 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:55:42.066307 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9bjst_b20b154f-1a4c-4b25-86db-aebe2d4ffc08/ovn-acl-logging/0.log" Jul 16 16:55:42.688169 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:55:42.688127 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ac435f2b-52gjn"] Jul 16 16:55:42.688574 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:55:42.688560 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="737d554c-df19-4b9e-87e1-8c5b5195d1d6" containerName="revoke-keys" Jul 16 16:55:42.688630 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:55:42.688575 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="737d554c-df19-4b9e-87e1-8c5b5195d1d6" containerName="revoke-keys" Jul 16 16:55:42.688630 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:55:42.688606 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="737d554c-df19-4b9e-87e1-8c5b5195d1d6" containerName="revoke-keys" Jul 16 16:55:42.688630 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:55:42.688611 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="737d554c-df19-4b9e-87e1-8c5b5195d1d6" containerName="revoke-keys" Jul 16 16:55:42.688724 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:55:42.688677 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="0ad76b56-3e04-49ff-9376-e8353e009e35" containerName="istio-proxy" Jul 16 16:55:42.688724 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:55:42.688685 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="737d554c-df19-4b9e-87e1-8c5b5195d1d6" containerName="revoke-keys" Jul 16 16:55:42.688724 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:55:42.688691 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="737d554c-df19-4b9e-87e1-8c5b5195d1d6" containerName="revoke-keys" Jul 16 16:55:42.688724 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:55:42.688700 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="737d554c-df19-4b9e-87e1-8c5b5195d1d6" containerName="revoke-keys" Jul 16 16:55:42.693044 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:55:42.693025 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ac435f2b-52gjn" Jul 16 16:55:42.695807 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:55:42.695785 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dssfh\"" Jul 16 16:55:42.695919 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:55:42.695791 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 16 16:55:42.695982 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:55:42.695950 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1b9cef5-e237-4a64-b504-5617c333924d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-ac435f2b-52gjn\" (UID: \"c1b9cef5-e237-4a64-b504-5617c333924d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ac435f2b-52gjn" Jul 16 16:55:42.701863 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:55:42.701838 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ac435f2b-52gjn"] Jul 16 16:55:42.796777 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:55:42.796735 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1b9cef5-e237-4a64-b504-5617c333924d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-ac435f2b-52gjn\" (UID: \"c1b9cef5-e237-4a64-b504-5617c333924d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ac435f2b-52gjn" Jul 16 16:55:42.797445 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:55:42.797377 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1b9cef5-e237-4a64-b504-5617c333924d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-ac435f2b-52gjn\" (UID: \"c1b9cef5-e237-4a64-b504-5617c333924d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ac435f2b-52gjn" Jul 16 16:55:43.005476 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:55:43.005345 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ac435f2b-52gjn" Jul 16 16:55:43.352450 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:55:43.352422 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ac435f2b-52gjn"] Jul 16 16:55:43.354849 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:55:43.354820 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1b9cef5_e237_4a64_b504_5617c333924d.slice/crio-3a7cfa47523698d98b33c40b1d5cde641d53676567b7f936b01a27a6bcb52092 WatchSource:0}: Error finding container 3a7cfa47523698d98b33c40b1d5cde641d53676567b7f936b01a27a6bcb52092: Status 404 returned error can't find the container with id 3a7cfa47523698d98b33c40b1d5cde641d53676567b7f936b01a27a6bcb52092 Jul 16 16:55:44.050284 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:55:44.050205 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ac435f2b-52gjn" event={"ID":"c1b9cef5-e237-4a64-b504-5617c333924d","Type":"ContainerStarted","Data":"8970e37a945153a0c803c279bccb7f3dbaf194363bbc60e7685a6fa661db7c87"} Jul 16 16:55:44.050284 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:55:44.050260 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ac435f2b-52gjn" event={"ID":"c1b9cef5-e237-4a64-b504-5617c333924d","Type":"ContainerStarted","Data":"3a7cfa47523698d98b33c40b1d5cde641d53676567b7f936b01a27a6bcb52092"} Jul 16 16:55:44.071216 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:55:44.071158 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ac435f2b-52gjn" podStartSLOduration=2.07114053 podStartE2EDuration="2.07114053s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 16:55:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 16:55:44.069283738 +0000 UTC m=+1802.573341593" watchObservedRunningTime="2026-07-16 16:55:44.07114053 +0000 UTC m=+1802.575198380" Jul 16 16:55:45.057666 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:55:45.057629 2577 generic.go:356] "Generic (PLEG): container finished" podID="c1b9cef5-e237-4a64-b504-5617c333924d" containerID="8970e37a945153a0c803c279bccb7f3dbaf194363bbc60e7685a6fa661db7c87" exitCode=0 Jul 16 16:55:45.058093 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:55:45.057695 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ac435f2b-52gjn" event={"ID":"c1b9cef5-e237-4a64-b504-5617c333924d","Type":"ContainerDied","Data":"8970e37a945153a0c803c279bccb7f3dbaf194363bbc60e7685a6fa661db7c87"} Jul 16 16:55:46.202718 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:55:46.202689 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ac435f2b-52gjn" Jul 16 16:55:46.223523 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:55:46.223495 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1b9cef5-e237-4a64-b504-5617c333924d-maas-api-service-ca\") pod \"c1b9cef5-e237-4a64-b504-5617c333924d\" (UID: \"c1b9cef5-e237-4a64-b504-5617c333924d\") " Jul 16 16:55:46.223912 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:55:46.223879 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1b9cef5-e237-4a64-b504-5617c333924d-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "c1b9cef5-e237-4a64-b504-5617c333924d" (UID: "c1b9cef5-e237-4a64-b504-5617c333924d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 16:55:46.324417 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:55:46.324362 2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1b9cef5-e237-4a64-b504-5617c333924d-maas-api-service-ca\") on node \"ip-10-0-142-101.ec2.internal\" DevicePath \"\"" Jul 16 16:55:47.069196 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:55:47.069166 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ac435f2b-52gjn" Jul 16 16:55:47.069373 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:55:47.069167 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ac435f2b-52gjn" event={"ID":"c1b9cef5-e237-4a64-b504-5617c333924d","Type":"ContainerDied","Data":"3a7cfa47523698d98b33c40b1d5cde641d53676567b7f936b01a27a6bcb52092"} Jul 16 16:55:47.069373 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:55:47.069294 2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3a7cfa47523698d98b33c40b1d5cde641d53676567b7f936b01a27a6bcb52092" Jul 16 16:56:07.991184 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:56:07.991142 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-ac435f2b-openshift-default-77c4b545cf-j8rq8"] Jul 16 16:56:07.991816 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:56:07.991444 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-ac435f2b-openshift-default-77c4b545cf-j8rq8" podUID="66594dbb-e95f-4449-a8bc-54ef2f289968" containerName="istio-proxy" containerID="cri-o://5e7ec071d606b5c145d9adc3e7f9633fcb3fda1491da3c37559e2f3bb3a26f6b" gracePeriod=30 Jul 16 16:56:13.209751 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:56:13.209717 2577 generic.go:356] "Generic (PLEG): container finished" podID="66594dbb-e95f-4449-a8bc-54ef2f289968" containerID="5e7ec071d606b5c145d9adc3e7f9633fcb3fda1491da3c37559e2f3bb3a26f6b" exitCode=0 Jul 16 16:56:13.210239 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:56:13.209796 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-ac435f2b-openshift-default-77c4b545cf-j8rq8" event={"ID":"66594dbb-e95f-4449-a8bc-54ef2f289968","Type":"ContainerDied","Data":"5e7ec071d606b5c145d9adc3e7f9633fcb3fda1491da3c37559e2f3bb3a26f6b"} Jul 16 16:56:13.210239 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:56:13.209857 2577 scope.go:117] "RemoveContainer" containerID="abcb4136e83cc3468c8a2652aee9687fd1375884d346e3ceab65fe16e88c09ed" Jul 16 16:56:13.245756 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:56:13.245727 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-ac435f2b-openshift-default-77c4b545cf-j8rq8" Jul 16 16:56:13.392861 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:56:13.392828 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66594dbb-e95f-4449-a8bc-54ef2f289968-istio-envoy\") pod \"66594dbb-e95f-4449-a8bc-54ef2f289968\" (UID: \"66594dbb-e95f-4449-a8bc-54ef2f289968\") " Jul 16 16:56:13.393717 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:56:13.392874 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66594dbb-e95f-4449-a8bc-54ef2f289968-istiod-ca-cert\") pod \"66594dbb-e95f-4449-a8bc-54ef2f289968\" (UID: \"66594dbb-e95f-4449-a8bc-54ef2f289968\") " Jul 16 16:56:13.393717 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:56:13.392917 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zgzg2\" (UniqueName: \"kubernetes.io/projected/66594dbb-e95f-4449-a8bc-54ef2f289968-kube-api-access-zgzg2\") pod \"66594dbb-e95f-4449-a8bc-54ef2f289968\" (UID: \"66594dbb-e95f-4449-a8bc-54ef2f289968\") " Jul 16 16:56:13.393717 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:56:13.392939 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66594dbb-e95f-4449-a8bc-54ef2f289968-istio-podinfo\") pod \"66594dbb-e95f-4449-a8bc-54ef2f289968\" (UID: \"66594dbb-e95f-4449-a8bc-54ef2f289968\") " Jul 16 16:56:13.393717 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:56:13.392970 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/66594dbb-e95f-4449-a8bc-54ef2f289968-istio-token\") pod \"66594dbb-e95f-4449-a8bc-54ef2f289968\" (UID: \"66594dbb-e95f-4449-a8bc-54ef2f289968\") " Jul 16 16:56:13.393717 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:56:13.393009 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66594dbb-e95f-4449-a8bc-54ef2f289968-credential-socket\") pod \"66594dbb-e95f-4449-a8bc-54ef2f289968\" (UID: \"66594dbb-e95f-4449-a8bc-54ef2f289968\") " Jul 16 16:56:13.393717 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:56:13.393091 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66594dbb-e95f-4449-a8bc-54ef2f289968-workload-certs\") pod \"66594dbb-e95f-4449-a8bc-54ef2f289968\" (UID: \"66594dbb-e95f-4449-a8bc-54ef2f289968\") " Jul 16 16:56:13.393717 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:56:13.393120 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66594dbb-e95f-4449-a8bc-54ef2f289968-workload-socket\") pod \"66594dbb-e95f-4449-a8bc-54ef2f289968\" (UID: \"66594dbb-e95f-4449-a8bc-54ef2f289968\") " Jul 16 16:56:13.393717 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:56:13.393170 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66594dbb-e95f-4449-a8bc-54ef2f289968-istio-data\") pod \"66594dbb-e95f-4449-a8bc-54ef2f289968\" (UID: \"66594dbb-e95f-4449-a8bc-54ef2f289968\") " Jul 16 16:56:13.393717 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:56:13.393355 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66594dbb-e95f-4449-a8bc-54ef2f289968-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "66594dbb-e95f-4449-a8bc-54ef2f289968" (UID: "66594dbb-e95f-4449-a8bc-54ef2f289968"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 16:56:13.393717 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:56:13.393374 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66594dbb-e95f-4449-a8bc-54ef2f289968-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "66594dbb-e95f-4449-a8bc-54ef2f289968" (UID: "66594dbb-e95f-4449-a8bc-54ef2f289968"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 16:56:13.393717 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:56:13.393445 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66594dbb-e95f-4449-a8bc-54ef2f289968-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "66594dbb-e95f-4449-a8bc-54ef2f289968" (UID: "66594dbb-e95f-4449-a8bc-54ef2f289968"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 16:56:13.393717 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:56:13.393605 2577 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66594dbb-e95f-4449-a8bc-54ef2f289968-workload-certs\") on node \"ip-10-0-142-101.ec2.internal\" DevicePath \"\"" Jul 16 16:56:13.393717 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:56:13.393612 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66594dbb-e95f-4449-a8bc-54ef2f289968-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "66594dbb-e95f-4449-a8bc-54ef2f289968" (UID: "66594dbb-e95f-4449-a8bc-54ef2f289968"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 16:56:13.393717 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:56:13.393627 2577 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66594dbb-e95f-4449-a8bc-54ef2f289968-istiod-ca-cert\") on node \"ip-10-0-142-101.ec2.internal\" DevicePath \"\"" Jul 16 16:56:13.393717 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:56:13.393641 2577 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66594dbb-e95f-4449-a8bc-54ef2f289968-credential-socket\") on node \"ip-10-0-142-101.ec2.internal\" DevicePath \"\"" Jul 16 16:56:13.394632 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:56:13.394020 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66594dbb-e95f-4449-a8bc-54ef2f289968-istio-data" (OuterVolumeSpecName: "istio-data") pod "66594dbb-e95f-4449-a8bc-54ef2f289968" (UID: "66594dbb-e95f-4449-a8bc-54ef2f289968"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 16:56:13.395173 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:56:13.395151 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66594dbb-e95f-4449-a8bc-54ef2f289968-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "66594dbb-e95f-4449-a8bc-54ef2f289968" (UID: "66594dbb-e95f-4449-a8bc-54ef2f289968"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 16:56:13.395433 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:56:13.395410 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66594dbb-e95f-4449-a8bc-54ef2f289968-istio-token" (OuterVolumeSpecName: "istio-token") pod "66594dbb-e95f-4449-a8bc-54ef2f289968" (UID: "66594dbb-e95f-4449-a8bc-54ef2f289968"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 16:56:13.395613 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:56:13.395589 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66594dbb-e95f-4449-a8bc-54ef2f289968-kube-api-access-zgzg2" (OuterVolumeSpecName: "kube-api-access-zgzg2") pod "66594dbb-e95f-4449-a8bc-54ef2f289968" (UID: "66594dbb-e95f-4449-a8bc-54ef2f289968"). InnerVolumeSpecName "kube-api-access-zgzg2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 16:56:13.395613 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:56:13.395594 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/66594dbb-e95f-4449-a8bc-54ef2f289968-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "66594dbb-e95f-4449-a8bc-54ef2f289968" (UID: "66594dbb-e95f-4449-a8bc-54ef2f289968"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 16 16:56:13.495147 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:56:13.495088 2577 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66594dbb-e95f-4449-a8bc-54ef2f289968-workload-socket\") on node \"ip-10-0-142-101.ec2.internal\" DevicePath \"\"" Jul 16 16:56:13.495147 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:56:13.495138 2577 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66594dbb-e95f-4449-a8bc-54ef2f289968-istio-data\") on node \"ip-10-0-142-101.ec2.internal\" DevicePath \"\"" Jul 16 16:56:13.495147 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:56:13.495148 2577 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66594dbb-e95f-4449-a8bc-54ef2f289968-istio-envoy\") on node \"ip-10-0-142-101.ec2.internal\" DevicePath \"\"" Jul 16 16:56:13.495147 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:56:13.495160 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zgzg2\" (UniqueName: \"kubernetes.io/projected/66594dbb-e95f-4449-a8bc-54ef2f289968-kube-api-access-zgzg2\") on node \"ip-10-0-142-101.ec2.internal\" DevicePath \"\"" Jul 16 16:56:13.495147 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:56:13.495168 2577 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66594dbb-e95f-4449-a8bc-54ef2f289968-istio-podinfo\") on node \"ip-10-0-142-101.ec2.internal\" DevicePath \"\"" Jul 16 16:56:13.495506 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:56:13.495176 2577 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/66594dbb-e95f-4449-a8bc-54ef2f289968-istio-token\") on node \"ip-10-0-142-101.ec2.internal\" DevicePath \"\"" Jul 16 16:56:14.216640 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:56:14.216610 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-ac435f2b-openshift-default-77c4b545cf-j8rq8" Jul 16 16:56:14.216640 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:56:14.216617 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-ac435f2b-openshift-default-77c4b545cf-j8rq8" event={"ID":"66594dbb-e95f-4449-a8bc-54ef2f289968","Type":"ContainerDied","Data":"3dd1890113dd3aaa045586a4279259500ff6e3149b244043348360f821ea2c69"} Jul 16 16:56:14.217177 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:56:14.216682 2577 scope.go:117] "RemoveContainer" containerID="5e7ec071d606b5c145d9adc3e7f9633fcb3fda1491da3c37559e2f3bb3a26f6b" Jul 16 16:56:14.245425 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:56:14.240263 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-ac435f2b-openshift-default-77c4b545cf-j8rq8"] Jul 16 16:56:14.249613 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:56:14.249581 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-ac435f2b-openshift-default-77c4b545cf-j8rq8"] Jul 16 16:56:16.017613 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:56:16.017570 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="66594dbb-e95f-4449-a8bc-54ef2f289968" path="/var/lib/kubelet/pods/66594dbb-e95f-4449-a8bc-54ef2f289968/volumes" Jul 16 16:57:42.146033 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:57:42.145949 2577 scope.go:117] "RemoveContainer" containerID="04ae557e51a785edbd5752b4c5f9b10d915ee3370337f2c48cf523c236bc48de" Jul 16 16:57:46.384339 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:57:46.384299 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-8a5e08ec-openshift-default-cdb5568f5-wnp95"] Jul 16 16:57:46.384720 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:57:46.384693 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c1b9cef5-e237-4a64-b504-5617c333924d" containerName="revoke-keys" Jul 16 16:57:46.384720 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:57:46.384704 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="c1b9cef5-e237-4a64-b504-5617c333924d" containerName="revoke-keys" Jul 16 16:57:46.384720 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:57:46.384714 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="66594dbb-e95f-4449-a8bc-54ef2f289968" containerName="istio-proxy" Jul 16 16:57:46.384720 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:57:46.384720 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="66594dbb-e95f-4449-a8bc-54ef2f289968" containerName="istio-proxy" Jul 16 16:57:46.384856 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:57:46.384740 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="737d554c-df19-4b9e-87e1-8c5b5195d1d6" containerName="revoke-keys" Jul 16 16:57:46.384856 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:57:46.384746 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="737d554c-df19-4b9e-87e1-8c5b5195d1d6" containerName="revoke-keys" Jul 16 16:57:46.384856 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:57:46.384756 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="66594dbb-e95f-4449-a8bc-54ef2f289968" containerName="istio-proxy" Jul 16 16:57:46.384856 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:57:46.384761 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="66594dbb-e95f-4449-a8bc-54ef2f289968" containerName="istio-proxy" Jul 16 16:57:46.384856 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:57:46.384823 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="66594dbb-e95f-4449-a8bc-54ef2f289968" containerName="istio-proxy" Jul 16 16:57:46.384856 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:57:46.384832 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="66594dbb-e95f-4449-a8bc-54ef2f289968" containerName="istio-proxy" Jul 16 16:57:46.384856 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:57:46.384839 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="c1b9cef5-e237-4a64-b504-5617c333924d" containerName="revoke-keys" Jul 16 16:57:46.388104 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:57:46.388078 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-8a5e08ec-openshift-default-cdb5568f5-wnp95" Jul 16 16:57:46.390832 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:57:46.390808 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-8a5e08ec-openshift-default-dockercfg-5bd2b\"" Jul 16 16:57:46.414758 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:57:46.414719 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-8a5e08ec-openshift-default-cdb5568f5-wnp95"] Jul 16 16:57:46.509960 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:57:46.509916 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09731438-b411-4610-85e3-91e1af09768c-credential-socket\") pod \"e2e-mt-8a5e08ec-openshift-default-cdb5568f5-wnp95\" (UID: \"09731438-b411-4610-85e3-91e1af09768c\") " pod="openshift-ingress/e2e-mt-8a5e08ec-openshift-default-cdb5568f5-wnp95" Jul 16 16:57:46.509960 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:57:46.509968 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09731438-b411-4610-85e3-91e1af09768c-istio-podinfo\") pod \"e2e-mt-8a5e08ec-openshift-default-cdb5568f5-wnp95\" (UID: \"09731438-b411-4610-85e3-91e1af09768c\") " pod="openshift-ingress/e2e-mt-8a5e08ec-openshift-default-cdb5568f5-wnp95" Jul 16 16:57:46.510181 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:57:46.509987 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p297t\" (UniqueName: \"kubernetes.io/projected/09731438-b411-4610-85e3-91e1af09768c-kube-api-access-p297t\") pod \"e2e-mt-8a5e08ec-openshift-default-cdb5568f5-wnp95\" (UID: \"09731438-b411-4610-85e3-91e1af09768c\") " pod="openshift-ingress/e2e-mt-8a5e08ec-openshift-default-cdb5568f5-wnp95" Jul 16 16:57:46.510181 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:57:46.510019 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09731438-b411-4610-85e3-91e1af09768c-istio-token\") pod \"e2e-mt-8a5e08ec-openshift-default-cdb5568f5-wnp95\" (UID: \"09731438-b411-4610-85e3-91e1af09768c\") " pod="openshift-ingress/e2e-mt-8a5e08ec-openshift-default-cdb5568f5-wnp95" Jul 16 16:57:46.510181 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:57:46.510112 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09731438-b411-4610-85e3-91e1af09768c-istio-envoy\") pod \"e2e-mt-8a5e08ec-openshift-default-cdb5568f5-wnp95\" (UID: \"09731438-b411-4610-85e3-91e1af09768c\") " pod="openshift-ingress/e2e-mt-8a5e08ec-openshift-default-cdb5568f5-wnp95" Jul 16 16:57:46.510181 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:57:46.510146 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09731438-b411-4610-85e3-91e1af09768c-workload-certs\") pod \"e2e-mt-8a5e08ec-openshift-default-cdb5568f5-wnp95\" (UID: \"09731438-b411-4610-85e3-91e1af09768c\") " pod="openshift-ingress/e2e-mt-8a5e08ec-openshift-default-cdb5568f5-wnp95" Jul 16 16:57:46.510313 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:57:46.510252 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09731438-b411-4610-85e3-91e1af09768c-workload-socket\") pod \"e2e-mt-8a5e08ec-openshift-default-cdb5568f5-wnp95\" (UID: \"09731438-b411-4610-85e3-91e1af09768c\") " pod="openshift-ingress/e2e-mt-8a5e08ec-openshift-default-cdb5568f5-wnp95" Jul 16 16:57:46.510313 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:57:46.510272 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09731438-b411-4610-85e3-91e1af09768c-istio-data\") pod \"e2e-mt-8a5e08ec-openshift-default-cdb5568f5-wnp95\" (UID: \"09731438-b411-4610-85e3-91e1af09768c\") " pod="openshift-ingress/e2e-mt-8a5e08ec-openshift-default-cdb5568f5-wnp95" Jul 16 16:57:46.510313 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:57:46.510291 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09731438-b411-4610-85e3-91e1af09768c-istiod-ca-cert\") pod \"e2e-mt-8a5e08ec-openshift-default-cdb5568f5-wnp95\" (UID: \"09731438-b411-4610-85e3-91e1af09768c\") " pod="openshift-ingress/e2e-mt-8a5e08ec-openshift-default-cdb5568f5-wnp95" Jul 16 16:57:46.611549 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:57:46.611506 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09731438-b411-4610-85e3-91e1af09768c-workload-socket\") pod \"e2e-mt-8a5e08ec-openshift-default-cdb5568f5-wnp95\" (UID: \"09731438-b411-4610-85e3-91e1af09768c\") " pod="openshift-ingress/e2e-mt-8a5e08ec-openshift-default-cdb5568f5-wnp95" Jul 16 16:57:46.611549 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:57:46.611550 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09731438-b411-4610-85e3-91e1af09768c-istio-data\") pod \"e2e-mt-8a5e08ec-openshift-default-cdb5568f5-wnp95\" (UID: \"09731438-b411-4610-85e3-91e1af09768c\") " pod="openshift-ingress/e2e-mt-8a5e08ec-openshift-default-cdb5568f5-wnp95" Jul 16 16:57:46.611825 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:57:46.611572 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09731438-b411-4610-85e3-91e1af09768c-istiod-ca-cert\") pod \"e2e-mt-8a5e08ec-openshift-default-cdb5568f5-wnp95\" (UID: \"09731438-b411-4610-85e3-91e1af09768c\") " pod="openshift-ingress/e2e-mt-8a5e08ec-openshift-default-cdb5568f5-wnp95" Jul 16 16:57:46.611825 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:57:46.611609 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09731438-b411-4610-85e3-91e1af09768c-credential-socket\") pod \"e2e-mt-8a5e08ec-openshift-default-cdb5568f5-wnp95\" (UID: \"09731438-b411-4610-85e3-91e1af09768c\") " pod="openshift-ingress/e2e-mt-8a5e08ec-openshift-default-cdb5568f5-wnp95" Jul 16 16:57:46.611825 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:57:46.611632 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09731438-b411-4610-85e3-91e1af09768c-istio-podinfo\") pod \"e2e-mt-8a5e08ec-openshift-default-cdb5568f5-wnp95\" (UID: \"09731438-b411-4610-85e3-91e1af09768c\") " pod="openshift-ingress/e2e-mt-8a5e08ec-openshift-default-cdb5568f5-wnp95" Jul 16 16:57:46.611825 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:57:46.611649 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p297t\" (UniqueName: \"kubernetes.io/projected/09731438-b411-4610-85e3-91e1af09768c-kube-api-access-p297t\") pod \"e2e-mt-8a5e08ec-openshift-default-cdb5568f5-wnp95\" (UID: \"09731438-b411-4610-85e3-91e1af09768c\") " pod="openshift-ingress/e2e-mt-8a5e08ec-openshift-default-cdb5568f5-wnp95" Jul 16 16:57:46.611825 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:57:46.611677 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09731438-b411-4610-85e3-91e1af09768c-istio-token\") pod \"e2e-mt-8a5e08ec-openshift-default-cdb5568f5-wnp95\" (UID: \"09731438-b411-4610-85e3-91e1af09768c\") " pod="openshift-ingress/e2e-mt-8a5e08ec-openshift-default-cdb5568f5-wnp95" Jul 16 16:57:46.611825 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:57:46.611804 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09731438-b411-4610-85e3-91e1af09768c-istio-envoy\") pod \"e2e-mt-8a5e08ec-openshift-default-cdb5568f5-wnp95\" (UID: \"09731438-b411-4610-85e3-91e1af09768c\") " pod="openshift-ingress/e2e-mt-8a5e08ec-openshift-default-cdb5568f5-wnp95" Jul 16 16:57:46.612134 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:57:46.611844 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09731438-b411-4610-85e3-91e1af09768c-workload-certs\") pod \"e2e-mt-8a5e08ec-openshift-default-cdb5568f5-wnp95\" (UID: \"09731438-b411-4610-85e3-91e1af09768c\") " pod="openshift-ingress/e2e-mt-8a5e08ec-openshift-default-cdb5568f5-wnp95" Jul 16 16:57:46.612134 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:57:46.612057 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09731438-b411-4610-85e3-91e1af09768c-istio-data\") pod \"e2e-mt-8a5e08ec-openshift-default-cdb5568f5-wnp95\" (UID: \"09731438-b411-4610-85e3-91e1af09768c\") " pod="openshift-ingress/e2e-mt-8a5e08ec-openshift-default-cdb5568f5-wnp95" Jul 16 16:57:46.612134 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:57:46.612082 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09731438-b411-4610-85e3-91e1af09768c-credential-socket\") pod \"e2e-mt-8a5e08ec-openshift-default-cdb5568f5-wnp95\" (UID: \"09731438-b411-4610-85e3-91e1af09768c\") " pod="openshift-ingress/e2e-mt-8a5e08ec-openshift-default-cdb5568f5-wnp95" Jul 16 16:57:46.612134 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:57:46.612103 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09731438-b411-4610-85e3-91e1af09768c-workload-socket\") pod \"e2e-mt-8a5e08ec-openshift-default-cdb5568f5-wnp95\" (UID: \"09731438-b411-4610-85e3-91e1af09768c\") " pod="openshift-ingress/e2e-mt-8a5e08ec-openshift-default-cdb5568f5-wnp95" Jul 16 16:57:46.612316 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:57:46.612294 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09731438-b411-4610-85e3-91e1af09768c-workload-certs\") pod \"e2e-mt-8a5e08ec-openshift-default-cdb5568f5-wnp95\" (UID: \"09731438-b411-4610-85e3-91e1af09768c\") " pod="openshift-ingress/e2e-mt-8a5e08ec-openshift-default-cdb5568f5-wnp95" Jul 16 16:57:46.612642 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:57:46.612614 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09731438-b411-4610-85e3-91e1af09768c-istiod-ca-cert\") pod \"e2e-mt-8a5e08ec-openshift-default-cdb5568f5-wnp95\" (UID: \"09731438-b411-4610-85e3-91e1af09768c\") " pod="openshift-ingress/e2e-mt-8a5e08ec-openshift-default-cdb5568f5-wnp95" Jul 16 16:57:46.613939 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:57:46.613917 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09731438-b411-4610-85e3-91e1af09768c-istio-envoy\") pod \"e2e-mt-8a5e08ec-openshift-default-cdb5568f5-wnp95\" (UID: \"09731438-b411-4610-85e3-91e1af09768c\") " pod="openshift-ingress/e2e-mt-8a5e08ec-openshift-default-cdb5568f5-wnp95" Jul 16 16:57:46.614344 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:57:46.614321 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09731438-b411-4610-85e3-91e1af09768c-istio-podinfo\") pod \"e2e-mt-8a5e08ec-openshift-default-cdb5568f5-wnp95\" (UID: \"09731438-b411-4610-85e3-91e1af09768c\") " pod="openshift-ingress/e2e-mt-8a5e08ec-openshift-default-cdb5568f5-wnp95" Jul 16 16:57:46.620079 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:57:46.620055 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09731438-b411-4610-85e3-91e1af09768c-istio-token\") pod \"e2e-mt-8a5e08ec-openshift-default-cdb5568f5-wnp95\" (UID: \"09731438-b411-4610-85e3-91e1af09768c\") " pod="openshift-ingress/e2e-mt-8a5e08ec-openshift-default-cdb5568f5-wnp95" Jul 16 16:57:46.620367 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:57:46.620343 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p297t\" (UniqueName: \"kubernetes.io/projected/09731438-b411-4610-85e3-91e1af09768c-kube-api-access-p297t\") pod \"e2e-mt-8a5e08ec-openshift-default-cdb5568f5-wnp95\" (UID: \"09731438-b411-4610-85e3-91e1af09768c\") " pod="openshift-ingress/e2e-mt-8a5e08ec-openshift-default-cdb5568f5-wnp95" Jul 16 16:57:46.698286 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:57:46.698190 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-8a5e08ec-openshift-default-cdb5568f5-wnp95" Jul 16 16:57:46.842264 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:57:46.842236 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-8a5e08ec-openshift-default-cdb5568f5-wnp95"] Jul 16 16:57:46.844779 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:57:46.844749 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09731438_b411_4610_85e3_91e1af09768c.slice/crio-31d706c6978cff94b877a77c234fda09af83846984bf95112c6e28c333e7b956 WatchSource:0}: Error finding container 31d706c6978cff94b877a77c234fda09af83846984bf95112c6e28c333e7b956: Status 404 returned error can't find the container with id 31d706c6978cff94b877a77c234fda09af83846984bf95112c6e28c333e7b956 Jul 16 16:57:46.847489 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:57:46.847451 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 16 16:57:46.847592 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:57:46.847538 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 16 16:57:46.847592 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:57:46.847581 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 16 16:57:47.722314 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:57:47.722274 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-8a5e08ec-openshift-default-cdb5568f5-wnp95" event={"ID":"09731438-b411-4610-85e3-91e1af09768c","Type":"ContainerStarted","Data":"51ae653877db1190c02f29c6f37a69502186686587d3be44c5cf40428eee222e"} Jul 16 16:57:47.722871 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:57:47.722331 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-8a5e08ec-openshift-default-cdb5568f5-wnp95" event={"ID":"09731438-b411-4610-85e3-91e1af09768c","Type":"ContainerStarted","Data":"31d706c6978cff94b877a77c234fda09af83846984bf95112c6e28c333e7b956"} Jul 16 16:57:47.744778 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:57:47.744714 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-8a5e08ec-openshift-default-cdb5568f5-wnp95" podStartSLOduration=1.744692827 podStartE2EDuration="1.744692827s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 16:57:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 16:57:47.742730219 +0000 UTC m=+1926.246788104" watchObservedRunningTime="2026-07-16 16:57:47.744692827 +0000 UTC m=+1926.248750679" Jul 16 16:57:48.699373 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:57:48.699332 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-8a5e08ec-openshift-default-cdb5568f5-wnp95" Jul 16 16:57:48.704558 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:57:48.704529 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-8a5e08ec-openshift-default-cdb5568f5-wnp95" Jul 16 16:57:48.729041 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:57:48.729002 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-8a5e08ec-openshift-default-cdb5568f5-wnp95" Jul 16 16:57:48.730093 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:57:48.730074 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-8a5e08ec-openshift-default-cdb5568f5-wnp95" Jul 16 16:58:01.893190 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:58:01.893150 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8a5e08ec-jn2ds"] Jul 16 16:58:01.897599 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:58:01.897571 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8a5e08ec-jn2ds" Jul 16 16:58:01.901099 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:58:01.901071 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dssfh\"" Jul 16 16:58:01.901258 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:58:01.901208 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 16 16:58:01.921661 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:58:01.921631 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8a5e08ec-jn2ds"] Jul 16 16:58:02.088170 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:58:02.088128 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/18cf139b-c345-4d42-badd-027e6931d112-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-8a5e08ec-jn2ds\" (UID: \"18cf139b-c345-4d42-badd-027e6931d112\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8a5e08ec-jn2ds" Jul 16 16:58:02.189689 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:58:02.189592 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/18cf139b-c345-4d42-badd-027e6931d112-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-8a5e08ec-jn2ds\" (UID: \"18cf139b-c345-4d42-badd-027e6931d112\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8a5e08ec-jn2ds" Jul 16 16:58:02.190242 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:58:02.190219 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/18cf139b-c345-4d42-badd-027e6931d112-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-8a5e08ec-jn2ds\" (UID: \"18cf139b-c345-4d42-badd-027e6931d112\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8a5e08ec-jn2ds" Jul 16 16:58:02.208547 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:58:02.208514 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8a5e08ec-jn2ds" Jul 16 16:58:02.368063 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:58:02.368033 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8a5e08ec-jn2ds"] Jul 16 16:58:02.368712 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:58:02.368684 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18cf139b_c345_4d42_badd_027e6931d112.slice/crio-c00e69f13f4c7b67f3cf42bc72b564e1e33dc429515e0c3587c1a4d9ede56fb9 WatchSource:0}: Error finding container c00e69f13f4c7b67f3cf42bc72b564e1e33dc429515e0c3587c1a4d9ede56fb9: Status 404 returned error can't find the container with id c00e69f13f4c7b67f3cf42bc72b564e1e33dc429515e0c3587c1a4d9ede56fb9 Jul 16 16:58:02.808770 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:58:02.808681 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8a5e08ec-jn2ds" event={"ID":"18cf139b-c345-4d42-badd-027e6931d112","Type":"ContainerStarted","Data":"45bc48caad6cd6afb6f857bfd11a38e7273919db3f18544c8a3810b3dc4cbd2b"} Jul 16 16:58:02.808770 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:58:02.808745 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8a5e08ec-jn2ds" event={"ID":"18cf139b-c345-4d42-badd-027e6931d112","Type":"ContainerStarted","Data":"c00e69f13f4c7b67f3cf42bc72b564e1e33dc429515e0c3587c1a4d9ede56fb9"} Jul 16 16:58:02.828095 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:58:02.828040 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8a5e08ec-jn2ds" podStartSLOduration=1.828023536 podStartE2EDuration="1.828023536s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 16:58:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 16:58:02.826365734 +0000 UTC m=+1941.330423586" watchObservedRunningTime="2026-07-16 16:58:02.828023536 +0000 UTC m=+1941.332081386" Jul 16 16:58:03.730953 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:58:03.730912 2577 patch_prober.go:28] interesting pod/e2e-mt-8a5e08ec-openshift-default-cdb5568f5-wnp95 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Jul 16 16:58:03.731377 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:58:03.730965 2577 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-8a5e08ec-openshift-default-cdb5568f5-wnp95" podUID="09731438-b411-4610-85e3-91e1af09768c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Jul 16 16:58:03.815795 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:58:03.815758 2577 generic.go:356] "Generic (PLEG): container finished" podID="09731438-b411-4610-85e3-91e1af09768c" containerID="51ae653877db1190c02f29c6f37a69502186686587d3be44c5cf40428eee222e" exitCode=0 Jul 16 16:58:03.815974 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:58:03.815833 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-8a5e08ec-openshift-default-cdb5568f5-wnp95" event={"ID":"09731438-b411-4610-85e3-91e1af09768c","Type":"ContainerDied","Data":"51ae653877db1190c02f29c6f37a69502186686587d3be44c5cf40428eee222e"} Jul 16 16:58:03.816334 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:58:03.816311 2577 scope.go:117] "RemoveContainer" containerID="51ae653877db1190c02f29c6f37a69502186686587d3be44c5cf40428eee222e" Jul 16 16:58:03.817453 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:58:03.817427 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 16 16:58:03.818059 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:58:03.817921 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 16 16:58:03.818059 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:58:03.817935 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8a5e08ec-jn2ds" event={"ID":"18cf139b-c345-4d42-badd-027e6931d112","Type":"ContainerDied","Data":"45bc48caad6cd6afb6f857bfd11a38e7273919db3f18544c8a3810b3dc4cbd2b"} Jul 16 16:58:03.818059 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:58:03.818049 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 16 16:58:03.818285 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:58:03.817921 2577 generic.go:356] "Generic (PLEG): container finished" podID="18cf139b-c345-4d42-badd-027e6931d112" containerID="45bc48caad6cd6afb6f857bfd11a38e7273919db3f18544c8a3810b3dc4cbd2b" exitCode=0 Jul 16 16:58:04.825921 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:58:04.825881 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-8a5e08ec-openshift-default-cdb5568f5-wnp95" event={"ID":"09731438-b411-4610-85e3-91e1af09768c","Type":"ContainerStarted","Data":"322b0a5467b46607142aa1847e0bb54961be044a38075fde47b78fbb74f2a059"} Jul 16 16:58:05.015884 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:58:05.015855 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8a5e08ec-jn2ds" Jul 16 16:58:05.017011 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:58:05.016987 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/18cf139b-c345-4d42-badd-027e6931d112-maas-api-service-ca\") pod \"18cf139b-c345-4d42-badd-027e6931d112\" (UID: \"18cf139b-c345-4d42-badd-027e6931d112\") " Jul 16 16:58:05.017351 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:58:05.017327 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18cf139b-c345-4d42-badd-027e6931d112-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "18cf139b-c345-4d42-badd-027e6931d112" (UID: "18cf139b-c345-4d42-badd-027e6931d112"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 16:58:05.118268 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:58:05.118173 2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/18cf139b-c345-4d42-badd-027e6931d112-maas-api-service-ca\") on node \"ip-10-0-142-101.ec2.internal\" DevicePath \"\"" Jul 16 16:58:05.698607 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:58:05.698566 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-8a5e08ec-openshift-default-cdb5568f5-wnp95" Jul 16 16:58:05.698847 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:58:05.698824 2577 patch_prober.go:28] interesting pod/e2e-mt-8a5e08ec-openshift-default-cdb5568f5-wnp95 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Jul 16 16:58:05.698897 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:58:05.698872 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-8a5e08ec-openshift-default-cdb5568f5-wnp95" podUID="09731438-b411-4610-85e3-91e1af09768c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Jul 16 16:58:05.832854 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:58:05.832820 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8a5e08ec-jn2ds" Jul 16 16:58:05.833258 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:58:05.832858 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8a5e08ec-jn2ds" event={"ID":"18cf139b-c345-4d42-badd-027e6931d112","Type":"ContainerDied","Data":"c00e69f13f4c7b67f3cf42bc72b564e1e33dc429515e0c3587c1a4d9ede56fb9"} Jul 16 16:58:05.833258 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:58:05.832910 2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c00e69f13f4c7b67f3cf42bc72b564e1e33dc429515e0c3587c1a4d9ede56fb9" Jul 16 16:58:06.699632 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:58:06.699598 2577 patch_prober.go:28] interesting pod/e2e-mt-8a5e08ec-openshift-default-cdb5568f5-wnp95 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Jul 16 16:58:06.699854 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:58:06.699673 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-8a5e08ec-openshift-default-cdb5568f5-wnp95" podUID="09731438-b411-4610-85e3-91e1af09768c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Jul 16 16:58:07.699372 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:58:07.699338 2577 patch_prober.go:28] interesting pod/e2e-mt-8a5e08ec-openshift-default-cdb5568f5-wnp95 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Jul 16 16:58:07.699786 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:58:07.699408 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-8a5e08ec-openshift-default-cdb5568f5-wnp95" podUID="09731438-b411-4610-85e3-91e1af09768c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Jul 16 16:58:08.699438 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:58:08.699366 2577 patch_prober.go:28] interesting pod/e2e-mt-8a5e08ec-openshift-default-cdb5568f5-wnp95 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Jul 16 16:58:08.699927 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:58:08.699477 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-8a5e08ec-openshift-default-cdb5568f5-wnp95" podUID="09731438-b411-4610-85e3-91e1af09768c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Jul 16 16:58:09.702650 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:58:09.702617 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-8a5e08ec-openshift-default-cdb5568f5-wnp95" Jul 16 16:58:09.703033 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:58:09.702875 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-8a5e08ec-openshift-default-cdb5568f5-wnp95" Jul 16 16:58:09.703631 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:58:09.703614 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-8a5e08ec-openshift-default-cdb5568f5-wnp95" Jul 16 16:58:27.722379 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:58:27.722343 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-8a5e08ec-openshift-default-cdb5568f5-wnp95"] Jul 16 16:58:27.722990 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:58:27.722591 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-8a5e08ec-openshift-default-cdb5568f5-wnp95" podUID="09731438-b411-4610-85e3-91e1af09768c" containerName="istio-proxy" containerID="cri-o://322b0a5467b46607142aa1847e0bb54961be044a38075fde47b78fbb74f2a059" gracePeriod=30 Jul 16 16:58:32.987637 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:58:32.987569 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-8a5e08ec-openshift-default-cdb5568f5-wnp95" Jul 16 16:58:33.000181 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:58:33.000144 2577 generic.go:356] "Generic (PLEG): container finished" podID="09731438-b411-4610-85e3-91e1af09768c" containerID="322b0a5467b46607142aa1847e0bb54961be044a38075fde47b78fbb74f2a059" exitCode=0 Jul 16 16:58:33.000337 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:58:33.000187 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-8a5e08ec-openshift-default-cdb5568f5-wnp95" Jul 16 16:58:33.000337 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:58:33.000196 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-8a5e08ec-openshift-default-cdb5568f5-wnp95" event={"ID":"09731438-b411-4610-85e3-91e1af09768c","Type":"ContainerDied","Data":"322b0a5467b46607142aa1847e0bb54961be044a38075fde47b78fbb74f2a059"} Jul 16 16:58:33.000337 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:58:33.000243 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-8a5e08ec-openshift-default-cdb5568f5-wnp95" event={"ID":"09731438-b411-4610-85e3-91e1af09768c","Type":"ContainerDied","Data":"31d706c6978cff94b877a77c234fda09af83846984bf95112c6e28c333e7b956"} Jul 16 16:58:33.000337 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:58:33.000266 2577 scope.go:117] "RemoveContainer" containerID="322b0a5467b46607142aa1847e0bb54961be044a38075fde47b78fbb74f2a059" Jul 16 16:58:33.010320 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:58:33.010289 2577 scope.go:117] "RemoveContainer" containerID="51ae653877db1190c02f29c6f37a69502186686587d3be44c5cf40428eee222e" Jul 16 16:58:33.023781 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:58:33.023758 2577 scope.go:117] "RemoveContainer" containerID="322b0a5467b46607142aa1847e0bb54961be044a38075fde47b78fbb74f2a059" Jul 16 16:58:33.024088 ip-10-0-142-101 kubenswrapper[2577]: E0716 16:58:33.024065 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"322b0a5467b46607142aa1847e0bb54961be044a38075fde47b78fbb74f2a059\": container with ID starting with 322b0a5467b46607142aa1847e0bb54961be044a38075fde47b78fbb74f2a059 not found: ID does not exist" containerID="322b0a5467b46607142aa1847e0bb54961be044a38075fde47b78fbb74f2a059" Jul 16 16:58:33.024188 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:58:33.024095 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"322b0a5467b46607142aa1847e0bb54961be044a38075fde47b78fbb74f2a059"} err="failed to get container status \"322b0a5467b46607142aa1847e0bb54961be044a38075fde47b78fbb74f2a059\": rpc error: code = NotFound desc = could not find container \"322b0a5467b46607142aa1847e0bb54961be044a38075fde47b78fbb74f2a059\": container with ID starting with 322b0a5467b46607142aa1847e0bb54961be044a38075fde47b78fbb74f2a059 not found: ID does not exist" Jul 16 16:58:33.024188 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:58:33.024114 2577 scope.go:117] "RemoveContainer" containerID="51ae653877db1190c02f29c6f37a69502186686587d3be44c5cf40428eee222e" Jul 16 16:58:33.024418 ip-10-0-142-101 kubenswrapper[2577]: E0716 16:58:33.024397 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"51ae653877db1190c02f29c6f37a69502186686587d3be44c5cf40428eee222e\": container with ID starting with 51ae653877db1190c02f29c6f37a69502186686587d3be44c5cf40428eee222e not found: ID does not exist" containerID="51ae653877db1190c02f29c6f37a69502186686587d3be44c5cf40428eee222e" Jul 16 16:58:33.024466 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:58:33.024424 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"51ae653877db1190c02f29c6f37a69502186686587d3be44c5cf40428eee222e"} err="failed to get container status \"51ae653877db1190c02f29c6f37a69502186686587d3be44c5cf40428eee222e\": rpc error: code = NotFound desc = could not find container \"51ae653877db1190c02f29c6f37a69502186686587d3be44c5cf40428eee222e\": container with ID starting with 51ae653877db1190c02f29c6f37a69502186686587d3be44c5cf40428eee222e not found: ID does not exist" Jul 16 16:58:33.095297 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:58:33.095265 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09731438-b411-4610-85e3-91e1af09768c-istio-envoy\") pod \"09731438-b411-4610-85e3-91e1af09768c\" (UID: \"09731438-b411-4610-85e3-91e1af09768c\") " Jul 16 16:58:33.095513 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:58:33.095313 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09731438-b411-4610-85e3-91e1af09768c-credential-socket\") pod \"09731438-b411-4610-85e3-91e1af09768c\" (UID: \"09731438-b411-4610-85e3-91e1af09768c\") " Jul 16 16:58:33.095513 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:58:33.095342 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09731438-b411-4610-85e3-91e1af09768c-workload-certs\") pod \"09731438-b411-4610-85e3-91e1af09768c\" (UID: \"09731438-b411-4610-85e3-91e1af09768c\") " Jul 16 16:58:33.095513 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:58:33.095427 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09731438-b411-4610-85e3-91e1af09768c-istiod-ca-cert\") pod \"09731438-b411-4610-85e3-91e1af09768c\" (UID: \"09731438-b411-4610-85e3-91e1af09768c\") " Jul 16 16:58:33.095513 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:58:33.095457 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p297t\" (UniqueName: \"kubernetes.io/projected/09731438-b411-4610-85e3-91e1af09768c-kube-api-access-p297t\") pod \"09731438-b411-4610-85e3-91e1af09768c\" (UID: \"09731438-b411-4610-85e3-91e1af09768c\") " Jul 16 16:58:33.095738 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:58:33.095516 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09731438-b411-4610-85e3-91e1af09768c-istio-data\") pod \"09731438-b411-4610-85e3-91e1af09768c\" (UID: \"09731438-b411-4610-85e3-91e1af09768c\") " Jul 16 16:58:33.095738 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:58:33.095551 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09731438-b411-4610-85e3-91e1af09768c-istio-token\") pod \"09731438-b411-4610-85e3-91e1af09768c\" (UID: \"09731438-b411-4610-85e3-91e1af09768c\") " Jul 16 16:58:33.095738 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:58:33.095588 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09731438-b411-4610-85e3-91e1af09768c-workload-socket\") pod \"09731438-b411-4610-85e3-91e1af09768c\" (UID: \"09731438-b411-4610-85e3-91e1af09768c\") " Jul 16 16:58:33.095738 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:58:33.095640 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09731438-b411-4610-85e3-91e1af09768c-istio-podinfo\") pod \"09731438-b411-4610-85e3-91e1af09768c\" (UID: \"09731438-b411-4610-85e3-91e1af09768c\") " Jul 16 16:58:33.095738 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:58:33.095674 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09731438-b411-4610-85e3-91e1af09768c-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "09731438-b411-4610-85e3-91e1af09768c" (UID: "09731438-b411-4610-85e3-91e1af09768c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 16:58:33.095738 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:58:33.095695 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09731438-b411-4610-85e3-91e1af09768c-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "09731438-b411-4610-85e3-91e1af09768c" (UID: "09731438-b411-4610-85e3-91e1af09768c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 16:58:33.096048 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:58:33.095901 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09731438-b411-4610-85e3-91e1af09768c-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "09731438-b411-4610-85e3-91e1af09768c" (UID: "09731438-b411-4610-85e3-91e1af09768c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 16:58:33.096048 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:58:33.095928 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09731438-b411-4610-85e3-91e1af09768c-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "09731438-b411-4610-85e3-91e1af09768c" (UID: "09731438-b411-4610-85e3-91e1af09768c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 16:58:33.096166 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:58:33.096098 2577 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09731438-b411-4610-85e3-91e1af09768c-credential-socket\") on node \"ip-10-0-142-101.ec2.internal\" DevicePath \"\"" Jul 16 16:58:33.096166 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:58:33.096117 2577 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09731438-b411-4610-85e3-91e1af09768c-workload-certs\") on node \"ip-10-0-142-101.ec2.internal\" DevicePath \"\"" Jul 16 16:58:33.096166 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:58:33.096131 2577 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09731438-b411-4610-85e3-91e1af09768c-istiod-ca-cert\") on node \"ip-10-0-142-101.ec2.internal\" DevicePath \"\"" Jul 16 16:58:33.096166 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:58:33.096143 2577 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09731438-b411-4610-85e3-91e1af09768c-workload-socket\") on node \"ip-10-0-142-101.ec2.internal\" DevicePath \"\"" Jul 16 16:58:33.096778 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:58:33.096745 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09731438-b411-4610-85e3-91e1af09768c-istio-data" (OuterVolumeSpecName: "istio-data") pod "09731438-b411-4610-85e3-91e1af09768c" (UID: "09731438-b411-4610-85e3-91e1af09768c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 16:58:33.097652 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:58:33.097633 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09731438-b411-4610-85e3-91e1af09768c-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "09731438-b411-4610-85e3-91e1af09768c" (UID: "09731438-b411-4610-85e3-91e1af09768c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 16:58:33.097874 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:58:33.097860 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/09731438-b411-4610-85e3-91e1af09768c-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "09731438-b411-4610-85e3-91e1af09768c" (UID: "09731438-b411-4610-85e3-91e1af09768c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 16 16:58:33.098253 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:58:33.098237 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09731438-b411-4610-85e3-91e1af09768c-istio-token" (OuterVolumeSpecName: "istio-token") pod "09731438-b411-4610-85e3-91e1af09768c" (UID: "09731438-b411-4610-85e3-91e1af09768c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 16:58:33.098321 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:58:33.098262 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09731438-b411-4610-85e3-91e1af09768c-kube-api-access-p297t" (OuterVolumeSpecName: "kube-api-access-p297t") pod "09731438-b411-4610-85e3-91e1af09768c" (UID: "09731438-b411-4610-85e3-91e1af09768c"). InnerVolumeSpecName "kube-api-access-p297t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 16:58:33.197221 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:58:33.197129 2577 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09731438-b411-4610-85e3-91e1af09768c-istio-podinfo\") on node \"ip-10-0-142-101.ec2.internal\" DevicePath \"\"" Jul 16 16:58:33.197221 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:58:33.197164 2577 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09731438-b411-4610-85e3-91e1af09768c-istio-envoy\") on node \"ip-10-0-142-101.ec2.internal\" DevicePath \"\"" Jul 16 16:58:33.197221 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:58:33.197178 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p297t\" (UniqueName: \"kubernetes.io/projected/09731438-b411-4610-85e3-91e1af09768c-kube-api-access-p297t\") on node \"ip-10-0-142-101.ec2.internal\" DevicePath \"\"" Jul 16 16:58:33.197221 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:58:33.197191 2577 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09731438-b411-4610-85e3-91e1af09768c-istio-data\") on node \"ip-10-0-142-101.ec2.internal\" DevicePath \"\"" Jul 16 16:58:33.197221 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:58:33.197203 2577 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09731438-b411-4610-85e3-91e1af09768c-istio-token\") on node \"ip-10-0-142-101.ec2.internal\" DevicePath \"\"" Jul 16 16:58:33.323704 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:58:33.323667 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-8a5e08ec-openshift-default-cdb5568f5-wnp95"] Jul 16 16:58:33.329809 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:58:33.329782 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-8a5e08ec-openshift-default-cdb5568f5-wnp95"] Jul 16 16:58:34.018292 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:58:34.018256 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="09731438-b411-4610-85e3-91e1af09768c" path="/var/lib/kubelet/pods/09731438-b411-4610-85e3-91e1af09768c/volumes" Jul 16 16:58:42.167315 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:58:42.167278 2577 scope.go:117] "RemoveContainer" containerID="1c1bef34ca10986cdb489bd597e99e382b01e19cff2e2e2d4852b2cf2f2c82e7" Jul 16 16:58:48.132607 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:58:48.132568 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-187946be-openshift-default-6555dfc579-pwrt9"] Jul 16 16:58:48.133170 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:58:48.133150 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="18cf139b-c345-4d42-badd-027e6931d112" containerName="revoke-keys" Jul 16 16:58:48.133217 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:58:48.133173 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="18cf139b-c345-4d42-badd-027e6931d112" containerName="revoke-keys" Jul 16 16:58:48.133217 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:58:48.133190 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="09731438-b411-4610-85e3-91e1af09768c" containerName="istio-proxy" Jul 16 16:58:48.133217 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:58:48.133198 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="09731438-b411-4610-85e3-91e1af09768c" containerName="istio-proxy" Jul 16 16:58:48.133314 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:58:48.133218 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="09731438-b411-4610-85e3-91e1af09768c" containerName="istio-proxy" Jul 16 16:58:48.133314 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:58:48.133226 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="09731438-b411-4610-85e3-91e1af09768c" containerName="istio-proxy" Jul 16 16:58:48.133314 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:58:48.133312 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="09731438-b411-4610-85e3-91e1af09768c" containerName="istio-proxy" Jul 16 16:58:48.133439 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:58:48.133322 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="09731438-b411-4610-85e3-91e1af09768c" containerName="istio-proxy" Jul 16 16:58:48.133439 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:58:48.133336 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="18cf139b-c345-4d42-badd-027e6931d112" containerName="revoke-keys" Jul 16 16:58:48.136630 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:58:48.136608 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-187946be-openshift-default-6555dfc579-pwrt9" Jul 16 16:58:48.140224 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:58:48.140202 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-187946be-openshift-default-dockercfg-k4sj9\"" Jul 16 16:58:48.164210 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:58:48.164175 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-187946be-openshift-default-6555dfc579-pwrt9"] Jul 16 16:58:48.238843 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:58:48.238801 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed53e0ee-2e8e-4694-8290-2754d392cbe8-istio-envoy\") pod \"e2e-mt-187946be-openshift-default-6555dfc579-pwrt9\" (UID: \"ed53e0ee-2e8e-4694-8290-2754d392cbe8\") " pod="openshift-ingress/e2e-mt-187946be-openshift-default-6555dfc579-pwrt9" Jul 16 16:58:48.239062 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:58:48.238846 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed53e0ee-2e8e-4694-8290-2754d392cbe8-workload-certs\") pod \"e2e-mt-187946be-openshift-default-6555dfc579-pwrt9\" (UID: \"ed53e0ee-2e8e-4694-8290-2754d392cbe8\") " pod="openshift-ingress/e2e-mt-187946be-openshift-default-6555dfc579-pwrt9" Jul 16 16:58:48.239062 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:58:48.238980 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7k946\" (UniqueName: \"kubernetes.io/projected/ed53e0ee-2e8e-4694-8290-2754d392cbe8-kube-api-access-7k946\") pod \"e2e-mt-187946be-openshift-default-6555dfc579-pwrt9\" (UID: \"ed53e0ee-2e8e-4694-8290-2754d392cbe8\") " pod="openshift-ingress/e2e-mt-187946be-openshift-default-6555dfc579-pwrt9" Jul 16 16:58:48.239062 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:58:48.239008 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed53e0ee-2e8e-4694-8290-2754d392cbe8-istiod-ca-cert\") pod \"e2e-mt-187946be-openshift-default-6555dfc579-pwrt9\" (UID: \"ed53e0ee-2e8e-4694-8290-2754d392cbe8\") " pod="openshift-ingress/e2e-mt-187946be-openshift-default-6555dfc579-pwrt9" Jul 16 16:58:48.239062 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:58:48.239035 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed53e0ee-2e8e-4694-8290-2754d392cbe8-istio-podinfo\") pod \"e2e-mt-187946be-openshift-default-6555dfc579-pwrt9\" (UID: \"ed53e0ee-2e8e-4694-8290-2754d392cbe8\") " pod="openshift-ingress/e2e-mt-187946be-openshift-default-6555dfc579-pwrt9" Jul 16 16:58:48.239299 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:58:48.239079 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed53e0ee-2e8e-4694-8290-2754d392cbe8-istio-token\") pod \"e2e-mt-187946be-openshift-default-6555dfc579-pwrt9\" (UID: \"ed53e0ee-2e8e-4694-8290-2754d392cbe8\") " pod="openshift-ingress/e2e-mt-187946be-openshift-default-6555dfc579-pwrt9" Jul 16 16:58:48.239299 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:58:48.239122 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed53e0ee-2e8e-4694-8290-2754d392cbe8-istio-data\") pod \"e2e-mt-187946be-openshift-default-6555dfc579-pwrt9\" (UID: \"ed53e0ee-2e8e-4694-8290-2754d392cbe8\") " pod="openshift-ingress/e2e-mt-187946be-openshift-default-6555dfc579-pwrt9" Jul 16 16:58:48.239299 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:58:48.239156 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed53e0ee-2e8e-4694-8290-2754d392cbe8-workload-socket\") pod \"e2e-mt-187946be-openshift-default-6555dfc579-pwrt9\" (UID: \"ed53e0ee-2e8e-4694-8290-2754d392cbe8\") " pod="openshift-ingress/e2e-mt-187946be-openshift-default-6555dfc579-pwrt9" Jul 16 16:58:48.239299 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:58:48.239184 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed53e0ee-2e8e-4694-8290-2754d392cbe8-credential-socket\") pod \"e2e-mt-187946be-openshift-default-6555dfc579-pwrt9\" (UID: \"ed53e0ee-2e8e-4694-8290-2754d392cbe8\") " pod="openshift-ingress/e2e-mt-187946be-openshift-default-6555dfc579-pwrt9" Jul 16 16:58:48.339709 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:58:48.339673 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7k946\" (UniqueName: \"kubernetes.io/projected/ed53e0ee-2e8e-4694-8290-2754d392cbe8-kube-api-access-7k946\") pod \"e2e-mt-187946be-openshift-default-6555dfc579-pwrt9\" (UID: \"ed53e0ee-2e8e-4694-8290-2754d392cbe8\") " pod="openshift-ingress/e2e-mt-187946be-openshift-default-6555dfc579-pwrt9" Jul 16 16:58:48.339709 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:58:48.339712 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed53e0ee-2e8e-4694-8290-2754d392cbe8-istiod-ca-cert\") pod \"e2e-mt-187946be-openshift-default-6555dfc579-pwrt9\" (UID: \"ed53e0ee-2e8e-4694-8290-2754d392cbe8\") " pod="openshift-ingress/e2e-mt-187946be-openshift-default-6555dfc579-pwrt9" Jul 16 16:58:48.339966 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:58:48.339735 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed53e0ee-2e8e-4694-8290-2754d392cbe8-istio-podinfo\") pod \"e2e-mt-187946be-openshift-default-6555dfc579-pwrt9\" (UID: \"ed53e0ee-2e8e-4694-8290-2754d392cbe8\") " pod="openshift-ingress/e2e-mt-187946be-openshift-default-6555dfc579-pwrt9" Jul 16 16:58:48.339966 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:58:48.339753 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed53e0ee-2e8e-4694-8290-2754d392cbe8-istio-token\") pod \"e2e-mt-187946be-openshift-default-6555dfc579-pwrt9\" (UID: \"ed53e0ee-2e8e-4694-8290-2754d392cbe8\") " pod="openshift-ingress/e2e-mt-187946be-openshift-default-6555dfc579-pwrt9" Jul 16 16:58:48.339966 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:58:48.339775 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed53e0ee-2e8e-4694-8290-2754d392cbe8-istio-data\") pod \"e2e-mt-187946be-openshift-default-6555dfc579-pwrt9\" (UID: \"ed53e0ee-2e8e-4694-8290-2754d392cbe8\") " pod="openshift-ingress/e2e-mt-187946be-openshift-default-6555dfc579-pwrt9" Jul 16 16:58:48.339966 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:58:48.339804 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed53e0ee-2e8e-4694-8290-2754d392cbe8-workload-socket\") pod \"e2e-mt-187946be-openshift-default-6555dfc579-pwrt9\" (UID: \"ed53e0ee-2e8e-4694-8290-2754d392cbe8\") " pod="openshift-ingress/e2e-mt-187946be-openshift-default-6555dfc579-pwrt9" Jul 16 16:58:48.339966 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:58:48.339830 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed53e0ee-2e8e-4694-8290-2754d392cbe8-credential-socket\") pod \"e2e-mt-187946be-openshift-default-6555dfc579-pwrt9\" (UID: \"ed53e0ee-2e8e-4694-8290-2754d392cbe8\") " pod="openshift-ingress/e2e-mt-187946be-openshift-default-6555dfc579-pwrt9" Jul 16 16:58:48.339966 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:58:48.339864 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed53e0ee-2e8e-4694-8290-2754d392cbe8-istio-envoy\") pod \"e2e-mt-187946be-openshift-default-6555dfc579-pwrt9\" (UID: \"ed53e0ee-2e8e-4694-8290-2754d392cbe8\") " pod="openshift-ingress/e2e-mt-187946be-openshift-default-6555dfc579-pwrt9" Jul 16 16:58:48.339966 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:58:48.339890 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed53e0ee-2e8e-4694-8290-2754d392cbe8-workload-certs\") pod \"e2e-mt-187946be-openshift-default-6555dfc579-pwrt9\" (UID: \"ed53e0ee-2e8e-4694-8290-2754d392cbe8\") " pod="openshift-ingress/e2e-mt-187946be-openshift-default-6555dfc579-pwrt9" Jul 16 16:58:48.340364 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:58:48.340276 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed53e0ee-2e8e-4694-8290-2754d392cbe8-workload-certs\") pod \"e2e-mt-187946be-openshift-default-6555dfc579-pwrt9\" (UID: \"ed53e0ee-2e8e-4694-8290-2754d392cbe8\") " pod="openshift-ingress/e2e-mt-187946be-openshift-default-6555dfc579-pwrt9" Jul 16 16:58:48.340364 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:58:48.340336 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed53e0ee-2e8e-4694-8290-2754d392cbe8-credential-socket\") pod \"e2e-mt-187946be-openshift-default-6555dfc579-pwrt9\" (UID: \"ed53e0ee-2e8e-4694-8290-2754d392cbe8\") " pod="openshift-ingress/e2e-mt-187946be-openshift-default-6555dfc579-pwrt9" Jul 16 16:58:48.340491 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:58:48.340424 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed53e0ee-2e8e-4694-8290-2754d392cbe8-istio-data\") pod \"e2e-mt-187946be-openshift-default-6555dfc579-pwrt9\" (UID: \"ed53e0ee-2e8e-4694-8290-2754d392cbe8\") " pod="openshift-ingress/e2e-mt-187946be-openshift-default-6555dfc579-pwrt9" Jul 16 16:58:48.340566 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:58:48.340546 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed53e0ee-2e8e-4694-8290-2754d392cbe8-workload-socket\") pod \"e2e-mt-187946be-openshift-default-6555dfc579-pwrt9\" (UID: \"ed53e0ee-2e8e-4694-8290-2754d392cbe8\") " pod="openshift-ingress/e2e-mt-187946be-openshift-default-6555dfc579-pwrt9" Jul 16 16:58:48.340605 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:58:48.340573 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed53e0ee-2e8e-4694-8290-2754d392cbe8-istiod-ca-cert\") pod \"e2e-mt-187946be-openshift-default-6555dfc579-pwrt9\" (UID: \"ed53e0ee-2e8e-4694-8290-2754d392cbe8\") " pod="openshift-ingress/e2e-mt-187946be-openshift-default-6555dfc579-pwrt9" Jul 16 16:58:48.342126 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:58:48.342105 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed53e0ee-2e8e-4694-8290-2754d392cbe8-istio-envoy\") pod \"e2e-mt-187946be-openshift-default-6555dfc579-pwrt9\" (UID: \"ed53e0ee-2e8e-4694-8290-2754d392cbe8\") " pod="openshift-ingress/e2e-mt-187946be-openshift-default-6555dfc579-pwrt9" Jul 16 16:58:48.342259 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:58:48.342243 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed53e0ee-2e8e-4694-8290-2754d392cbe8-istio-podinfo\") pod \"e2e-mt-187946be-openshift-default-6555dfc579-pwrt9\" (UID: \"ed53e0ee-2e8e-4694-8290-2754d392cbe8\") " pod="openshift-ingress/e2e-mt-187946be-openshift-default-6555dfc579-pwrt9" Jul 16 16:58:48.349210 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:58:48.349185 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed53e0ee-2e8e-4694-8290-2754d392cbe8-istio-token\") pod \"e2e-mt-187946be-openshift-default-6555dfc579-pwrt9\" (UID: \"ed53e0ee-2e8e-4694-8290-2754d392cbe8\") " pod="openshift-ingress/e2e-mt-187946be-openshift-default-6555dfc579-pwrt9" Jul 16 16:58:48.349509 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:58:48.349484 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7k946\" (UniqueName: \"kubernetes.io/projected/ed53e0ee-2e8e-4694-8290-2754d392cbe8-kube-api-access-7k946\") pod \"e2e-mt-187946be-openshift-default-6555dfc579-pwrt9\" (UID: \"ed53e0ee-2e8e-4694-8290-2754d392cbe8\") " pod="openshift-ingress/e2e-mt-187946be-openshift-default-6555dfc579-pwrt9" Jul 16 16:58:48.446990 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:58:48.446893 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-187946be-openshift-default-6555dfc579-pwrt9" Jul 16 16:58:48.600829 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:58:48.600797 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-187946be-openshift-default-6555dfc579-pwrt9"] Jul 16 16:58:48.602930 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:58:48.602887 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded53e0ee_2e8e_4694_8290_2754d392cbe8.slice/crio-bc706bbc59008fe9adc67581380ea490898c46a24ac7ff3d2d1b684fb276d1b6 WatchSource:0}: Error finding container bc706bbc59008fe9adc67581380ea490898c46a24ac7ff3d2d1b684fb276d1b6: Status 404 returned error can't find the container with id bc706bbc59008fe9adc67581380ea490898c46a24ac7ff3d2d1b684fb276d1b6 Jul 16 16:58:48.605000 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:58:48.604971 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 16 16:58:48.605063 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:58:48.605033 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 16 16:58:48.605100 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:58:48.605063 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 16 16:58:49.085416 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:58:49.085369 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-187946be-openshift-default-6555dfc579-pwrt9" event={"ID":"ed53e0ee-2e8e-4694-8290-2754d392cbe8","Type":"ContainerStarted","Data":"41e8016196dd7f2587663ca798d1057da42fa64ef69a8466152f327576e81413"} Jul 16 16:58:49.085595 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:58:49.085438 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-187946be-openshift-default-6555dfc579-pwrt9" event={"ID":"ed53e0ee-2e8e-4694-8290-2754d392cbe8","Type":"ContainerStarted","Data":"bc706bbc59008fe9adc67581380ea490898c46a24ac7ff3d2d1b684fb276d1b6"} Jul 16 16:58:49.106718 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:58:49.106658 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-187946be-openshift-default-6555dfc579-pwrt9" podStartSLOduration=1.106637546 podStartE2EDuration="1.106637546s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 16:58:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 16:58:49.104524391 +0000 UTC m=+1987.608582245" watchObservedRunningTime="2026-07-16 16:58:49.106637546 +0000 UTC m=+1987.610695402" Jul 16 16:58:49.447623 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:58:49.447527 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-187946be-openshift-default-6555dfc579-pwrt9" Jul 16 16:58:49.452513 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:58:49.452484 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-187946be-openshift-default-6555dfc579-pwrt9" Jul 16 16:58:50.090927 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:58:50.090895 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-187946be-openshift-default-6555dfc579-pwrt9" Jul 16 16:58:50.091873 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:58:50.091855 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-187946be-openshift-default-6555dfc579-pwrt9" Jul 16 16:59:05.542184 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:59:05.542146 2577 patch_prober.go:28] interesting pod/e2e-mt-187946be-openshift-default-6555dfc579-pwrt9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": read tcp 10.134.0.2:38924->10.134.0.48:15021: read: connection reset by peer" start-of-body= Jul 16 16:59:05.542594 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:59:05.542233 2577 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-187946be-openshift-default-6555dfc579-pwrt9" podUID="ed53e0ee-2e8e-4694-8290-2754d392cbe8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": read tcp 10.134.0.2:38924->10.134.0.48:15021: read: connection reset by peer" Jul 16 16:59:06.197082 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:59:06.197047 2577 generic.go:356] "Generic (PLEG): container finished" podID="ed53e0ee-2e8e-4694-8290-2754d392cbe8" containerID="41e8016196dd7f2587663ca798d1057da42fa64ef69a8466152f327576e81413" exitCode=0 Jul 16 16:59:06.197252 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:59:06.197094 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-187946be-openshift-default-6555dfc579-pwrt9" event={"ID":"ed53e0ee-2e8e-4694-8290-2754d392cbe8","Type":"ContainerDied","Data":"41e8016196dd7f2587663ca798d1057da42fa64ef69a8466152f327576e81413"} Jul 16 16:59:06.197529 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:59:06.197514 2577 scope.go:117] "RemoveContainer" containerID="41e8016196dd7f2587663ca798d1057da42fa64ef69a8466152f327576e81413" Jul 16 16:59:06.198626 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:59:06.198594 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 16 16:59:06.198699 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:59:06.198681 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 16 16:59:06.198738 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:59:06.198728 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 16 16:59:07.204220 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:59:07.204185 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-187946be-openshift-default-6555dfc579-pwrt9" event={"ID":"ed53e0ee-2e8e-4694-8290-2754d392cbe8","Type":"ContainerStarted","Data":"29e31fe6f4a1288d911cb6c4aa3a250d23363abd5bac8e370c9f28af8a4a77c8"} Jul 16 16:59:07.447898 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:59:07.447855 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-187946be-openshift-default-6555dfc579-pwrt9" Jul 16 16:59:07.448127 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:59:07.448000 2577 patch_prober.go:28] interesting pod/e2e-mt-187946be-openshift-default-6555dfc579-pwrt9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Jul 16 16:59:07.448127 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:59:07.448091 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-187946be-openshift-default-6555dfc579-pwrt9" podUID="ed53e0ee-2e8e-4694-8290-2754d392cbe8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Jul 16 16:59:08.448193 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:59:08.448137 2577 patch_prober.go:28] interesting pod/e2e-mt-187946be-openshift-default-6555dfc579-pwrt9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Jul 16 16:59:08.448737 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:59:08.448232 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-187946be-openshift-default-6555dfc579-pwrt9" podUID="ed53e0ee-2e8e-4694-8290-2754d392cbe8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Jul 16 16:59:09.447855 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:59:09.447812 2577 patch_prober.go:28] interesting pod/e2e-mt-187946be-openshift-default-6555dfc579-pwrt9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Jul 16 16:59:09.448163 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:59:09.448127 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-187946be-openshift-default-6555dfc579-pwrt9" podUID="ed53e0ee-2e8e-4694-8290-2754d392cbe8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Jul 16 16:59:10.447830 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:59:10.447756 2577 patch_prober.go:28] interesting pod/e2e-mt-187946be-openshift-default-6555dfc579-pwrt9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Jul 16 16:59:10.448350 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:59:10.447888 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-187946be-openshift-default-6555dfc579-pwrt9" podUID="ed53e0ee-2e8e-4694-8290-2754d392cbe8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Jul 16 16:59:11.450660 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:59:11.450618 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-187946be-openshift-default-6555dfc579-pwrt9" Jul 16 16:59:11.451085 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:59:11.451003 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-187946be-openshift-default-6555dfc579-pwrt9" Jul 16 16:59:11.451749 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:59:11.451727 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-187946be-openshift-default-6555dfc579-pwrt9" Jul 16 16:59:39.343933 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:59:39.343898 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-187946be-kphw9"] Jul 16 16:59:39.347599 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:59:39.347580 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-187946be-kphw9" Jul 16 16:59:39.350247 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:59:39.350218 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 16 16:59:39.350371 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:59:39.350260 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dssfh\"" Jul 16 16:59:39.361806 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:59:39.357411 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-187946be-kphw9"] Jul 16 16:59:39.463676 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:59:39.463609 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/78781c24-f2d6-4408-ac09-1522f8a771a8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-187946be-kphw9\" (UID: \"78781c24-f2d6-4408-ac09-1522f8a771a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-187946be-kphw9" Jul 16 16:59:39.564801 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:59:39.564758 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/78781c24-f2d6-4408-ac09-1522f8a771a8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-187946be-kphw9\" (UID: \"78781c24-f2d6-4408-ac09-1522f8a771a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-187946be-kphw9" Jul 16 16:59:39.565446 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:59:39.565425 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/78781c24-f2d6-4408-ac09-1522f8a771a8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-187946be-kphw9\" (UID: \"78781c24-f2d6-4408-ac09-1522f8a771a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-187946be-kphw9" Jul 16 16:59:39.662524 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:59:39.662436 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-187946be-kphw9" Jul 16 16:59:39.800472 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:59:39.800446 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-187946be-kphw9"] Jul 16 16:59:39.802962 ip-10-0-142-101 kubenswrapper[2577]: W0716 16:59:39.802923 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78781c24_f2d6_4408_ac09_1522f8a771a8.slice/crio-1bd5886c3cf90df92c32ee9a4b9e6d326f19821fa08d2a91038a853e517f918f WatchSource:0}: Error finding container 1bd5886c3cf90df92c32ee9a4b9e6d326f19821fa08d2a91038a853e517f918f: Status 404 returned error can't find the container with id 1bd5886c3cf90df92c32ee9a4b9e6d326f19821fa08d2a91038a853e517f918f Jul 16 16:59:40.383474 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:59:40.383433 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-187946be-kphw9" event={"ID":"78781c24-f2d6-4408-ac09-1522f8a771a8","Type":"ContainerStarted","Data":"9d82f140a6c9d4adb09185a80986b895a10a5dcd0cc55ee3101ec8c6fdedc563"} Jul 16 16:59:40.383896 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:59:40.383492 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-187946be-kphw9" event={"ID":"78781c24-f2d6-4408-ac09-1522f8a771a8","Type":"ContainerStarted","Data":"1bd5886c3cf90df92c32ee9a4b9e6d326f19821fa08d2a91038a853e517f918f"} Jul 16 16:59:40.402844 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:59:40.402787 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-187946be-kphw9" podStartSLOduration=1.402734925 podStartE2EDuration="1.402734925s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 16:59:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 16:59:40.401765044 +0000 UTC m=+2038.905822896" watchObservedRunningTime="2026-07-16 16:59:40.402734925 +0000 UTC m=+2038.906792776" Jul 16 16:59:41.390053 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:59:41.390018 2577 generic.go:356] "Generic (PLEG): container finished" podID="78781c24-f2d6-4408-ac09-1522f8a771a8" containerID="9d82f140a6c9d4adb09185a80986b895a10a5dcd0cc55ee3101ec8c6fdedc563" exitCode=0 Jul 16 16:59:41.390496 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:59:41.390110 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-187946be-kphw9" event={"ID":"78781c24-f2d6-4408-ac09-1522f8a771a8","Type":"ContainerDied","Data":"9d82f140a6c9d4adb09185a80986b895a10a5dcd0cc55ee3101ec8c6fdedc563"} Jul 16 16:59:42.190577 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:59:42.190543 2577 scope.go:117] "RemoveContainer" containerID="797a0cdb57d9f3d1d06e24bef5eb8b44ec3dbca8ff7b4f56a10db88a271ea4e3" Jul 16 16:59:42.536857 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:59:42.536698 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-187946be-kphw9" Jul 16 16:59:42.695748 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:59:42.695709 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/78781c24-f2d6-4408-ac09-1522f8a771a8-maas-api-service-ca\") pod \"78781c24-f2d6-4408-ac09-1522f8a771a8\" (UID: \"78781c24-f2d6-4408-ac09-1522f8a771a8\") " Jul 16 16:59:42.696097 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:59:42.696075 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78781c24-f2d6-4408-ac09-1522f8a771a8-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "78781c24-f2d6-4408-ac09-1522f8a771a8" (UID: "78781c24-f2d6-4408-ac09-1522f8a771a8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 16:59:42.797217 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:59:42.797127 2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/78781c24-f2d6-4408-ac09-1522f8a771a8-maas-api-service-ca\") on node \"ip-10-0-142-101.ec2.internal\" DevicePath \"\"" Jul 16 16:59:43.401159 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:59:43.401134 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-187946be-kphw9" Jul 16 16:59:43.401339 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:59:43.401174 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-187946be-kphw9" event={"ID":"78781c24-f2d6-4408-ac09-1522f8a771a8","Type":"ContainerDied","Data":"1bd5886c3cf90df92c32ee9a4b9e6d326f19821fa08d2a91038a853e517f918f"} Jul 16 16:59:43.401339 ip-10-0-142-101 kubenswrapper[2577]: I0716 16:59:43.401221 2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1bd5886c3cf90df92c32ee9a4b9e6d326f19821fa08d2a91038a853e517f918f" Jul 16 17:00:00.140315 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:00:00.140273 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29737020-xrbqn"] Jul 16 17:00:00.140745 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:00:00.140731 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="78781c24-f2d6-4408-ac09-1522f8a771a8" containerName="revoke-keys" Jul 16 17:00:00.140790 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:00:00.140746 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="78781c24-f2d6-4408-ac09-1522f8a771a8" containerName="revoke-keys" Jul 16 17:00:00.140832 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:00:00.140824 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="78781c24-f2d6-4408-ac09-1522f8a771a8" containerName="revoke-keys" Jul 16 17:00:00.144038 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:00:00.144019 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29737020-xrbqn" Jul 16 17:00:00.147093 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:00:00.147071 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dssfh\"" Jul 16 17:00:00.154628 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:00:00.154603 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29737020-xrbqn"] Jul 16 17:00:00.155285 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:00:00.155264 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29737020-xrbqn" Jul 16 17:00:00.291310 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:00:00.291282 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29737020-xrbqn"] Jul 16 17:00:00.293175 ip-10-0-142-101 kubenswrapper[2577]: W0716 17:00:00.293149 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c521842_ea00_41ff_91a6_3cb7b12f125c.slice/crio-a77c4c717add487e94bb31ceabd2e0f53ee57e37e5e9dabaa4b594b8e072fba4 WatchSource:0}: Error finding container a77c4c717add487e94bb31ceabd2e0f53ee57e37e5e9dabaa4b594b8e072fba4: Status 404 returned error can't find the container with id a77c4c717add487e94bb31ceabd2e0f53ee57e37e5e9dabaa4b594b8e072fba4 Jul 16 17:00:00.491065 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:00:00.491026 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29737020-xrbqn" event={"ID":"3c521842-ea00-41ff-91a6-3cb7b12f125c","Type":"ContainerStarted","Data":"a77c4c717add487e94bb31ceabd2e0f53ee57e37e5e9dabaa4b594b8e072fba4"} Jul 16 17:00:01.497525 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:00:01.497437 2577 generic.go:356] "Generic (PLEG): container finished" podID="3c521842-ea00-41ff-91a6-3cb7b12f125c" containerID="89f93e245ed5c032012527f3f53812557c4e54a8ae66b7dda1b7f22652c94366" exitCode=0 Jul 16 17:00:01.497893 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:00:01.497527 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29737020-xrbqn" event={"ID":"3c521842-ea00-41ff-91a6-3cb7b12f125c","Type":"ContainerDied","Data":"89f93e245ed5c032012527f3f53812557c4e54a8ae66b7dda1b7f22652c94366"} Jul 16 17:00:02.645795 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:00:02.645767 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29737020-xrbqn" Jul 16 17:00:03.509281 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:00:03.509201 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29737020-xrbqn" Jul 16 17:00:03.509471 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:00:03.509200 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29737020-xrbqn" event={"ID":"3c521842-ea00-41ff-91a6-3cb7b12f125c","Type":"ContainerDied","Data":"a77c4c717add487e94bb31ceabd2e0f53ee57e37e5e9dabaa4b594b8e072fba4"} Jul 16 17:00:03.509471 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:00:03.509344 2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a77c4c717add487e94bb31ceabd2e0f53ee57e37e5e9dabaa4b594b8e072fba4" Jul 16 17:00:42.099072 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:00:42.099043 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9bjst_b20b154f-1a4c-4b25-86db-aebe2d4ffc08/ovn-acl-logging/0.log" Jul 16 17:00:42.104112 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:00:42.104090 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9bjst_b20b154f-1a4c-4b25-86db-aebe2d4ffc08/ovn-acl-logging/0.log" Jul 16 17:00:46.833987 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:00:46.833955 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-187946be-openshift-default-6555dfc579-pwrt9"] Jul 16 17:00:46.834397 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:00:46.834186 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-187946be-openshift-default-6555dfc579-pwrt9" podUID="ed53e0ee-2e8e-4694-8290-2754d392cbe8" containerName="istio-proxy" containerID="cri-o://29e31fe6f4a1288d911cb6c4aa3a250d23363abd5bac8e370c9f28af8a4a77c8" gracePeriod=30 Jul 16 17:00:52.089253 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:00:52.089226 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-187946be-openshift-default-6555dfc579-pwrt9" Jul 16 17:00:52.169855 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:00:52.169771 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed53e0ee-2e8e-4694-8290-2754d392cbe8-credential-socket\") pod \"ed53e0ee-2e8e-4694-8290-2754d392cbe8\" (UID: \"ed53e0ee-2e8e-4694-8290-2754d392cbe8\") " Jul 16 17:00:52.169855 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:00:52.169822 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed53e0ee-2e8e-4694-8290-2754d392cbe8-istiod-ca-cert\") pod \"ed53e0ee-2e8e-4694-8290-2754d392cbe8\" (UID: \"ed53e0ee-2e8e-4694-8290-2754d392cbe8\") " Jul 16 17:00:52.170110 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:00:52.169863 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7k946\" (UniqueName: \"kubernetes.io/projected/ed53e0ee-2e8e-4694-8290-2754d392cbe8-kube-api-access-7k946\") pod \"ed53e0ee-2e8e-4694-8290-2754d392cbe8\" (UID: \"ed53e0ee-2e8e-4694-8290-2754d392cbe8\") " Jul 16 17:00:52.170110 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:00:52.169879 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed53e0ee-2e8e-4694-8290-2754d392cbe8-workload-socket\") pod \"ed53e0ee-2e8e-4694-8290-2754d392cbe8\" (UID: \"ed53e0ee-2e8e-4694-8290-2754d392cbe8\") " Jul 16 17:00:52.170110 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:00:52.169933 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed53e0ee-2e8e-4694-8290-2754d392cbe8-istio-podinfo\") pod \"ed53e0ee-2e8e-4694-8290-2754d392cbe8\" (UID: \"ed53e0ee-2e8e-4694-8290-2754d392cbe8\") " Jul 16 17:00:52.170110 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:00:52.169985 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed53e0ee-2e8e-4694-8290-2754d392cbe8-workload-certs\") pod \"ed53e0ee-2e8e-4694-8290-2754d392cbe8\" (UID: \"ed53e0ee-2e8e-4694-8290-2754d392cbe8\") " Jul 16 17:00:52.170110 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:00:52.170013 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed53e0ee-2e8e-4694-8290-2754d392cbe8-istio-token\") pod \"ed53e0ee-2e8e-4694-8290-2754d392cbe8\" (UID: \"ed53e0ee-2e8e-4694-8290-2754d392cbe8\") " Jul 16 17:00:52.170110 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:00:52.170048 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed53e0ee-2e8e-4694-8290-2754d392cbe8-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "ed53e0ee-2e8e-4694-8290-2754d392cbe8" (UID: "ed53e0ee-2e8e-4694-8290-2754d392cbe8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 17:00:52.170110 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:00:52.170063 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed53e0ee-2e8e-4694-8290-2754d392cbe8-istio-envoy\") pod \"ed53e0ee-2e8e-4694-8290-2754d392cbe8\" (UID: \"ed53e0ee-2e8e-4694-8290-2754d392cbe8\") " Jul 16 17:00:52.170527 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:00:52.170148 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed53e0ee-2e8e-4694-8290-2754d392cbe8-istio-data\") pod \"ed53e0ee-2e8e-4694-8290-2754d392cbe8\" (UID: \"ed53e0ee-2e8e-4694-8290-2754d392cbe8\") " Jul 16 17:00:52.170527 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:00:52.170220 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed53e0ee-2e8e-4694-8290-2754d392cbe8-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "ed53e0ee-2e8e-4694-8290-2754d392cbe8" (UID: "ed53e0ee-2e8e-4694-8290-2754d392cbe8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 17:00:52.170527 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:00:52.170255 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed53e0ee-2e8e-4694-8290-2754d392cbe8-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "ed53e0ee-2e8e-4694-8290-2754d392cbe8" (UID: "ed53e0ee-2e8e-4694-8290-2754d392cbe8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 17:00:52.170527 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:00:52.170276 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed53e0ee-2e8e-4694-8290-2754d392cbe8-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "ed53e0ee-2e8e-4694-8290-2754d392cbe8" (UID: "ed53e0ee-2e8e-4694-8290-2754d392cbe8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 17:00:52.170748 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:00:52.170581 2577 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed53e0ee-2e8e-4694-8290-2754d392cbe8-workload-certs\") on node \"ip-10-0-142-101.ec2.internal\" DevicePath \"\"" Jul 16 17:00:52.170748 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:00:52.170603 2577 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed53e0ee-2e8e-4694-8290-2754d392cbe8-credential-socket\") on node \"ip-10-0-142-101.ec2.internal\" DevicePath \"\"" Jul 16 17:00:52.170748 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:00:52.170617 2577 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed53e0ee-2e8e-4694-8290-2754d392cbe8-istiod-ca-cert\") on node \"ip-10-0-142-101.ec2.internal\" DevicePath \"\"" Jul 16 17:00:52.170748 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:00:52.170630 2577 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed53e0ee-2e8e-4694-8290-2754d392cbe8-workload-socket\") on node \"ip-10-0-142-101.ec2.internal\" DevicePath \"\"" Jul 16 17:00:52.171283 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:00:52.171255 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed53e0ee-2e8e-4694-8290-2754d392cbe8-istio-data" (OuterVolumeSpecName: "istio-data") pod "ed53e0ee-2e8e-4694-8290-2754d392cbe8" (UID: "ed53e0ee-2e8e-4694-8290-2754d392cbe8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 17:00:52.172365 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:00:52.172332 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed53e0ee-2e8e-4694-8290-2754d392cbe8-kube-api-access-7k946" (OuterVolumeSpecName: "kube-api-access-7k946") pod "ed53e0ee-2e8e-4694-8290-2754d392cbe8" (UID: "ed53e0ee-2e8e-4694-8290-2754d392cbe8"). InnerVolumeSpecName "kube-api-access-7k946". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 17:00:52.172365 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:00:52.172339 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed53e0ee-2e8e-4694-8290-2754d392cbe8-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "ed53e0ee-2e8e-4694-8290-2754d392cbe8" (UID: "ed53e0ee-2e8e-4694-8290-2754d392cbe8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 17:00:52.172522 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:00:52.172408 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ed53e0ee-2e8e-4694-8290-2754d392cbe8-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "ed53e0ee-2e8e-4694-8290-2754d392cbe8" (UID: "ed53e0ee-2e8e-4694-8290-2754d392cbe8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 16 17:00:52.172522 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:00:52.172510 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed53e0ee-2e8e-4694-8290-2754d392cbe8-istio-token" (OuterVolumeSpecName: "istio-token") pod "ed53e0ee-2e8e-4694-8290-2754d392cbe8" (UID: "ed53e0ee-2e8e-4694-8290-2754d392cbe8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 17:00:52.272140 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:00:52.272103 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7k946\" (UniqueName: \"kubernetes.io/projected/ed53e0ee-2e8e-4694-8290-2754d392cbe8-kube-api-access-7k946\") on node \"ip-10-0-142-101.ec2.internal\" DevicePath \"\"" Jul 16 17:00:52.272140 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:00:52.272135 2577 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed53e0ee-2e8e-4694-8290-2754d392cbe8-istio-podinfo\") on node \"ip-10-0-142-101.ec2.internal\" DevicePath \"\"" Jul 16 17:00:52.272140 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:00:52.272145 2577 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed53e0ee-2e8e-4694-8290-2754d392cbe8-istio-token\") on node \"ip-10-0-142-101.ec2.internal\" DevicePath \"\"" Jul 16 17:00:52.272409 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:00:52.272153 2577 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed53e0ee-2e8e-4694-8290-2754d392cbe8-istio-envoy\") on node \"ip-10-0-142-101.ec2.internal\" DevicePath \"\"" Jul 16 17:00:52.272409 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:00:52.272162 2577 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed53e0ee-2e8e-4694-8290-2754d392cbe8-istio-data\") on node \"ip-10-0-142-101.ec2.internal\" DevicePath \"\"" Jul 16 17:00:52.773469 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:00:52.773434 2577 generic.go:356] "Generic (PLEG): container finished" podID="ed53e0ee-2e8e-4694-8290-2754d392cbe8" containerID="29e31fe6f4a1288d911cb6c4aa3a250d23363abd5bac8e370c9f28af8a4a77c8" exitCode=0 Jul 16 17:00:52.773469 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:00:52.773472 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-187946be-openshift-default-6555dfc579-pwrt9" event={"ID":"ed53e0ee-2e8e-4694-8290-2754d392cbe8","Type":"ContainerDied","Data":"29e31fe6f4a1288d911cb6c4aa3a250d23363abd5bac8e370c9f28af8a4a77c8"} Jul 16 17:00:52.773680 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:00:52.773486 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-187946be-openshift-default-6555dfc579-pwrt9" Jul 16 17:00:52.773680 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:00:52.773510 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-187946be-openshift-default-6555dfc579-pwrt9" event={"ID":"ed53e0ee-2e8e-4694-8290-2754d392cbe8","Type":"ContainerDied","Data":"bc706bbc59008fe9adc67581380ea490898c46a24ac7ff3d2d1b684fb276d1b6"} Jul 16 17:00:52.773680 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:00:52.773533 2577 scope.go:117] "RemoveContainer" containerID="29e31fe6f4a1288d911cb6c4aa3a250d23363abd5bac8e370c9f28af8a4a77c8" Jul 16 17:00:52.783209 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:00:52.783191 2577 scope.go:117] "RemoveContainer" containerID="41e8016196dd7f2587663ca798d1057da42fa64ef69a8466152f327576e81413" Jul 16 17:00:52.791456 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:00:52.791435 2577 scope.go:117] "RemoveContainer" containerID="29e31fe6f4a1288d911cb6c4aa3a250d23363abd5bac8e370c9f28af8a4a77c8" Jul 16 17:00:52.791740 ip-10-0-142-101 kubenswrapper[2577]: E0716 17:00:52.791719 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"29e31fe6f4a1288d911cb6c4aa3a250d23363abd5bac8e370c9f28af8a4a77c8\": container with ID starting with 29e31fe6f4a1288d911cb6c4aa3a250d23363abd5bac8e370c9f28af8a4a77c8 not found: ID does not exist" containerID="29e31fe6f4a1288d911cb6c4aa3a250d23363abd5bac8e370c9f28af8a4a77c8" Jul 16 17:00:52.791793 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:00:52.791751 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"29e31fe6f4a1288d911cb6c4aa3a250d23363abd5bac8e370c9f28af8a4a77c8"} err="failed to get container status \"29e31fe6f4a1288d911cb6c4aa3a250d23363abd5bac8e370c9f28af8a4a77c8\": rpc error: code = NotFound desc = could not find container \"29e31fe6f4a1288d911cb6c4aa3a250d23363abd5bac8e370c9f28af8a4a77c8\": container with ID starting with 29e31fe6f4a1288d911cb6c4aa3a250d23363abd5bac8e370c9f28af8a4a77c8 not found: ID does not exist" Jul 16 17:00:52.791793 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:00:52.791771 2577 scope.go:117] "RemoveContainer" containerID="41e8016196dd7f2587663ca798d1057da42fa64ef69a8466152f327576e81413" Jul 16 17:00:52.791995 ip-10-0-142-101 kubenswrapper[2577]: E0716 17:00:52.791976 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"41e8016196dd7f2587663ca798d1057da42fa64ef69a8466152f327576e81413\": container with ID starting with 41e8016196dd7f2587663ca798d1057da42fa64ef69a8466152f327576e81413 not found: ID does not exist" containerID="41e8016196dd7f2587663ca798d1057da42fa64ef69a8466152f327576e81413" Jul 16 17:00:52.792042 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:00:52.792001 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41e8016196dd7f2587663ca798d1057da42fa64ef69a8466152f327576e81413"} err="failed to get container status \"41e8016196dd7f2587663ca798d1057da42fa64ef69a8466152f327576e81413\": rpc error: code = NotFound desc = could not find container \"41e8016196dd7f2587663ca798d1057da42fa64ef69a8466152f327576e81413\": container with ID starting with 41e8016196dd7f2587663ca798d1057da42fa64ef69a8466152f327576e81413 not found: ID does not exist" Jul 16 17:00:52.801745 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:00:52.801712 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-187946be-openshift-default-6555dfc579-pwrt9"] Jul 16 17:00:52.820036 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:00:52.820001 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-187946be-openshift-default-6555dfc579-pwrt9"] Jul 16 17:00:54.017251 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:00:54.017216 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ed53e0ee-2e8e-4694-8290-2754d392cbe8" path="/var/lib/kubelet/pods/ed53e0ee-2e8e-4694-8290-2754d392cbe8/volumes" Jul 16 17:01:42.218365 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:01:42.218283 2577 scope.go:117] "RemoveContainer" containerID="2a75387222194b18cda407593cea741153e28f49814b64821b0608e70bff2cbd" Jul 16 17:01:42.686031 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:01:42.685995 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-fa0ccd-openshift-default-5d77f5f75f-t99j9"] Jul 16 17:01:42.686460 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:01:42.686446 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ed53e0ee-2e8e-4694-8290-2754d392cbe8" containerName="istio-proxy" Jul 16 17:01:42.686519 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:01:42.686462 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="ed53e0ee-2e8e-4694-8290-2754d392cbe8" containerName="istio-proxy" Jul 16 17:01:42.686519 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:01:42.686485 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3c521842-ea00-41ff-91a6-3cb7b12f125c" containerName="cleanup" Jul 16 17:01:42.686519 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:01:42.686491 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="3c521842-ea00-41ff-91a6-3cb7b12f125c" containerName="cleanup" Jul 16 17:01:42.686623 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:01:42.686570 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="3c521842-ea00-41ff-91a6-3cb7b12f125c" containerName="cleanup" Jul 16 17:01:42.686623 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:01:42.686579 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="ed53e0ee-2e8e-4694-8290-2754d392cbe8" containerName="istio-proxy" Jul 16 17:01:42.691183 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:01:42.691163 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-fa0ccd-openshift-default-5d77f5f75f-t99j9" Jul 16 17:01:42.693851 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:01:42.693818 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-fa0ccd-openshift-default-dockercfg-qgtj6\"" Jul 16 17:01:42.702590 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:01:42.702563 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-fa0ccd-openshift-default-5d77f5f75f-t99j9"] Jul 16 17:01:42.861119 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:01:42.861075 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ade7b933-de22-4ac2-8ef8-a9838f5e9787-istiod-ca-cert\") pod \"e2e-inf-a-fa0ccd-openshift-default-5d77f5f75f-t99j9\" (UID: \"ade7b933-de22-4ac2-8ef8-a9838f5e9787\") " pod="openshift-ingress/e2e-inf-a-fa0ccd-openshift-default-5d77f5f75f-t99j9" Jul 16 17:01:42.861334 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:01:42.861130 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ade7b933-de22-4ac2-8ef8-a9838f5e9787-credential-socket\") pod \"e2e-inf-a-fa0ccd-openshift-default-5d77f5f75f-t99j9\" (UID: \"ade7b933-de22-4ac2-8ef8-a9838f5e9787\") " pod="openshift-ingress/e2e-inf-a-fa0ccd-openshift-default-5d77f5f75f-t99j9" Jul 16 17:01:42.861334 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:01:42.861204 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ade7b933-de22-4ac2-8ef8-a9838f5e9787-istio-envoy\") pod \"e2e-inf-a-fa0ccd-openshift-default-5d77f5f75f-t99j9\" (UID: \"ade7b933-de22-4ac2-8ef8-a9838f5e9787\") " pod="openshift-ingress/e2e-inf-a-fa0ccd-openshift-default-5d77f5f75f-t99j9" Jul 16 17:01:42.861334 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:01:42.861233 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ade7b933-de22-4ac2-8ef8-a9838f5e9787-istio-data\") pod \"e2e-inf-a-fa0ccd-openshift-default-5d77f5f75f-t99j9\" (UID: \"ade7b933-de22-4ac2-8ef8-a9838f5e9787\") " pod="openshift-ingress/e2e-inf-a-fa0ccd-openshift-default-5d77f5f75f-t99j9" Jul 16 17:01:42.861334 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:01:42.861263 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ade7b933-de22-4ac2-8ef8-a9838f5e9787-workload-socket\") pod \"e2e-inf-a-fa0ccd-openshift-default-5d77f5f75f-t99j9\" (UID: \"ade7b933-de22-4ac2-8ef8-a9838f5e9787\") " pod="openshift-ingress/e2e-inf-a-fa0ccd-openshift-default-5d77f5f75f-t99j9" Jul 16 17:01:42.861334 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:01:42.861312 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-swwt2\" (UniqueName: \"kubernetes.io/projected/ade7b933-de22-4ac2-8ef8-a9838f5e9787-kube-api-access-swwt2\") pod \"e2e-inf-a-fa0ccd-openshift-default-5d77f5f75f-t99j9\" (UID: \"ade7b933-de22-4ac2-8ef8-a9838f5e9787\") " pod="openshift-ingress/e2e-inf-a-fa0ccd-openshift-default-5d77f5f75f-t99j9" Jul 16 17:01:42.861638 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:01:42.861357 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ade7b933-de22-4ac2-8ef8-a9838f5e9787-istio-podinfo\") pod \"e2e-inf-a-fa0ccd-openshift-default-5d77f5f75f-t99j9\" (UID: \"ade7b933-de22-4ac2-8ef8-a9838f5e9787\") " pod="openshift-ingress/e2e-inf-a-fa0ccd-openshift-default-5d77f5f75f-t99j9" Jul 16 17:01:42.861638 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:01:42.861428 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ade7b933-de22-4ac2-8ef8-a9838f5e9787-workload-certs\") pod \"e2e-inf-a-fa0ccd-openshift-default-5d77f5f75f-t99j9\" (UID: \"ade7b933-de22-4ac2-8ef8-a9838f5e9787\") " pod="openshift-ingress/e2e-inf-a-fa0ccd-openshift-default-5d77f5f75f-t99j9" Jul 16 17:01:42.861638 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:01:42.861523 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ade7b933-de22-4ac2-8ef8-a9838f5e9787-istio-token\") pod \"e2e-inf-a-fa0ccd-openshift-default-5d77f5f75f-t99j9\" (UID: \"ade7b933-de22-4ac2-8ef8-a9838f5e9787\") " pod="openshift-ingress/e2e-inf-a-fa0ccd-openshift-default-5d77f5f75f-t99j9" Jul 16 17:01:42.963111 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:01:42.963011 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ade7b933-de22-4ac2-8ef8-a9838f5e9787-istiod-ca-cert\") pod \"e2e-inf-a-fa0ccd-openshift-default-5d77f5f75f-t99j9\" (UID: \"ade7b933-de22-4ac2-8ef8-a9838f5e9787\") " pod="openshift-ingress/e2e-inf-a-fa0ccd-openshift-default-5d77f5f75f-t99j9" Jul 16 17:01:42.963111 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:01:42.963064 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ade7b933-de22-4ac2-8ef8-a9838f5e9787-credential-socket\") pod \"e2e-inf-a-fa0ccd-openshift-default-5d77f5f75f-t99j9\" (UID: \"ade7b933-de22-4ac2-8ef8-a9838f5e9787\") " pod="openshift-ingress/e2e-inf-a-fa0ccd-openshift-default-5d77f5f75f-t99j9" Jul 16 17:01:42.963111 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:01:42.963109 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ade7b933-de22-4ac2-8ef8-a9838f5e9787-istio-envoy\") pod \"e2e-inf-a-fa0ccd-openshift-default-5d77f5f75f-t99j9\" (UID: \"ade7b933-de22-4ac2-8ef8-a9838f5e9787\") " pod="openshift-ingress/e2e-inf-a-fa0ccd-openshift-default-5d77f5f75f-t99j9" Jul 16 17:01:42.963433 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:01:42.963124 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ade7b933-de22-4ac2-8ef8-a9838f5e9787-istio-data\") pod \"e2e-inf-a-fa0ccd-openshift-default-5d77f5f75f-t99j9\" (UID: \"ade7b933-de22-4ac2-8ef8-a9838f5e9787\") " pod="openshift-ingress/e2e-inf-a-fa0ccd-openshift-default-5d77f5f75f-t99j9" Jul 16 17:01:42.963433 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:01:42.963154 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ade7b933-de22-4ac2-8ef8-a9838f5e9787-workload-socket\") pod \"e2e-inf-a-fa0ccd-openshift-default-5d77f5f75f-t99j9\" (UID: \"ade7b933-de22-4ac2-8ef8-a9838f5e9787\") " pod="openshift-ingress/e2e-inf-a-fa0ccd-openshift-default-5d77f5f75f-t99j9" Jul 16 17:01:42.963433 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:01:42.963201 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-swwt2\" (UniqueName: \"kubernetes.io/projected/ade7b933-de22-4ac2-8ef8-a9838f5e9787-kube-api-access-swwt2\") pod \"e2e-inf-a-fa0ccd-openshift-default-5d77f5f75f-t99j9\" (UID: \"ade7b933-de22-4ac2-8ef8-a9838f5e9787\") " pod="openshift-ingress/e2e-inf-a-fa0ccd-openshift-default-5d77f5f75f-t99j9" Jul 16 17:01:42.963433 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:01:42.963234 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ade7b933-de22-4ac2-8ef8-a9838f5e9787-istio-podinfo\") pod \"e2e-inf-a-fa0ccd-openshift-default-5d77f5f75f-t99j9\" (UID: \"ade7b933-de22-4ac2-8ef8-a9838f5e9787\") " pod="openshift-ingress/e2e-inf-a-fa0ccd-openshift-default-5d77f5f75f-t99j9" Jul 16 17:01:42.963433 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:01:42.963259 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ade7b933-de22-4ac2-8ef8-a9838f5e9787-workload-certs\") pod \"e2e-inf-a-fa0ccd-openshift-default-5d77f5f75f-t99j9\" (UID: \"ade7b933-de22-4ac2-8ef8-a9838f5e9787\") " pod="openshift-ingress/e2e-inf-a-fa0ccd-openshift-default-5d77f5f75f-t99j9" Jul 16 17:01:42.963433 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:01:42.963350 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ade7b933-de22-4ac2-8ef8-a9838f5e9787-istio-token\") pod \"e2e-inf-a-fa0ccd-openshift-default-5d77f5f75f-t99j9\" (UID: \"ade7b933-de22-4ac2-8ef8-a9838f5e9787\") " pod="openshift-ingress/e2e-inf-a-fa0ccd-openshift-default-5d77f5f75f-t99j9" Jul 16 17:01:42.963768 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:01:42.963574 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ade7b933-de22-4ac2-8ef8-a9838f5e9787-credential-socket\") pod \"e2e-inf-a-fa0ccd-openshift-default-5d77f5f75f-t99j9\" (UID: \"ade7b933-de22-4ac2-8ef8-a9838f5e9787\") " pod="openshift-ingress/e2e-inf-a-fa0ccd-openshift-default-5d77f5f75f-t99j9" Jul 16 17:01:42.963768 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:01:42.963619 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ade7b933-de22-4ac2-8ef8-a9838f5e9787-workload-socket\") pod \"e2e-inf-a-fa0ccd-openshift-default-5d77f5f75f-t99j9\" (UID: \"ade7b933-de22-4ac2-8ef8-a9838f5e9787\") " pod="openshift-ingress/e2e-inf-a-fa0ccd-openshift-default-5d77f5f75f-t99j9" Jul 16 17:01:42.963768 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:01:42.963650 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ade7b933-de22-4ac2-8ef8-a9838f5e9787-istio-data\") pod \"e2e-inf-a-fa0ccd-openshift-default-5d77f5f75f-t99j9\" (UID: \"ade7b933-de22-4ac2-8ef8-a9838f5e9787\") " pod="openshift-ingress/e2e-inf-a-fa0ccd-openshift-default-5d77f5f75f-t99j9" Jul 16 17:01:42.963916 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:01:42.963833 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ade7b933-de22-4ac2-8ef8-a9838f5e9787-workload-certs\") pod \"e2e-inf-a-fa0ccd-openshift-default-5d77f5f75f-t99j9\" (UID: \"ade7b933-de22-4ac2-8ef8-a9838f5e9787\") " pod="openshift-ingress/e2e-inf-a-fa0ccd-openshift-default-5d77f5f75f-t99j9" Jul 16 17:01:42.963959 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:01:42.963923 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ade7b933-de22-4ac2-8ef8-a9838f5e9787-istiod-ca-cert\") pod \"e2e-inf-a-fa0ccd-openshift-default-5d77f5f75f-t99j9\" (UID: \"ade7b933-de22-4ac2-8ef8-a9838f5e9787\") " pod="openshift-ingress/e2e-inf-a-fa0ccd-openshift-default-5d77f5f75f-t99j9" Jul 16 17:01:42.965716 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:01:42.965696 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ade7b933-de22-4ac2-8ef8-a9838f5e9787-istio-envoy\") pod \"e2e-inf-a-fa0ccd-openshift-default-5d77f5f75f-t99j9\" (UID: \"ade7b933-de22-4ac2-8ef8-a9838f5e9787\") " pod="openshift-ingress/e2e-inf-a-fa0ccd-openshift-default-5d77f5f75f-t99j9" Jul 16 17:01:42.966034 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:01:42.966016 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ade7b933-de22-4ac2-8ef8-a9838f5e9787-istio-podinfo\") pod \"e2e-inf-a-fa0ccd-openshift-default-5d77f5f75f-t99j9\" (UID: \"ade7b933-de22-4ac2-8ef8-a9838f5e9787\") " pod="openshift-ingress/e2e-inf-a-fa0ccd-openshift-default-5d77f5f75f-t99j9" Jul 16 17:01:42.972781 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:01:42.972758 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-swwt2\" (UniqueName: \"kubernetes.io/projected/ade7b933-de22-4ac2-8ef8-a9838f5e9787-kube-api-access-swwt2\") pod \"e2e-inf-a-fa0ccd-openshift-default-5d77f5f75f-t99j9\" (UID: \"ade7b933-de22-4ac2-8ef8-a9838f5e9787\") " pod="openshift-ingress/e2e-inf-a-fa0ccd-openshift-default-5d77f5f75f-t99j9" Jul 16 17:01:42.973968 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:01:42.973926 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ade7b933-de22-4ac2-8ef8-a9838f5e9787-istio-token\") pod \"e2e-inf-a-fa0ccd-openshift-default-5d77f5f75f-t99j9\" (UID: \"ade7b933-de22-4ac2-8ef8-a9838f5e9787\") " pod="openshift-ingress/e2e-inf-a-fa0ccd-openshift-default-5d77f5f75f-t99j9" Jul 16 17:01:43.002335 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:01:43.002297 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-fa0ccd-openshift-default-5d77f5f75f-t99j9" Jul 16 17:01:43.155769 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:01:43.155738 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-fa0ccd-openshift-default-5d77f5f75f-t99j9"] Jul 16 17:01:43.157243 ip-10-0-142-101 kubenswrapper[2577]: W0716 17:01:43.157216 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podade7b933_de22_4ac2_8ef8_a9838f5e9787.slice/crio-8d5ea3db9ea6971d643065cb7ee3f8ee179bf9e8d334428f68dd4e7006140685 WatchSource:0}: Error finding container 8d5ea3db9ea6971d643065cb7ee3f8ee179bf9e8d334428f68dd4e7006140685: Status 404 returned error can't find the container with id 8d5ea3db9ea6971d643065cb7ee3f8ee179bf9e8d334428f68dd4e7006140685 Jul 16 17:01:43.159369 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:01:43.159339 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 16 17:01:43.159474 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:01:43.159429 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 16 17:01:43.159516 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:01:43.159460 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 16 17:01:44.051593 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:01:44.051560 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-fa0ccd-openshift-default-5d77f5f75f-t99j9" event={"ID":"ade7b933-de22-4ac2-8ef8-a9838f5e9787","Type":"ContainerStarted","Data":"e0808b2718689a1d22d3e5bcf11880a41eadef8eda4fdd03624e4b62f2dc13ad"} Jul 16 17:01:44.051999 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:01:44.051611 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-fa0ccd-openshift-default-5d77f5f75f-t99j9" event={"ID":"ade7b933-de22-4ac2-8ef8-a9838f5e9787","Type":"ContainerStarted","Data":"8d5ea3db9ea6971d643065cb7ee3f8ee179bf9e8d334428f68dd4e7006140685"} Jul 16 17:01:44.077705 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:01:44.077651 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-fa0ccd-openshift-default-5d77f5f75f-t99j9" podStartSLOduration=2.077633487 podStartE2EDuration="2.077633487s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:01:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:01:44.075084114 +0000 UTC m=+2162.579141966" watchObservedRunningTime="2026-07-16 17:01:44.077633487 +0000 UTC m=+2162.581691350" Jul 16 17:01:45.003422 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:01:45.003340 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-fa0ccd-openshift-default-5d77f5f75f-t99j9" Jul 16 17:01:45.008340 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:01:45.008310 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-fa0ccd-openshift-default-5d77f5f75f-t99j9" Jul 16 17:01:45.057153 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:01:45.057118 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-fa0ccd-openshift-default-5d77f5f75f-t99j9" Jul 16 17:01:45.058196 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:01:45.058175 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-fa0ccd-openshift-default-5d77f5f75f-t99j9" Jul 16 17:02:01.098553 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:02:01.098505 2577 patch_prober.go:28] interesting pod/e2e-inf-a-fa0ccd-openshift-default-5d77f5f75f-t99j9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 16 17:02:01.098963 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:02:01.098592 2577 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-fa0ccd-openshift-default-5d77f5f75f-t99j9" podUID="ade7b933-de22-4ac2-8ef8-a9838f5e9787" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 16 17:02:02.149487 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:02:02.149452 2577 generic.go:356] "Generic (PLEG): container finished" podID="ade7b933-de22-4ac2-8ef8-a9838f5e9787" containerID="e0808b2718689a1d22d3e5bcf11880a41eadef8eda4fdd03624e4b62f2dc13ad" exitCode=0 Jul 16 17:02:02.149975 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:02:02.149501 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-fa0ccd-openshift-default-5d77f5f75f-t99j9" event={"ID":"ade7b933-de22-4ac2-8ef8-a9838f5e9787","Type":"ContainerDied","Data":"e0808b2718689a1d22d3e5bcf11880a41eadef8eda4fdd03624e4b62f2dc13ad"} Jul 16 17:02:02.149975 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:02:02.149896 2577 scope.go:117] "RemoveContainer" containerID="e0808b2718689a1d22d3e5bcf11880a41eadef8eda4fdd03624e4b62f2dc13ad" Jul 16 17:02:02.150926 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:02:02.150896 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 16 17:02:02.151017 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:02:02.150962 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 16 17:02:02.151017 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:02:02.150992 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 16 17:02:03.160956 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:02:03.160908 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-fa0ccd-openshift-default-5d77f5f75f-t99j9" event={"ID":"ade7b933-de22-4ac2-8ef8-a9838f5e9787","Type":"ContainerStarted","Data":"8dd44b807a2aaaa0d950e698d74ca5169a9a6b79afa18a8ce5aec4f375089df4"} Jul 16 17:02:04.003069 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:02:04.003033 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-fa0ccd-openshift-default-5d77f5f75f-t99j9" Jul 16 17:02:04.003459 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:02:04.003432 2577 patch_prober.go:28] interesting pod/e2e-inf-a-fa0ccd-openshift-default-5d77f5f75f-t99j9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body= Jul 16 17:02:04.003579 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:02:04.003488 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-fa0ccd-openshift-default-5d77f5f75f-t99j9" podUID="ade7b933-de22-4ac2-8ef8-a9838f5e9787" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" Jul 16 17:02:05.003454 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:02:05.003417 2577 patch_prober.go:28] interesting pod/e2e-inf-a-fa0ccd-openshift-default-5d77f5f75f-t99j9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body= Jul 16 17:02:05.003846 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:02:05.003488 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-fa0ccd-openshift-default-5d77f5f75f-t99j9" podUID="ade7b933-de22-4ac2-8ef8-a9838f5e9787" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" Jul 16 17:02:06.003124 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:02:06.003078 2577 patch_prober.go:28] interesting pod/e2e-inf-a-fa0ccd-openshift-default-5d77f5f75f-t99j9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body= Jul 16 17:02:06.003318 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:02:06.003141 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-fa0ccd-openshift-default-5d77f5f75f-t99j9" podUID="ade7b933-de22-4ac2-8ef8-a9838f5e9787" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" Jul 16 17:02:07.003570 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:02:07.003531 2577 patch_prober.go:28] interesting pod/e2e-inf-a-fa0ccd-openshift-default-5d77f5f75f-t99j9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body= Jul 16 17:02:07.003983 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:02:07.003598 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-fa0ccd-openshift-default-5d77f5f75f-t99j9" podUID="ade7b933-de22-4ac2-8ef8-a9838f5e9787" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" Jul 16 17:02:08.006823 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:02:08.006793 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-fa0ccd-openshift-default-5d77f5f75f-t99j9" Jul 16 17:02:08.007309 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:02:08.007220 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-fa0ccd-openshift-default-5d77f5f75f-t99j9" Jul 16 17:02:08.007966 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:02:08.007947 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-fa0ccd-openshift-default-5d77f5f75f-t99j9" Jul 16 17:03:29.908940 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:03:29.908856 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fa0ccd-npnj2"] Jul 16 17:03:29.909432 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:03:29.909302 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ed53e0ee-2e8e-4694-8290-2754d392cbe8" containerName="istio-proxy" Jul 16 17:03:29.909432 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:03:29.909314 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="ed53e0ee-2e8e-4694-8290-2754d392cbe8" containerName="istio-proxy" Jul 16 17:03:29.909432 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:03:29.909395 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="ed53e0ee-2e8e-4694-8290-2754d392cbe8" containerName="istio-proxy" Jul 16 17:03:29.917201 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:03:29.917020 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fa0ccd-npnj2" Jul 16 17:03:29.919918 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:03:29.919888 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dssfh\"" Jul 16 17:03:29.920084 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:03:29.919955 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 16 17:03:29.920462 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:03:29.920441 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fa0ccd-npnj2"] Jul 16 17:03:30.033081 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:03:30.033043 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb31a6f3-3163-47bd-bbc7-0c24b5182edd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-fa0ccd-npnj2\" (UID: \"eb31a6f3-3163-47bd-bbc7-0c24b5182edd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fa0ccd-npnj2" Jul 16 17:03:30.134046 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:03:30.134003 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb31a6f3-3163-47bd-bbc7-0c24b5182edd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-fa0ccd-npnj2\" (UID: \"eb31a6f3-3163-47bd-bbc7-0c24b5182edd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fa0ccd-npnj2" Jul 16 17:03:30.134695 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:03:30.134676 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb31a6f3-3163-47bd-bbc7-0c24b5182edd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-fa0ccd-npnj2\" (UID: \"eb31a6f3-3163-47bd-bbc7-0c24b5182edd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fa0ccd-npnj2" Jul 16 17:03:30.230198 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:03:30.230123 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fa0ccd-npnj2" Jul 16 17:03:30.370715 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:03:30.370687 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fa0ccd-npnj2"] Jul 16 17:03:30.623840 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:03:30.623801 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fa0ccd-npnj2" event={"ID":"eb31a6f3-3163-47bd-bbc7-0c24b5182edd","Type":"ContainerStarted","Data":"141ccbbde02c25495d6496b353da9bf2648c8490fc5684707f49f4d1e774f918"} Jul 16 17:03:31.630689 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:03:31.630649 2577 generic.go:356] "Generic (PLEG): container finished" podID="eb31a6f3-3163-47bd-bbc7-0c24b5182edd" containerID="672179eab805afe2ea91d66efc587f3c785d6ed68766045a6c75db26037ae77b" exitCode=0 Jul 16 17:03:31.631112 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:03:31.630733 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fa0ccd-npnj2" event={"ID":"eb31a6f3-3163-47bd-bbc7-0c24b5182edd","Type":"ContainerDied","Data":"672179eab805afe2ea91d66efc587f3c785d6ed68766045a6c75db26037ae77b"} Jul 16 17:03:32.771499 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:03:32.771475 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fa0ccd-npnj2" Jul 16 17:03:32.861920 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:03:32.861891 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb31a6f3-3163-47bd-bbc7-0c24b5182edd-maas-api-service-ca\") pod \"eb31a6f3-3163-47bd-bbc7-0c24b5182edd\" (UID: \"eb31a6f3-3163-47bd-bbc7-0c24b5182edd\") " Jul 16 17:03:32.862247 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:03:32.862224 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb31a6f3-3163-47bd-bbc7-0c24b5182edd-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "eb31a6f3-3163-47bd-bbc7-0c24b5182edd" (UID: "eb31a6f3-3163-47bd-bbc7-0c24b5182edd"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 17:03:32.963312 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:03:32.963237 2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb31a6f3-3163-47bd-bbc7-0c24b5182edd-maas-api-service-ca\") on node \"ip-10-0-142-101.ec2.internal\" DevicePath \"\"" Jul 16 17:03:33.643107 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:03:33.643074 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fa0ccd-npnj2" Jul 16 17:03:33.643257 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:03:33.643101 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fa0ccd-npnj2" event={"ID":"eb31a6f3-3163-47bd-bbc7-0c24b5182edd","Type":"ContainerDied","Data":"141ccbbde02c25495d6496b353da9bf2648c8490fc5684707f49f4d1e774f918"} Jul 16 17:03:33.643257 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:03:33.643153 2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="141ccbbde02c25495d6496b353da9bf2648c8490fc5684707f49f4d1e774f918" Jul 16 17:03:55.772020 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:03:55.771987 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-fa0ccd-openshift-default-5d77f5f75f-t99j9"] Jul 16 17:03:55.772433 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:03:55.772234 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-fa0ccd-openshift-default-5d77f5f75f-t99j9" podUID="ade7b933-de22-4ac2-8ef8-a9838f5e9787" containerName="istio-proxy" containerID="cri-o://8dd44b807a2aaaa0d950e698d74ca5169a9a6b79afa18a8ce5aec4f375089df4" gracePeriod=30 Jul 16 17:03:56.178488 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:03:56.178425 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f37c57-q75wl"] Jul 16 17:03:56.178955 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:03:56.178935 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eb31a6f3-3163-47bd-bbc7-0c24b5182edd" containerName="revoke-keys" Jul 16 17:03:56.179012 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:03:56.178957 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="eb31a6f3-3163-47bd-bbc7-0c24b5182edd" containerName="revoke-keys" Jul 16 17:03:56.179053 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:03:56.179035 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="eb31a6f3-3163-47bd-bbc7-0c24b5182edd" containerName="revoke-keys" Jul 16 17:03:56.181095 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:03:56.181078 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f37c57-q75wl" Jul 16 17:03:56.184974 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:03:56.184948 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dssfh\"" Jul 16 17:03:56.185085 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:03:56.185049 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 16 17:03:56.192622 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:03:56.192600 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f37c57-q75wl"] Jul 16 17:03:56.305891 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:03:56.305848 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/389cfd7e-d0ed-48e5-96fb-89396b873d09-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-f37c57-q75wl\" (UID: \"389cfd7e-d0ed-48e5-96fb-89396b873d09\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f37c57-q75wl" Jul 16 17:03:56.407478 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:03:56.407375 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/389cfd7e-d0ed-48e5-96fb-89396b873d09-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-f37c57-q75wl\" (UID: \"389cfd7e-d0ed-48e5-96fb-89396b873d09\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f37c57-q75wl" Jul 16 17:03:56.407977 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:03:56.407955 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/389cfd7e-d0ed-48e5-96fb-89396b873d09-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-f37c57-q75wl\" (UID: \"389cfd7e-d0ed-48e5-96fb-89396b873d09\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f37c57-q75wl" Jul 16 17:03:56.491467 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:03:56.491356 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f37c57-q75wl" Jul 16 17:03:56.627927 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:03:56.627902 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f37c57-q75wl"] Jul 16 17:03:56.630043 ip-10-0-142-101 kubenswrapper[2577]: W0716 17:03:56.630011 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod389cfd7e_d0ed_48e5_96fb_89396b873d09.slice/crio-e094499853b5e70b214d765f7d2e4900c2f94c4eec624bbd2604723253628a1f WatchSource:0}: Error finding container e094499853b5e70b214d765f7d2e4900c2f94c4eec624bbd2604723253628a1f: Status 404 returned error can't find the container with id e094499853b5e70b214d765f7d2e4900c2f94c4eec624bbd2604723253628a1f Jul 16 17:03:56.766357 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:03:56.766258 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f37c57-q75wl" event={"ID":"389cfd7e-d0ed-48e5-96fb-89396b873d09","Type":"ContainerStarted","Data":"e094499853b5e70b214d765f7d2e4900c2f94c4eec624bbd2604723253628a1f"} Jul 16 17:03:57.773250 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:03:57.773160 2577 generic.go:356] "Generic (PLEG): container finished" podID="389cfd7e-d0ed-48e5-96fb-89396b873d09" containerID="986d2726bad86869717d04a19b139afa5ff6b491c364273e451d77335c3ab1fd" exitCode=0 Jul 16 17:03:57.773250 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:03:57.773199 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f37c57-q75wl" event={"ID":"389cfd7e-d0ed-48e5-96fb-89396b873d09","Type":"ContainerDied","Data":"986d2726bad86869717d04a19b139afa5ff6b491c364273e451d77335c3ab1fd"} Jul 16 17:03:58.909983 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:03:58.909957 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f37c57-q75wl" Jul 16 17:03:59.033443 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:03:59.033336 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/389cfd7e-d0ed-48e5-96fb-89396b873d09-maas-api-service-ca\") pod \"389cfd7e-d0ed-48e5-96fb-89396b873d09\" (UID: \"389cfd7e-d0ed-48e5-96fb-89396b873d09\") " Jul 16 17:03:59.033700 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:03:59.033676 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/389cfd7e-d0ed-48e5-96fb-89396b873d09-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "389cfd7e-d0ed-48e5-96fb-89396b873d09" (UID: "389cfd7e-d0ed-48e5-96fb-89396b873d09"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 17:03:59.134740 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:03:59.134698 2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/389cfd7e-d0ed-48e5-96fb-89396b873d09-maas-api-service-ca\") on node \"ip-10-0-142-101.ec2.internal\" DevicePath \"\"" Jul 16 17:03:59.785032 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:03:59.784993 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f37c57-q75wl" event={"ID":"389cfd7e-d0ed-48e5-96fb-89396b873d09","Type":"ContainerDied","Data":"e094499853b5e70b214d765f7d2e4900c2f94c4eec624bbd2604723253628a1f"} Jul 16 17:03:59.785205 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:03:59.785037 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f37c57-q75wl" Jul 16 17:03:59.785205 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:03:59.785085 2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e094499853b5e70b214d765f7d2e4900c2f94c4eec624bbd2604723253628a1f" Jul 16 17:04:01.029955 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:04:01.029928 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-fa0ccd-openshift-default-5d77f5f75f-t99j9" Jul 16 17:04:01.157832 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:04:01.157796 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ade7b933-de22-4ac2-8ef8-a9838f5e9787-istio-data\") pod \"ade7b933-de22-4ac2-8ef8-a9838f5e9787\" (UID: \"ade7b933-de22-4ac2-8ef8-a9838f5e9787\") " Jul 16 17:04:01.158039 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:04:01.157875 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ade7b933-de22-4ac2-8ef8-a9838f5e9787-istio-envoy\") pod \"ade7b933-de22-4ac2-8ef8-a9838f5e9787\" (UID: \"ade7b933-de22-4ac2-8ef8-a9838f5e9787\") " Jul 16 17:04:01.158039 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:04:01.157911 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ade7b933-de22-4ac2-8ef8-a9838f5e9787-istiod-ca-cert\") pod \"ade7b933-de22-4ac2-8ef8-a9838f5e9787\" (UID: \"ade7b933-de22-4ac2-8ef8-a9838f5e9787\") " Jul 16 17:04:01.158170 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:04:01.158068 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ade7b933-de22-4ac2-8ef8-a9838f5e9787-workload-socket\") pod \"ade7b933-de22-4ac2-8ef8-a9838f5e9787\" (UID: \"ade7b933-de22-4ac2-8ef8-a9838f5e9787\") " Jul 16 17:04:01.158170 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:04:01.158091 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-swwt2\" (UniqueName: \"kubernetes.io/projected/ade7b933-de22-4ac2-8ef8-a9838f5e9787-kube-api-access-swwt2\") pod \"ade7b933-de22-4ac2-8ef8-a9838f5e9787\" (UID: \"ade7b933-de22-4ac2-8ef8-a9838f5e9787\") " Jul 16 17:04:01.158170 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:04:01.158121 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ade7b933-de22-4ac2-8ef8-a9838f5e9787-workload-certs\") pod \"ade7b933-de22-4ac2-8ef8-a9838f5e9787\" (UID: \"ade7b933-de22-4ac2-8ef8-a9838f5e9787\") " Jul 16 17:04:01.158170 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:04:01.158162 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ade7b933-de22-4ac2-8ef8-a9838f5e9787-credential-socket\") pod \"ade7b933-de22-4ac2-8ef8-a9838f5e9787\" (UID: \"ade7b933-de22-4ac2-8ef8-a9838f5e9787\") " Jul 16 17:04:01.158423 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:04:01.158194 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ade7b933-de22-4ac2-8ef8-a9838f5e9787-istio-token\") pod \"ade7b933-de22-4ac2-8ef8-a9838f5e9787\" (UID: \"ade7b933-de22-4ac2-8ef8-a9838f5e9787\") " Jul 16 17:04:01.158423 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:04:01.158237 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ade7b933-de22-4ac2-8ef8-a9838f5e9787-istio-podinfo\") pod \"ade7b933-de22-4ac2-8ef8-a9838f5e9787\" (UID: \"ade7b933-de22-4ac2-8ef8-a9838f5e9787\") " Jul 16 17:04:01.158423 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:04:01.158238 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ade7b933-de22-4ac2-8ef8-a9838f5e9787-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "ade7b933-de22-4ac2-8ef8-a9838f5e9787" (UID: "ade7b933-de22-4ac2-8ef8-a9838f5e9787"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 17:04:01.158423 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:04:01.158308 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ade7b933-de22-4ac2-8ef8-a9838f5e9787-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "ade7b933-de22-4ac2-8ef8-a9838f5e9787" (UID: "ade7b933-de22-4ac2-8ef8-a9838f5e9787"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 17:04:01.158637 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:04:01.158488 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ade7b933-de22-4ac2-8ef8-a9838f5e9787-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "ade7b933-de22-4ac2-8ef8-a9838f5e9787" (UID: "ade7b933-de22-4ac2-8ef8-a9838f5e9787"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 17:04:01.158637 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:04:01.158522 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ade7b933-de22-4ac2-8ef8-a9838f5e9787-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "ade7b933-de22-4ac2-8ef8-a9838f5e9787" (UID: "ade7b933-de22-4ac2-8ef8-a9838f5e9787"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 17:04:01.158744 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:04:01.158675 2577 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ade7b933-de22-4ac2-8ef8-a9838f5e9787-istiod-ca-cert\") on node \"ip-10-0-142-101.ec2.internal\" DevicePath \"\"" Jul 16 17:04:01.158744 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:04:01.158692 2577 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ade7b933-de22-4ac2-8ef8-a9838f5e9787-workload-socket\") on node \"ip-10-0-142-101.ec2.internal\" DevicePath \"\"" Jul 16 17:04:01.158744 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:04:01.158707 2577 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ade7b933-de22-4ac2-8ef8-a9838f5e9787-workload-certs\") on node \"ip-10-0-142-101.ec2.internal\" DevicePath \"\"" Jul 16 17:04:01.158744 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:04:01.158721 2577 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ade7b933-de22-4ac2-8ef8-a9838f5e9787-credential-socket\") on node \"ip-10-0-142-101.ec2.internal\" DevicePath \"\"" Jul 16 17:04:01.158744 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:04:01.158716 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ade7b933-de22-4ac2-8ef8-a9838f5e9787-istio-data" (OuterVolumeSpecName: "istio-data") pod "ade7b933-de22-4ac2-8ef8-a9838f5e9787" (UID: "ade7b933-de22-4ac2-8ef8-a9838f5e9787"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 17:04:01.160310 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:04:01.160280 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ade7b933-de22-4ac2-8ef8-a9838f5e9787-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "ade7b933-de22-4ac2-8ef8-a9838f5e9787" (UID: "ade7b933-de22-4ac2-8ef8-a9838f5e9787"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 17:04:01.160310 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:04:01.160296 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ade7b933-de22-4ac2-8ef8-a9838f5e9787-kube-api-access-swwt2" (OuterVolumeSpecName: "kube-api-access-swwt2") pod "ade7b933-de22-4ac2-8ef8-a9838f5e9787" (UID: "ade7b933-de22-4ac2-8ef8-a9838f5e9787"). InnerVolumeSpecName "kube-api-access-swwt2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 17:04:01.160505 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:04:01.160363 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ade7b933-de22-4ac2-8ef8-a9838f5e9787-istio-token" (OuterVolumeSpecName: "istio-token") pod "ade7b933-de22-4ac2-8ef8-a9838f5e9787" (UID: "ade7b933-de22-4ac2-8ef8-a9838f5e9787"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 17:04:01.160505 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:04:01.160409 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ade7b933-de22-4ac2-8ef8-a9838f5e9787-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "ade7b933-de22-4ac2-8ef8-a9838f5e9787" (UID: "ade7b933-de22-4ac2-8ef8-a9838f5e9787"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 16 17:04:01.259509 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:04:01.259480 2577 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ade7b933-de22-4ac2-8ef8-a9838f5e9787-istio-envoy\") on node \"ip-10-0-142-101.ec2.internal\" DevicePath \"\"" Jul 16 17:04:01.259509 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:04:01.259507 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-swwt2\" (UniqueName: \"kubernetes.io/projected/ade7b933-de22-4ac2-8ef8-a9838f5e9787-kube-api-access-swwt2\") on node \"ip-10-0-142-101.ec2.internal\" DevicePath \"\"" Jul 16 17:04:01.259698 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:04:01.259520 2577 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ade7b933-de22-4ac2-8ef8-a9838f5e9787-istio-token\") on node \"ip-10-0-142-101.ec2.internal\" DevicePath \"\"" Jul 16 17:04:01.259698 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:04:01.259532 2577 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ade7b933-de22-4ac2-8ef8-a9838f5e9787-istio-podinfo\") on node \"ip-10-0-142-101.ec2.internal\" DevicePath \"\"" Jul 16 17:04:01.259698 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:04:01.259541 2577 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ade7b933-de22-4ac2-8ef8-a9838f5e9787-istio-data\") on node \"ip-10-0-142-101.ec2.internal\" DevicePath \"\"" Jul 16 17:04:01.798209 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:04:01.798175 2577 generic.go:356] "Generic (PLEG): container finished" podID="ade7b933-de22-4ac2-8ef8-a9838f5e9787" containerID="8dd44b807a2aaaa0d950e698d74ca5169a9a6b79afa18a8ce5aec4f375089df4" exitCode=0 Jul 16 17:04:01.798209 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:04:01.798218 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-fa0ccd-openshift-default-5d77f5f75f-t99j9" Jul 16 17:04:01.798461 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:04:01.798255 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-fa0ccd-openshift-default-5d77f5f75f-t99j9" event={"ID":"ade7b933-de22-4ac2-8ef8-a9838f5e9787","Type":"ContainerDied","Data":"8dd44b807a2aaaa0d950e698d74ca5169a9a6b79afa18a8ce5aec4f375089df4"} Jul 16 17:04:01.798461 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:04:01.798317 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-fa0ccd-openshift-default-5d77f5f75f-t99j9" event={"ID":"ade7b933-de22-4ac2-8ef8-a9838f5e9787","Type":"ContainerDied","Data":"8d5ea3db9ea6971d643065cb7ee3f8ee179bf9e8d334428f68dd4e7006140685"} Jul 16 17:04:01.798461 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:04:01.798335 2577 scope.go:117] "RemoveContainer" containerID="8dd44b807a2aaaa0d950e698d74ca5169a9a6b79afa18a8ce5aec4f375089df4" Jul 16 17:04:01.808028 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:04:01.808009 2577 scope.go:117] "RemoveContainer" containerID="e0808b2718689a1d22d3e5bcf11880a41eadef8eda4fdd03624e4b62f2dc13ad" Jul 16 17:04:01.816016 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:04:01.815993 2577 scope.go:117] "RemoveContainer" containerID="8dd44b807a2aaaa0d950e698d74ca5169a9a6b79afa18a8ce5aec4f375089df4" Jul 16 17:04:01.816284 ip-10-0-142-101 kubenswrapper[2577]: E0716 17:04:01.816268 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8dd44b807a2aaaa0d950e698d74ca5169a9a6b79afa18a8ce5aec4f375089df4\": container with ID starting with 8dd44b807a2aaaa0d950e698d74ca5169a9a6b79afa18a8ce5aec4f375089df4 not found: ID does not exist" containerID="8dd44b807a2aaaa0d950e698d74ca5169a9a6b79afa18a8ce5aec4f375089df4" Jul 16 17:04:01.816323 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:04:01.816293 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8dd44b807a2aaaa0d950e698d74ca5169a9a6b79afa18a8ce5aec4f375089df4"} err="failed to get container status \"8dd44b807a2aaaa0d950e698d74ca5169a9a6b79afa18a8ce5aec4f375089df4\": rpc error: code = NotFound desc = could not find container \"8dd44b807a2aaaa0d950e698d74ca5169a9a6b79afa18a8ce5aec4f375089df4\": container with ID starting with 8dd44b807a2aaaa0d950e698d74ca5169a9a6b79afa18a8ce5aec4f375089df4 not found: ID does not exist" Jul 16 17:04:01.816323 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:04:01.816308 2577 scope.go:117] "RemoveContainer" containerID="e0808b2718689a1d22d3e5bcf11880a41eadef8eda4fdd03624e4b62f2dc13ad" Jul 16 17:04:01.816636 ip-10-0-142-101 kubenswrapper[2577]: E0716 17:04:01.816617 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e0808b2718689a1d22d3e5bcf11880a41eadef8eda4fdd03624e4b62f2dc13ad\": container with ID starting with e0808b2718689a1d22d3e5bcf11880a41eadef8eda4fdd03624e4b62f2dc13ad not found: ID does not exist" containerID="e0808b2718689a1d22d3e5bcf11880a41eadef8eda4fdd03624e4b62f2dc13ad" Jul 16 17:04:01.816679 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:04:01.816643 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e0808b2718689a1d22d3e5bcf11880a41eadef8eda4fdd03624e4b62f2dc13ad"} err="failed to get container status \"e0808b2718689a1d22d3e5bcf11880a41eadef8eda4fdd03624e4b62f2dc13ad\": rpc error: code = NotFound desc = could not find container \"e0808b2718689a1d22d3e5bcf11880a41eadef8eda4fdd03624e4b62f2dc13ad\": container with ID starting with e0808b2718689a1d22d3e5bcf11880a41eadef8eda4fdd03624e4b62f2dc13ad not found: ID does not exist" Jul 16 17:04:01.822569 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:04:01.822545 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-fa0ccd-openshift-default-5d77f5f75f-t99j9"] Jul 16 17:04:01.831116 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:04:01.831092 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-fa0ccd-openshift-default-5d77f5f75f-t99j9"] Jul 16 17:04:02.019398 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:04:02.019355 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ade7b933-de22-4ac2-8ef8-a9838f5e9787" path="/var/lib/kubelet/pods/ade7b933-de22-4ac2-8ef8-a9838f5e9787/volumes" Jul 16 17:04:23.156300 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:04:23.156263 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-6f111d-openshift-default-68f84858d9-z5fjb"] Jul 16 17:04:23.156692 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:04:23.156666 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="389cfd7e-d0ed-48e5-96fb-89396b873d09" containerName="revoke-keys" Jul 16 17:04:23.156692 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:04:23.156678 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="389cfd7e-d0ed-48e5-96fb-89396b873d09" containerName="revoke-keys" Jul 16 17:04:23.156765 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:04:23.156693 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ade7b933-de22-4ac2-8ef8-a9838f5e9787" containerName="istio-proxy" Jul 16 17:04:23.156765 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:04:23.156700 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="ade7b933-de22-4ac2-8ef8-a9838f5e9787" containerName="istio-proxy" Jul 16 17:04:23.156765 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:04:23.156711 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ade7b933-de22-4ac2-8ef8-a9838f5e9787" containerName="istio-proxy" Jul 16 17:04:23.156765 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:04:23.156717 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="ade7b933-de22-4ac2-8ef8-a9838f5e9787" containerName="istio-proxy" Jul 16 17:04:23.156893 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:04:23.156776 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="ade7b933-de22-4ac2-8ef8-a9838f5e9787" containerName="istio-proxy" Jul 16 17:04:23.156893 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:04:23.156785 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="ade7b933-de22-4ac2-8ef8-a9838f5e9787" containerName="istio-proxy" Jul 16 17:04:23.156893 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:04:23.156792 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="389cfd7e-d0ed-48e5-96fb-89396b873d09" containerName="revoke-keys" Jul 16 17:04:23.158893 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:04:23.158875 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-6f111d-openshift-default-68f84858d9-z5fjb" Jul 16 17:04:23.164551 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:04:23.164521 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-6f111d-openshift-default-dockercfg-xqcv6\"" Jul 16 17:04:23.181825 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:04:23.181791 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-6f111d-openshift-default-68f84858d9-z5fjb"] Jul 16 17:04:23.260005 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:04:23.259969 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wdknr\" (UniqueName: \"kubernetes.io/projected/6c13a348-581f-4ea1-b44d-5e9bf37c98a2-kube-api-access-wdknr\") pod \"e2e-api-a-6f111d-openshift-default-68f84858d9-z5fjb\" (UID: \"6c13a348-581f-4ea1-b44d-5e9bf37c98a2\") " pod="openshift-ingress/e2e-api-a-6f111d-openshift-default-68f84858d9-z5fjb" Jul 16 17:04:23.260005 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:04:23.260011 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c13a348-581f-4ea1-b44d-5e9bf37c98a2-credential-socket\") pod \"e2e-api-a-6f111d-openshift-default-68f84858d9-z5fjb\" (UID: \"6c13a348-581f-4ea1-b44d-5e9bf37c98a2\") " pod="openshift-ingress/e2e-api-a-6f111d-openshift-default-68f84858d9-z5fjb" Jul 16 17:04:23.260241 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:04:23.260028 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c13a348-581f-4ea1-b44d-5e9bf37c98a2-istiod-ca-cert\") pod \"e2e-api-a-6f111d-openshift-default-68f84858d9-z5fjb\" (UID: \"6c13a348-581f-4ea1-b44d-5e9bf37c98a2\") " pod="openshift-ingress/e2e-api-a-6f111d-openshift-default-68f84858d9-z5fjb" Jul 16 17:04:23.260241 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:04:23.260050 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c13a348-581f-4ea1-b44d-5e9bf37c98a2-istio-data\") pod \"e2e-api-a-6f111d-openshift-default-68f84858d9-z5fjb\" (UID: \"6c13a348-581f-4ea1-b44d-5e9bf37c98a2\") " pod="openshift-ingress/e2e-api-a-6f111d-openshift-default-68f84858d9-z5fjb" Jul 16 17:04:23.260241 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:04:23.260106 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c13a348-581f-4ea1-b44d-5e9bf37c98a2-workload-certs\") pod \"e2e-api-a-6f111d-openshift-default-68f84858d9-z5fjb\" (UID: \"6c13a348-581f-4ea1-b44d-5e9bf37c98a2\") " pod="openshift-ingress/e2e-api-a-6f111d-openshift-default-68f84858d9-z5fjb" Jul 16 17:04:23.260241 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:04:23.260128 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c13a348-581f-4ea1-b44d-5e9bf37c98a2-workload-socket\") pod \"e2e-api-a-6f111d-openshift-default-68f84858d9-z5fjb\" (UID: \"6c13a348-581f-4ea1-b44d-5e9bf37c98a2\") " pod="openshift-ingress/e2e-api-a-6f111d-openshift-default-68f84858d9-z5fjb" Jul 16 17:04:23.260241 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:04:23.260142 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c13a348-581f-4ea1-b44d-5e9bf37c98a2-istio-envoy\") pod \"e2e-api-a-6f111d-openshift-default-68f84858d9-z5fjb\" (UID: \"6c13a348-581f-4ea1-b44d-5e9bf37c98a2\") " pod="openshift-ingress/e2e-api-a-6f111d-openshift-default-68f84858d9-z5fjb" Jul 16 17:04:23.260241 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:04:23.260208 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c13a348-581f-4ea1-b44d-5e9bf37c98a2-istio-token\") pod \"e2e-api-a-6f111d-openshift-default-68f84858d9-z5fjb\" (UID: \"6c13a348-581f-4ea1-b44d-5e9bf37c98a2\") " pod="openshift-ingress/e2e-api-a-6f111d-openshift-default-68f84858d9-z5fjb" Jul 16 17:04:23.260545 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:04:23.260299 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c13a348-581f-4ea1-b44d-5e9bf37c98a2-istio-podinfo\") pod \"e2e-api-a-6f111d-openshift-default-68f84858d9-z5fjb\" (UID: \"6c13a348-581f-4ea1-b44d-5e9bf37c98a2\") " pod="openshift-ingress/e2e-api-a-6f111d-openshift-default-68f84858d9-z5fjb" Jul 16 17:04:23.361594 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:04:23.361556 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c13a348-581f-4ea1-b44d-5e9bf37c98a2-credential-socket\") pod \"e2e-api-a-6f111d-openshift-default-68f84858d9-z5fjb\" (UID: \"6c13a348-581f-4ea1-b44d-5e9bf37c98a2\") " pod="openshift-ingress/e2e-api-a-6f111d-openshift-default-68f84858d9-z5fjb" Jul 16 17:04:23.361594 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:04:23.361594 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c13a348-581f-4ea1-b44d-5e9bf37c98a2-istiod-ca-cert\") pod \"e2e-api-a-6f111d-openshift-default-68f84858d9-z5fjb\" (UID: \"6c13a348-581f-4ea1-b44d-5e9bf37c98a2\") " pod="openshift-ingress/e2e-api-a-6f111d-openshift-default-68f84858d9-z5fjb" Jul 16 17:04:23.361818 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:04:23.361618 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c13a348-581f-4ea1-b44d-5e9bf37c98a2-istio-data\") pod \"e2e-api-a-6f111d-openshift-default-68f84858d9-z5fjb\" (UID: \"6c13a348-581f-4ea1-b44d-5e9bf37c98a2\") " pod="openshift-ingress/e2e-api-a-6f111d-openshift-default-68f84858d9-z5fjb" Jul 16 17:04:23.361818 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:04:23.361676 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c13a348-581f-4ea1-b44d-5e9bf37c98a2-workload-certs\") pod \"e2e-api-a-6f111d-openshift-default-68f84858d9-z5fjb\" (UID: \"6c13a348-581f-4ea1-b44d-5e9bf37c98a2\") " pod="openshift-ingress/e2e-api-a-6f111d-openshift-default-68f84858d9-z5fjb" Jul 16 17:04:23.361818 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:04:23.361712 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c13a348-581f-4ea1-b44d-5e9bf37c98a2-workload-socket\") pod \"e2e-api-a-6f111d-openshift-default-68f84858d9-z5fjb\" (UID: \"6c13a348-581f-4ea1-b44d-5e9bf37c98a2\") " pod="openshift-ingress/e2e-api-a-6f111d-openshift-default-68f84858d9-z5fjb" Jul 16 17:04:23.361818 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:04:23.361738 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c13a348-581f-4ea1-b44d-5e9bf37c98a2-istio-envoy\") pod \"e2e-api-a-6f111d-openshift-default-68f84858d9-z5fjb\" (UID: \"6c13a348-581f-4ea1-b44d-5e9bf37c98a2\") " pod="openshift-ingress/e2e-api-a-6f111d-openshift-default-68f84858d9-z5fjb" Jul 16 17:04:23.361818 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:04:23.361786 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c13a348-581f-4ea1-b44d-5e9bf37c98a2-istio-token\") pod \"e2e-api-a-6f111d-openshift-default-68f84858d9-z5fjb\" (UID: \"6c13a348-581f-4ea1-b44d-5e9bf37c98a2\") " pod="openshift-ingress/e2e-api-a-6f111d-openshift-default-68f84858d9-z5fjb" Jul 16 17:04:23.362029 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:04:23.361842 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c13a348-581f-4ea1-b44d-5e9bf37c98a2-istio-podinfo\") pod \"e2e-api-a-6f111d-openshift-default-68f84858d9-z5fjb\" (UID: \"6c13a348-581f-4ea1-b44d-5e9bf37c98a2\") " pod="openshift-ingress/e2e-api-a-6f111d-openshift-default-68f84858d9-z5fjb" Jul 16 17:04:23.362029 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:04:23.361884 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wdknr\" (UniqueName: \"kubernetes.io/projected/6c13a348-581f-4ea1-b44d-5e9bf37c98a2-kube-api-access-wdknr\") pod \"e2e-api-a-6f111d-openshift-default-68f84858d9-z5fjb\" (UID: \"6c13a348-581f-4ea1-b44d-5e9bf37c98a2\") " pod="openshift-ingress/e2e-api-a-6f111d-openshift-default-68f84858d9-z5fjb" Jul 16 17:04:23.362136 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:04:23.362037 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c13a348-581f-4ea1-b44d-5e9bf37c98a2-credential-socket\") pod \"e2e-api-a-6f111d-openshift-default-68f84858d9-z5fjb\" (UID: \"6c13a348-581f-4ea1-b44d-5e9bf37c98a2\") " pod="openshift-ingress/e2e-api-a-6f111d-openshift-default-68f84858d9-z5fjb" Jul 16 17:04:23.362136 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:04:23.362057 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c13a348-581f-4ea1-b44d-5e9bf37c98a2-istio-data\") pod \"e2e-api-a-6f111d-openshift-default-68f84858d9-z5fjb\" (UID: \"6c13a348-581f-4ea1-b44d-5e9bf37c98a2\") " pod="openshift-ingress/e2e-api-a-6f111d-openshift-default-68f84858d9-z5fjb" Jul 16 17:04:23.362240 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:04:23.362182 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c13a348-581f-4ea1-b44d-5e9bf37c98a2-workload-socket\") pod \"e2e-api-a-6f111d-openshift-default-68f84858d9-z5fjb\" (UID: \"6c13a348-581f-4ea1-b44d-5e9bf37c98a2\") " pod="openshift-ingress/e2e-api-a-6f111d-openshift-default-68f84858d9-z5fjb" Jul 16 17:04:23.362303 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:04:23.362264 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c13a348-581f-4ea1-b44d-5e9bf37c98a2-workload-certs\") pod \"e2e-api-a-6f111d-openshift-default-68f84858d9-z5fjb\" (UID: \"6c13a348-581f-4ea1-b44d-5e9bf37c98a2\") " pod="openshift-ingress/e2e-api-a-6f111d-openshift-default-68f84858d9-z5fjb" Jul 16 17:04:23.362504 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:04:23.362470 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c13a348-581f-4ea1-b44d-5e9bf37c98a2-istiod-ca-cert\") pod \"e2e-api-a-6f111d-openshift-default-68f84858d9-z5fjb\" (UID: \"6c13a348-581f-4ea1-b44d-5e9bf37c98a2\") " pod="openshift-ingress/e2e-api-a-6f111d-openshift-default-68f84858d9-z5fjb" Jul 16 17:04:23.364235 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:04:23.364214 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c13a348-581f-4ea1-b44d-5e9bf37c98a2-istio-envoy\") pod \"e2e-api-a-6f111d-openshift-default-68f84858d9-z5fjb\" (UID: \"6c13a348-581f-4ea1-b44d-5e9bf37c98a2\") " pod="openshift-ingress/e2e-api-a-6f111d-openshift-default-68f84858d9-z5fjb" Jul 16 17:04:23.364372 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:04:23.364356 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c13a348-581f-4ea1-b44d-5e9bf37c98a2-istio-podinfo\") pod \"e2e-api-a-6f111d-openshift-default-68f84858d9-z5fjb\" (UID: \"6c13a348-581f-4ea1-b44d-5e9bf37c98a2\") " pod="openshift-ingress/e2e-api-a-6f111d-openshift-default-68f84858d9-z5fjb" Jul 16 17:04:23.371930 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:04:23.371903 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c13a348-581f-4ea1-b44d-5e9bf37c98a2-istio-token\") pod \"e2e-api-a-6f111d-openshift-default-68f84858d9-z5fjb\" (UID: \"6c13a348-581f-4ea1-b44d-5e9bf37c98a2\") " pod="openshift-ingress/e2e-api-a-6f111d-openshift-default-68f84858d9-z5fjb" Jul 16 17:04:23.375331 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:04:23.375304 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wdknr\" (UniqueName: \"kubernetes.io/projected/6c13a348-581f-4ea1-b44d-5e9bf37c98a2-kube-api-access-wdknr\") pod \"e2e-api-a-6f111d-openshift-default-68f84858d9-z5fjb\" (UID: \"6c13a348-581f-4ea1-b44d-5e9bf37c98a2\") " pod="openshift-ingress/e2e-api-a-6f111d-openshift-default-68f84858d9-z5fjb" Jul 16 17:04:23.468854 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:04:23.468764 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-6f111d-openshift-default-68f84858d9-z5fjb" Jul 16 17:04:23.625727 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:04:23.625696 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-6f111d-openshift-default-68f84858d9-z5fjb"] Jul 16 17:04:23.626351 ip-10-0-142-101 kubenswrapper[2577]: W0716 17:04:23.626328 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c13a348_581f_4ea1_b44d_5e9bf37c98a2.slice/crio-0cd074abf36c5c93d0f2ade4e1eaee8efb41edaa7013e4182bae7da472a2ebf7 WatchSource:0}: Error finding container 0cd074abf36c5c93d0f2ade4e1eaee8efb41edaa7013e4182bae7da472a2ebf7: Status 404 returned error can't find the container with id 0cd074abf36c5c93d0f2ade4e1eaee8efb41edaa7013e4182bae7da472a2ebf7 Jul 16 17:04:23.628452 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:04:23.628422 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 16 17:04:23.628535 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:04:23.628491 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 16 17:04:23.628535 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:04:23.628520 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 16 17:04:23.916491 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:04:23.916461 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-6f111d-openshift-default-68f84858d9-z5fjb" event={"ID":"6c13a348-581f-4ea1-b44d-5e9bf37c98a2","Type":"ContainerStarted","Data":"ad7359e29d99821c5752f1bb8fa00195762c164f129ec5771ad13d0ff22071eb"} Jul 16 17:04:23.916682 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:04:23.916513 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-6f111d-openshift-default-68f84858d9-z5fjb" event={"ID":"6c13a348-581f-4ea1-b44d-5e9bf37c98a2","Type":"ContainerStarted","Data":"0cd074abf36c5c93d0f2ade4e1eaee8efb41edaa7013e4182bae7da472a2ebf7"} Jul 16 17:04:23.947261 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:04:23.947206 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-6f111d-openshift-default-68f84858d9-z5fjb" podStartSLOduration=0.947188912 podStartE2EDuration="947.188912ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:04:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:04:23.944890894 +0000 UTC m=+2322.448948745" watchObservedRunningTime="2026-07-16 17:04:23.947188912 +0000 UTC m=+2322.451246760" Jul 16 17:04:24.469428 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:04:24.469366 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-6f111d-openshift-default-68f84858d9-z5fjb" Jul 16 17:04:24.474543 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:04:24.474516 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-6f111d-openshift-default-68f84858d9-z5fjb" Jul 16 17:04:24.921700 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:04:24.921656 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-6f111d-openshift-default-68f84858d9-z5fjb" Jul 16 17:04:24.922567 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:04:24.922549 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-6f111d-openshift-default-68f84858d9-z5fjb" Jul 16 17:04:39.010447 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:04:39.010411 2577 generic.go:356] "Generic (PLEG): container finished" podID="6c13a348-581f-4ea1-b44d-5e9bf37c98a2" containerID="ad7359e29d99821c5752f1bb8fa00195762c164f129ec5771ad13d0ff22071eb" exitCode=0 Jul 16 17:04:39.010985 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:04:39.010485 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-6f111d-openshift-default-68f84858d9-z5fjb" event={"ID":"6c13a348-581f-4ea1-b44d-5e9bf37c98a2","Type":"ContainerDied","Data":"ad7359e29d99821c5752f1bb8fa00195762c164f129ec5771ad13d0ff22071eb"} Jul 16 17:04:39.011055 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:04:39.011009 2577 scope.go:117] "RemoveContainer" containerID="ad7359e29d99821c5752f1bb8fa00195762c164f129ec5771ad13d0ff22071eb" Jul 16 17:04:39.012013 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:04:39.011989 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 16 17:04:39.012081 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:04:39.012064 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 16 17:04:39.012118 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:04:39.012100 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 16 17:04:39.922218 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:04:39.922179 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-6f111d-openshift-default-68f84858d9-z5fjb" Jul 16 17:04:40.018512 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:04:40.018467 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-6f111d-openshift-default-68f84858d9-z5fjb" event={"ID":"6c13a348-581f-4ea1-b44d-5e9bf37c98a2","Type":"ContainerStarted","Data":"3ff3ea7e2deaba2906f0756cbddfc41229f6f124fab7c502eb38ef03eeefd945"} Jul 16 17:04:40.469519 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:04:40.469469 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-6f111d-openshift-default-68f84858d9-z5fjb" Jul 16 17:04:40.470752 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:04:40.470727 2577 patch_prober.go:28] interesting pod/e2e-api-a-6f111d-openshift-default-68f84858d9-z5fjb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body= Jul 16 17:04:40.470907 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:04:40.470793 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-6f111d-openshift-default-68f84858d9-z5fjb" podUID="6c13a348-581f-4ea1-b44d-5e9bf37c98a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" Jul 16 17:04:41.469541 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:04:41.469507 2577 patch_prober.go:28] interesting pod/e2e-api-a-6f111d-openshift-default-68f84858d9-z5fjb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body= Jul 16 17:04:41.469949 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:04:41.469565 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-6f111d-openshift-default-68f84858d9-z5fjb" podUID="6c13a348-581f-4ea1-b44d-5e9bf37c98a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" Jul 16 17:04:42.469814 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:04:42.469764 2577 patch_prober.go:28] interesting pod/e2e-api-a-6f111d-openshift-default-68f84858d9-z5fjb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body= Jul 16 17:04:42.470295 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:04:42.469844 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-6f111d-openshift-default-68f84858d9-z5fjb" podUID="6c13a348-581f-4ea1-b44d-5e9bf37c98a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" Jul 16 17:04:43.470067 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:04:43.470023 2577 patch_prober.go:28] interesting pod/e2e-api-a-6f111d-openshift-default-68f84858d9-z5fjb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body= Jul 16 17:04:43.470500 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:04:43.470090 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-6f111d-openshift-default-68f84858d9-z5fjb" podUID="6c13a348-581f-4ea1-b44d-5e9bf37c98a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" Jul 16 17:04:44.473808 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:04:44.473777 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-6f111d-openshift-default-68f84858d9-z5fjb" Jul 16 17:04:44.474184 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:04:44.474153 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-6f111d-openshift-default-68f84858d9-z5fjb" Jul 16 17:04:44.474810 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:04:44.474778 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-6f111d-openshift-default-68f84858d9-z5fjb" Jul 16 17:04:47.068584 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:04:47.068554 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6f111d-82gc7"] Jul 16 17:04:47.071018 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:04:47.071000 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6f111d-82gc7" Jul 16 17:04:47.073870 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:04:47.073845 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 16 17:04:47.074008 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:04:47.073912 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dssfh\"" Jul 16 17:04:47.083226 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:04:47.083199 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6f111d-82gc7"] Jul 16 17:04:47.199378 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:04:47.199337 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/14c15a24-306b-405e-bca2-873d7eeb9393-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-6f111d-82gc7\" (UID: \"14c15a24-306b-405e-bca2-873d7eeb9393\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6f111d-82gc7" Jul 16 17:04:47.299998 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:04:47.299961 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/14c15a24-306b-405e-bca2-873d7eeb9393-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-6f111d-82gc7\" (UID: \"14c15a24-306b-405e-bca2-873d7eeb9393\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6f111d-82gc7" Jul 16 17:04:47.300625 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:04:47.300606 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/14c15a24-306b-405e-bca2-873d7eeb9393-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-6f111d-82gc7\" (UID: \"14c15a24-306b-405e-bca2-873d7eeb9393\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6f111d-82gc7" Jul 16 17:04:47.382592 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:04:47.382555 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6f111d-82gc7" Jul 16 17:04:47.534029 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:04:47.533996 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6f111d-82gc7"] Jul 16 17:04:47.535314 ip-10-0-142-101 kubenswrapper[2577]: W0716 17:04:47.535281 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14c15a24_306b_405e_bca2_873d7eeb9393.slice/crio-cd2996d636bd902f63e7dec3a897fbc24ef65040d1aba3240481befc621e0510 WatchSource:0}: Error finding container cd2996d636bd902f63e7dec3a897fbc24ef65040d1aba3240481befc621e0510: Status 404 returned error can't find the container with id cd2996d636bd902f63e7dec3a897fbc24ef65040d1aba3240481befc621e0510 Jul 16 17:04:48.065559 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:04:48.065462 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6f111d-82gc7" event={"ID":"14c15a24-306b-405e-bca2-873d7eeb9393","Type":"ContainerStarted","Data":"44dcdcd46222bb1a9f2531eb61703e65de6ffd6549e84517e4dfea02561c0e55"} Jul 16 17:04:48.065559 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:04:48.065529 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6f111d-82gc7" event={"ID":"14c15a24-306b-405e-bca2-873d7eeb9393","Type":"ContainerStarted","Data":"cd2996d636bd902f63e7dec3a897fbc24ef65040d1aba3240481befc621e0510"} Jul 16 17:04:48.085491 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:04:48.085441 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6f111d-82gc7" podStartSLOduration=1.085425431 podStartE2EDuration="1.085425431s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:04:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:04:48.083149882 +0000 UTC m=+2346.587207734" watchObservedRunningTime="2026-07-16 17:04:48.085425431 +0000 UTC m=+2346.589483307" Jul 16 17:04:49.072291 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:04:49.072255 2577 generic.go:356] "Generic (PLEG): container finished" podID="14c15a24-306b-405e-bca2-873d7eeb9393" containerID="44dcdcd46222bb1a9f2531eb61703e65de6ffd6549e84517e4dfea02561c0e55" exitCode=0 Jul 16 17:04:49.072476 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:04:49.072339 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6f111d-82gc7" event={"ID":"14c15a24-306b-405e-bca2-873d7eeb9393","Type":"ContainerDied","Data":"44dcdcd46222bb1a9f2531eb61703e65de6ffd6549e84517e4dfea02561c0e55"} Jul 16 17:04:50.212127 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:04:50.212103 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6f111d-82gc7" Jul 16 17:04:50.222566 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:04:50.222533 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/14c15a24-306b-405e-bca2-873d7eeb9393-maas-api-service-ca\") pod \"14c15a24-306b-405e-bca2-873d7eeb9393\" (UID: \"14c15a24-306b-405e-bca2-873d7eeb9393\") " Jul 16 17:04:50.222881 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:04:50.222858 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14c15a24-306b-405e-bca2-873d7eeb9393-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "14c15a24-306b-405e-bca2-873d7eeb9393" (UID: "14c15a24-306b-405e-bca2-873d7eeb9393"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 17:04:50.222946 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:04:50.222929 2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/14c15a24-306b-405e-bca2-873d7eeb9393-maas-api-service-ca\") on node \"ip-10-0-142-101.ec2.internal\" DevicePath \"\"" Jul 16 17:04:51.084468 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:04:51.084434 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6f111d-82gc7" Jul 16 17:04:51.084468 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:04:51.084452 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6f111d-82gc7" event={"ID":"14c15a24-306b-405e-bca2-873d7eeb9393","Type":"ContainerDied","Data":"cd2996d636bd902f63e7dec3a897fbc24ef65040d1aba3240481befc621e0510"} Jul 16 17:04:51.084698 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:04:51.084504 2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cd2996d636bd902f63e7dec3a897fbc24ef65040d1aba3240481befc621e0510" Jul 16 17:05:12.908841 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:05:12.908807 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-6f111d-openshift-default-68f84858d9-z5fjb"] Jul 16 17:05:12.909379 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:05:12.909022 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-6f111d-openshift-default-68f84858d9-z5fjb" podUID="6c13a348-581f-4ea1-b44d-5e9bf37c98a2" containerName="istio-proxy" containerID="cri-o://3ff3ea7e2deaba2906f0756cbddfc41229f6f124fab7c502eb38ef03eeefd945" gracePeriod=30 Jul 16 17:05:13.343368 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:05:13.343325 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6da193-k9sq7"] Jul 16 17:05:13.343979 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:05:13.343962 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="14c15a24-306b-405e-bca2-873d7eeb9393" containerName="revoke-keys" Jul 16 17:05:13.344025 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:05:13.343984 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="14c15a24-306b-405e-bca2-873d7eeb9393" containerName="revoke-keys" Jul 16 17:05:13.344079 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:05:13.344068 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="14c15a24-306b-405e-bca2-873d7eeb9393" containerName="revoke-keys" Jul 16 17:05:13.347712 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:05:13.347688 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6da193-k9sq7" Jul 16 17:05:13.350355 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:05:13.350331 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dssfh\"" Jul 16 17:05:13.350498 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:05:13.350366 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 16 17:05:13.355050 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:05:13.355020 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6da193-k9sq7"] Jul 16 17:05:13.454964 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:05:13.454920 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f86a2759-c090-4b7b-bbcb-3b3036d51dab-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-6da193-k9sq7\" (UID: \"f86a2759-c090-4b7b-bbcb-3b3036d51dab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6da193-k9sq7" Jul 16 17:05:13.556345 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:05:13.556308 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f86a2759-c090-4b7b-bbcb-3b3036d51dab-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-6da193-k9sq7\" (UID: \"f86a2759-c090-4b7b-bbcb-3b3036d51dab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6da193-k9sq7" Jul 16 17:05:13.557016 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:05:13.556995 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f86a2759-c090-4b7b-bbcb-3b3036d51dab-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-6da193-k9sq7\" (UID: \"f86a2759-c090-4b7b-bbcb-3b3036d51dab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6da193-k9sq7" Jul 16 17:05:13.661115 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:05:13.661011 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6da193-k9sq7" Jul 16 17:05:13.804002 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:05:13.803976 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6da193-k9sq7"] Jul 16 17:05:13.806158 ip-10-0-142-101 kubenswrapper[2577]: W0716 17:05:13.806127 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf86a2759_c090_4b7b_bbcb_3b3036d51dab.slice/crio-22e47c4526d2542b0d8af61ed38d2cfacc32c098b5ed6fa2980cb8ca01de24d9 WatchSource:0}: Error finding container 22e47c4526d2542b0d8af61ed38d2cfacc32c098b5ed6fa2980cb8ca01de24d9: Status 404 returned error can't find the container with id 22e47c4526d2542b0d8af61ed38d2cfacc32c098b5ed6fa2980cb8ca01de24d9 Jul 16 17:05:14.208595 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:05:14.208560 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6da193-k9sq7" event={"ID":"f86a2759-c090-4b7b-bbcb-3b3036d51dab","Type":"ContainerStarted","Data":"74bd0b64180cd282921e858b9ed03d2a1b9fdbaa48fdf61e0111f7b33ea0d12d"} Jul 16 17:05:14.208981 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:05:14.208621 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6da193-k9sq7" event={"ID":"f86a2759-c090-4b7b-bbcb-3b3036d51dab","Type":"ContainerStarted","Data":"22e47c4526d2542b0d8af61ed38d2cfacc32c098b5ed6fa2980cb8ca01de24d9"} Jul 16 17:05:14.229270 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:05:14.229214 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6da193-k9sq7" podStartSLOduration=1.229198536 podStartE2EDuration="1.229198536s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:05:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:05:14.227668792 +0000 UTC m=+2372.731726642" watchObservedRunningTime="2026-07-16 17:05:14.229198536 +0000 UTC m=+2372.733256387" Jul 16 17:05:14.476345 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:05:14.476313 2577 patch_prober.go:28] interesting pod/e2e-api-a-6f111d-openshift-default-68f84858d9-z5fjb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 16 17:05:14.476556 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:05:14.476372 2577 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-6f111d-openshift-default-68f84858d9-z5fjb" podUID="6c13a348-581f-4ea1-b44d-5e9bf37c98a2" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 16 17:05:15.215538 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:05:15.215501 2577 generic.go:356] "Generic (PLEG): container finished" podID="f86a2759-c090-4b7b-bbcb-3b3036d51dab" containerID="74bd0b64180cd282921e858b9ed03d2a1b9fdbaa48fdf61e0111f7b33ea0d12d" exitCode=0 Jul 16 17:05:15.215945 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:05:15.215587 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6da193-k9sq7" event={"ID":"f86a2759-c090-4b7b-bbcb-3b3036d51dab","Type":"ContainerDied","Data":"74bd0b64180cd282921e858b9ed03d2a1b9fdbaa48fdf61e0111f7b33ea0d12d"} Jul 16 17:05:16.361419 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:05:16.361376 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6da193-k9sq7" Jul 16 17:05:16.489927 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:05:16.489824 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f86a2759-c090-4b7b-bbcb-3b3036d51dab-maas-api-service-ca\") pod \"f86a2759-c090-4b7b-bbcb-3b3036d51dab\" (UID: \"f86a2759-c090-4b7b-bbcb-3b3036d51dab\") " Jul 16 17:05:16.490197 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:05:16.490174 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f86a2759-c090-4b7b-bbcb-3b3036d51dab-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "f86a2759-c090-4b7b-bbcb-3b3036d51dab" (UID: "f86a2759-c090-4b7b-bbcb-3b3036d51dab"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 17:05:16.591459 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:05:16.591416 2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f86a2759-c090-4b7b-bbcb-3b3036d51dab-maas-api-service-ca\") on node \"ip-10-0-142-101.ec2.internal\" DevicePath \"\"" Jul 16 17:05:17.227623 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:05:17.227593 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6da193-k9sq7" Jul 16 17:05:17.227796 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:05:17.227624 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6da193-k9sq7" event={"ID":"f86a2759-c090-4b7b-bbcb-3b3036d51dab","Type":"ContainerDied","Data":"22e47c4526d2542b0d8af61ed38d2cfacc32c098b5ed6fa2980cb8ca01de24d9"} Jul 16 17:05:17.227796 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:05:17.227667 2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="22e47c4526d2542b0d8af61ed38d2cfacc32c098b5ed6fa2980cb8ca01de24d9" Jul 16 17:05:18.172589 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:05:18.172530 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-6f111d-openshift-default-68f84858d9-z5fjb" Jul 16 17:05:18.234319 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:05:18.234289 2577 generic.go:356] "Generic (PLEG): container finished" podID="6c13a348-581f-4ea1-b44d-5e9bf37c98a2" containerID="3ff3ea7e2deaba2906f0756cbddfc41229f6f124fab7c502eb38ef03eeefd945" exitCode=0 Jul 16 17:05:18.234532 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:05:18.234339 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-6f111d-openshift-default-68f84858d9-z5fjb" Jul 16 17:05:18.234532 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:05:18.234355 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-6f111d-openshift-default-68f84858d9-z5fjb" event={"ID":"6c13a348-581f-4ea1-b44d-5e9bf37c98a2","Type":"ContainerDied","Data":"3ff3ea7e2deaba2906f0756cbddfc41229f6f124fab7c502eb38ef03eeefd945"} Jul 16 17:05:18.234532 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:05:18.234415 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-6f111d-openshift-default-68f84858d9-z5fjb" event={"ID":"6c13a348-581f-4ea1-b44d-5e9bf37c98a2","Type":"ContainerDied","Data":"0cd074abf36c5c93d0f2ade4e1eaee8efb41edaa7013e4182bae7da472a2ebf7"} Jul 16 17:05:18.234532 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:05:18.234439 2577 scope.go:117] "RemoveContainer" containerID="3ff3ea7e2deaba2906f0756cbddfc41229f6f124fab7c502eb38ef03eeefd945" Jul 16 17:05:18.245172 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:05:18.245149 2577 scope.go:117] "RemoveContainer" containerID="ad7359e29d99821c5752f1bb8fa00195762c164f129ec5771ad13d0ff22071eb" Jul 16 17:05:18.253330 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:05:18.253300 2577 scope.go:117] "RemoveContainer" containerID="3ff3ea7e2deaba2906f0756cbddfc41229f6f124fab7c502eb38ef03eeefd945" Jul 16 17:05:18.253608 ip-10-0-142-101 kubenswrapper[2577]: E0716 17:05:18.253590 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ff3ea7e2deaba2906f0756cbddfc41229f6f124fab7c502eb38ef03eeefd945\": container with ID starting with 3ff3ea7e2deaba2906f0756cbddfc41229f6f124fab7c502eb38ef03eeefd945 not found: ID does not exist" containerID="3ff3ea7e2deaba2906f0756cbddfc41229f6f124fab7c502eb38ef03eeefd945" Jul 16 17:05:18.253671 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:05:18.253620 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ff3ea7e2deaba2906f0756cbddfc41229f6f124fab7c502eb38ef03eeefd945"} err="failed to get container status \"3ff3ea7e2deaba2906f0756cbddfc41229f6f124fab7c502eb38ef03eeefd945\": rpc error: code = NotFound desc = could not find container \"3ff3ea7e2deaba2906f0756cbddfc41229f6f124fab7c502eb38ef03eeefd945\": container with ID starting with 3ff3ea7e2deaba2906f0756cbddfc41229f6f124fab7c502eb38ef03eeefd945 not found: ID does not exist" Jul 16 17:05:18.253671 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:05:18.253638 2577 scope.go:117] "RemoveContainer" containerID="ad7359e29d99821c5752f1bb8fa00195762c164f129ec5771ad13d0ff22071eb" Jul 16 17:05:18.253862 ip-10-0-142-101 kubenswrapper[2577]: E0716 17:05:18.253845 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad7359e29d99821c5752f1bb8fa00195762c164f129ec5771ad13d0ff22071eb\": container with ID starting with ad7359e29d99821c5752f1bb8fa00195762c164f129ec5771ad13d0ff22071eb not found: ID does not exist" containerID="ad7359e29d99821c5752f1bb8fa00195762c164f129ec5771ad13d0ff22071eb" Jul 16 17:05:18.253910 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:05:18.253866 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad7359e29d99821c5752f1bb8fa00195762c164f129ec5771ad13d0ff22071eb"} err="failed to get container status \"ad7359e29d99821c5752f1bb8fa00195762c164f129ec5771ad13d0ff22071eb\": rpc error: code = NotFound desc = could not find container \"ad7359e29d99821c5752f1bb8fa00195762c164f129ec5771ad13d0ff22071eb\": container with ID starting with ad7359e29d99821c5752f1bb8fa00195762c164f129ec5771ad13d0ff22071eb not found: ID does not exist" Jul 16 17:05:18.307450 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:05:18.307415 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c13a348-581f-4ea1-b44d-5e9bf37c98a2-credential-socket\") pod \"6c13a348-581f-4ea1-b44d-5e9bf37c98a2\" (UID: \"6c13a348-581f-4ea1-b44d-5e9bf37c98a2\") " Jul 16 17:05:18.307650 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:05:18.307463 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c13a348-581f-4ea1-b44d-5e9bf37c98a2-workload-certs\") pod \"6c13a348-581f-4ea1-b44d-5e9bf37c98a2\" (UID: \"6c13a348-581f-4ea1-b44d-5e9bf37c98a2\") " Jul 16 17:05:18.307650 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:05:18.307483 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c13a348-581f-4ea1-b44d-5e9bf37c98a2-istiod-ca-cert\") pod \"6c13a348-581f-4ea1-b44d-5e9bf37c98a2\" (UID: \"6c13a348-581f-4ea1-b44d-5e9bf37c98a2\") " Jul 16 17:05:18.307650 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:05:18.307504 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c13a348-581f-4ea1-b44d-5e9bf37c98a2-istio-podinfo\") pod \"6c13a348-581f-4ea1-b44d-5e9bf37c98a2\" (UID: \"6c13a348-581f-4ea1-b44d-5e9bf37c98a2\") " Jul 16 17:05:18.307650 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:05:18.307629 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c13a348-581f-4ea1-b44d-5e9bf37c98a2-istio-data\") pod \"6c13a348-581f-4ea1-b44d-5e9bf37c98a2\" (UID: \"6c13a348-581f-4ea1-b44d-5e9bf37c98a2\") " Jul 16 17:05:18.307875 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:05:18.307693 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c13a348-581f-4ea1-b44d-5e9bf37c98a2-workload-socket\") pod \"6c13a348-581f-4ea1-b44d-5e9bf37c98a2\" (UID: \"6c13a348-581f-4ea1-b44d-5e9bf37c98a2\") " Jul 16 17:05:18.307875 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:05:18.307733 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c13a348-581f-4ea1-b44d-5e9bf37c98a2-istio-token\") pod \"6c13a348-581f-4ea1-b44d-5e9bf37c98a2\" (UID: \"6c13a348-581f-4ea1-b44d-5e9bf37c98a2\") " Jul 16 17:05:18.307875 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:05:18.307762 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c13a348-581f-4ea1-b44d-5e9bf37c98a2-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "6c13a348-581f-4ea1-b44d-5e9bf37c98a2" (UID: "6c13a348-581f-4ea1-b44d-5e9bf37c98a2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 17:05:18.307875 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:05:18.307782 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c13a348-581f-4ea1-b44d-5e9bf37c98a2-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "6c13a348-581f-4ea1-b44d-5e9bf37c98a2" (UID: "6c13a348-581f-4ea1-b44d-5e9bf37c98a2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 17:05:18.307875 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:05:18.307796 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wdknr\" (UniqueName: \"kubernetes.io/projected/6c13a348-581f-4ea1-b44d-5e9bf37c98a2-kube-api-access-wdknr\") pod \"6c13a348-581f-4ea1-b44d-5e9bf37c98a2\" (UID: \"6c13a348-581f-4ea1-b44d-5e9bf37c98a2\") " Jul 16 17:05:18.307875 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:05:18.307857 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c13a348-581f-4ea1-b44d-5e9bf37c98a2-istio-envoy\") pod \"6c13a348-581f-4ea1-b44d-5e9bf37c98a2\" (UID: \"6c13a348-581f-4ea1-b44d-5e9bf37c98a2\") " Jul 16 17:05:18.308216 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:05:18.307914 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c13a348-581f-4ea1-b44d-5e9bf37c98a2-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "6c13a348-581f-4ea1-b44d-5e9bf37c98a2" (UID: "6c13a348-581f-4ea1-b44d-5e9bf37c98a2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 17:05:18.308216 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:05:18.307990 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c13a348-581f-4ea1-b44d-5e9bf37c98a2-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "6c13a348-581f-4ea1-b44d-5e9bf37c98a2" (UID: "6c13a348-581f-4ea1-b44d-5e9bf37c98a2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 17:05:18.308307 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:05:18.308234 2577 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c13a348-581f-4ea1-b44d-5e9bf37c98a2-credential-socket\") on node \"ip-10-0-142-101.ec2.internal\" DevicePath \"\"" Jul 16 17:05:18.308307 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:05:18.308252 2577 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c13a348-581f-4ea1-b44d-5e9bf37c98a2-workload-certs\") on node \"ip-10-0-142-101.ec2.internal\" DevicePath \"\"" Jul 16 17:05:18.308307 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:05:18.308265 2577 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c13a348-581f-4ea1-b44d-5e9bf37c98a2-istiod-ca-cert\") on node \"ip-10-0-142-101.ec2.internal\" DevicePath \"\"" Jul 16 17:05:18.308307 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:05:18.308278 2577 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c13a348-581f-4ea1-b44d-5e9bf37c98a2-workload-socket\") on node \"ip-10-0-142-101.ec2.internal\" DevicePath \"\"" Jul 16 17:05:18.308573 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:05:18.308555 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c13a348-581f-4ea1-b44d-5e9bf37c98a2-istio-data" (OuterVolumeSpecName: "istio-data") pod "6c13a348-581f-4ea1-b44d-5e9bf37c98a2" (UID: "6c13a348-581f-4ea1-b44d-5e9bf37c98a2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 17:05:18.310195 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:05:18.310173 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6c13a348-581f-4ea1-b44d-5e9bf37c98a2-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "6c13a348-581f-4ea1-b44d-5e9bf37c98a2" (UID: "6c13a348-581f-4ea1-b44d-5e9bf37c98a2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 16 17:05:18.310287 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:05:18.310222 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c13a348-581f-4ea1-b44d-5e9bf37c98a2-istio-token" (OuterVolumeSpecName: "istio-token") pod "6c13a348-581f-4ea1-b44d-5e9bf37c98a2" (UID: "6c13a348-581f-4ea1-b44d-5e9bf37c98a2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 17:05:18.310401 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:05:18.310364 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c13a348-581f-4ea1-b44d-5e9bf37c98a2-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "6c13a348-581f-4ea1-b44d-5e9bf37c98a2" (UID: "6c13a348-581f-4ea1-b44d-5e9bf37c98a2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 17:05:18.310471 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:05:18.310363 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c13a348-581f-4ea1-b44d-5e9bf37c98a2-kube-api-access-wdknr" (OuterVolumeSpecName: "kube-api-access-wdknr") pod "6c13a348-581f-4ea1-b44d-5e9bf37c98a2" (UID: "6c13a348-581f-4ea1-b44d-5e9bf37c98a2"). InnerVolumeSpecName "kube-api-access-wdknr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 17:05:18.409776 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:05:18.409736 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wdknr\" (UniqueName: \"kubernetes.io/projected/6c13a348-581f-4ea1-b44d-5e9bf37c98a2-kube-api-access-wdknr\") on node \"ip-10-0-142-101.ec2.internal\" DevicePath \"\"" Jul 16 17:05:18.409776 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:05:18.409773 2577 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c13a348-581f-4ea1-b44d-5e9bf37c98a2-istio-envoy\") on node \"ip-10-0-142-101.ec2.internal\" DevicePath \"\"" Jul 16 17:05:18.409776 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:05:18.409782 2577 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c13a348-581f-4ea1-b44d-5e9bf37c98a2-istio-podinfo\") on node \"ip-10-0-142-101.ec2.internal\" DevicePath \"\"" Jul 16 17:05:18.410025 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:05:18.409790 2577 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c13a348-581f-4ea1-b44d-5e9bf37c98a2-istio-data\") on node \"ip-10-0-142-101.ec2.internal\" DevicePath \"\"" Jul 16 17:05:18.410025 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:05:18.409799 2577 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c13a348-581f-4ea1-b44d-5e9bf37c98a2-istio-token\") on node \"ip-10-0-142-101.ec2.internal\" DevicePath \"\"" Jul 16 17:05:18.559195 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:05:18.559153 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-6f111d-openshift-default-68f84858d9-z5fjb"] Jul 16 17:05:18.561770 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:05:18.561747 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-6f111d-openshift-default-68f84858d9-z5fjb"] Jul 16 17:05:20.023985 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:05:20.023943 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6c13a348-581f-4ea1-b44d-5e9bf37c98a2" path="/var/lib/kubelet/pods/6c13a348-581f-4ea1-b44d-5e9bf37c98a2/volumes" Jul 16 17:05:40.092816 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:05:40.092775 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-0cb2ac-openshift-default-7b64f4b4ff-wgjdt"] Jul 16 17:05:40.093206 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:05:40.093192 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6c13a348-581f-4ea1-b44d-5e9bf37c98a2" containerName="istio-proxy" Jul 16 17:05:40.093206 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:05:40.093204 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="6c13a348-581f-4ea1-b44d-5e9bf37c98a2" containerName="istio-proxy" Jul 16 17:05:40.093279 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:05:40.093214 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6c13a348-581f-4ea1-b44d-5e9bf37c98a2" containerName="istio-proxy" Jul 16 17:05:40.093279 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:05:40.093221 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="6c13a348-581f-4ea1-b44d-5e9bf37c98a2" containerName="istio-proxy" Jul 16 17:05:40.093279 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:05:40.093248 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f86a2759-c090-4b7b-bbcb-3b3036d51dab" containerName="revoke-keys" Jul 16 17:05:40.093279 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:05:40.093256 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="f86a2759-c090-4b7b-bbcb-3b3036d51dab" containerName="revoke-keys" Jul 16 17:05:40.093440 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:05:40.093327 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="6c13a348-581f-4ea1-b44d-5e9bf37c98a2" containerName="istio-proxy" Jul 16 17:05:40.093440 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:05:40.093335 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="f86a2759-c090-4b7b-bbcb-3b3036d51dab" containerName="revoke-keys" Jul 16 17:05:40.093440 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:05:40.093342 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="6c13a348-581f-4ea1-b44d-5e9bf37c98a2" containerName="istio-proxy" Jul 16 17:05:40.098095 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:05:40.098073 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-0cb2ac-openshift-default-7b64f4b4ff-wgjdt" Jul 16 17:05:40.102927 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:05:40.102904 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-0cb2ac-openshift-default-dockercfg-gdwm8\"" Jul 16 17:05:40.111830 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:05:40.111799 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-0cb2ac-openshift-default-7b64f4b4ff-wgjdt"] Jul 16 17:05:40.199651 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:05:40.199610 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b03d37c-5beb-485f-87d6-5cac22ef2716-istio-data\") pod \"e2e-shared-a-0cb2ac-openshift-default-7b64f4b4ff-wgjdt\" (UID: \"0b03d37c-5beb-485f-87d6-5cac22ef2716\") " pod="openshift-ingress/e2e-shared-a-0cb2ac-openshift-default-7b64f4b4ff-wgjdt" Jul 16 17:05:40.199835 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:05:40.199662 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b03d37c-5beb-485f-87d6-5cac22ef2716-istiod-ca-cert\") pod \"e2e-shared-a-0cb2ac-openshift-default-7b64f4b4ff-wgjdt\" (UID: \"0b03d37c-5beb-485f-87d6-5cac22ef2716\") " pod="openshift-ingress/e2e-shared-a-0cb2ac-openshift-default-7b64f4b4ff-wgjdt" Jul 16 17:05:40.199835 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:05:40.199748 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b03d37c-5beb-485f-87d6-5cac22ef2716-istio-token\") pod \"e2e-shared-a-0cb2ac-openshift-default-7b64f4b4ff-wgjdt\" (UID: \"0b03d37c-5beb-485f-87d6-5cac22ef2716\") " pod="openshift-ingress/e2e-shared-a-0cb2ac-openshift-default-7b64f4b4ff-wgjdt" Jul 16 17:05:40.199835 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:05:40.199800 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b03d37c-5beb-485f-87d6-5cac22ef2716-istio-envoy\") pod \"e2e-shared-a-0cb2ac-openshift-default-7b64f4b4ff-wgjdt\" (UID: \"0b03d37c-5beb-485f-87d6-5cac22ef2716\") " pod="openshift-ingress/e2e-shared-a-0cb2ac-openshift-default-7b64f4b4ff-wgjdt" Jul 16 17:05:40.199990 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:05:40.199837 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b03d37c-5beb-485f-87d6-5cac22ef2716-workload-certs\") pod \"e2e-shared-a-0cb2ac-openshift-default-7b64f4b4ff-wgjdt\" (UID: \"0b03d37c-5beb-485f-87d6-5cac22ef2716\") " pod="openshift-ingress/e2e-shared-a-0cb2ac-openshift-default-7b64f4b4ff-wgjdt" Jul 16 17:05:40.199990 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:05:40.199913 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b03d37c-5beb-485f-87d6-5cac22ef2716-istio-podinfo\") pod \"e2e-shared-a-0cb2ac-openshift-default-7b64f4b4ff-wgjdt\" (UID: \"0b03d37c-5beb-485f-87d6-5cac22ef2716\") " pod="openshift-ingress/e2e-shared-a-0cb2ac-openshift-default-7b64f4b4ff-wgjdt" Jul 16 17:05:40.199990 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:05:40.199947 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b03d37c-5beb-485f-87d6-5cac22ef2716-workload-socket\") pod \"e2e-shared-a-0cb2ac-openshift-default-7b64f4b4ff-wgjdt\" (UID: \"0b03d37c-5beb-485f-87d6-5cac22ef2716\") " pod="openshift-ingress/e2e-shared-a-0cb2ac-openshift-default-7b64f4b4ff-wgjdt" Jul 16 17:05:40.199990 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:05:40.199972 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-926hb\" (UniqueName: \"kubernetes.io/projected/0b03d37c-5beb-485f-87d6-5cac22ef2716-kube-api-access-926hb\") pod \"e2e-shared-a-0cb2ac-openshift-default-7b64f4b4ff-wgjdt\" (UID: \"0b03d37c-5beb-485f-87d6-5cac22ef2716\") " pod="openshift-ingress/e2e-shared-a-0cb2ac-openshift-default-7b64f4b4ff-wgjdt" Jul 16 17:05:40.200154 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:05:40.200001 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b03d37c-5beb-485f-87d6-5cac22ef2716-credential-socket\") pod \"e2e-shared-a-0cb2ac-openshift-default-7b64f4b4ff-wgjdt\" (UID: \"0b03d37c-5beb-485f-87d6-5cac22ef2716\") " pod="openshift-ingress/e2e-shared-a-0cb2ac-openshift-default-7b64f4b4ff-wgjdt" Jul 16 17:05:40.301155 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:05:40.301113 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-926hb\" (UniqueName: \"kubernetes.io/projected/0b03d37c-5beb-485f-87d6-5cac22ef2716-kube-api-access-926hb\") pod \"e2e-shared-a-0cb2ac-openshift-default-7b64f4b4ff-wgjdt\" (UID: \"0b03d37c-5beb-485f-87d6-5cac22ef2716\") " pod="openshift-ingress/e2e-shared-a-0cb2ac-openshift-default-7b64f4b4ff-wgjdt" Jul 16 17:05:40.301155 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:05:40.301159 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b03d37c-5beb-485f-87d6-5cac22ef2716-credential-socket\") pod \"e2e-shared-a-0cb2ac-openshift-default-7b64f4b4ff-wgjdt\" (UID: \"0b03d37c-5beb-485f-87d6-5cac22ef2716\") " pod="openshift-ingress/e2e-shared-a-0cb2ac-openshift-default-7b64f4b4ff-wgjdt" Jul 16 17:05:40.301456 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:05:40.301214 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b03d37c-5beb-485f-87d6-5cac22ef2716-istio-data\") pod \"e2e-shared-a-0cb2ac-openshift-default-7b64f4b4ff-wgjdt\" (UID: \"0b03d37c-5beb-485f-87d6-5cac22ef2716\") " pod="openshift-ingress/e2e-shared-a-0cb2ac-openshift-default-7b64f4b4ff-wgjdt" Jul 16 17:05:40.301456 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:05:40.301253 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b03d37c-5beb-485f-87d6-5cac22ef2716-istiod-ca-cert\") pod \"e2e-shared-a-0cb2ac-openshift-default-7b64f4b4ff-wgjdt\" (UID: \"0b03d37c-5beb-485f-87d6-5cac22ef2716\") " pod="openshift-ingress/e2e-shared-a-0cb2ac-openshift-default-7b64f4b4ff-wgjdt" Jul 16 17:05:40.301456 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:05:40.301283 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b03d37c-5beb-485f-87d6-5cac22ef2716-istio-token\") pod \"e2e-shared-a-0cb2ac-openshift-default-7b64f4b4ff-wgjdt\" (UID: \"0b03d37c-5beb-485f-87d6-5cac22ef2716\") " pod="openshift-ingress/e2e-shared-a-0cb2ac-openshift-default-7b64f4b4ff-wgjdt" Jul 16 17:05:40.301456 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:05:40.301318 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b03d37c-5beb-485f-87d6-5cac22ef2716-istio-envoy\") pod \"e2e-shared-a-0cb2ac-openshift-default-7b64f4b4ff-wgjdt\" (UID: \"0b03d37c-5beb-485f-87d6-5cac22ef2716\") " pod="openshift-ingress/e2e-shared-a-0cb2ac-openshift-default-7b64f4b4ff-wgjdt" Jul 16 17:05:40.301456 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:05:40.301350 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b03d37c-5beb-485f-87d6-5cac22ef2716-workload-certs\") pod \"e2e-shared-a-0cb2ac-openshift-default-7b64f4b4ff-wgjdt\" (UID: \"0b03d37c-5beb-485f-87d6-5cac22ef2716\") " pod="openshift-ingress/e2e-shared-a-0cb2ac-openshift-default-7b64f4b4ff-wgjdt" Jul 16 17:05:40.301456 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:05:40.301445 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b03d37c-5beb-485f-87d6-5cac22ef2716-istio-podinfo\") pod \"e2e-shared-a-0cb2ac-openshift-default-7b64f4b4ff-wgjdt\" (UID: \"0b03d37c-5beb-485f-87d6-5cac22ef2716\") " pod="openshift-ingress/e2e-shared-a-0cb2ac-openshift-default-7b64f4b4ff-wgjdt" Jul 16 17:05:40.301792 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:05:40.301498 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b03d37c-5beb-485f-87d6-5cac22ef2716-workload-socket\") pod \"e2e-shared-a-0cb2ac-openshift-default-7b64f4b4ff-wgjdt\" (UID: \"0b03d37c-5beb-485f-87d6-5cac22ef2716\") " pod="openshift-ingress/e2e-shared-a-0cb2ac-openshift-default-7b64f4b4ff-wgjdt" Jul 16 17:05:40.301792 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:05:40.301757 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b03d37c-5beb-485f-87d6-5cac22ef2716-istio-data\") pod \"e2e-shared-a-0cb2ac-openshift-default-7b64f4b4ff-wgjdt\" (UID: \"0b03d37c-5beb-485f-87d6-5cac22ef2716\") " pod="openshift-ingress/e2e-shared-a-0cb2ac-openshift-default-7b64f4b4ff-wgjdt" Jul 16 17:05:40.301903 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:05:40.301844 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b03d37c-5beb-485f-87d6-5cac22ef2716-credential-socket\") pod \"e2e-shared-a-0cb2ac-openshift-default-7b64f4b4ff-wgjdt\" (UID: \"0b03d37c-5beb-485f-87d6-5cac22ef2716\") " pod="openshift-ingress/e2e-shared-a-0cb2ac-openshift-default-7b64f4b4ff-wgjdt" Jul 16 17:05:40.301986 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:05:40.301956 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b03d37c-5beb-485f-87d6-5cac22ef2716-workload-certs\") pod \"e2e-shared-a-0cb2ac-openshift-default-7b64f4b4ff-wgjdt\" (UID: \"0b03d37c-5beb-485f-87d6-5cac22ef2716\") " pod="openshift-ingress/e2e-shared-a-0cb2ac-openshift-default-7b64f4b4ff-wgjdt" Jul 16 17:05:40.302104 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:05:40.302080 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b03d37c-5beb-485f-87d6-5cac22ef2716-workload-socket\") pod \"e2e-shared-a-0cb2ac-openshift-default-7b64f4b4ff-wgjdt\" (UID: \"0b03d37c-5beb-485f-87d6-5cac22ef2716\") " pod="openshift-ingress/e2e-shared-a-0cb2ac-openshift-default-7b64f4b4ff-wgjdt" Jul 16 17:05:40.302318 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:05:40.302297 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b03d37c-5beb-485f-87d6-5cac22ef2716-istiod-ca-cert\") pod \"e2e-shared-a-0cb2ac-openshift-default-7b64f4b4ff-wgjdt\" (UID: \"0b03d37c-5beb-485f-87d6-5cac22ef2716\") " pod="openshift-ingress/e2e-shared-a-0cb2ac-openshift-default-7b64f4b4ff-wgjdt" Jul 16 17:05:40.303672 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:05:40.303649 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b03d37c-5beb-485f-87d6-5cac22ef2716-istio-envoy\") pod \"e2e-shared-a-0cb2ac-openshift-default-7b64f4b4ff-wgjdt\" (UID: \"0b03d37c-5beb-485f-87d6-5cac22ef2716\") " pod="openshift-ingress/e2e-shared-a-0cb2ac-openshift-default-7b64f4b4ff-wgjdt" Jul 16 17:05:40.304047 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:05:40.304025 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b03d37c-5beb-485f-87d6-5cac22ef2716-istio-podinfo\") pod \"e2e-shared-a-0cb2ac-openshift-default-7b64f4b4ff-wgjdt\" (UID: \"0b03d37c-5beb-485f-87d6-5cac22ef2716\") " pod="openshift-ingress/e2e-shared-a-0cb2ac-openshift-default-7b64f4b4ff-wgjdt" Jul 16 17:05:40.310396 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:05:40.310352 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-926hb\" (UniqueName: \"kubernetes.io/projected/0b03d37c-5beb-485f-87d6-5cac22ef2716-kube-api-access-926hb\") pod \"e2e-shared-a-0cb2ac-openshift-default-7b64f4b4ff-wgjdt\" (UID: \"0b03d37c-5beb-485f-87d6-5cac22ef2716\") " pod="openshift-ingress/e2e-shared-a-0cb2ac-openshift-default-7b64f4b4ff-wgjdt" Jul 16 17:05:40.310503 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:05:40.310416 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b03d37c-5beb-485f-87d6-5cac22ef2716-istio-token\") pod \"e2e-shared-a-0cb2ac-openshift-default-7b64f4b4ff-wgjdt\" (UID: \"0b03d37c-5beb-485f-87d6-5cac22ef2716\") " pod="openshift-ingress/e2e-shared-a-0cb2ac-openshift-default-7b64f4b4ff-wgjdt" Jul 16 17:05:40.409440 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:05:40.409327 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-0cb2ac-openshift-default-7b64f4b4ff-wgjdt" Jul 16 17:05:40.578186 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:05:40.578161 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-0cb2ac-openshift-default-7b64f4b4ff-wgjdt"] Jul 16 17:05:40.580104 ip-10-0-142-101 kubenswrapper[2577]: W0716 17:05:40.580074 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b03d37c_5beb_485f_87d6_5cac22ef2716.slice/crio-e3e935f995b523e4efc19eb432bec24cc74d987b9145e54f29bfb2c849459d62 WatchSource:0}: Error finding container e3e935f995b523e4efc19eb432bec24cc74d987b9145e54f29bfb2c849459d62: Status 404 returned error can't find the container with id e3e935f995b523e4efc19eb432bec24cc74d987b9145e54f29bfb2c849459d62 Jul 16 17:05:40.588651 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:05:40.586510 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 16 17:05:40.588651 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:05:40.586641 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 16 17:05:40.588651 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:05:40.586727 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 16 17:05:41.359474 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:05:41.359429 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-0cb2ac-openshift-default-7b64f4b4ff-wgjdt" event={"ID":"0b03d37c-5beb-485f-87d6-5cac22ef2716","Type":"ContainerStarted","Data":"f09156435823bd48904349f7159e5b30ce1e4cf90b20235141beb0d81c17be5c"} Jul 16 17:05:41.359851 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:05:41.359495 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-0cb2ac-openshift-default-7b64f4b4ff-wgjdt" event={"ID":"0b03d37c-5beb-485f-87d6-5cac22ef2716","Type":"ContainerStarted","Data":"e3e935f995b523e4efc19eb432bec24cc74d987b9145e54f29bfb2c849459d62"} Jul 16 17:05:41.385223 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:05:41.385168 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-0cb2ac-openshift-default-7b64f4b4ff-wgjdt" podStartSLOduration=1.385152374 podStartE2EDuration="1.385152374s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:05:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:05:41.382943364 +0000 UTC m=+2399.887001216" watchObservedRunningTime="2026-07-16 17:05:41.385152374 +0000 UTC m=+2399.889210225" Jul 16 17:05:41.409664 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:05:41.409626 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-0cb2ac-openshift-default-7b64f4b4ff-wgjdt" Jul 16 17:05:41.414908 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:05:41.414882 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-0cb2ac-openshift-default-7b64f4b4ff-wgjdt" Jul 16 17:05:42.134946 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:05:42.134918 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9bjst_b20b154f-1a4c-4b25-86db-aebe2d4ffc08/ovn-acl-logging/0.log" Jul 16 17:05:42.141186 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:05:42.141161 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9bjst_b20b154f-1a4c-4b25-86db-aebe2d4ffc08/ovn-acl-logging/0.log" Jul 16 17:05:42.366479 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:05:42.366443 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-0cb2ac-openshift-default-7b64f4b4ff-wgjdt" Jul 16 17:05:42.367438 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:05:42.367423 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-0cb2ac-openshift-default-7b64f4b4ff-wgjdt" Jul 16 17:05:48.550318 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:05:48.550272 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-734b32-openshift-default-7d6f8b88c5-4rb8c"] Jul 16 17:05:48.553991 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:05:48.553970 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-734b32-openshift-default-7d6f8b88c5-4rb8c" Jul 16 17:05:48.556846 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:05:48.556824 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-734b32-openshift-default-dockercfg-9h2k4\"" Jul 16 17:05:48.567743 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:05:48.567712 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-734b32-openshift-default-7d6f8b88c5-4rb8c"] Jul 16 17:05:48.582859 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:05:48.582828 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gz5ts\" (UniqueName: \"kubernetes.io/projected/37f54670-7bd6-46c3-bb0e-3218fb0de41d-kube-api-access-gz5ts\") pod \"e2e-shared-b-734b32-openshift-default-7d6f8b88c5-4rb8c\" (UID: \"37f54670-7bd6-46c3-bb0e-3218fb0de41d\") " pod="openshift-ingress/e2e-shared-b-734b32-openshift-default-7d6f8b88c5-4rb8c" Jul 16 17:05:48.583018 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:05:48.582878 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37f54670-7bd6-46c3-bb0e-3218fb0de41d-credential-socket\") pod \"e2e-shared-b-734b32-openshift-default-7d6f8b88c5-4rb8c\" (UID: \"37f54670-7bd6-46c3-bb0e-3218fb0de41d\") " pod="openshift-ingress/e2e-shared-b-734b32-openshift-default-7d6f8b88c5-4rb8c" Jul 16 17:05:48.583066 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:05:48.583027 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37f54670-7bd6-46c3-bb0e-3218fb0de41d-workload-certs\") pod \"e2e-shared-b-734b32-openshift-default-7d6f8b88c5-4rb8c\" (UID: \"37f54670-7bd6-46c3-bb0e-3218fb0de41d\") " pod="openshift-ingress/e2e-shared-b-734b32-openshift-default-7d6f8b88c5-4rb8c" Jul 16 17:05:48.583066 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:05:48.583058 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37f54670-7bd6-46c3-bb0e-3218fb0de41d-istiod-ca-cert\") pod \"e2e-shared-b-734b32-openshift-default-7d6f8b88c5-4rb8c\" (UID: \"37f54670-7bd6-46c3-bb0e-3218fb0de41d\") " pod="openshift-ingress/e2e-shared-b-734b32-openshift-default-7d6f8b88c5-4rb8c" Jul 16 17:05:48.583179 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:05:48.583083 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37f54670-7bd6-46c3-bb0e-3218fb0de41d-istio-data\") pod \"e2e-shared-b-734b32-openshift-default-7d6f8b88c5-4rb8c\" (UID: \"37f54670-7bd6-46c3-bb0e-3218fb0de41d\") " pod="openshift-ingress/e2e-shared-b-734b32-openshift-default-7d6f8b88c5-4rb8c" Jul 16 17:05:48.583179 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:05:48.583127 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37f54670-7bd6-46c3-bb0e-3218fb0de41d-istio-token\") pod \"e2e-shared-b-734b32-openshift-default-7d6f8b88c5-4rb8c\" (UID: \"37f54670-7bd6-46c3-bb0e-3218fb0de41d\") " pod="openshift-ingress/e2e-shared-b-734b32-openshift-default-7d6f8b88c5-4rb8c" Jul 16 17:05:48.583264 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:05:48.583244 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37f54670-7bd6-46c3-bb0e-3218fb0de41d-workload-socket\") pod \"e2e-shared-b-734b32-openshift-default-7d6f8b88c5-4rb8c\" (UID: \"37f54670-7bd6-46c3-bb0e-3218fb0de41d\") " pod="openshift-ingress/e2e-shared-b-734b32-openshift-default-7d6f8b88c5-4rb8c" Jul 16 17:05:48.583307 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:05:48.583287 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37f54670-7bd6-46c3-bb0e-3218fb0de41d-istio-podinfo\") pod \"e2e-shared-b-734b32-openshift-default-7d6f8b88c5-4rb8c\" (UID: \"37f54670-7bd6-46c3-bb0e-3218fb0de41d\") " pod="openshift-ingress/e2e-shared-b-734b32-openshift-default-7d6f8b88c5-4rb8c" Jul 16 17:05:48.583352 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:05:48.583318 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37f54670-7bd6-46c3-bb0e-3218fb0de41d-istio-envoy\") pod \"e2e-shared-b-734b32-openshift-default-7d6f8b88c5-4rb8c\" (UID: \"37f54670-7bd6-46c3-bb0e-3218fb0de41d\") " pod="openshift-ingress/e2e-shared-b-734b32-openshift-default-7d6f8b88c5-4rb8c" Jul 16 17:05:48.684777 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:05:48.684733 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37f54670-7bd6-46c3-bb0e-3218fb0de41d-istio-data\") pod \"e2e-shared-b-734b32-openshift-default-7d6f8b88c5-4rb8c\" (UID: \"37f54670-7bd6-46c3-bb0e-3218fb0de41d\") " pod="openshift-ingress/e2e-shared-b-734b32-openshift-default-7d6f8b88c5-4rb8c" Jul 16 17:05:48.684980 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:05:48.684779 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37f54670-7bd6-46c3-bb0e-3218fb0de41d-istio-token\") pod \"e2e-shared-b-734b32-openshift-default-7d6f8b88c5-4rb8c\" (UID: \"37f54670-7bd6-46c3-bb0e-3218fb0de41d\") " pod="openshift-ingress/e2e-shared-b-734b32-openshift-default-7d6f8b88c5-4rb8c" Jul 16 17:05:48.684980 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:05:48.684829 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37f54670-7bd6-46c3-bb0e-3218fb0de41d-workload-socket\") pod \"e2e-shared-b-734b32-openshift-default-7d6f8b88c5-4rb8c\" (UID: \"37f54670-7bd6-46c3-bb0e-3218fb0de41d\") " pod="openshift-ingress/e2e-shared-b-734b32-openshift-default-7d6f8b88c5-4rb8c" Jul 16 17:05:48.684980 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:05:48.684858 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37f54670-7bd6-46c3-bb0e-3218fb0de41d-istio-podinfo\") pod \"e2e-shared-b-734b32-openshift-default-7d6f8b88c5-4rb8c\" (UID: \"37f54670-7bd6-46c3-bb0e-3218fb0de41d\") " pod="openshift-ingress/e2e-shared-b-734b32-openshift-default-7d6f8b88c5-4rb8c" Jul 16 17:05:48.684980 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:05:48.684883 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37f54670-7bd6-46c3-bb0e-3218fb0de41d-istio-envoy\") pod \"e2e-shared-b-734b32-openshift-default-7d6f8b88c5-4rb8c\" (UID: \"37f54670-7bd6-46c3-bb0e-3218fb0de41d\") " pod="openshift-ingress/e2e-shared-b-734b32-openshift-default-7d6f8b88c5-4rb8c" Jul 16 17:05:48.684980 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:05:48.684927 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gz5ts\" (UniqueName: \"kubernetes.io/projected/37f54670-7bd6-46c3-bb0e-3218fb0de41d-kube-api-access-gz5ts\") pod \"e2e-shared-b-734b32-openshift-default-7d6f8b88c5-4rb8c\" (UID: \"37f54670-7bd6-46c3-bb0e-3218fb0de41d\") " pod="openshift-ingress/e2e-shared-b-734b32-openshift-default-7d6f8b88c5-4rb8c" Jul 16 17:05:48.684980 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:05:48.684961 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37f54670-7bd6-46c3-bb0e-3218fb0de41d-credential-socket\") pod \"e2e-shared-b-734b32-openshift-default-7d6f8b88c5-4rb8c\" (UID: \"37f54670-7bd6-46c3-bb0e-3218fb0de41d\") " pod="openshift-ingress/e2e-shared-b-734b32-openshift-default-7d6f8b88c5-4rb8c" Jul 16 17:05:48.685312 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:05:48.685086 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37f54670-7bd6-46c3-bb0e-3218fb0de41d-workload-certs\") pod \"e2e-shared-b-734b32-openshift-default-7d6f8b88c5-4rb8c\" (UID: \"37f54670-7bd6-46c3-bb0e-3218fb0de41d\") " pod="openshift-ingress/e2e-shared-b-734b32-openshift-default-7d6f8b88c5-4rb8c" Jul 16 17:05:48.685312 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:05:48.685113 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37f54670-7bd6-46c3-bb0e-3218fb0de41d-istiod-ca-cert\") pod \"e2e-shared-b-734b32-openshift-default-7d6f8b88c5-4rb8c\" (UID: \"37f54670-7bd6-46c3-bb0e-3218fb0de41d\") " pod="openshift-ingress/e2e-shared-b-734b32-openshift-default-7d6f8b88c5-4rb8c" Jul 16 17:05:48.685312 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:05:48.685219 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37f54670-7bd6-46c3-bb0e-3218fb0de41d-istio-data\") pod \"e2e-shared-b-734b32-openshift-default-7d6f8b88c5-4rb8c\" (UID: \"37f54670-7bd6-46c3-bb0e-3218fb0de41d\") " pod="openshift-ingress/e2e-shared-b-734b32-openshift-default-7d6f8b88c5-4rb8c" Jul 16 17:05:48.685526 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:05:48.685432 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37f54670-7bd6-46c3-bb0e-3218fb0de41d-workload-socket\") pod \"e2e-shared-b-734b32-openshift-default-7d6f8b88c5-4rb8c\" (UID: \"37f54670-7bd6-46c3-bb0e-3218fb0de41d\") " pod="openshift-ingress/e2e-shared-b-734b32-openshift-default-7d6f8b88c5-4rb8c" Jul 16 17:05:48.685526 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:05:48.685463 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37f54670-7bd6-46c3-bb0e-3218fb0de41d-workload-certs\") pod \"e2e-shared-b-734b32-openshift-default-7d6f8b88c5-4rb8c\" (UID: \"37f54670-7bd6-46c3-bb0e-3218fb0de41d\") " pod="openshift-ingress/e2e-shared-b-734b32-openshift-default-7d6f8b88c5-4rb8c" Jul 16 17:05:48.685526 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:05:48.685509 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37f54670-7bd6-46c3-bb0e-3218fb0de41d-credential-socket\") pod \"e2e-shared-b-734b32-openshift-default-7d6f8b88c5-4rb8c\" (UID: \"37f54670-7bd6-46c3-bb0e-3218fb0de41d\") " pod="openshift-ingress/e2e-shared-b-734b32-openshift-default-7d6f8b88c5-4rb8c" Jul 16 17:05:48.685787 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:05:48.685767 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37f54670-7bd6-46c3-bb0e-3218fb0de41d-istiod-ca-cert\") pod \"e2e-shared-b-734b32-openshift-default-7d6f8b88c5-4rb8c\" (UID: \"37f54670-7bd6-46c3-bb0e-3218fb0de41d\") " pod="openshift-ingress/e2e-shared-b-734b32-openshift-default-7d6f8b88c5-4rb8c" Jul 16 17:05:48.687308 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:05:48.687289 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37f54670-7bd6-46c3-bb0e-3218fb0de41d-istio-envoy\") pod \"e2e-shared-b-734b32-openshift-default-7d6f8b88c5-4rb8c\" (UID: \"37f54670-7bd6-46c3-bb0e-3218fb0de41d\") " pod="openshift-ingress/e2e-shared-b-734b32-openshift-default-7d6f8b88c5-4rb8c" Jul 16 17:05:48.687533 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:05:48.687513 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37f54670-7bd6-46c3-bb0e-3218fb0de41d-istio-podinfo\") pod \"e2e-shared-b-734b32-openshift-default-7d6f8b88c5-4rb8c\" (UID: \"37f54670-7bd6-46c3-bb0e-3218fb0de41d\") " pod="openshift-ingress/e2e-shared-b-734b32-openshift-default-7d6f8b88c5-4rb8c" Jul 16 17:05:48.693468 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:05:48.693440 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37f54670-7bd6-46c3-bb0e-3218fb0de41d-istio-token\") pod \"e2e-shared-b-734b32-openshift-default-7d6f8b88c5-4rb8c\" (UID: \"37f54670-7bd6-46c3-bb0e-3218fb0de41d\") " pod="openshift-ingress/e2e-shared-b-734b32-openshift-default-7d6f8b88c5-4rb8c" Jul 16 17:05:48.693681 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:05:48.693663 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gz5ts\" (UniqueName: \"kubernetes.io/projected/37f54670-7bd6-46c3-bb0e-3218fb0de41d-kube-api-access-gz5ts\") pod \"e2e-shared-b-734b32-openshift-default-7d6f8b88c5-4rb8c\" (UID: \"37f54670-7bd6-46c3-bb0e-3218fb0de41d\") " pod="openshift-ingress/e2e-shared-b-734b32-openshift-default-7d6f8b88c5-4rb8c" Jul 16 17:05:48.864271 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:05:48.864232 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-734b32-openshift-default-7d6f8b88c5-4rb8c" Jul 16 17:05:49.012098 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:05:49.012067 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-734b32-openshift-default-7d6f8b88c5-4rb8c"] Jul 16 17:05:49.013229 ip-10-0-142-101 kubenswrapper[2577]: W0716 17:05:49.013205 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37f54670_7bd6_46c3_bb0e_3218fb0de41d.slice/crio-ece42829aea266bd9c4bc34a3e8894a8ec8915fab7d1346559e0a08fc77f33b2 WatchSource:0}: Error finding container ece42829aea266bd9c4bc34a3e8894a8ec8915fab7d1346559e0a08fc77f33b2: Status 404 returned error can't find the container with id ece42829aea266bd9c4bc34a3e8894a8ec8915fab7d1346559e0a08fc77f33b2 Jul 16 17:05:49.015667 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:05:49.015637 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 16 17:05:49.015757 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:05:49.015703 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 16 17:05:49.015757 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:05:49.015731 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 16 17:05:49.408010 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:05:49.407971 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-734b32-openshift-default-7d6f8b88c5-4rb8c" event={"ID":"37f54670-7bd6-46c3-bb0e-3218fb0de41d","Type":"ContainerStarted","Data":"ebd9a81baf09efeac6271f27deaec97294519f0ff968c23fca4c78ca1b27ccb7"} Jul 16 17:05:49.408198 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:05:49.408034 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-734b32-openshift-default-7d6f8b88c5-4rb8c" event={"ID":"37f54670-7bd6-46c3-bb0e-3218fb0de41d","Type":"ContainerStarted","Data":"ece42829aea266bd9c4bc34a3e8894a8ec8915fab7d1346559e0a08fc77f33b2"} Jul 16 17:05:49.429821 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:05:49.429772 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-734b32-openshift-default-7d6f8b88c5-4rb8c" podStartSLOduration=1.429755243 podStartE2EDuration="1.429755243s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:05:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:05:49.428183748 +0000 UTC m=+2407.932241599" watchObservedRunningTime="2026-07-16 17:05:49.429755243 +0000 UTC m=+2407.933813093" Jul 16 17:05:49.864700 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:05:49.864662 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-734b32-openshift-default-7d6f8b88c5-4rb8c" Jul 16 17:05:50.870026 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:05:50.869994 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-734b32-openshift-default-7d6f8b88c5-4rb8c" Jul 16 17:05:51.420274 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:05:51.420246 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-734b32-openshift-default-7d6f8b88c5-4rb8c" Jul 16 17:05:51.421204 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:05:51.421184 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-734b32-openshift-default-7d6f8b88c5-4rb8c" Jul 16 17:06:01.489426 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:06:01.489371 2577 generic.go:356] "Generic (PLEG): container finished" podID="0b03d37c-5beb-485f-87d6-5cac22ef2716" containerID="f09156435823bd48904349f7159e5b30ce1e4cf90b20235141beb0d81c17be5c" exitCode=0 Jul 16 17:06:01.489982 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:06:01.489449 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-0cb2ac-openshift-default-7b64f4b4ff-wgjdt" event={"ID":"0b03d37c-5beb-485f-87d6-5cac22ef2716","Type":"ContainerDied","Data":"f09156435823bd48904349f7159e5b30ce1e4cf90b20235141beb0d81c17be5c"} Jul 16 17:06:01.490054 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:06:01.490017 2577 scope.go:117] "RemoveContainer" containerID="f09156435823bd48904349f7159e5b30ce1e4cf90b20235141beb0d81c17be5c" Jul 16 17:06:01.491122 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:06:01.491097 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 16 17:06:01.491193 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:06:01.491168 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 16 17:06:01.491239 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:06:01.491200 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 16 17:06:02.497618 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:06:02.497575 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-0cb2ac-openshift-default-7b64f4b4ff-wgjdt" event={"ID":"0b03d37c-5beb-485f-87d6-5cac22ef2716","Type":"ContainerStarted","Data":"a58f490144679db795a5a0b0e83e24ca4eb5224ab96051382cbdb4fa772097df"} Jul 16 17:06:03.410812 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:06:03.410719 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-0cb2ac-openshift-default-7b64f4b4ff-wgjdt" Jul 16 17:06:03.410956 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:06:03.410840 2577 patch_prober.go:28] interesting pod/e2e-shared-a-0cb2ac-openshift-default-7b64f4b4ff-wgjdt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Jul 16 17:06:03.410956 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:06:03.410877 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-0cb2ac-openshift-default-7b64f4b4ff-wgjdt" podUID="0b03d37c-5beb-485f-87d6-5cac22ef2716" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Jul 16 17:06:04.410519 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:06:04.410483 2577 patch_prober.go:28] interesting pod/e2e-shared-a-0cb2ac-openshift-default-7b64f4b4ff-wgjdt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Jul 16 17:06:04.410910 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:06:04.410539 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-0cb2ac-openshift-default-7b64f4b4ff-wgjdt" podUID="0b03d37c-5beb-485f-87d6-5cac22ef2716" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Jul 16 17:06:05.410715 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:06:05.410673 2577 patch_prober.go:28] interesting pod/e2e-shared-a-0cb2ac-openshift-default-7b64f4b4ff-wgjdt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Jul 16 17:06:05.411206 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:06:05.410727 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-0cb2ac-openshift-default-7b64f4b4ff-wgjdt" podUID="0b03d37c-5beb-485f-87d6-5cac22ef2716" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Jul 16 17:06:06.410317 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:06:06.410271 2577 patch_prober.go:28] interesting pod/e2e-shared-a-0cb2ac-openshift-default-7b64f4b4ff-wgjdt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Jul 16 17:06:06.410527 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:06:06.410338 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-0cb2ac-openshift-default-7b64f4b4ff-wgjdt" podUID="0b03d37c-5beb-485f-87d6-5cac22ef2716" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Jul 16 17:06:07.413834 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:06:07.413791 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-0cb2ac-openshift-default-7b64f4b4ff-wgjdt" Jul 16 17:06:07.414379 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:06:07.414357 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-0cb2ac-openshift-default-7b64f4b4ff-wgjdt" Jul 16 17:06:07.415186 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:06:07.415164 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-0cb2ac-openshift-default-7b64f4b4ff-wgjdt" Jul 16 17:06:07.462590 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:06:07.462538 2577 patch_prober.go:28] interesting pod/e2e-shared-b-734b32-openshift-default-7d6f8b88c5-4rb8c container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 16 17:06:07.462811 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:06:07.462614 2577 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-734b32-openshift-default-7d6f8b88c5-4rb8c" podUID="37f54670-7bd6-46c3-bb0e-3218fb0de41d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 16 17:06:08.534104 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:06:08.533991 2577 generic.go:356] "Generic (PLEG): container finished" podID="37f54670-7bd6-46c3-bb0e-3218fb0de41d" containerID="ebd9a81baf09efeac6271f27deaec97294519f0ff968c23fca4c78ca1b27ccb7" exitCode=0 Jul 16 17:06:08.534104 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:06:08.534073 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-734b32-openshift-default-7d6f8b88c5-4rb8c" event={"ID":"37f54670-7bd6-46c3-bb0e-3218fb0de41d","Type":"ContainerDied","Data":"ebd9a81baf09efeac6271f27deaec97294519f0ff968c23fca4c78ca1b27ccb7"} Jul 16 17:06:08.534652 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:06:08.534636 2577 scope.go:117] "RemoveContainer" containerID="ebd9a81baf09efeac6271f27deaec97294519f0ff968c23fca4c78ca1b27ccb7" Jul 16 17:06:08.535771 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:06:08.535735 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 16 17:06:08.535880 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:06:08.535842 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 16 17:06:08.535923 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:06:08.535886 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 16 17:06:09.544658 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:06:09.544607 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-734b32-openshift-default-7d6f8b88c5-4rb8c" event={"ID":"37f54670-7bd6-46c3-bb0e-3218fb0de41d","Type":"ContainerStarted","Data":"06da9e2f7375f8e43c5bb8120d821a186bc16256d90f4c9dd69fc866887b2637"} Jul 16 17:06:09.865523 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:06:09.865465 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-734b32-openshift-default-7d6f8b88c5-4rb8c" Jul 16 17:06:09.865791 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:06:09.865753 2577 patch_prober.go:28] interesting pod/e2e-shared-b-734b32-openshift-default-7d6f8b88c5-4rb8c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Jul 16 17:06:09.865932 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:06:09.865818 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-734b32-openshift-default-7d6f8b88c5-4rb8c" podUID="37f54670-7bd6-46c3-bb0e-3218fb0de41d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Jul 16 17:06:10.865594 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:06:10.865557 2577 patch_prober.go:28] interesting pod/e2e-shared-b-734b32-openshift-default-7d6f8b88c5-4rb8c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Jul 16 17:06:10.866174 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:06:10.866134 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-734b32-openshift-default-7d6f8b88c5-4rb8c" podUID="37f54670-7bd6-46c3-bb0e-3218fb0de41d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Jul 16 17:06:11.865458 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:06:11.865421 2577 patch_prober.go:28] interesting pod/e2e-shared-b-734b32-openshift-default-7d6f8b88c5-4rb8c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Jul 16 17:06:11.865636 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:06:11.865493 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-734b32-openshift-default-7d6f8b88c5-4rb8c" podUID="37f54670-7bd6-46c3-bb0e-3218fb0de41d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Jul 16 17:06:12.865116 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:06:12.865071 2577 patch_prober.go:28] interesting pod/e2e-shared-b-734b32-openshift-default-7d6f8b88c5-4rb8c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Jul 16 17:06:12.865347 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:06:12.865144 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-734b32-openshift-default-7d6f8b88c5-4rb8c" podUID="37f54670-7bd6-46c3-bb0e-3218fb0de41d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Jul 16 17:06:13.868545 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:06:13.868512 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-734b32-openshift-default-7d6f8b88c5-4rb8c" Jul 16 17:06:13.868969 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:06:13.868767 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-734b32-openshift-default-7d6f8b88c5-4rb8c" Jul 16 17:06:13.869478 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:06:13.869459 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-734b32-openshift-default-7d6f8b88c5-4rb8c" Jul 16 17:10:42.172968 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:10:42.172885 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9bjst_b20b154f-1a4c-4b25-86db-aebe2d4ffc08/ovn-acl-logging/0.log" Jul 16 17:10:42.179450 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:10:42.179429 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9bjst_b20b154f-1a4c-4b25-86db-aebe2d4ffc08/ovn-acl-logging/0.log" Jul 16 17:11:45.659548 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:11:45.659506 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-0cb2ac-b9z2h"] Jul 16 17:11:45.663315 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:11:45.663290 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-0cb2ac-b9z2h" Jul 16 17:11:45.665914 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:11:45.665883 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 16 17:11:45.666032 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:11:45.665943 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dssfh\"" Jul 16 17:11:45.674114 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:11:45.674089 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-0cb2ac-b9z2h"] Jul 16 17:11:45.827118 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:11:45.827081 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ebce8a2-b83e-4d98-88aa-90e2981f472f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-0cb2ac-b9z2h\" (UID: \"1ebce8a2-b83e-4d98-88aa-90e2981f472f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-0cb2ac-b9z2h" Jul 16 17:11:45.928274 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:11:45.928181 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ebce8a2-b83e-4d98-88aa-90e2981f472f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-0cb2ac-b9z2h\" (UID: \"1ebce8a2-b83e-4d98-88aa-90e2981f472f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-0cb2ac-b9z2h" Jul 16 17:11:45.928868 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:11:45.928845 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ebce8a2-b83e-4d98-88aa-90e2981f472f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-0cb2ac-b9z2h\" (UID: \"1ebce8a2-b83e-4d98-88aa-90e2981f472f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-0cb2ac-b9z2h" Jul 16 17:11:45.975100 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:11:45.975066 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-0cb2ac-b9z2h" Jul 16 17:11:46.120690 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:11:46.120661 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-0cb2ac-b9z2h"] Jul 16 17:11:46.122566 ip-10-0-142-101 kubenswrapper[2577]: W0716 17:11:46.122535 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ebce8a2_b83e_4d98_88aa_90e2981f472f.slice/crio-8337c2c42a4fbcfccbc684e8796875d146d9d0694dcd79ff1c2432a04714ad6e WatchSource:0}: Error finding container 8337c2c42a4fbcfccbc684e8796875d146d9d0694dcd79ff1c2432a04714ad6e: Status 404 returned error can't find the container with id 8337c2c42a4fbcfccbc684e8796875d146d9d0694dcd79ff1c2432a04714ad6e Jul 16 17:11:46.445357 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:11:46.445316 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-0cb2ac-b9z2h" event={"ID":"1ebce8a2-b83e-4d98-88aa-90e2981f472f","Type":"ContainerStarted","Data":"8337c2c42a4fbcfccbc684e8796875d146d9d0694dcd79ff1c2432a04714ad6e"} Jul 16 17:11:47.452019 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:11:47.451983 2577 generic.go:356] "Generic (PLEG): container finished" podID="1ebce8a2-b83e-4d98-88aa-90e2981f472f" containerID="30b2f9cba66e67a59431a33e0b3d5582e665b17aa5b3c5790267180fb24398fd" exitCode=0 Jul 16 17:11:47.452451 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:11:47.452050 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-0cb2ac-b9z2h" event={"ID":"1ebce8a2-b83e-4d98-88aa-90e2981f472f","Type":"ContainerDied","Data":"30b2f9cba66e67a59431a33e0b3d5582e665b17aa5b3c5790267180fb24398fd"} Jul 16 17:11:48.592903 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:11:48.592878 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-0cb2ac-b9z2h" Jul 16 17:11:48.755625 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:11:48.755515 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ebce8a2-b83e-4d98-88aa-90e2981f472f-maas-api-service-ca\") pod \"1ebce8a2-b83e-4d98-88aa-90e2981f472f\" (UID: \"1ebce8a2-b83e-4d98-88aa-90e2981f472f\") " Jul 16 17:11:48.755876 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:11:48.755853 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ebce8a2-b83e-4d98-88aa-90e2981f472f-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "1ebce8a2-b83e-4d98-88aa-90e2981f472f" (UID: "1ebce8a2-b83e-4d98-88aa-90e2981f472f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 17:11:48.856747 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:11:48.856716 2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ebce8a2-b83e-4d98-88aa-90e2981f472f-maas-api-service-ca\") on node \"ip-10-0-142-101.ec2.internal\" DevicePath \"\"" Jul 16 17:11:49.464547 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:11:49.464511 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-0cb2ac-b9z2h" event={"ID":"1ebce8a2-b83e-4d98-88aa-90e2981f472f","Type":"ContainerDied","Data":"8337c2c42a4fbcfccbc684e8796875d146d9d0694dcd79ff1c2432a04714ad6e"} Jul 16 17:11:49.464722 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:11:49.464560 2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8337c2c42a4fbcfccbc684e8796875d146d9d0694dcd79ff1c2432a04714ad6e" Jul 16 17:11:49.464722 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:11:49.464519 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-0cb2ac-b9z2h" Jul 16 17:12:11.598169 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:12:11.598083 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-0cb2ac-openshift-default-7b64f4b4ff-wgjdt"] Jul 16 17:12:11.598630 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:12:11.598343 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-0cb2ac-openshift-default-7b64f4b4ff-wgjdt" podUID="0b03d37c-5beb-485f-87d6-5cac22ef2716" containerName="istio-proxy" containerID="cri-o://a58f490144679db795a5a0b0e83e24ca4eb5224ab96051382cbdb4fa772097df" gracePeriod=30 Jul 16 17:12:16.855962 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:12:16.855934 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-0cb2ac-openshift-default-7b64f4b4ff-wgjdt" Jul 16 17:12:16.929431 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:12:16.929329 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b03d37c-5beb-485f-87d6-5cac22ef2716-credential-socket\") pod \"0b03d37c-5beb-485f-87d6-5cac22ef2716\" (UID: \"0b03d37c-5beb-485f-87d6-5cac22ef2716\") " Jul 16 17:12:16.929431 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:12:16.929374 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b03d37c-5beb-485f-87d6-5cac22ef2716-istio-podinfo\") pod \"0b03d37c-5beb-485f-87d6-5cac22ef2716\" (UID: \"0b03d37c-5beb-485f-87d6-5cac22ef2716\") " Jul 16 17:12:16.929658 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:12:16.929435 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b03d37c-5beb-485f-87d6-5cac22ef2716-workload-certs\") pod \"0b03d37c-5beb-485f-87d6-5cac22ef2716\" (UID: \"0b03d37c-5beb-485f-87d6-5cac22ef2716\") " Jul 16 17:12:16.929658 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:12:16.929452 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b03d37c-5beb-485f-87d6-5cac22ef2716-istio-envoy\") pod \"0b03d37c-5beb-485f-87d6-5cac22ef2716\" (UID: \"0b03d37c-5beb-485f-87d6-5cac22ef2716\") " Jul 16 17:12:16.929658 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:12:16.929476 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b03d37c-5beb-485f-87d6-5cac22ef2716-istio-token\") pod \"0b03d37c-5beb-485f-87d6-5cac22ef2716\" (UID: \"0b03d37c-5beb-485f-87d6-5cac22ef2716\") " Jul 16 17:12:16.929658 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:12:16.929503 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-926hb\" (UniqueName: \"kubernetes.io/projected/0b03d37c-5beb-485f-87d6-5cac22ef2716-kube-api-access-926hb\") pod \"0b03d37c-5beb-485f-87d6-5cac22ef2716\" (UID: \"0b03d37c-5beb-485f-87d6-5cac22ef2716\") " Jul 16 17:12:16.929658 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:12:16.929532 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b03d37c-5beb-485f-87d6-5cac22ef2716-workload-socket\") pod \"0b03d37c-5beb-485f-87d6-5cac22ef2716\" (UID: \"0b03d37c-5beb-485f-87d6-5cac22ef2716\") " Jul 16 17:12:16.929658 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:12:16.929630 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b03d37c-5beb-485f-87d6-5cac22ef2716-istio-data\") pod \"0b03d37c-5beb-485f-87d6-5cac22ef2716\" (UID: \"0b03d37c-5beb-485f-87d6-5cac22ef2716\") " Jul 16 17:12:16.929972 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:12:16.929662 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b03d37c-5beb-485f-87d6-5cac22ef2716-istiod-ca-cert\") pod \"0b03d37c-5beb-485f-87d6-5cac22ef2716\" (UID: \"0b03d37c-5beb-485f-87d6-5cac22ef2716\") " Jul 16 17:12:16.929972 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:12:16.929667 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b03d37c-5beb-485f-87d6-5cac22ef2716-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "0b03d37c-5beb-485f-87d6-5cac22ef2716" (UID: "0b03d37c-5beb-485f-87d6-5cac22ef2716"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 17:12:16.929972 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:12:16.929675 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b03d37c-5beb-485f-87d6-5cac22ef2716-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "0b03d37c-5beb-485f-87d6-5cac22ef2716" (UID: "0b03d37c-5beb-485f-87d6-5cac22ef2716"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 17:12:16.929972 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:12:16.929853 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b03d37c-5beb-485f-87d6-5cac22ef2716-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "0b03d37c-5beb-485f-87d6-5cac22ef2716" (UID: "0b03d37c-5beb-485f-87d6-5cac22ef2716"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 17:12:16.930179 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:12:16.930039 2577 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b03d37c-5beb-485f-87d6-5cac22ef2716-workload-socket\") on node \"ip-10-0-142-101.ec2.internal\" DevicePath \"\"" Jul 16 17:12:16.930179 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:12:16.930057 2577 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b03d37c-5beb-485f-87d6-5cac22ef2716-credential-socket\") on node \"ip-10-0-142-101.ec2.internal\" DevicePath \"\"" Jul 16 17:12:16.930179 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:12:16.930069 2577 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b03d37c-5beb-485f-87d6-5cac22ef2716-workload-certs\") on node \"ip-10-0-142-101.ec2.internal\" DevicePath \"\"" Jul 16 17:12:16.930179 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:12:16.930144 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b03d37c-5beb-485f-87d6-5cac22ef2716-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "0b03d37c-5beb-485f-87d6-5cac22ef2716" (UID: "0b03d37c-5beb-485f-87d6-5cac22ef2716"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 17:12:16.930532 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:12:16.930512 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b03d37c-5beb-485f-87d6-5cac22ef2716-istio-data" (OuterVolumeSpecName: "istio-data") pod "0b03d37c-5beb-485f-87d6-5cac22ef2716" (UID: "0b03d37c-5beb-485f-87d6-5cac22ef2716"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 17:12:16.931831 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:12:16.931803 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b03d37c-5beb-485f-87d6-5cac22ef2716-kube-api-access-926hb" (OuterVolumeSpecName: "kube-api-access-926hb") pod "0b03d37c-5beb-485f-87d6-5cac22ef2716" (UID: "0b03d37c-5beb-485f-87d6-5cac22ef2716"). InnerVolumeSpecName "kube-api-access-926hb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 17:12:16.931967 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:12:16.931848 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0b03d37c-5beb-485f-87d6-5cac22ef2716-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "0b03d37c-5beb-485f-87d6-5cac22ef2716" (UID: "0b03d37c-5beb-485f-87d6-5cac22ef2716"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 16 17:12:16.932144 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:12:16.932123 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b03d37c-5beb-485f-87d6-5cac22ef2716-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "0b03d37c-5beb-485f-87d6-5cac22ef2716" (UID: "0b03d37c-5beb-485f-87d6-5cac22ef2716"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 17:12:16.932191 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:12:16.932146 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b03d37c-5beb-485f-87d6-5cac22ef2716-istio-token" (OuterVolumeSpecName: "istio-token") pod "0b03d37c-5beb-485f-87d6-5cac22ef2716" (UID: "0b03d37c-5beb-485f-87d6-5cac22ef2716"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 17:12:17.031141 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:12:17.031103 2577 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b03d37c-5beb-485f-87d6-5cac22ef2716-istio-envoy\") on node \"ip-10-0-142-101.ec2.internal\" DevicePath \"\"" Jul 16 17:12:17.031141 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:12:17.031135 2577 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b03d37c-5beb-485f-87d6-5cac22ef2716-istio-token\") on node \"ip-10-0-142-101.ec2.internal\" DevicePath \"\"" Jul 16 17:12:17.031141 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:12:17.031144 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-926hb\" (UniqueName: \"kubernetes.io/projected/0b03d37c-5beb-485f-87d6-5cac22ef2716-kube-api-access-926hb\") on node \"ip-10-0-142-101.ec2.internal\" DevicePath \"\"" Jul 16 17:12:17.031414 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:12:17.031155 2577 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b03d37c-5beb-485f-87d6-5cac22ef2716-istio-data\") on node \"ip-10-0-142-101.ec2.internal\" DevicePath \"\"" Jul 16 17:12:17.031414 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:12:17.031164 2577 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b03d37c-5beb-485f-87d6-5cac22ef2716-istiod-ca-cert\") on node \"ip-10-0-142-101.ec2.internal\" DevicePath \"\"" Jul 16 17:12:17.031414 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:12:17.031172 2577 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b03d37c-5beb-485f-87d6-5cac22ef2716-istio-podinfo\") on node \"ip-10-0-142-101.ec2.internal\" DevicePath \"\"" Jul 16 17:12:17.626888 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:12:17.626851 2577 generic.go:356] "Generic (PLEG): container finished" podID="0b03d37c-5beb-485f-87d6-5cac22ef2716" containerID="a58f490144679db795a5a0b0e83e24ca4eb5224ab96051382cbdb4fa772097df" exitCode=0 Jul 16 17:12:17.627077 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:12:17.626903 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-0cb2ac-openshift-default-7b64f4b4ff-wgjdt" Jul 16 17:12:17.627077 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:12:17.626918 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-0cb2ac-openshift-default-7b64f4b4ff-wgjdt" event={"ID":"0b03d37c-5beb-485f-87d6-5cac22ef2716","Type":"ContainerDied","Data":"a58f490144679db795a5a0b0e83e24ca4eb5224ab96051382cbdb4fa772097df"} Jul 16 17:12:17.627077 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:12:17.626971 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-0cb2ac-openshift-default-7b64f4b4ff-wgjdt" event={"ID":"0b03d37c-5beb-485f-87d6-5cac22ef2716","Type":"ContainerDied","Data":"e3e935f995b523e4efc19eb432bec24cc74d987b9145e54f29bfb2c849459d62"} Jul 16 17:12:17.627077 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:12:17.627001 2577 scope.go:117] "RemoveContainer" containerID="a58f490144679db795a5a0b0e83e24ca4eb5224ab96051382cbdb4fa772097df" Jul 16 17:12:17.637347 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:12:17.637324 2577 scope.go:117] "RemoveContainer" containerID="f09156435823bd48904349f7159e5b30ce1e4cf90b20235141beb0d81c17be5c" Jul 16 17:12:17.645508 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:12:17.645484 2577 scope.go:117] "RemoveContainer" containerID="a58f490144679db795a5a0b0e83e24ca4eb5224ab96051382cbdb4fa772097df" Jul 16 17:12:17.645775 ip-10-0-142-101 kubenswrapper[2577]: E0716 17:12:17.645756 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a58f490144679db795a5a0b0e83e24ca4eb5224ab96051382cbdb4fa772097df\": container with ID starting with a58f490144679db795a5a0b0e83e24ca4eb5224ab96051382cbdb4fa772097df not found: ID does not exist" containerID="a58f490144679db795a5a0b0e83e24ca4eb5224ab96051382cbdb4fa772097df" Jul 16 17:12:17.645854 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:12:17.645788 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a58f490144679db795a5a0b0e83e24ca4eb5224ab96051382cbdb4fa772097df"} err="failed to get container status \"a58f490144679db795a5a0b0e83e24ca4eb5224ab96051382cbdb4fa772097df\": rpc error: code = NotFound desc = could not find container \"a58f490144679db795a5a0b0e83e24ca4eb5224ab96051382cbdb4fa772097df\": container with ID starting with a58f490144679db795a5a0b0e83e24ca4eb5224ab96051382cbdb4fa772097df not found: ID does not exist" Jul 16 17:12:17.645854 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:12:17.645814 2577 scope.go:117] "RemoveContainer" containerID="f09156435823bd48904349f7159e5b30ce1e4cf90b20235141beb0d81c17be5c" Jul 16 17:12:17.646044 ip-10-0-142-101 kubenswrapper[2577]: E0716 17:12:17.646023 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f09156435823bd48904349f7159e5b30ce1e4cf90b20235141beb0d81c17be5c\": container with ID starting with f09156435823bd48904349f7159e5b30ce1e4cf90b20235141beb0d81c17be5c not found: ID does not exist" containerID="f09156435823bd48904349f7159e5b30ce1e4cf90b20235141beb0d81c17be5c" Jul 16 17:12:17.646086 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:12:17.646051 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f09156435823bd48904349f7159e5b30ce1e4cf90b20235141beb0d81c17be5c"} err="failed to get container status \"f09156435823bd48904349f7159e5b30ce1e4cf90b20235141beb0d81c17be5c\": rpc error: code = NotFound desc = could not find container \"f09156435823bd48904349f7159e5b30ce1e4cf90b20235141beb0d81c17be5c\": container with ID starting with f09156435823bd48904349f7159e5b30ce1e4cf90b20235141beb0d81c17be5c not found: ID does not exist" Jul 16 17:12:17.661445 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:12:17.661413 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-0cb2ac-openshift-default-7b64f4b4ff-wgjdt"] Jul 16 17:12:17.663848 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:12:17.663823 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-0cb2ac-openshift-default-7b64f4b4ff-wgjdt"] Jul 16 17:12:18.017440 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:12:18.017330 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0b03d37c-5beb-485f-87d6-5cac22ef2716" path="/var/lib/kubelet/pods/0b03d37c-5beb-485f-87d6-5cac22ef2716/volumes" Jul 16 17:12:37.877293 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:12:37.877259 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-734b32-openshift-default-7d6f8b88c5-4rb8c"] Jul 16 17:12:37.877742 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:12:37.877473 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-734b32-openshift-default-7d6f8b88c5-4rb8c" podUID="37f54670-7bd6-46c3-bb0e-3218fb0de41d" containerName="istio-proxy" containerID="cri-o://06da9e2f7375f8e43c5bb8120d821a186bc16256d90f4c9dd69fc866887b2637" gracePeriod=30 Jul 16 17:12:43.135216 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:12:43.135191 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-734b32-openshift-default-7d6f8b88c5-4rb8c" Jul 16 17:12:43.176028 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:12:43.175945 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37f54670-7bd6-46c3-bb0e-3218fb0de41d-istio-token\") pod \"37f54670-7bd6-46c3-bb0e-3218fb0de41d\" (UID: \"37f54670-7bd6-46c3-bb0e-3218fb0de41d\") " Jul 16 17:12:43.176028 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:12:43.176000 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37f54670-7bd6-46c3-bb0e-3218fb0de41d-istio-envoy\") pod \"37f54670-7bd6-46c3-bb0e-3218fb0de41d\" (UID: \"37f54670-7bd6-46c3-bb0e-3218fb0de41d\") " Jul 16 17:12:43.176028 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:12:43.176029 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37f54670-7bd6-46c3-bb0e-3218fb0de41d-istiod-ca-cert\") pod \"37f54670-7bd6-46c3-bb0e-3218fb0de41d\" (UID: \"37f54670-7bd6-46c3-bb0e-3218fb0de41d\") " Jul 16 17:12:43.176318 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:12:43.176126 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37f54670-7bd6-46c3-bb0e-3218fb0de41d-workload-certs\") pod \"37f54670-7bd6-46c3-bb0e-3218fb0de41d\" (UID: \"37f54670-7bd6-46c3-bb0e-3218fb0de41d\") " Jul 16 17:12:43.176318 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:12:43.176154 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37f54670-7bd6-46c3-bb0e-3218fb0de41d-istio-podinfo\") pod \"37f54670-7bd6-46c3-bb0e-3218fb0de41d\" (UID: \"37f54670-7bd6-46c3-bb0e-3218fb0de41d\") " Jul 16 17:12:43.176318 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:12:43.176194 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gz5ts\" (UniqueName: \"kubernetes.io/projected/37f54670-7bd6-46c3-bb0e-3218fb0de41d-kube-api-access-gz5ts\") pod \"37f54670-7bd6-46c3-bb0e-3218fb0de41d\" (UID: \"37f54670-7bd6-46c3-bb0e-3218fb0de41d\") " Jul 16 17:12:43.176318 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:12:43.176236 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37f54670-7bd6-46c3-bb0e-3218fb0de41d-istio-data\") pod \"37f54670-7bd6-46c3-bb0e-3218fb0de41d\" (UID: \"37f54670-7bd6-46c3-bb0e-3218fb0de41d\") " Jul 16 17:12:43.176318 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:12:43.176279 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37f54670-7bd6-46c3-bb0e-3218fb0de41d-credential-socket\") pod \"37f54670-7bd6-46c3-bb0e-3218fb0de41d\" (UID: \"37f54670-7bd6-46c3-bb0e-3218fb0de41d\") " Jul 16 17:12:43.176318 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:12:43.176315 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37f54670-7bd6-46c3-bb0e-3218fb0de41d-workload-socket\") pod \"37f54670-7bd6-46c3-bb0e-3218fb0de41d\" (UID: \"37f54670-7bd6-46c3-bb0e-3218fb0de41d\") " Jul 16 17:12:43.176671 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:12:43.176602 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/37f54670-7bd6-46c3-bb0e-3218fb0de41d-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "37f54670-7bd6-46c3-bb0e-3218fb0de41d" (UID: "37f54670-7bd6-46c3-bb0e-3218fb0de41d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 17:12:43.176775 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:12:43.176737 2577 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37f54670-7bd6-46c3-bb0e-3218fb0de41d-istiod-ca-cert\") on node \"ip-10-0-142-101.ec2.internal\" DevicePath \"\"" Jul 16 17:12:43.176880 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:12:43.176862 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37f54670-7bd6-46c3-bb0e-3218fb0de41d-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "37f54670-7bd6-46c3-bb0e-3218fb0de41d" (UID: "37f54670-7bd6-46c3-bb0e-3218fb0de41d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 17:12:43.176948 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:12:43.176922 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37f54670-7bd6-46c3-bb0e-3218fb0de41d-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "37f54670-7bd6-46c3-bb0e-3218fb0de41d" (UID: "37f54670-7bd6-46c3-bb0e-3218fb0de41d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 17:12:43.177341 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:12:43.177045 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37f54670-7bd6-46c3-bb0e-3218fb0de41d-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "37f54670-7bd6-46c3-bb0e-3218fb0de41d" (UID: "37f54670-7bd6-46c3-bb0e-3218fb0de41d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 17:12:43.177641 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:12:43.177607 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37f54670-7bd6-46c3-bb0e-3218fb0de41d-istio-data" (OuterVolumeSpecName: "istio-data") pod "37f54670-7bd6-46c3-bb0e-3218fb0de41d" (UID: "37f54670-7bd6-46c3-bb0e-3218fb0de41d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 17:12:43.178760 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:12:43.178723 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37f54670-7bd6-46c3-bb0e-3218fb0de41d-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "37f54670-7bd6-46c3-bb0e-3218fb0de41d" (UID: "37f54670-7bd6-46c3-bb0e-3218fb0de41d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 17:12:43.179351 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:12:43.179318 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/37f54670-7bd6-46c3-bb0e-3218fb0de41d-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "37f54670-7bd6-46c3-bb0e-3218fb0de41d" (UID: "37f54670-7bd6-46c3-bb0e-3218fb0de41d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 16 17:12:43.179478 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:12:43.179351 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37f54670-7bd6-46c3-bb0e-3218fb0de41d-kube-api-access-gz5ts" (OuterVolumeSpecName: "kube-api-access-gz5ts") pod "37f54670-7bd6-46c3-bb0e-3218fb0de41d" (UID: "37f54670-7bd6-46c3-bb0e-3218fb0de41d"). InnerVolumeSpecName "kube-api-access-gz5ts". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 17:12:43.179478 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:12:43.179365 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37f54670-7bd6-46c3-bb0e-3218fb0de41d-istio-token" (OuterVolumeSpecName: "istio-token") pod "37f54670-7bd6-46c3-bb0e-3218fb0de41d" (UID: "37f54670-7bd6-46c3-bb0e-3218fb0de41d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 17:12:43.277807 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:12:43.277765 2577 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37f54670-7bd6-46c3-bb0e-3218fb0de41d-credential-socket\") on node \"ip-10-0-142-101.ec2.internal\" DevicePath \"\"" Jul 16 17:12:43.277807 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:12:43.277796 2577 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37f54670-7bd6-46c3-bb0e-3218fb0de41d-workload-socket\") on node \"ip-10-0-142-101.ec2.internal\" DevicePath \"\"" Jul 16 17:12:43.277807 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:12:43.277810 2577 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37f54670-7bd6-46c3-bb0e-3218fb0de41d-istio-token\") on node \"ip-10-0-142-101.ec2.internal\" DevicePath \"\"" Jul 16 17:12:43.278055 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:12:43.277823 2577 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37f54670-7bd6-46c3-bb0e-3218fb0de41d-istio-envoy\") on node \"ip-10-0-142-101.ec2.internal\" DevicePath \"\"" Jul 16 17:12:43.278055 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:12:43.277834 2577 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37f54670-7bd6-46c3-bb0e-3218fb0de41d-workload-certs\") on node \"ip-10-0-142-101.ec2.internal\" DevicePath \"\"" Jul 16 17:12:43.278055 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:12:43.277844 2577 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37f54670-7bd6-46c3-bb0e-3218fb0de41d-istio-podinfo\") on node \"ip-10-0-142-101.ec2.internal\" DevicePath \"\"" Jul 16 17:12:43.278055 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:12:43.277855 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gz5ts\" (UniqueName: \"kubernetes.io/projected/37f54670-7bd6-46c3-bb0e-3218fb0de41d-kube-api-access-gz5ts\") on node \"ip-10-0-142-101.ec2.internal\" DevicePath \"\"" Jul 16 17:12:43.278055 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:12:43.277865 2577 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37f54670-7bd6-46c3-bb0e-3218fb0de41d-istio-data\") on node \"ip-10-0-142-101.ec2.internal\" DevicePath \"\"" Jul 16 17:12:43.788706 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:12:43.788665 2577 generic.go:356] "Generic (PLEG): container finished" podID="37f54670-7bd6-46c3-bb0e-3218fb0de41d" containerID="06da9e2f7375f8e43c5bb8120d821a186bc16256d90f4c9dd69fc866887b2637" exitCode=0 Jul 16 17:12:43.788892 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:12:43.788726 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-734b32-openshift-default-7d6f8b88c5-4rb8c" Jul 16 17:12:43.788892 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:12:43.788750 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-734b32-openshift-default-7d6f8b88c5-4rb8c" event={"ID":"37f54670-7bd6-46c3-bb0e-3218fb0de41d","Type":"ContainerDied","Data":"06da9e2f7375f8e43c5bb8120d821a186bc16256d90f4c9dd69fc866887b2637"} Jul 16 17:12:43.788892 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:12:43.788839 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-734b32-openshift-default-7d6f8b88c5-4rb8c" event={"ID":"37f54670-7bd6-46c3-bb0e-3218fb0de41d","Type":"ContainerDied","Data":"ece42829aea266bd9c4bc34a3e8894a8ec8915fab7d1346559e0a08fc77f33b2"} Jul 16 17:12:43.788892 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:12:43.788867 2577 scope.go:117] "RemoveContainer" containerID="06da9e2f7375f8e43c5bb8120d821a186bc16256d90f4c9dd69fc866887b2637" Jul 16 17:12:43.798220 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:12:43.798202 2577 scope.go:117] "RemoveContainer" containerID="ebd9a81baf09efeac6271f27deaec97294519f0ff968c23fca4c78ca1b27ccb7" Jul 16 17:12:43.806667 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:12:43.806642 2577 scope.go:117] "RemoveContainer" containerID="06da9e2f7375f8e43c5bb8120d821a186bc16256d90f4c9dd69fc866887b2637" Jul 16 17:12:43.806928 ip-10-0-142-101 kubenswrapper[2577]: E0716 17:12:43.806905 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"06da9e2f7375f8e43c5bb8120d821a186bc16256d90f4c9dd69fc866887b2637\": container with ID starting with 06da9e2f7375f8e43c5bb8120d821a186bc16256d90f4c9dd69fc866887b2637 not found: ID does not exist" containerID="06da9e2f7375f8e43c5bb8120d821a186bc16256d90f4c9dd69fc866887b2637" Jul 16 17:12:43.806976 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:12:43.806936 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"06da9e2f7375f8e43c5bb8120d821a186bc16256d90f4c9dd69fc866887b2637"} err="failed to get container status \"06da9e2f7375f8e43c5bb8120d821a186bc16256d90f4c9dd69fc866887b2637\": rpc error: code = NotFound desc = could not find container \"06da9e2f7375f8e43c5bb8120d821a186bc16256d90f4c9dd69fc866887b2637\": container with ID starting with 06da9e2f7375f8e43c5bb8120d821a186bc16256d90f4c9dd69fc866887b2637 not found: ID does not exist" Jul 16 17:12:43.806976 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:12:43.806953 2577 scope.go:117] "RemoveContainer" containerID="ebd9a81baf09efeac6271f27deaec97294519f0ff968c23fca4c78ca1b27ccb7" Jul 16 17:12:43.807190 ip-10-0-142-101 kubenswrapper[2577]: E0716 17:12:43.807173 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ebd9a81baf09efeac6271f27deaec97294519f0ff968c23fca4c78ca1b27ccb7\": container with ID starting with ebd9a81baf09efeac6271f27deaec97294519f0ff968c23fca4c78ca1b27ccb7 not found: ID does not exist" containerID="ebd9a81baf09efeac6271f27deaec97294519f0ff968c23fca4c78ca1b27ccb7" Jul 16 17:12:43.807233 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:12:43.807195 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ebd9a81baf09efeac6271f27deaec97294519f0ff968c23fca4c78ca1b27ccb7"} err="failed to get container status \"ebd9a81baf09efeac6271f27deaec97294519f0ff968c23fca4c78ca1b27ccb7\": rpc error: code = NotFound desc = could not find container \"ebd9a81baf09efeac6271f27deaec97294519f0ff968c23fca4c78ca1b27ccb7\": container with ID starting with ebd9a81baf09efeac6271f27deaec97294519f0ff968c23fca4c78ca1b27ccb7 not found: ID does not exist" Jul 16 17:12:43.812153 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:12:43.812131 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-734b32-openshift-default-7d6f8b88c5-4rb8c"] Jul 16 17:12:43.819167 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:12:43.819145 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-734b32-openshift-default-7d6f8b88c5-4rb8c"] Jul 16 17:12:44.018316 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:12:44.018280 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="37f54670-7bd6-46c3-bb0e-3218fb0de41d" path="/var/lib/kubelet/pods/37f54670-7bd6-46c3-bb0e-3218fb0de41d/volumes" Jul 16 17:12:45.077331 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:12:45.077305 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-846bd4c774-wlzp8_2dd9c4bf-dd03-4c43-8f5f-b5fa4a178b6d/manager/0.log" Jul 16 17:12:45.086225 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:12:45.086203 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-846bd4c774-wlzp8_2dd9c4bf-dd03-4c43-8f5f-b5fa4a178b6d/copy-manifests/0.log" Jul 16 17:12:45.719073 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:12:45.719046 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29737005-t72lz_7e211b39-1283-45ca-8b0d-6bce4939e466/cleanup/0.log" Jul 16 17:12:45.837155 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:12:45.837131 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29737020-xrbqn_3c521842-ea00-41ff-91a6-3cb7b12f125c/cleanup/0.log" Jul 16 17:12:45.956643 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:12:45.956616 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-ait-7a95ef74-7xgr9_86d20cca-30d2-4ed9-bed9-659c339429ea/revoke-keys/0.log" Jul 16 17:12:46.080217 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:12:46.080189 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-ait-fb78c5b0-zv8kw_c116afa5-da9b-4c3d-9022-7826683596c8/revoke-keys/0.log" Jul 16 17:12:46.209148 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:12:46.209119 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-api-a-6f111d-82gc7_14c15a24-306b-405e-bca2-873d7eeb9393/revoke-keys/0.log" Jul 16 17:12:46.346258 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:12:46.346170 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-api-b-6da193-k9sq7_f86a2759-c090-4b7b-bbcb-3b3036d51dab/revoke-keys/0.log" Jul 16 17:12:46.477298 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:12:46.477267 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-derive-ac435f2b-52gjn_c1b9cef5-e237-4a64-b504-5617c333924d/revoke-keys/0.log" Jul 16 17:12:46.597634 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:12:46.597552 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-inf-a-fa0ccd-npnj2_eb31a6f3-3163-47bd-bbc7-0c24b5182edd/revoke-keys/0.log" Jul 16 17:12:46.716854 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:12:46.716812 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-inf-b-f37c57-q75wl_389cfd7e-d0ed-48e5-96fb-89396b873d09/revoke-keys/0.log" Jul 16 17:12:46.847154 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:12:46.847124 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-migrate-c61ec96a-mswhf_c4df3ac0-fff6-41e5-a4fd-82d06f73226f/revoke-keys/0.log" Jul 16 17:12:46.974804 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:12:46.974725 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-mt-187946be-kphw9_78781c24-f2d6-4408-ac09-1522f8a771a8/revoke-keys/0.log" Jul 16 17:12:47.099164 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:12:47.099125 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-mt-8a5e08ec-jn2ds_18cf139b-c345-4d42-badd-027e6931d112/revoke-keys/0.log" Jul 16 17:12:47.211194 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:12:47.211164 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-shared-a-0cb2ac-b9z2h_1ebce8a2-b83e-4d98-88aa-90e2981f472f/revoke-keys/0.log" Jul 16 17:12:49.477264 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:12:49.477232 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-8jf9w_30c25bef-1662-4e8f-8389-f075eda51313/manager/0.log" Jul 16 17:12:50.107980 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:12:50.107945 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4hr8c9_1be39f1d-1d72-4e13-b292-8d7cc8389fc1/istio-proxy/0.log" Jul 16 17:12:50.238404 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:12:50.238365 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-j5pgj_f161514a-da71-4244-88f3-1a42a7c4a87d/discovery/0.log" Jul 16 17:12:50.737067 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:12:50.737026 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-74584bb9f8-ht4px_998f1366-d52e-447f-8465-1dfc4b4e6c2e/payload-pre-processing/0.log" Jul 16 17:12:51.573302 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:12:51.573272 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-n76ht_416f1afc-880f-4e22-83da-bf4d8ad0e5d5/main/0.log" Jul 16 17:13:00.655885 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:13:00.655845 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-rqzn4_e2d675d8-edf6-4765-a240-4b723cdd2d38/global-pull-secret-syncer/0.log" Jul 16 17:13:00.830302 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:13:00.830268 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-xljrg_a2cb45f0-0fb3-4761-98b0-a1797a00e647/konnectivity-agent/0.log" Jul 16 17:13:00.913214 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:13:00.913123 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-142-101.ec2.internal_077fca01a64486b516cac217b95ed2fa/haproxy/0.log" Jul 16 17:13:06.106821 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:13:06.106786 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-8jf9w_30c25bef-1662-4e8f-8389-f075eda51313/manager/0.log" Jul 16 17:13:07.814034 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:13:07.814005 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_8c88b029-8a81-402c-90e4-77eff554183c/alertmanager/0.log" Jul 16 17:13:07.847654 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:13:07.847620 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_8c88b029-8a81-402c-90e4-77eff554183c/config-reloader/0.log" Jul 16 17:13:07.881429 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:13:07.881329 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_8c88b029-8a81-402c-90e4-77eff554183c/kube-rbac-proxy-web/0.log" Jul 16 17:13:07.912574 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:13:07.912539 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_8c88b029-8a81-402c-90e4-77eff554183c/kube-rbac-proxy/0.log" Jul 16 17:13:07.949606 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:13:07.949567 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_8c88b029-8a81-402c-90e4-77eff554183c/kube-rbac-proxy-metric/0.log" Jul 16 17:13:07.981060 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:13:07.981037 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_8c88b029-8a81-402c-90e4-77eff554183c/prom-label-proxy/0.log" Jul 16 17:13:08.017662 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:13:08.017637 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_8c88b029-8a81-402c-90e4-77eff554183c/init-config-reloader/0.log" Jul 16 17:13:08.098596 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:13:08.098564 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-785778ff74-vfxfr_10bc3cb6-aa0e-4d0f-a5a9-75f494f7fffb/kube-state-metrics/0.log" Jul 16 17:13:08.117628 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:13:08.117602 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-785778ff74-vfxfr_10bc3cb6-aa0e-4d0f-a5a9-75f494f7fffb/kube-rbac-proxy-main/0.log" Jul 16 17:13:08.146234 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:13:08.146157 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-785778ff74-vfxfr_10bc3cb6-aa0e-4d0f-a5a9-75f494f7fffb/kube-rbac-proxy-self/0.log" Jul 16 17:13:08.180984 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:13:08.180953 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-6fd5bb7967-8t82m_3ed7defb-10df-42c6-af68-8a2145ad3539/metrics-server/0.log" Jul 16 17:13:08.210589 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:13:08.210563 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-5f68df7f58-rrhsm_342fe0d4-70b8-43b9-b052-93a1068deb59/monitoring-plugin/0.log" Jul 16 17:13:08.450667 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:13:08.450632 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-hvwxd_42c6e795-fc08-4367-8e7f-9c65cb84c814/node-exporter/0.log" Jul 16 17:13:08.489210 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:13:08.489184 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-hvwxd_42c6e795-fc08-4367-8e7f-9c65cb84c814/kube-rbac-proxy/0.log" Jul 16 17:13:08.525075 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:13:08.525052 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-hvwxd_42c6e795-fc08-4367-8e7f-9c65cb84c814/init-textfile/0.log" Jul 16 17:13:08.658834 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:13:08.658793 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-84bd68b896-xfk8h_3253b8ec-f532-448b-9a7e-6fcb71b0d3fd/kube-rbac-proxy-main/0.log" Jul 16 17:13:08.701342 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:13:08.701268 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-84bd68b896-xfk8h_3253b8ec-f532-448b-9a7e-6fcb71b0d3fd/kube-rbac-proxy-self/0.log" Jul 16 17:13:08.745223 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:13:08.745192 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-84bd68b896-xfk8h_3253b8ec-f532-448b-9a7e-6fcb71b0d3fd/openshift-state-metrics/0.log" Jul 16 17:13:08.794578 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:13:08.794544 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_838dbd8b-f75f-4739-9052-56ecfc1cbe5e/prometheus/0.log" Jul 16 17:13:08.822019 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:13:08.821989 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_838dbd8b-f75f-4739-9052-56ecfc1cbe5e/config-reloader/0.log" Jul 16 17:13:08.864264 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:13:08.864238 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_838dbd8b-f75f-4739-9052-56ecfc1cbe5e/thanos-sidecar/0.log" Jul 16 17:13:08.890758 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:13:08.890733 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_838dbd8b-f75f-4739-9052-56ecfc1cbe5e/kube-rbac-proxy-web/0.log" Jul 16 17:13:08.913889 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:13:08.913863 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_838dbd8b-f75f-4739-9052-56ecfc1cbe5e/kube-rbac-proxy/0.log" Jul 16 17:13:08.977829 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:13:08.977741 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_838dbd8b-f75f-4739-9052-56ecfc1cbe5e/kube-rbac-proxy-thanos/0.log" Jul 16 17:13:09.021030 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:13:09.020982 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_838dbd8b-f75f-4739-9052-56ecfc1cbe5e/init-config-reloader/0.log" Jul 16 17:13:09.145488 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:13:09.145453 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-64785b6675-vp8d9_7fe140c1-9268-46ef-aa31-3283957b0fff/prometheus-operator/0.log" Jul 16 17:13:09.204438 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:13:09.204403 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-64785b6675-vp8d9_7fe140c1-9268-46ef-aa31-3283957b0fff/kube-rbac-proxy/0.log" Jul 16 17:13:09.251493 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:13:09.251415 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-77d98bc786-875t5_7ca62a57-8939-42e8-a31d-d8fd4e1595bb/prometheus-operator-admission-webhook/0.log" Jul 16 17:13:09.302291 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:13:09.302247 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-568889b88c-4488z_ce591e34-5de4-45d1-ab53-e9963ffae504/telemeter-client/0.log" Jul 16 17:13:09.353306 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:13:09.353278 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-568889b88c-4488z_ce591e34-5de4-45d1-ab53-e9963ffae504/reload/0.log" Jul 16 17:13:09.387697 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:13:09.387659 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-568889b88c-4488z_ce591e34-5de4-45d1-ab53-e9963ffae504/kube-rbac-proxy/0.log" Jul 16 17:13:09.433194 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:13:09.433158 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-b957bd756-6cvr2_2e58ee24-b3bc-4e3a-828c-6ac801ec7fd7/thanos-query/0.log" Jul 16 17:13:09.458989 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:13:09.458952 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-b957bd756-6cvr2_2e58ee24-b3bc-4e3a-828c-6ac801ec7fd7/kube-rbac-proxy-web/0.log" Jul 16 17:13:09.488291 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:13:09.488261 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-b957bd756-6cvr2_2e58ee24-b3bc-4e3a-828c-6ac801ec7fd7/kube-rbac-proxy/0.log" Jul 16 17:13:09.515861 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:13:09.515783 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-b957bd756-6cvr2_2e58ee24-b3bc-4e3a-828c-6ac801ec7fd7/prom-label-proxy/0.log" Jul 16 17:13:09.541912 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:13:09.541884 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-b957bd756-6cvr2_2e58ee24-b3bc-4e3a-828c-6ac801ec7fd7/kube-rbac-proxy-rules/0.log" Jul 16 17:13:09.568784 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:13:09.568726 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-b957bd756-6cvr2_2e58ee24-b3bc-4e3a-828c-6ac801ec7fd7/kube-rbac-proxy-metrics/0.log" Jul 16 17:13:10.989324 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:13:10.989294 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-84tj4/perf-node-gather-daemonset-gx965"] Jul 16 17:13:10.989756 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:13:10.989676 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0b03d37c-5beb-485f-87d6-5cac22ef2716" containerName="istio-proxy" Jul 16 17:13:10.989756 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:13:10.989687 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="0b03d37c-5beb-485f-87d6-5cac22ef2716" containerName="istio-proxy" Jul 16 17:13:10.989756 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:13:10.989700 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="37f54670-7bd6-46c3-bb0e-3218fb0de41d" containerName="istio-proxy" Jul 16 17:13:10.989756 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:13:10.989706 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="37f54670-7bd6-46c3-bb0e-3218fb0de41d" containerName="istio-proxy" Jul 16 17:13:10.989756 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:13:10.989718 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0b03d37c-5beb-485f-87d6-5cac22ef2716" containerName="istio-proxy" Jul 16 17:13:10.989756 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:13:10.989723 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="0b03d37c-5beb-485f-87d6-5cac22ef2716" containerName="istio-proxy" Jul 16 17:13:10.989756 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:13:10.989729 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1ebce8a2-b83e-4d98-88aa-90e2981f472f" containerName="revoke-keys" Jul 16 17:13:10.989756 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:13:10.989735 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="1ebce8a2-b83e-4d98-88aa-90e2981f472f" containerName="revoke-keys" Jul 16 17:13:10.990069 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:13:10.989803 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="37f54670-7bd6-46c3-bb0e-3218fb0de41d" containerName="istio-proxy" Jul 16 17:13:10.990069 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:13:10.989812 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="1ebce8a2-b83e-4d98-88aa-90e2981f472f" containerName="revoke-keys" Jul 16 17:13:10.990069 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:13:10.989819 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="37f54670-7bd6-46c3-bb0e-3218fb0de41d" containerName="istio-proxy" Jul 16 17:13:10.990069 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:13:10.989825 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="0b03d37c-5beb-485f-87d6-5cac22ef2716" containerName="istio-proxy" Jul 16 17:13:10.990069 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:13:10.989831 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="0b03d37c-5beb-485f-87d6-5cac22ef2716" containerName="istio-proxy" Jul 16 17:13:10.993213 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:13:10.993192 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-84tj4/perf-node-gather-daemonset-gx965" Jul 16 17:13:10.995969 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:13:10.995946 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-84tj4\"/\"kube-root-ca.crt\"" Jul 16 17:13:10.996232 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:13:10.996214 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-84tj4\"/\"openshift-service-ca.crt\"" Jul 16 17:13:10.996984 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:13:10.996970 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-84tj4\"/\"default-dockercfg-c7czs\"" Jul 16 17:13:11.009565 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:13:11.009537 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-84tj4/perf-node-gather-daemonset-gx965"] Jul 16 17:13:11.151310 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:13:11.151261 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2e5f72db-8ac9-4d53-9dcb-6ab50586f829-sys\") pod \"perf-node-gather-daemonset-gx965\" (UID: \"2e5f72db-8ac9-4d53-9dcb-6ab50586f829\") " pod="openshift-must-gather-84tj4/perf-node-gather-daemonset-gx965" Jul 16 17:13:11.151530 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:13:11.151416 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/2e5f72db-8ac9-4d53-9dcb-6ab50586f829-podres\") pod \"perf-node-gather-daemonset-gx965\" (UID: \"2e5f72db-8ac9-4d53-9dcb-6ab50586f829\") " pod="openshift-must-gather-84tj4/perf-node-gather-daemonset-gx965" Jul 16 17:13:11.151530 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:13:11.151445 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/2e5f72db-8ac9-4d53-9dcb-6ab50586f829-proc\") pod \"perf-node-gather-daemonset-gx965\" (UID: \"2e5f72db-8ac9-4d53-9dcb-6ab50586f829\") " pod="openshift-must-gather-84tj4/perf-node-gather-daemonset-gx965" Jul 16 17:13:11.151530 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:13:11.151493 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-58jgf\" (UniqueName: \"kubernetes.io/projected/2e5f72db-8ac9-4d53-9dcb-6ab50586f829-kube-api-access-58jgf\") pod \"perf-node-gather-daemonset-gx965\" (UID: \"2e5f72db-8ac9-4d53-9dcb-6ab50586f829\") " pod="openshift-must-gather-84tj4/perf-node-gather-daemonset-gx965" Jul 16 17:13:11.151530 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:13:11.151519 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2e5f72db-8ac9-4d53-9dcb-6ab50586f829-lib-modules\") pod \"perf-node-gather-daemonset-gx965\" (UID: \"2e5f72db-8ac9-4d53-9dcb-6ab50586f829\") " pod="openshift-must-gather-84tj4/perf-node-gather-daemonset-gx965" Jul 16 17:13:11.252932 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:13:11.252844 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/2e5f72db-8ac9-4d53-9dcb-6ab50586f829-podres\") pod \"perf-node-gather-daemonset-gx965\" (UID: \"2e5f72db-8ac9-4d53-9dcb-6ab50586f829\") " pod="openshift-must-gather-84tj4/perf-node-gather-daemonset-gx965" Jul 16 17:13:11.252932 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:13:11.252884 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/2e5f72db-8ac9-4d53-9dcb-6ab50586f829-proc\") pod \"perf-node-gather-daemonset-gx965\" (UID: \"2e5f72db-8ac9-4d53-9dcb-6ab50586f829\") " pod="openshift-must-gather-84tj4/perf-node-gather-daemonset-gx965" Jul 16 17:13:11.253137 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:13:11.253013 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58jgf\" (UniqueName: \"kubernetes.io/projected/2e5f72db-8ac9-4d53-9dcb-6ab50586f829-kube-api-access-58jgf\") pod \"perf-node-gather-daemonset-gx965\" (UID: \"2e5f72db-8ac9-4d53-9dcb-6ab50586f829\") " pod="openshift-must-gather-84tj4/perf-node-gather-daemonset-gx965" Jul 16 17:13:11.253137 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:13:11.253035 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/2e5f72db-8ac9-4d53-9dcb-6ab50586f829-podres\") pod \"perf-node-gather-daemonset-gx965\" (UID: \"2e5f72db-8ac9-4d53-9dcb-6ab50586f829\") " pod="openshift-must-gather-84tj4/perf-node-gather-daemonset-gx965" Jul 16 17:13:11.253137 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:13:11.253058 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2e5f72db-8ac9-4d53-9dcb-6ab50586f829-lib-modules\") pod \"perf-node-gather-daemonset-gx965\" (UID: \"2e5f72db-8ac9-4d53-9dcb-6ab50586f829\") " pod="openshift-must-gather-84tj4/perf-node-gather-daemonset-gx965" Jul 16 17:13:11.253137 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:13:11.253098 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/2e5f72db-8ac9-4d53-9dcb-6ab50586f829-proc\") pod \"perf-node-gather-daemonset-gx965\" (UID: \"2e5f72db-8ac9-4d53-9dcb-6ab50586f829\") " pod="openshift-must-gather-84tj4/perf-node-gather-daemonset-gx965" Jul 16 17:13:11.253287 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:13:11.253153 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2e5f72db-8ac9-4d53-9dcb-6ab50586f829-sys\") pod \"perf-node-gather-daemonset-gx965\" (UID: \"2e5f72db-8ac9-4d53-9dcb-6ab50586f829\") " pod="openshift-must-gather-84tj4/perf-node-gather-daemonset-gx965" Jul 16 17:13:11.253287 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:13:11.253185 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2e5f72db-8ac9-4d53-9dcb-6ab50586f829-lib-modules\") pod \"perf-node-gather-daemonset-gx965\" (UID: \"2e5f72db-8ac9-4d53-9dcb-6ab50586f829\") " pod="openshift-must-gather-84tj4/perf-node-gather-daemonset-gx965" Jul 16 17:13:11.253287 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:13:11.253229 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2e5f72db-8ac9-4d53-9dcb-6ab50586f829-sys\") pod \"perf-node-gather-daemonset-gx965\" (UID: \"2e5f72db-8ac9-4d53-9dcb-6ab50586f829\") " pod="openshift-must-gather-84tj4/perf-node-gather-daemonset-gx965" Jul 16 17:13:11.263132 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:13:11.263105 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-58jgf\" (UniqueName: \"kubernetes.io/projected/2e5f72db-8ac9-4d53-9dcb-6ab50586f829-kube-api-access-58jgf\") pod \"perf-node-gather-daemonset-gx965\" (UID: \"2e5f72db-8ac9-4d53-9dcb-6ab50586f829\") " pod="openshift-must-gather-84tj4/perf-node-gather-daemonset-gx965" Jul 16 17:13:11.303043 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:13:11.303005 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-84tj4/perf-node-gather-daemonset-gx965" Jul 16 17:13:11.444856 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:13:11.444825 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-84tj4/perf-node-gather-daemonset-gx965"] Jul 16 17:13:11.446825 ip-10-0-142-101 kubenswrapper[2577]: W0716 17:13:11.446791 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod2e5f72db_8ac9_4d53_9dcb_6ab50586f829.slice/crio-0c9fddbf3a18e7c7d7335521f5b5af69b24d1d66bd9810f1605f30918510774f WatchSource:0}: Error finding container 0c9fddbf3a18e7c7d7335521f5b5af69b24d1d66bd9810f1605f30918510774f: Status 404 returned error can't find the container with id 0c9fddbf3a18e7c7d7335521f5b5af69b24d1d66bd9810f1605f30918510774f Jul 16 17:13:11.943435 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:13:11.943375 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-84tj4/perf-node-gather-daemonset-gx965" event={"ID":"2e5f72db-8ac9-4d53-9dcb-6ab50586f829","Type":"ContainerStarted","Data":"504e7e8bdb52e094204a7e303a6433833f6bd8d977e0219fc68efa543c406efc"} Jul 16 17:13:11.943610 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:13:11.943448 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-84tj4/perf-node-gather-daemonset-gx965" event={"ID":"2e5f72db-8ac9-4d53-9dcb-6ab50586f829","Type":"ContainerStarted","Data":"0c9fddbf3a18e7c7d7335521f5b5af69b24d1d66bd9810f1605f30918510774f"} Jul 16 17:13:11.943610 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:13:11.943475 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-84tj4/perf-node-gather-daemonset-gx965" Jul 16 17:13:11.963733 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:13:11.963682 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-84tj4/perf-node-gather-daemonset-gx965" podStartSLOduration=1.9636664179999999 podStartE2EDuration="1.963666418s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 17:13:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:13:11.961181155 +0000 UTC m=+2850.465239019" watchObservedRunningTime="2026-07-16 17:13:11.963666418 +0000 UTC m=+2850.467724269" Jul 16 17:13:12.035278 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:13:12.035244 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-67f5cb746f-bn5m2_5439ffd6-38a7-4d37-a229-38d02ee0e167/console/0.log" Jul 16 17:13:13.826743 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:13:13.826714 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-qhj4x_984479f9-02a8-4fe9-ad59-23f9e3946e3a/dns/0.log" Jul 16 17:13:13.848021 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:13:13.847996 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-qhj4x_984479f9-02a8-4fe9-ad59-23f9e3946e3a/kube-rbac-proxy/0.log" Jul 16 17:13:13.921818 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:13:13.921789 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-6sc67_93f30808-1a75-44b5-801f-8faeceb72afb/dns-node-resolver/0.log" Jul 16 17:13:14.646941 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:13:14.646910 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-hwq4x_cbd027ac-97d9-4769-a867-47b7b8256cd0/node-ca/0.log" Jul 16 17:13:15.630541 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:13:15.630490 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4hr8c9_1be39f1d-1d72-4e13-b292-8d7cc8389fc1/istio-proxy/0.log" Jul 16 17:13:15.848905 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:13:15.848870 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-j5pgj_f161514a-da71-4244-88f3-1a42a7c4a87d/discovery/0.log" Jul 16 17:13:15.993913 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:13:15.993816 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-74584bb9f8-ht4px_998f1366-d52e-447f-8465-1dfc4b4e6c2e/payload-pre-processing/0.log" Jul 16 17:13:16.813839 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:13:16.813804 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-6jhs5_533bf5ba-0e8a-40a2-b11a-44584469eae2/serve-healthcheck-canary/0.log" Jul 16 17:13:17.484080 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:13:17.484054 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-lf2qz_9964ff62-4edc-4638-adf2-bbc37c39d29c/kube-rbac-proxy/0.log" Jul 16 17:13:17.508807 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:13:17.508769 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-lf2qz_9964ff62-4edc-4638-adf2-bbc37c39d29c/exporter/0.log" Jul 16 17:13:17.533566 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:13:17.533524 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-lf2qz_9964ff62-4edc-4638-adf2-bbc37c39d29c/extractor/0.log" Jul 16 17:13:17.959585 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:13:17.959553 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-84tj4/perf-node-gather-daemonset-gx965" Jul 16 17:13:20.395216 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:13:20.395185 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-846bd4c774-wlzp8_2dd9c4bf-dd03-4c43-8f5f-b5fa4a178b6d/manager/0.log" Jul 16 17:13:20.422027 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:13:20.422004 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-846bd4c774-wlzp8_2dd9c4bf-dd03-4c43-8f5f-b5fa4a178b6d/copy-manifests/0.log" Jul 16 17:13:21.906176 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:13:21.906140 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4hr8c9_1be39f1d-1d72-4e13-b292-8d7cc8389fc1/istio-proxy/0.log" Jul 16 17:13:21.908924 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:13:21.908903 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-6697dd89c8-whsxg_bf9f4e29-a305-490b-b161-ae5f9890dd38/manager/0.log" Jul 16 17:13:22.115201 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:13:22.115169 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-j5pgj_f161514a-da71-4244-88f3-1a42a7c4a87d/discovery/0.log" Jul 16 17:13:22.236694 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:13:22.236662 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-74584bb9f8-ht4px_998f1366-d52e-447f-8465-1dfc4b4e6c2e/payload-pre-processing/0.log" Jul 16 17:13:23.588657 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:13:23.588627 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4hr8c9_1be39f1d-1d72-4e13-b292-8d7cc8389fc1/istio-proxy/0.log" Jul 16 17:13:23.796595 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:13:23.796559 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-j5pgj_f161514a-da71-4244-88f3-1a42a7c4a87d/discovery/0.log" Jul 16 17:13:23.935116 ip-10-0-142-101 kubenswrapper[2577]: I0716 17:13:23.935024 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-74584bb9f8-ht4px_998f1366-d52e-447f-8465-1dfc4b4e6c2e/payload-pre-processing/0.log"